Accelerating VA IT Modernization through DevOps

Driving improved outcomes for VA

In 2016, OIT embarked on a sweeping IT transformation to collaborate with our business partners and create the best experience for all Veterans. We stood up the Enterprise Program Management Office (EPMO), a control tower for all IT solutions in development. We realigned our service delivery folks in the field to the IT Operations and Services (ITOPS) organization to standardize our customer service and bolster visibility of trending IT challenges at VA. EPMO manages the lifecycle of our technologies, from the original business request through the transition, release, and sustainment phases. ITOPS implements those technologies and provides technical support to manage them. We’ve since built on that transformation with a focus on IT modernization — creating a VA IT organization that stands toe-to-toe with the top tech companies in private industry.

This year, we accelerated this IT modernization, announcing the integration of the EPMO and ITOPS divisions into one Development Operations (DevOps) environment.

In a world in which IT and business processes change at a moment’s notice, teams need to align staff efficiently and adapt quickly to changing needs and requirements. It isn’t enough to just be agile; adherence to Agile methods such as Scrum, XP, or Kanban will only take an organization so far. Agile success requires more than just development and testing. It requires true collaboration between the development and operations teams throughout the entire deployment pipeline. Shifting to a DevOps culture and framework fundamentally changes the way we operate. DevOps is entirely about delivering the best outcomes at the best value to our customers.

Challenging conventions at VA

People

VA OIT is committed to a culture where collaborative, high-trust teams can pursue innovation, without fear of condemnation, and where standardized processes are practiced. This objective is achieved by (1) institutionalizing predictable development, delivery, and deployment activities, and (2) establishing self-organizing teams that have a shared purpose and are afforded the autonomy to master leading practices, while pursing continuous improvement.
In the past, our IT team structures encouraged functional specialization and optimization of individual steps of a process, but not so much organizational collaboration. This not only impedes cultural change, but also makes it more difficult to hold individuals responsible for service delivery outcomes.

To change this, our new DevOps structure requires our staff at each step of the development process to work together to champion products throughout the lifecycle. In other words, developers can no longer hand their work off to testers, and testers can no longer hand off their work to operators to deal with any release crises. Those integrated project teams must be mindful of operational requirements from release planning, logging, and exception handling—to monitoring, service management, and security. When planning, teams must incorporate requirements, such as system stability, availability of test data, suitability of the solution to its operating environment, and performance. The delivery outcomes improve because many parts of the organization are collaborating early and often, and the whole group feels a sense of responsibility for the final product.

Culture

Changing the culture means challenging conventional thinking. It doesn’t just mean keeping the lights on or keeping the trains running on time. A true DevOps environment is a change of IT culture that demands strategic risk and investment decisions to achieve rapid IT service delivery, in a lean and Agile systems-oriented approach.

DevOps extends Agile methods and responsibilities beyond development and testing to include collaboration with operations personnel. This collaboration entices teams to reduce feedback loops, enhance product testability, improve product delivery predictability, and define success in terms of shared product deployment and quality goals, rather than hand-off completion.

Technology

Changing our IT culture is not enough. We need to reimagine the way we test—and we also need end-to-end visibility into our tools and interfaces at the program level. To achieve this, our integrated team is developing a deployment pipeline that provides visibility into all build, deploy, test, and release activities. It will also deliver actionable feedback so that staff can resolve problems quickly, with minimal intervention.

That kind of integration and collaboration enables both Continuous Integration (CI) and Continuous Delivery (CD) practices. CI improves the flow of work from developers and testers through operations to the customer; enables timely, frequent feedback from customers and operations to developers and testers; and improves team member coordination while standardizing predictable development, delivery, and deployment activities.

CD extends CI practices by automating the deployment of CI build products to downstream production-like environments that test the system the way it will be operated in real life. This shortens feedback loops and makes certain that we do not promote and test builds that seem to work in test environments yet fail when deployed to the field and across VA. When this entire process is automated, culminating in the actual deployment of the product to the field, this is called “Continuous Deployment”—an end-to-end automation and feedback practice, from development, to testing, to deployment to the production environment.

Process

No organizational realignment comes without some level of uncertainty and friction. To reduce that friction, we are instituting effective processes that provide the kind of feedback that helps us prevent new and recurring problems and enables us to detect, remediate, and mitigate the roadblocks that inevitably arise. Those Agile processes will also allow us to create quality at the source, helping us fend off expensive and disruptive unplanned work. These kinds of processes must include automation of error-prone and laborious manual tasks, test suite automation, shared goals and rewards for development and operations personnel, and continuous production monitoring.

Designing with the customer, instead of for the customer

Elements of the DevOps environment noted above, namely CI, CD, and Continuous Deployment, focus on integrating the customer’s experience as a key process, from beginning to end, versus bolting on fixes or enhancements at the end. They allow us to succeed in areas where legacy methods have failed in the past, bringing our IT organization closer to the VA customer, the user. An exceptional customer service experience begins with a fundamental understanding of the needs of the user—but that understanding cannot simply begin and end as a conversation between the VA business partner and the IT project owner; the customers’ needs must be integrated at every step of the development, testing, deployment, and operations cycle. This new DevOps environment facilitates that seamless, continuous exchange of customer feedback, enabling our IT organization to design, deliver, and support solutions that meet our customers’ needs—solutions created with the customer, not for the customer.