Continuous Delivery (CD) has been around for a few years now and we see increased adoption of CD practices within every organization. CD improves an organization’s time to market by shortening the feedback cycle, improves quality by introducing automation into the Software Development Life cycle (SDLC) process, and is an enabler for experimentation and faster validation of new business ideas.

Continuous Delivery

However, one area where most of the organization do not have insight is about measuring the value that “Continuous Delivery” practice delivers. This is one topic that has not been clearly outlined in some of the best publications of our time , though they make a few implied references. Having made that bold statement, this is an honest attempt at identifying the value that Continuous Delivery adds to an organization.

Before we jump to the details, there are some key points to consider:

• Business and Technology alignment — Start with business and customer needs/problems first. Buy- in from business and technology management is important to understand the customer problems and business challenges that CD could solve before you even start.
• Organization readiness — While CD can bring a lot of value to the organization, it is also vital to assess the organization readiness on the people-process-technology triad. The Continuous Delivery Maturity Model by InfoQ is a very good reference point in this case.
• Critical Success factors — Identify and track what you want to measure and improve (Speed, Quality, Cost and Customer satisfaction, …). Deep dive into your current development process to identify how these factors are impacted with lack of automated build, test, integration and deployment processes. CD is beyond automation, you might want to eliminate a few (non- value add) processes on your journey to achieve Continuous delivery.

Measure the metrics that matter: The above pointers should provide you insight to decide why CD is needed, at which stage and what to expect out of it. Some of the KPI’s (opinionated) that can help you measure the value delivered by CD are listed below:

Speed & Time –

• Improve the Lead time (lead time = cycle time + wait time) from “code commit to deployment” and how that enables a faster delivery of a feature request to customer.
• Measure how much of “wait time” ( by eliminating non-value add activity Refer “Muda” from Lean) is reduced after implementing CD.
• Measure the increase in efficiency of SDLC processes by visualization each step in the CD process(build, test, deployment) with the Lean flow approach. CD also provides an organization the speed to experiment their new product ideas faster with the “fail fast & learn fast” approach.

Quality –

• Change in % or number of failed deployments/bugs generated after implementing CD
• Measure the up-time and time to recover and how these contribute to customer retention and satisfaction using the roll forward strategy enabled by CD.
• Reduction in impact of risks, namely; security, cost, infrastructure to name a few by releasing smaller features

Cost –

• Cost saved by speeding up product teams by shortening product releases cycles
• Number of experiments run ; faster validation enabled by CD resulting in reduced cost with shorter “build measure and learn” cycle

Satisfaction scores –

• Change in customer satisfaction scores impacted by timeliness to develop a new feature faster, improved availability, less bugs/defects in production using CD practices
• Measure product team’s satisfaction score and ask explicitly how CD impacts

Continuous Delivery, if implemented well can bring some qualitative benefits as well for an organization such as improved morale of the product teams (the State of DevOps Report by Puppet provides good insight on some of these metrics) and increase in the organization maturity in the overall SDLC process.

We would be glad to hear your opinion and experience on how you are measuring the effectiveness of the continuous delivery practices within the organization. Please share your feedback and inputs.

Special note of thanks to my colleague Rupert Broad who shared valuable inputs and feedback.