What is Deployment Frequency & How to Measure it?

75% of tech leaders rely on deployment frequency as their measure of DevOps success. Deployment frequency is one of the key value stream metrics, and of course, one of the core DORA metrics to benchmark your software delivery efficiency. 

In general, engineering managers & practitioners use all 4 DORA metrics to assess the efficiency of their SDLC processes and engineering practices. 

But Deployment Frequency (75%) is by far the most tracked engineering metric compared to Mean Time To Recovery (55%), Lead Time (53%), and Change Failure Rate (48%). 

Read this insight to find out-

• What is deployment frequency?

• Should you measure deployment frequency to benchmark your DevOps performance? If yes, then how do you measure deployment frequency?

• How can you improve deployment frequency if you want to?

What is Deployment Frequency (DF)?

In simple terms, Deployment Frequency is the count of ‘how many times you deploy code to production in a given time frame’. 

Okay. That’s simple, but not clear. Doesn’t it sound vague and a little cryptic?

For a more descriptive & vivid definition of deployment frequency, it’s important to understand what qualifies as a successful deployment.

In an org that has embraced DevOps practices, here’s how the code gets deployed to production-

• A developer or team pushes the code to the repository i.e., raises a PR (Pull request). The PR gets subjected to all sorts of SDLC processes— code review, functionality testing, etc. It’s recommended to follow the PR best practices.

• Continuous Integration (CI) server detects the code changes in the repository and triggers automated build & testing processes.

• The code gets compiled into executable binaries/artifacts (for example - docker images for containerized applications).

• The software application artifact is then pushed to the staging environment where automated tests are carried out, and the QA team validates the new artifact’s performance, security, and functional aspects.

• If critical bugs or issues are identified in the QA process, the developers are requested to fix them. If not, the artifacts are in a production-ready state and enter the deployment pipeline.

• The deployment pipeline provisions pre-production or the production environment using GitOps— IaC scripts, and the defined deployment process (Canary-releases, Blue-green, Shadow, Rolling, A/B Testing, etcetera). Ideally, it’s a good practice to test all aspects of software applications in a pre-production environment for improved software reliability.

• At last, the artifacts get deployed to multiple servers & regions to reduce redundancy. After all, multi-cloud DevOps is the new norm.

• Thereafter, the monitoring, logging & alert systems get into action to detect any anomalies in the software performance, and trigger alerts to the concerned teams for timely troubleshooting the issues & lowering the risks.

• If everything goes well, it is considered a successful deployment.

• Else, if there are critical incidents, performance loopholes, outages, or security risks, often the deployment is rolled back, reverting to the last working artifact. Such deployments don’t qualify as successful.

Now, many engineering leaders calculate total deployments (successful or unsuccessful) in a given time frame as deployment frequency. Some exclude rollbacks and consider only successfully deployed artifacts to calculate deployment frequency. 

But what’s the right approach to measuring deployment frequency? Wouldn’t considering unsuccessful deployment artifacts to calculate deployment frequency skew the performance report and paint a misleading narrative? What is the point if the deployment frequency DORA metric is way too good, but the business KPIs are bathing in red?

Well, keep reading and we shall figure that out. But a pertaining question at the moment is whether we should measure deployment frequency at all.

Why Is Deployment Frequency Important?

The benefits of measuring deployment frequency transcends beyond gauging software delivery speed or operational efficiency. When used right, it not only helps you enhance the overall SDLC processes but also improves business outcomes-

Deployment frequency facilitates agility, adaptability, and quick value delivery

High deployment frequency, with the right engineering practices in place (rigorous testing, and code reviews) would mean feedback-driven rapid value delivery to end users through iterative software development cycles. 

Again, the gambit here is that an increase in deployment frequency shouldn’t deteriorate reliability metrics. 

• MTTF- how long software systems operate without failing, MTBF- the average time between system failure, System Uptime, Error Recovery Rate, and Test Success Rate shouldn’t decrease with an increase in deployment frequency. 

• Similarly, Failure Density, MTTR - how long it takes to fix system failures, Incident Rate, and Open Tickets metrics shouldn’t spike up.

1. Deployment Frequency Helps in Effective Risk Mitigation

High deployment frequency, could also mean quick roll-out of security patches, and bug fixes, and thus help in timely mitigation of software vulnerabilities. The lower the batch sizes, the higher the deployment frequency. Hence, it is easily manageable. 

Continuous deployments mean rapid feedback from end customers. Thus, issues can be detected in almost real-time. You can pinpoint specific deployments that might have triggered the problem. And developers too can work on it effectively, as they might have the context fresh in their memory. 

This is not possible in case of large and infrequent deployments i.e., low deployment frequency, where you identify issues after weeks of time, and probably the developers would have lost the context, and may even have switched teams or companies (thanks to the great resignation waves). 

2. Deployment Frequency Can Help Improve Business KPIs

Frequent deployments, feedback-driven rapid value delivery, quick issue remediation, etcetera enhance the overall user experience. 

It also improves the innovation quotient of your organization and helps you capture a larger share of the market. 

Besides, continuous shipping of new features to the screens in the hand not only delights users, but also improves user engagement, brand perception, and customer loyalty. And of course, improved user satisfaction is followed by more business and higher revenue. 

Should You Measure Deployment Frequency to Benchmark Your DevOps Performance?

If utilized right, deployment frequency does allow you to gain greater visibility into your engineering processes. It helps you assess how efficiently & effectively you develop, deliver, and operate software applications. 

But you can’t rely on deployment frequency as a sole engineering performance metric. 

• You need to look at it in tandem with other software delivery metrics to evaluate your DevOps performance & engineering efficiency. 

• And then, as highlighted in Google’s State of DevOps 2022 report, you also need to map DevOps performance metrics (esp deployment frequency) to the bottom line and see how it impacts the operational performance i.e., reliability, and subsequently organizational performance i.e., better user experience, upsells, revenue breaching the highs, etcetera. 

The same report says-

“We have evidence that suggests that delivery performance can be detrimental to organizational performance if not paired with strong operational performance”.

So, a sensible approach would be to definitely consider deployment frequency to assess your engineering efficiency, but you must also factor in the engineering metrics that reflect software reliability— 

• Mean Time To Failure (MTTF) 

• Mean Time Between Failures (MTBF)

• Mean Time To Repair (MTTR)

• System Uptime 

• Failure Density

• Error/Incident Rate 

• Error Recovery Rate 

• Code Coverage

• Test Success Rate 

• Failover Time 

• Open Tickets (issues), etcetera.

The goal is to find the right balance between software reliability metrics & deployment frequency. It’s about both— speed & quality.

How to Measure Deployment Frequency?

Now, we understand that measuring software delivery metrics is not to do gorilla-style chest-thumping or boasting about high deployment frequency, but to improve your overall engineering health and business outcomes. So, it doesn’t make sense to include rollbacks in calculating deployment frequency. Ideally, you should only include successful deployments while measuring deployment frequency.

A question that emanates is how to measure deployment frequency. Should it be done manually? Or, are there tools to measure deployment frequency?

Well, you can manually count the number of successful deployments by delving into the logs history of your CI/CD tools. Divide it by the number of days, weeks, or months depending on how you’re calculating deployment frequency. 

But it is only feasible for a very small team of developers. And it is NOT RECOMMENDED. Yes, in bold & caps. Because then there are high chance that bias will be introduced for obvious incentives. Besides, manually you can’t measure and map deployment frequency to other software delivery & reliability metrics.

A better alternative is to use an engineering analytics tool (say ‘Hi’ to Hatica). Hatica is an engineering analytics tool that integrates well with your existing SDLC tech stack, aka toolchain and helps you gain complete visibility into your engineering processes by tracking 130+ engineering KPIs & metrics. 

Hatica not only helps you gain insights into your engineering health but also enables you to optimize your processes for enhanced developer well-being by enabling you to keep a tab on developer experience (devEx) metrics like developer burnout, coding days, maker time, etcetera. 

Anyway, here’s how ideally deployment frequency gets calculated automatically using Hatica.

• Hatica integrates with your CI/CD tool chain to track events like code commits, build initiations, testing, and deployments. This can be done using ping, and poll mechanisms for event tracking. 

• The deployment data is recorded in real-time, and based on how you configure it, deployment frequency is calculated. It is computed by dividing the total count of deployments by the time interval i.e., a week or a month. 

• Next, the result gets sanitized by normalizing it against various variables like the number of working days in a week/month, unsuccessful deployments & rollbacks, deployments that resulted in service degradation, etcetera. Normalization is possible, because, as stated earlier, Hatica tracks 130+ engineering metrics. Without normalization, you would be often left with skewed information, aka data noise. 

• Additionally, Hatica also helps you collate the deployment frequency of specific development teams distributed across business verticals and geographical regions. This helps you further understand & improve the efficiency of your engineering teams at a granular level.

Here’s a Hatica insight to help you better understand how it works: Deployment tracking in Github.

Okay! So now we understand the nuances of deployment frequency, and we also know what tool to use for measuring deployment frequency, but how do you know if your deployment frequency is too low or good when compared to the industry?

What's An Ideal Deployment Frequency?

The answer is the usual cliche, ‘it depends’. 

Large organizations with 10s of applications in their suite of software solutions, and 100s of developers committing code will have a very high deployment frequency. 

As per the DevOps report 2022:

• High performering teams tend to deploy multiple times a day, and for them, yearly deployment frequency may look as good as 1460/year or 4/day.

Of course, the numbers will vary greatly from one team to another, but this is a normalized average using inferential predictive methods on top of latent constructs. 

You can zero in on what’s the ideal deployment frequency for your team by carefully balancing the software delivery metrics and system reliability metrics. There will be an inflection point beyond which if you attempt to increase the delivery frequency, it will negatively impact the reliability metrics. That’s where you would know, you have reached your DF cap.

How to Improve Deployment Frequency?

Well, let’s say you have low deployment frequency. Then the first step toward improving deployment frequency is to diagnose whether it is-

• Engineering problem (legacy infrastructure, complex deployment pipelines)

• Productivity problem (low coding days, maker time)

• Culture problem (not tracking & improving engineering metrics & business KPIs, lack of effective communication & collaboration)

• Talent problem (not having adequate number of developers in the team, skill adequacy) 

Accordingly, you’ll find the right solution to improve your deployment frequency and hence engineering efficiency. In general, here are some suggestions that will help you improve deployment frequency:

1. Refine Your Engineering Silos

For instance, if you’re on a monolithic architecture and follow the waterfall model of software development, then obviously, it is tough for you to release often. In such cases, you need to audit whether monolithic is a necessity for your business niche, or is it a bottleneck for you to compete in the market. And accordingly, you need to think if you should move to relatively newer SDLC methodologies- 

• 85% of organizations with 5000 or more employees say that they are using microservices architecture for one or more applications.

• 47% of the organizations state adopting DevOps & DevSecOps software development methodologies to improve engineering outcomes like faster time to market, improved code quality, security, and improved team collaboration & communication.

2. Go aggressive on Automation with CI/CD Tools

Build automated software delivery pipelines i.e., make use of continuous integration & continuous delivery platforms like Circle CI & Gitlab that listen to your version control system tools & code repositories to automatically trigger code review, artifact building, automation testing, and deployment processes.
This not only saves the development & operations teams time but also reduces human-introduced errors that may lead to deployment failure or cause unnecessary delays.
It also means that programmers have more time now to focus on developing features and fixing bugs.
Hence, embracing CI/CD tools can significantly help accelerate the deployment frequency.
Besides, you can easily roll back deployments in case anything goes wrong.

3. Reduce the Deployment Size, aka Batch Size

Beware, reducing batch size doesn’t mean pushing code that has no significant impact on the existing software. Feature toggling, infrastructure as code (IaC) modifications, and artifact re-deployment without any code changes are all considered non-production deployments.

Thus, these are not considered in calculating deployment frequency. Ideally, reducing batch size for improving deployment frequency means, one deployment should cater to one of the following:

• Unit new feature release

• Enhancement to existing feature

• Bug fixes

• Hot fixes

• Security updates

• Performance improvement

• Configuration changes like database configuration, or third-party API integration

• Regulatory compliance updates

4. Foster Agility

Inculcate an agile culture that promotes-

• Continuous software delivery

• Self-organizing and self-driven teams

• Shorter sprint lengths

• Embracing tools & practices that facilitate collaboration & effective communication across teams

• Iterative development with rapid feedback loops from the end customers.

5. Go Heavy on Automation Testing

Embrace automation testing at every SDLC stage. Especially in the deployment pipeline. This is to ensure that deployments are successful and won’t break the production environment resulting in outages.
In general, increased automation testing & improved code coverage help you detect issues early in the development cycle and help you ship only features and not bugs to the production. Thus, minimizes non-production deployments, and improves deployment frequency. 

6. Use an Engineering Analytics Tool

As mentioned earlier, Hatica tracks key engineering metrics and KPIs like Cycle Time, Code Churn, PR size, Comment count per PR, reviewer reaction time, deployment time, etcetera, which helps you spot inefficiencies into your engineering processes and mitigate the same.
For instance, if code churn is high, it means low-quality code is making it to production. Diving deep, it is possible that the review process is inefficient. Thus, you can fix the process inefficiency to improve the software delivery speed and quality.

7. Reduce Developer Friction to Increase Deployment Frequency

You can facilitate developers to be at their creative best by minimizing friction in the SDLC processes. Embrace GitOps (IaC tools) & containerized deployments to automate infrastructure orchestration and provisioning.
Empower developers with AI-powered coding tools, and invest in establishing an engineering culture that prioritizes code documentation, code reviews, pair programming, and developer well-being. To reduce developer friction, you can again make use of Hatica.

Maximize Software Delivery with Deployment Frequency

Deployment frequency is a highly useful & impactful software delivery metric when looked at in conjunction with the reliability metrics.
Harnessed right, it can guide you to improve your software delivery efficiency by enhancing the speed & quality of code you push to the production environment i.e., how frequently you deliver value to the end-users.

But it demands SDLC automation, and improvement in how effectively teams communicate & collaborate. In short, the way to improve and make the most of deployment frequency is to root for cultural transformation. But it isn’t that tough either. 

Let Hatica & engineering metrics be your light. Adhere to the tips shared in this insight and soon you will be toasting to the awesome deployment frequency numbers.

Read more insights on the Hatica Blog to build winning engineering teams.