Measuring Engineering Effectiveness as a Director of Engineering

How can we ship better software faster? 

One question that is encountered by every Director of Engineering in their conversations with both the ground team, and C-sec executives. The premise can be different but the question persists and often without a concrete answer.

These conversations often fall short of making any tangible difference -  primarily  because of a lack of quantifiable, leading, and actionable metrics crucial to spot bottlenecks, or map engineering efforts with business goals, or reduce development costs. 

This lack of data topped with lack of workflow visibility is antithetical to the role of an engineering director – who is directly responsible for tracking, monitoring, measuring engineering effectiveness and delivering for overall engineering success. 

Powerful leaders have historically backed their decisions with objective data, and that's where this blog comes in. In this blog we are going to walk you through key engineering KPIs and metrics that help define success for a Director of Engineering. This will give you a preview into metrics and actionable KPIs (derived from these metrics) that you must track and communicate to your team. We'll break down the blog into:

• How to define success with these engineering metrics by connecting the dots between data and team performance.

• Their impact on your engineering squad, and how data insights drive improvement.

• Objective tracking methods to leave no room for guesswork here

But, before we dive into the nitty-gritty of success metrics, let's take a quick detour to understand your role as a  DOE.

Director of Engineering - The Role - Bridging Gap Between Development Teams and Leadership

A director of engineering is the architect of any organization's software strategy, and truly the voice of engineering. 

But, how? 

They translate the impact of engineering efforts into business outcomes, keep the teams under their purview productive and aligned, and ultimately deliver on goals set by the executive leadership.

Stepping into the role means bringing alignment into engineering teams by breaking down work silos, ensuring everyone understands the project's roadmap, and bringing clarity, and communication, between engineering, and business teams. 

As much as alignment is a vital KPI for a DOE, they also have to juggle a high-wire act: Balancing delivery velocity, and software quality to produce bug-free, customer-centric and sustainable product in a given frame of time. The best of both worlds ensures that DOEs are able to maintain the pace of innovation, and keep their product ahead of the curve. 

Over and above that, a thriving team fueled by high focus time is a DOEs' greatest asset, and they are responsible for maintaining it consistently working towards people's health. They invest in developer experience, to build an engineering culture of high focus time to their devs and address burnout before it becomes a performance drain. 

The buck doesn’t stop here. The role of a Director of Engineering comes with its own share of operation hustle - from compliance, budgeting, hiring, to setting up processes and  ensuring best practices are followed in delivering high-quality work.

However, as exhilarating and fancy as the title sounds, being an engineering director isn’t easy. It's demanding, it's challenging, and it's constantly evolving. Given the constant pressures engineering teams face, the role of a DOE keeps getting more specialized, and evolved. 

This effectiveness in the delivery pipeline hinges on proactive visibility. Can you pilot a high-powered rocket, blindfolded? That's what managing the delivery pipeline feels like without a data-driven engineering culture. Without having a clear picture of where your SDLC is moving, DOEs will keep seeing high divergence in planned vs delivered tasks, impacting both quality and velocity of the code being shipped. 
And that’s where adopting a data-driven approach for engineering kicks in. So, here we put together the top software metrics that DOEs could track to make data-informed decisions, and define success for their engineering teams.

We have further classified these metrics into 4 Key categories (that we call Pillars of Engineering Productivity):

• Team Alignment

• Velocity

• Code Quality

• People and Process Health

Key Metrics - Team Alignment 

With the engineering team often being the biggest investment in your company, ensuring every dollar drives maximum impact is crucial, making it all the way important to align all teams involved in product development: 

Resource Allocation

The metric tracks the ROI of your engineering investments and how much of them is actually spent building business-critical roadmap vs hunting bugs, KTLO, and technical debt. It essentially ties engineering activities to business requirements.

The signal also helps you track the value created by each dollar, and whether a business initiative is turning into a cost center than profit, or investment center. 

Hatica’s resource allocation dashboard allows you to identify work/effort  breakdown ina a given sprint window across projects, task types, epics or custom built labels suited to your organizational needs. For example, DoEs can check the effort invested on bugs across projects through the bug filter in a given month. 

Cost Control

Once you have figured out the effort allocation, it is important to put a dollar figure on your engineering spend across all the effort and map it back to the ROI. 

While it's still easier to put a monetary value on visible costs of development like cloud infrastructure optimization, or money spent in licenses, and API, the hidden costs like technical debt, code churn, and SaaS sprawl remains mostly untrackable, and difficult to quantify. Moreover, it needs to be seen in tandem with team well-being, and long-term sustainability. 

Hatica does that for you by collating mull data from your toolstack, and measuring FTE efforts to calculate the total engineering cost of each project, or team. 

PS: We also deliver customizable cost capitalization reports with visibility into costs that can be treated as capex, and how to strategically place them for healthier balance sheets. Check here: Automating Software Capitalization for Engineering Teams with Hatica

Key Metrics - Velocity 

Velocity Metrics are a measure of how much work an engineering team can complete in a development cycle, aka iteration, sprint, and are often used by a Director of Engineering to diagnose roadblocks, adjust project scope, and estimate better sprint timelines. Here are crucial velocity metrics to measure engineering success: 

Developer Throughput 

It is a crucial metric to define delivery velocity and efficiency of an engineering team. Developer Throughput measures the new work, along with changed and refactored code to understand the volume of work produced by the team. The signal also lets you tie the team (managed and custom) and individual work with the overall development cycle. 

Qualitatively speaking, comparing throughput of your team across weeks also helps see any hidden pattern, identify gaps, and put a check on the health of the working environment, and overall engineering culture. The sharper the variations in your week over week throughput, you and your team are more likely to cater to the complex, and hidden inefficiencies as a team.  

Hatica measures throughput by tracking PRs created, and merged in a time period (date range of your choice, ofcourse). 

We also measure throughput per contributor, and active contributors involved in the process. 

My personal favorites are the Code Churn trend and Code Churn breakdown as per task-types. See image below.

Deployment Frequency

Deployment Frequency is the number of deployments to production. It also flags the effectiveness, and integrity of your release processes, and whether the current workflow is compatible with the engineering team size, structure, and how they generally deliver work. 

High deployment frequency, with the right engineering practices in place (rigorous testing, and code reviews) would mean feedback-driven rapid value delivery to end users through iterative software development cycles. However, lower deployment frequency could be an indicator of something broken or inefficient hindering the process across the development lifecycle. 

Hatica helps you keep a pulse on your release and deployment cycle by pulling data from your CI/CD tools to track deployments made in a period, and even rationing failed vs successful deployments.  

Cycle Time

Talking about the software development lifecycle, cycle time has always been #1 on every engineering team’s and engineering leader’s radar. As we know, cycle time is the time taken to release a feature, and measures your delivery flow at every step of the development process by tracking coding time, review time, and rework, and deployment time. The signal is a full visibility marker (from first commit to final deployment) and shows how feasible your software delivery timeline is, and the delivery trends underneath. 

Unfortunately, if you do not have a view into these developer workflows, the chances are you are going to walk off without being able to identify what’s causing your workflows to fall apart and eventually impacting overall cycle time.

Hatica helps you track cycle time across projects, repo, issues, teams, and timezones so you are always on top of your engineering efforts, and how the work is authentically flowing across the SDLC. 

Key Metrics - Software Quality  

Change Failure Rate 

It goes without saying that engineering teams are hard pressed for short timelines and the pressure to deliver fast is immense. However this fast delivery of software cannot come at the expense of quality. But it is a no hidden fact that engineering teams often get themselves into a situation where the code pushed to the production environment for deployment has to be rolled back and further investigated for apt fixes. 

In that sense Change failure rate highlights the ROI of your testing process by analyzing the percentage of deployments causing issues requiring rollbacks or fixes. With one metric, DoEs can quantify the quality, reliability and stability of the software delivered.

Thankfully, its empirically possible to calculate CFR through: 

CFR = (Count of deployments that resulted in degraded services / Total deployment count) * 100

But what counts as “degraded services”? Hatica tracks it as a percentage of changes that led to downtime, errors, or negative impact on users.

MTTR 

Once you have identified the change failure rate, the next is to resolve the bugs ensuring stability of the code. MTTR is one of the crucial DORA Metrics that measure your team’s efficiency of incident response i.e., average time it takes to restore services. 

Directors of engineering are the face of the engineering ecosystem in an organization, meaning they bear the brunt of downtimes, customer disapproval, and soaring opportunity costs that happen with every such incident. In 2023 alone engineering orgs lost upto $450,000 to every downtime that happened. 

So, it becomes imperative for you to track how fast your systems are back to a fully functional state, and have a minimal impact on customers. 

Key Metrics - People and Process Health 

Now that we have looked into key engineering metrics that define - Alignment, Velocity, Code Quality - success for a Director of Engineering and their team; let’s take a look into how they go on to build a healthy and productive work environment for their engineering teams!

In rapidly changing work dynamics, a Director of Engineering is not eluded to maintaining a healthy engineering culture within the team both wrt people and processes.

In this section, we are going to highlight key metrics that define people/process health for an engineering org.

Workload Distribution 

While this metric comes in more handy for an Engineering manager in their day-to-day work routines and daily standups, it empowers the director of engineering to keep a tab on individual workload, and how their efforts map to the overall business goals. 

Keenly tracking workload across your engineering cycle can help you focus on what truly matters, and how to deliver more on the roadmap and optimize time spent on shadow projects, unplanned work, and non-core tasks. This ensures that no individual or team is over burdened or swamped with workload exceeding the overall team capacity and all of this can be well taken into account at the time of work planning meetings.

Hatica facilitates data-driven visibility to allocate and balance workload sustainably by accounting for existing workshare, location/time zone, and IC’s strengths, and capabilities. We help teams to move beyond story point estimation, and use actionable data to allocate PR reviews, incident management or even interview load across engineers so they can balance productive time, and individual growth. 

Code Reviews

Workload distribution, Maker time along with other key metrics are good indicators of healthy processes being followed within the team ensuring healthy developers (and people) in the team. It is an inextricable process where one leads to another. 

When it comes to process health, the code review process has always been touted as one of the toughest challenges to handle. However, Hatica allows you to track the efficacy of your review cycle, by understanding the involvement of commit owner, and reviewer, identifying long PRs, and unreviewed but merged PRs hampering the integrity of your engineering workflow, and the overall software pipeline. 

It’s also a great collaboration metric to ensure your team has effective communication, high-level dynamics, and has equal command over speed, and thoroughness. 

Measuring engineering effectiveness is a never ending process (and at times a long fought battle) but an unavoidable one. However, with these data metrics for key pillars of engineering productivity - Alignment, Velocity, Code Quality and People/Process health - being available, Directors of Engineering are equipped with the right tools and information to take effective and informed decisions.

Bottom Line: Metric-Driven Engineering For Forward-Thinking Leaders 

Directors of engineering today are under constant pressure to optimize their processes, and delivery cycles. The race to stay ahead of the competition is real, and executives are leaving no stone unturned in making engineering deliver faster than ever. This percolates down to VPEs, and DOEs to move beyond gut feeling, and adopt data-driven decision making for sustainable, long-term, and feedback driven engineering ecosystems.   

For many years, they have struggled with finding meaningful ways to track, and measure engineering success, and show that stakeholders can rely on the team to deliver on time, and maintain all quality standards. 

We at Hatica want to change that by becoming your single source of truth of all things engineering, identifying areas of improvement, and delivering value through insightful, contextual, and actionable metrics proactively. 

It's not just damage control; it's engineering excellence, sculpted by data.

If you are a director of engineering struggling to measure engineering effectiveness for your team, please feel free to reach out to us and our productivity experts can help you guide your way through.