Ambuj Agrawal loves to call himself a Fractional CTO who consults tech companies to build and run effective engineering organizations. He was previously Head of Engineering at Ola and recently owned product charter along with engineering org at Zeta.
In a recent conversation Naomi Chopra, our Founder & CEO at Hatica had with Ambuj, we were able to dig deeper into the nuances of leading modern-day engineering teams which need to be agile, scalable, and innovate faster in this competitive landscape. His journey has been no less than a roller coaster and a mix of tough unique experiences which he lavishly shares with Naomi over a candid conversation.
“What do you need to succeed as an engineering leader today?”
The question has kept so many young (and senior) leaders awake.
20 years ago, the role of engineering leadership was largely defined by a mindset of preservation, controlled certainty, and even authority.
But just as everything in technology has evolved, so too has the role of an engineering leader.
As technology is changing at a breakneck speed, engineering leaders are trying to spearhead these changes with agility, and a data-driven mindset, while amplifying the performance of their people, and products.
Now, their role seems to converge more as architects, creating new value from a context of abundance, and discovery, while generating growth for all stakeholders.
For engineering leadership, there isn’t just any finish line.
We spoke with Ambuj Agrawal, Ex-VP of Product & Engineering at Zeta, an engineering leader for more than 20 years.
Ambuj has done it all he’s been an IC, a team lead, transitioned to an engineering manager, and got back again into the role of an IC. He later went on to lead Zeta as their VP of Product and Engineering, Ola Pay, and NNN.now as the Engineering Head.
In this deep dive, we dig into his engineering vision, getting buy-ins from executives, the radical need to structure engineering teams, and his candid take on the future of software engineering.
From an IC to a VP of Engineering: Taking The Leap
My journey started in a small setup, with 6-7 engineers in total. We started off, even setting up conference rooms, and wires with our own hands. Fast-forward to my last gig, I had a team of 240+ members working with me hand in hand. I have seen both sides of the story.
Back in 2008, I had a team of 30-odd people. I realized it was probably too soon for me. So I left it all and became an IC again. Working as an independent developer for years made me realize engineers need managers who can understand their perspective. And that's why I choose to move up in management.
I kept experimenting and observing– as an IC, pair programmer, architect, manager, and product engineer. There was a time when I was only delivering for engineering. Then, I started delivering for the business; I have seen and donned multiple hats in this career span.
Running an Engineering Team: Then vs Now
Back in the 2010s, we were more concerned about agile scorecards. It was important to know:
- What is the sprint backlog?
- How much is the sprint velocity, or is our sprint delivery effective?
Teams are now moving to more metric-driven engineering, and benchmarking their process to know what’s working and what’s not. Their conversations flow around concentrated engineering signals like the mean time to deploy or the mean time to resolve.
Now the whole engineering ecosystem is looking to find critical engineering metrics that directly correlate with team success.
For example, if a team’s merging time is 2 weeks when it should have been a few days, does it imply they are doing something wrong? So that's like checking the temperature of a person. A high temperature means there's something wrong. It might be a seasonal flu or Covid-19. But the data is telling something.
Once we reach a point of critically analyzing these signals, and symptoms holistically through our machines and tools, we can definitely direct good data-driven decision-making.
The Challenges That Continued and Remained
Businesses move fast, even in larger ships like Amazon, Salesforce, or SAP, and we definitely don’t work on 100% of the items we plan for. This is where shifting gears from the topmost to the bottommost is critical, especially to close the loose threads.
These threads aren’t tracked well and sometimes translate into different projections of velocity and team success. Another challenge for engineering teams here is to gauge the timeframe required to gather sufficient data.
It also puts a break on the effectiveness of decision-making as the top leadership brass need to consider the appropriate timing of the next set of actions based on the availability of adequate metrics.
Only when these challenges are resolved, engineering leadership can be truly enabled to make the next set of decisions.
This is, however, a 60,000 ft view. Now, if I talk from granular retrospectives, the top problems will be:
- Our sprint plan got changed in the middle
- We encountered ad-hoc priorities
- The product team kept getting new customer requirements
- Customer asked for a faster TAT on a specific problem
Engineering teams aren’t bothered by the new work they encounter, but when they work on something for 2 months, it changes and goes down the drain.
Imagine a physical setup, the civil engineering setup. You're setting up something. And then someone asks, throw it out, do something else. That’s a killjoy.
Here again, I’ll highlight the lack of data-driven development. Engineering leaders don’t have an inherent metric to track how long to wait, and why to wait. This misalignment creates friction between the management and the ground-level employee.
It then translates into a very pushy work culture, and even mistrust between management, and ground-level teams.
On Growing, and Structuring an Engineering Team
Do you think size impacts how engineering teams are run and led?
Let's imagine the evolution and anatomy of an engineering team.
It starts with having an idea, followed by a Proof of Concept (POC), and assembling a small, 1-pizza, 2-pizza team. We are now building, but the moment the rubber hits the road, we have customer challenges and expansion challenges. This is the point where we must strategize on growing amid unpredictability.
We start with creating a few parallel initiatives and keep branching them out as and when a challenge arises. That's how the whole entire thing starts doubling itself like an amoeba.
To tackle the problem, it's always important to focus on the 80-20: Find an easier, and faster way to reduce bugs, and cognitive overload for the engineering team.
Building from 0-1 is tough in its own way. But scaling from 1-10 comes with a whole new set of challenges that teams might be unprepared for. That is why I have always preferred sustainable growth over overnight exponential growth.
Coming to the latter part of how an engineering team should be structured, I believe it's a manifestation of how many parallel initiatives a leader wants to run.
We can know a lot about how an engineering team is structured through simple 2 or 3 questions:
- How many war rooms do they have in a month?
- How many initiatives are run from the CEO-CTO-CXOs office?
There’s a straightforward correlation between these answers and the structure of teams.
The evolution of engineering teams is like a stem cell, we evolve based on needs; sometimes we become a skin cell, other times a specialized cell–each role tailored to address specific requirements.
DevOps, ProdOps, SRE, and QA
In many engineering teams, backend, frontend, mobile, and QA are seen as one arm, and only DevOps is considered separately.
In my personal experience, QA should be separate, but the development and deployment department should be kept with one person. We can't just build and assume it works on our machines—it must run seamlessly from our computers to the customer's screen.
That “one” person should be responsible for reporting to the higher-ups in the organization and deciding whether a release should go live or not.
But one size doesn’t fit all. No single pattern works with all engineering teams. Amazon has a dedicated SRE team for code replication, but production solely lies with the engineering team.
Now, let's dig deeper into SRE and DevOps. Developers cannot work in isolation. We need IT-vetted libraries, internal tools, and various projects to import, and use. Moreover, all of us have a certain way to test, compile, and deploy code in an ephemeral environment or a dev staging environment.
Traditionally, the setup is seen as an extension of DevOps, but it should not. A dev should not assume that they can live within their ID, and just go with it.
Our code works as a small piece of the puzzle in a larger setup, and it is our responsibility to certify the code, so it not only runs beautifully in a CPU 512 MB docker, but gets delivered to DevOps hassle-free.
We definitely need a separate ProdOps to ensure nothing brings the system down.
Leading Engineering Through Experiments
Our narratives of leading, and building effective engineering teams are based on observations accumulated over the period of years.
In my team, I've had individuals with 25-26 years of experience from established places like Yodlee and Yahoo, bringing diverse leadership experiences. Meanwhile, I've also worked with professionals from hyper-growth startups. What succeeded for one cohort doesn’t always work for the other.
In my setups, I started treating management like engineering—a continuous experiment. We started proposing a hypothesis with guidelines rather than assumptions, tested it, and then validated the results.
It's a process where we may or may not face favorable odds. But conclusions should only be drawn when concrete, objective results are right in front of us.
In this metric-driven business, how many initiatives you are running sometimes cannot be controlled. The magic ingredient is to track them and understand the breakdown of your team’s efforts spent on these initiatives. That’s how your team can move fast, and experiment faster. Unfortunately, we have failed to arrive here effectively in the past.
Taking Buy-Ins for Engineering Experiments
This is a tough one – to get a buy-in for an engineering experiment.
Absolutely, you can't impose decisions on your team; it begins with self-conviction. Identify the right stakeholders, and understand their motives and the problems they aim to solve. Timing is crucial – be ready when the opportunity aligns when stakeholders resonate with the problem.
I have seen people failing miserably in their projects because they are hell-bent on securing unanimous buy-in. One thing we need to understand about engineering leadership is–we cannot do a 100% collective buy-in, decisions won't be made otherwise.
It's always about enabling leadership with your vision, the execution, and the transformation it can bring to the entire engineering workflow, team dynamics, and overall processes.
With rapid innovation cycles and the need to push things to production faster than the competition, leadership cycles have become shorter. But, at the same time, they too want to go through multiple learning points before agreeing to engage in an engineering experiment.
Hence, there is a paramount need to educate the leadership as much as enable the engineering team with the right tools and mindset…and we need to accelerate the process with empathy.
On Metric-Driven Engineering
As a natural evolution, we've become more metric-driven than ever. Engineering teams today are using metrics to gauge team alignment, code quality, process efficiency, and whether they meet industry standards for delivery speed.
A decade back, engineering teams focused more on sticking to the processes they deemed fit. Today, the cognitive bias has shifted towards measuring the process effectiveness using the right metrics and tied Key Performance Indicators (KPIs). Teams use these metrics as a north star to stay on the right track.
Engineering teams must move beyond measuring mere system efficiency, or the depth of engineering processes. Now, more than ever, we need to keep a vigilant eye on the overall health of the SDLC, and engineering well-being.
Defining "the right metric" is indeed a challenge. While every engineering team has its fundamental or first-principal metrics, these often become less precise and relevant in complex projects.
The key is balance– to have a rich mix of objective, measurable metrics coming from both top-tier and ground-level engineers.
The Biggest Barriers to Becoming Data-Driven
Engineering leadership craves data-driven development. Unfortunately, even without the presence of contextual data, they must act for business progress. To capture this dataset and future-proof their decision-making process, a leader has to be willing to get everybody on board and create an action plan from scratch.
The plan, though, can slip at two levels despite all the instilled efforts. It stalls either due to the lack of the right tools in capturing datasets, or because the team lacks insights into how to collect, and consume the avalanche of data without hampering development velocity.
In some cases, the challenge isn't collecting data but the trust teams can put in a metric. A lot of times, these metrics falter on precision and accuracy. Engineers operate in binary terms – if a system is up, it's up; if it's down, it's down. Achieving precision in system uptime, even up to 5 or 6 decimal points, is standard.
However, can we attain similar trustworthiness, and precision even to just one decimal point, in matters related to people? I am still trying to be sure about that. There's definitely a lack of consensus among engineering leadership on a standard error range or scope of acceptance in metric measurement.
Also, data-driven engineering is a cultural shift. If you truly want it to work for your team, both sides of the house must be willing to cooperate in the behavioral modifications necessary to effect such change. The shift has to stem from grassroots efforts; and without buy-in at the engineer's level, the powerful idea of doing data-driven development may lose momentum.
Only a handful of decision-makers actively work to establish data-informed processes or use tools for measurement.
So, even if data exists, willingness does not.
This is why many people follow the status quo rather than making the effort to integrate metrics into their engineering workflows.
Metric-driven engineering culture should come from the bottom-up, just as much as from the top-down. To break all these silos, engineering leaders must create a participatory culture where individual contributors can voice their concerns without the top tier constantly asking. There! We build a movement with rhythm and self-organization.
The One Metric That Makes All The Difference
Meantime for maximum throughput achieved.
Once you start an initiative, it cannot reach maximum throughput on a linear scale. It's akin to an engine; once ignited, you keep adjusting the gear for optimal performance. Success, here, is measured by how quickly the engine attains maximum speed on the changed gear.
For engineering teams, the critical metric is the average time taken to achieve maximum work output after addressing an event, such as a failure or incident. This "meantime to achieve maximum throughput post a change" serves as a tangible measure of the team's responsiveness and adaptability during disruptions.
Helping Engineering Teams Embrace Cultural Shifts
Small wins have the power to bring greater cultural transformation.
Keep taking baby steps in the right direction, and over time you’ll be amazed by the progress your engineering team has achieved.
I’ll take the example of GitHub here. GitHub literally forces and pushes devs to write a commit message with every push. While not foolproof, this practice has nudged developers into the habit of writing comments. And I have tried to follow the methodology as an engineering leader.
One time, I asked my ICs to update Jira tickets before coming on the standup. The team had their reservations—What's in it for me? Will it streamline our processes?
I made a promise to my team: if they provide detailed updates on their ticket progress, blockers, and plans, we'd shorten our standups from 45 minutes to maybe 10 minutes. The plan worked, my team and I, we all delivered on our commitments. Now, the ICs had more coding and grooming time, and our standups too turned into productive catchups!
As a leader, it's crucial that I commit to something and deliver on it promptly. This sets the expectation right for my team too.
Orchestrating Change from the Bottom Up
In one of my last stints, I was overseeing a large engineering team. There were sub-teams as well, let’s say Team A vs Team B.
Team A took the initiative to deploy their own engineering metrics tools, even bringing them into their appraisal discussions. They had well-defined goals, associated KPIs, and local metrics for code push, review time, and deployment success.
This proactive approach left Team B feeling somewhat left out. To address this, Team B started exploring empirical ways to track their progress. Once the tracking process was shared and embraced by other teams, we could see a real bottom-up change taking place in the engineering ecosystem.
The motivation has to come from within developers. Engineers need to introspect why they invest their time and effort into something, rather than doing something to satisfy the leadership. This genuine sense of ownership and purpose makes all the difference!
During my time at Amazon, we had a centralized directory displaying the hierarchy up to Jeff Bezos, along with scores based on project health. All I did was show my team the page, saying, "Guys, here's how we fare in these parameters." I didn't have to do much else. The team autonomously reorganized themselves, generated proposals, and executed them. In no time, our numbers improved—it was a collective team win! I did nothing.
The team’s willingness to learn and improvise brought cultural change.
The Future Of Engineering Leadership
It’s hilarious that being engineers and students of math, we lack empirical methods to measure our own work. Sadly, this absence of inductivism in our work is the reason why engineering teams face misalignment, or and often miss the mark.
As Fredrick Brooks says:
Adding people to an already late software project makes it even later.
I cannot agree more.
The idea of generating reports with an excess of reporting tools and an overload of metrics solely for the sake of reporting needs to change. Instead, our engineering efforts should center on identifying touchpoints where our systems/processes are faltering, and understanding the root causes of delays to improve the overall software delivery process.
In the next 10 years, this entire system of “reporting” to management will topple. Future belongs to organizations and tools that can seamlessly instrument data from the bottom up, and convert this raw data into actionable insights without friction.
To insinuate the change, software development analytics tools too should be designed and implemented with a primary focus on improving delivery and quality at its core, and move on from the old-age narrative of merely reporting engineering effectiveness.
One Message For Engineering Teams
Find a metric that is frictionless to capture. Whether it's deployment time, mean time to resolve, deployment frequency, or as I mentioned, mean time to maximize throughput. Focus on one that is transparent, precise, and consistently trackable for the future.
Stick to this metric and ask your team to solve their reference problems in a backward fashion. If their output is slow, don't dive into abstractions.
Instead measure just one signal: How often they deploy. Let's say it's once every two weeks. Why, once every 2 weeks? Why not once a week?
Run team meetings dedicated to this single topic, nothing else. Use these catchups to understand the reasons behind delays– slow machines, unstable staging environments, or delayed contracts. Just focus on that metric, and work with each IC to drive continuous improvement.
There, you just created an atomic habit for your team! Since you have now established a positive cycle by solving one metric– devs will naturally step up as and when new metrics are introduced, and solved.
That's what building a culture means. It's training yourself to consistently do the right things.
