3 Key Metrics to Measure Engineering Success
Two decades of leading software teams taught me that leaders tend to put too much emphasis on engineering metrics alone, without thinking about the bigger picture.
After speaking to a range of engineering industry leaders and analyzing millions of jobs processed from software teams across the globe, my team found that the most insightful and relevant metrics fall into three categories:
- Velocity metrics
- Morale metrics
- Business metrics
Velocity Metrics
Engineering velocity metrics measure the speed and efficiency of software delivery pipelines—it’s the category that a lot of managers pay attention to. Velocity metrics are critical in helping teams identify slowdowns and optimize their overall performance.
Some of the most common velocity metrics are:
- Throughput
- Change lead time
- Sprint velocity
- Duration
- Mean time-to-recovery
- Success rate
Customize Throughput to Your Team Structure
“Throughput helps us identify and understand speed, but the throughput of a system is also bound by its primary constraint,” says Moses Mendoza, former head of engineering at data processing and review platform Zapproved. “Throughput will show you what the slowest issue is in a chain of events, but it won’t show you how to fix it to speed up your work.”
Graeme Harvey, an engineering manager on my team, emphasizes that it’s important for all engineers to customize throughput measurement.
“Because our team practices pair programming, measuring throughput isn’t something that can be tied to an individual’s productivity,” Harvey says.
When it comes to throughput, engineers on Harvey’s team optimize for the whole team rather than the individual. Pair programming might feel like it’s impeding the progress of an individual, but it refocuses energy on what’s most important for the team and, ultimately, the business.
Measure Lead Time to Identify Pipeline Quality
“One way to measure change lead time is to look at how long it takes to get an idea out and for the idea to go full cycle,” says Alex Bilmes, former VP of growth at software configuration tool Puppet. “The other way is to look at deployment lead time, which measures how long it takes to get to production after a developer has pushed the change to production.”
Full change lead time will point out issues in the depth of your backlog. Deployment lead time is more likely to show the quality of your pipelines and tooling.
Sprint Velocity can Help Plan Workload
The amount of work a team can tackle during a single sprint is known as sprint velocity, and it can be used for planning and measuring team performance.
Tom Forlini, CTO at video conferencing platform Livestorm, dives even deeper when measuring velocity. He focuses on three smaller metrics:
- Number of issues and story points
- Number of issues done versus planned
- The percentage of issues distributed depending on type
“Livestorm engineers work on two-week sprints and have 50 story points per sprint,” says Forlini. “We track the number of issues done versus planned because it gives us a good indication of the sprint planning quality between product and tech.”
His team then looks at the percentage of issues by type. “When a sprint contains only new feature issues, we know from the start that it might be quite a challenging sprint to tackle,” Forlini adds.
Morale Metrics
Morale metrics are, quite possibly, the most overlooked metrics category in engineering. They tell you how engineers feel about the quality of their work and their job satisfaction, which is a big factor in retaining employees. Keeping retention high means keeping morale high.
Some common morale metrics are:
- Individual and team morale
- Code quality confidence
- Direct feedback from engineers
At Zapproved, Mendoza tracks morale to better monitor employee retention. “We measured morale at work using surveys, having conversations and asking managers to dive deeper in one-on-one meetings to find out how employees felt,” he says.
Focus on Confidence Over Coverage
“As we measured code quality confidence over two or three sprints, if we saw code quality tanking, it meant something was wrong with how the teams planned their individual investment with the work,” says Mendoza.
The engineering managers that I lead also measure work using confidence as a metric. Harvey’s team focuses on delivering small iterations quickly. This provides the confidence that the team is building something of quality, nothing is broken and they’ve made the right choices in building tools for the dashboard.
Business Metrics
Everything an engineer does should propel the company forward. That’s why it’s also essential to track business metrics.
Some common business metrics include:
- Company growth
- Funnel metrics
- End-user value
Track how Fast Your Business is Scaling
Business metrics allow your team to accommodate for user growth. According to Yixin Zhu, formerly of Uber, while it’s essential to look at engineering execution metrics, it’s also important to be dialed in to the business’s goals and to measure company growth.
As Uber grew exponentially, tracking business metrics was crucial for Zhu’s engineering team to succeed. “When you’re talking about doubling every six months, you have to be tracking that to know what you need to build, what degradations to expect, how many data centers you need, how many boxes, etc.,” Zhu says.
In short, engineers have to keep an eye on real-time business metrics to project and plan accurately. “You have to be proactive,” Zhu adds.
How to Get the Most Value out of Your Engineering Efforts
Define business goals. Your engineers should know what your business is trying to accomplish. When engineers understand the direction of the company, they understand what they’re working toward.
Review metrics often. Review metrics at the start of every meeting so the team is aligned on what you’re doing right and what needs improvement.
Start small. If you’re intentional about tracking simple metrics, experimenting with different approaches and seeing those metrics evolve, you can add more metrics over time.
Look beyond execution metrics. The success of an engineering team hinges on more than engineering metrics. Pairing engineering metrics with business metrics will help you understand why you’re building what you’re building.
To hear more about cloud-native topics, join the Cloud Native Computing Foundation and cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021