#56 Resolving DORA Metrics Mistakes

We're already well into 2024 and it’s sad that people still have enough fuel to complain about various aspects of their engineering life.

DORA seems to be turning into one of those problem areas.

Not at every organization, but some places are turning it into a case of “hitting metrics” without caring for the underlying capabilities and conversations.

Nathen Harvey is no stranger to this problem.

He used to talk a lot about SRE at Google as a developer advocate. Then, he became the lead advocate for DORA when Google acquired it in 2018.

His focus has been on questions like:

How do we help teams get better at delivering and operating software?

You and I can agree that this is an important question to ask.

I’d listen to what he has to say about DORA because he’s got a wealth of experience behind him, having also run community engineering at Chef Software.

Before we continue, let’s explore What is DORA? in Nathen’s (paraphrased) words:

DORA is a software research program that's been running since 2015.

This research program looks to figure out:

How do teams get good at delivering, operating, building, and running software?

The researchers were able to draw out the concept of the metrics based on correlating teams that have good technology practices with highly robust software delivery outcomes.

They found that this positively impacted organizational outcomes like profitability, revenue, and customer satisfaction.

Essentially, all those things that matter to the business.

One of the challenges the researchers found over the last decade was working out: how do you measure something like software delivery?

It's not the same as a factory system where you can go and count the widgets that we're delivering necessarily.

The unfortunate problem is that the factory mindset I think still leaks in. I’ve personally noted some silly metrics over the years like lines of code.

Imagine being asked constantly: “How many lines of code did you write this week?”

You might not have to imagine. It might be a reality for you.

DORA’s researchers agreed that the factory mode of metrics cannot determine whether or not you are a productive engineer.

They settled on and validated 4 key measures for software delivery performance.

Nathen elaborated that 2 of these measures look at throughput:

[Those] two [that] look at throughput really ask two questions:

* How long does it take for a change of any kind, whether it's a code change, configuration change, whatever, a change to go from the developer's workstation. right through to production?

And then the second question on throughput is:

* How frequently are you updating production?

In plain English, these 2 metrics are:

* Deployment Frequency. How often code is deployed to production? This metric reflects the team's ability to deliver new features or updates quickly.

* Lead Time for Changes: Measures the time it takes from code being committed to being deployed to production.

Nathen recounted his experience of working at organizations that differed in how often they update production from once every six months to multiple times a day.

They're both very different types of organizations, so their perspective on throughput metrics will be wildly different.

This has some implications for the speed of software delivery.

Of course, everyone wants to move faster, but there’s this other thing that comes in and that's stability.

And so, the other two stability-oriented metrics look at:

What happens when you do update production and... something's gone horribly wrong. “Yeah, we need to roll that back quickly or push a hot fix.”

In plain English, they are:

* Change Failure Rate: Measures the percentage of deployments that cause a failure in production (e.g., outages, bugs).

* Failed Deployment Recovery Time: Measures how long it takes to recover from a failure in production.

You might be thinking the same thing as me. These stability metrics might be a lot more interesting to reliability folks than the first 2 throughput metrics.

But keep in mind, it’s about balancing all 4 metrics.

Nathen believes it’s fair to say today that across many organizations, they look at these concepts of throughput and stability as tradeoffs of one another.

We can either be fast or we can be stable.

But the interesting thing that the DORA researchers have learned from their decade of collecting data is that throughput and stability aren't trade-offs of one another.

They tend to move together. They’ve seen organizations of every shap