agile metrics and measurements2

25 Agile metrics and measurements - Measure one damn thing!

Share on linkedin
Share on twitter
Share on facebook
Share on email

You and I know it: There are loads of agile metrics and measurements. But you always have to keep one thing in mind:

“Tell me how you measure me, and I will tell you how I will behave.”

Dr. Eli Goldratt

Therefore, one question has to be asked: Are all of these metrics relevant? And then there is this other citation:

“Simplicity is the soul of efficiency.”

Austin Freeman

So the next question arises: If we would want to measure agile as simply as possible, how would we measure it? What agile metrics and measurements matter most? Or if we would measure only one damn thing, what would it be?

Don’t get me wrong: Of course you can have more complicated models for agile metrics, such as the Agilometer. But I was fascinated by the goal to measure one thing only. Therefore, this article investigates this goal. Agilometer. But I find the thought experiment of only wanting to measure one KPI quite exciting. Therefore, this article pursues this goal. 

We use the following agile metrics as a basis.

Agile KPI

Explanation

Based on

Correlation to future value

Easiness to measure

Output

Sprint Burndown chart

Shows the progress within the sprint towards reaching the sprint goal

Subjectively estimated effort

Low 🛑

Partly manual ⚠️

Productivity ⛹🏾‍♀️

Velocity

Indication of how much work has been done during a sprint

Sum of subjectively estimated storypoints (completely delivered user stories)

Low 🛑

Partly manual ⚠️

Productivity ⛹🏾‍♀️

Epic and Release Burndown

Track progress over a large work body towards an epic

Subjectively estimated storypoints

Medium ⚠️

Partly manual ⚠️

Productivity ⛹🏾‍♀️

Control Chart

Time duration from the “in progress” to “complete” status of tasks

Objective time measurement

Low 🛑

Automated

Productivity ⛹🏾‍♀️

Cumulative Flow Diagram

Number of issues/tasks/work items in your backlog compared to time (left)

Objective Number of issues/tasks/work items & time

Medium ⚠️

Automated

Productivity ⛹🏾‍♀️

Lead Time

Time period between making a request for delivering a product and the actual delivery of the product

Objective timeline

Medium ⚠️

Automated

Productivity ⛹🏾‍♀️

Value Delivered

Customer value of a requirement in $ or points.

Subjectively estimated and assigned by a product manager / owner

Medium ⚠️

Partly manual ⚠️

Value 🍭

Net Promoter Score

Willingness to recommend your product

Subjectively by customer

High

Partly manual & tools needed 🛑

Value 🍭

Work Item Age

Indication for time that passes between the start and completion of the current task

Objectively by a tool

Low 🛑

Automated

Productivity ⛹🏾‍♀️

Throughput

Average tasks processed in a time unit

Objectively by a tool

Medium ⚠️

Automated

Productivity ⛹🏾‍♀️

Blocked Time

Number and duration of tasks that cannot be proceeded for dependency reasons

Objectively by a tool

Medium ⚠️

Automated

Predictability 🎲

Escaped Defects

Indication for number of bugs when a release enters production

Objectively by a tool

Medium ⚠️

Automated

Quality 🏆

Failed Deployments

Number of overall deployments

Objectively by a tool

Medium ⚠️

Automated

Quality 🏆

Code Coverage

Degree to which the source code of a program is executed (indicates software quality)

Objectively by a tool

Medium ⚠️

Automated

Quality 🏆

Quality Intelligence

Helps to identify recent code changes (indicates software quality)

Objectively by a tool

Medium ⚠️

Automated

Quality 🏆

Cycle time

Work in progress divided with the average completion rate of a task (indicates workflow health)

Objectively by a tool

Medium ⚠️

Automated

Productivity ⛹🏾‍♀️

Customer satisfaction

Satisfaction of the customer / user with your product or service

Subjectively by customer

High

Partly manual & tools needed 🛑

Value 🍭

Planned-to-done-ratio

Ratio between planned vs. delivered user stories

Objectively by a tool

Medium ⚠️

Automated

Predictability 🎲

Usage index

Shows what features customers are using & their intensity

Objectively by a tool

High

Tools needed ⚠️

Value 🍭

Innovation rate

Capacity of a team to build value-driven features versus non-discretionary work like detect fixes and application support

Objectively by a tool

Medium ⚠️

Partly manual & tools needed 🛑

Quality 🏆

Business value

The value of your business

Objectively or subjectively by your stakeholders

Medium ⚠️

Tools needed ⚠️

Value 🍭

Psychological safety

Probability of your employees to speak up freely

Subjectively through a (survey) tool

High

Partly manual & tools needed 🛑

Culture 👯‍♀️

Purpose

A bold affirmation of your companies (or departments etc.) reason for being in business

Subjectively through a (survey) tool

High

Partly manual & tools needed 🛑

Culture 👯‍♀️

Vision

An emotional image of a better version of your company (or department etc.) in the future

Subjectively through a (survey) tool

High

Partly manual & tools needed 🛑

Culture 👯‍♀️

Employee satisfaction / happiness

Employees satisfaction regarding their work

Subjectively through a (survey) tool

High

Partly manual & tools needed 🛑

Culture 👯‍♀️

Do we even want agile metrics and measurements?

Before we go deeper, one thing that I regularly hear or read on LinkedIn when I talk to agile coaches, scrum masters or scaled agile frameworks consultants: Do you even want to measure agile?

“There are lies, there are damn lies and then there are statistics.”

Mark Twain

Mark Twain is a little dramatic about it. But he has a point. A point that Albert Einstein put in a Nutshell.

Not everything that counts can be counted, and not everything that can be counted counts.

Albert Einstein

What is "KPIs" in agile - Velocity, Burndown charts, number of failed deployments? Are these metrics key for agile success? I doubt that. 

But abandon all metrics? That would also be a mistake.

In today’s world, economical circumstances change fast. In good times, decision makers start thinking about an agile transformation because they have the resources and safety to do so. And in bad times? 

In bad times, senior executives will fall back into traditional thinking. They will move to old behavioral patterns: e.g., top down decisions that are not compatible with modern agile thinking.

If you would accept not having agile metrics in bad times, you shouldn’t even start an agile transformation. Because at that point, any progress made within that transformation would be destroyed.

Therefore, the only chance that scaled agile frameworks survive difficult times is by beating the system with its own weapons: by providing metrics. Metrics that help to manage in times of uncertainty.

And I am sure about it: there are agile metrics that matter. 

“In retrospect, I consider it one of my biggest mistakes to have always categorically rejected metrics for the agile transformation.“

Marcus Raitner

Einstein also says it in his citation: There are things that count. Those are the ones we are looking for in this article.

Agile metrics and measurements - What makes a good metric?

Let’s say, the question you ask in your daily stand-up typically is: What did you achieve today? This is, therefore, how you “measure” progress in your team.

Good question, right? No, not right. This question makes you want to show you are busy. It pressures you to get your “To do list” done, so that you can proudly update on your metric: Yes, I was very busy in the last 24 hours!

Getting your “To do list” done, is good, is it not? Well, it depends. Far more important is something else: reaching your goal. This is typically - in case of agile teams - delivering value to your customers.

Agile KPIs - how to measure agile success

Therefore, a better question (or metric) within your Daily Stand-up would be: “How did I help my team or organization achieve our (sprint) goal in the last 24 hours?”

Change the question (or the metric), change how people think and act: effective first, efficient second - to use the words of Peter Drucker. Peter Drucker accept.

Change the metric, change how people think and act.

To use Einstein’s wording, we have to find the one thing (if we want to measure one thing) that can be counted - and that really counts.

What does really count in an agile transformation?

How agile metrics and measurements should be viewed

The goal of your agile transformation is definitely not an agile transformation. Why do you do the agile transformation? Let’s try a “3-why’s technique” to understand that.

  • Why are you doing an agile transformation? Because you want to be more productive and efficient, even in times of uncertainty.

  • Why? Because you want to provide the maximum amount of value to your customer in the shortest amount of time.

  • Why? Because you want to survive long term as a business.

So the core reason for your agile transformation: you don’t want to end up like Blockbuster, ignoring industry trends and customer needs, unopen to change and finally running out of money.

You want to end up like Netflix, constantly evolving your business model around the core needs of your customer. More on the Blockbuster vs. Netflix here.

How can you translate that into measuring one thing in an agile transformation? , Well that’s tough.

What companies do instead: measuring parameters because they are easy to track. Because tools out there throw them at you. The probability that these are the right metrics is pretty low.

Because for example, it is not about improving sprint velocity. That is a common mistake: monitoring and measuring effort, or efficiency. Instead, it is about solving customer needs. 

Agile metrics that matter - a key takeaway

Having laid down all of this, if we want to measure one thing in agile, we have to rate the validity of this metric against one thing.

We will have to measure outputs, not outcomes. We have to measure our agile transformation by how much it helps us achieve our goal. We have to measure people not by their effort, but in terms of their contribution to a shared vision or goal - we have to measure by the value created for the customer!

If we want to measure by value, we have to deeply understand the customer’s needs.

For example, a railroad company has to understand that it is not in the railroad business. It has to understand that it is in the transportation business. Customers do not care if they are transported by train or by plane.

One other thing about agile metrics that matter - timing

One other thought is important here. At best, metrics depend on the timing and/or at what stage you are in your agile transformation. 

Let’s imagine that you already know that transforming to scaled agile frameworks or other agile working practices is the right move for you (That is by the way the first thing that many companies forget to check).

In that case, at the beginning of your agile transformation your focus should be on one thing: The agile mindset of the leadership team.

Is the leadership team really ready to change? Does it understand the implications of a transformation? Is it ready to be the first team of the company to implement agile methods by heart - e.g., Kanban & transparency, agile retrospectives & continuous self reflection?

Theoretically, an initial agile metric should focus on the "readiness of the leadership team". By the way, in this blog article my colleague Jean has 7 tips about the role of the leadership team in agile transformations.

The next most important question to ask in your transformation: Do we have the right processes in place to understand and monitor our customers’ needs continuously? This is where the next agile metric could focus on.

But stop. This does not help us reach the goal of this text - to measure one thing. a Thing to measure.

No. It serves to inspire you for your agile transformation.

So let us now have a look at what typical agile metrics there are and how much they correlate to the most important output your agile transformation is about: Value.

Agile metrics that matter - a ranking

You can find a table above with the agile metrics that matterwithin scaled agile frameworks and agile transformations. I ordered them by five areas, five goals that you generally want to reach through your agile transformation:

  • Customer value: Are you meeting the customer’s needs?
  • Predictability: Are you delivering on time with smooth processes?
  • Productivity: Are you getting more and more done at the same time and with the same resources?
  • Quality: Do you deliver a product free of bugs and other issues?
  • Culture: Are the people in your organization satisfied, continuously learning and do they feel free to innovate, making sure you can maintain your pace?

Agile KPIs - how to measure agile success

The table above also gives an indication of

  • how easy it is to measure the metric (based on my personal experience)
  • how much that metric is correlated to our core goal with a long term perspective: future customer value (based on my experience).

So which agile metric from the table above is the one?

Interesting to note is that there does not seem to be an agile metric that is easy to measure and provides the right kind of value. Damn it.

I would say that the “Easiness to measure” is not as important as “Correlation to future value”. Therefore, the most valid metrics seem to be “Usage Index”, “Customer Satisfaction” or the “Net Promoter Score”.

Scaled agile framework - Measure one thing!

It does not matter if you use the Scaled Agile Framework (SAFe®) to measure one thing, or a different framework to do your agile transformation.

Given these three metrics, what correlates most with future customer value is probably customer satisfaction. Or, to be more specific, top-box customer satisfaction. You can find more on that topic following this link.

Customer satisfaction probably also has the highest correlation of all of these metrics with the ROI of your agile transformation. It shows you whether to pivot, persevere or perish.

Scaled agile framework - Really measure one thing?

But stop. By focussing only on customer satisfaction, how do you make sure you are staying ahead of your competitors long term ? How do you make it possible that innovative and disruptive ideas can flourish in your company? Which is your long term goal...

Given these questions, I think we are getting back to the basics: to continuously provide value and to innovate around it, you need two other things: smooth agile processes and a great company culture.

Your company culture makes sure that employees feel psychologically safe , are open to fail, raise their voice and share their ideas. And your agile processes make sure you implement these ideas faster than your competitors.

Agile, like a leader, aims at people-first approach; putting people above things.
Vikram Verma

The following graphic illustrates this in an easy way. If value is your long term goal, then our input is your working methods times culture.

I once read here that “agile transformation requires a cultural change, not a process change.”

I disagree with that. It requires both.

Scaled agile framework - Measure one thing? Measure three things!

In conclusion, if you only want to measure one thing in the Scaled Agile Frameworks (SAFe®) or other frameworks, it would be customer satisfaction. It is the agile KPI to measure agile success. But I cannot recommend measuring only one thing - sorry for disappointing you.

If you really want to keep it as simple as possible with your metrics, I recommend that you measure a minimum of three things that will help you thrive long term:

  • Measure customer value - through customer satisfaction. 
  • Measure corporate culture - through psychological safety as an indicator for learning and innovation. 
  • Measure agile effectiveness - through planned-to-done-ratio as an indicator for how well you are able to incrementally deliver value.

After measuring, build. Then learn. And then iterate on your metrics...

That’s so easy, right? Nope. Are you serious about implementing agile frameworks throughout your department or company?

What is a good KPI in agile and can I do about it?

We are currently interviewing dozens of experts - release train engineers, agile coaches, Scaled Agile Framework consultants - around metrics in scaled agile frameworks and agile methods in general.

Based on these interviews, we will develop a quick and simple online video training that will help you implement agile metrics the right way.

Sign up for the free "Scaled Agile" video-training

So, if you are serious about your agile transformation, I recommend you fill in the form below so we can let you know about the online training as soon as we have it ready. See you then!

Articles you may also be interested in

Book a 20-minute online demo now and find out how you can use Echometer to promote agile work in your organization.