You and I know it: There are loads of agile metrics and measurements. But you always have to keep one thing in mind:
“Tell me how you measure me, and I will tell you how I will behave.”
Dr. Eli Goldratt
Therefore, one question has to be asked: Are all of these metrics relevant? And then there is this other citation:
“Simplicity is the soul of efficiency.”
Austin Freeman
So the next question arises: If we would want to measure agile as simply as possible, how would we measure it? What agile metrics and measurements matter most? Or if we would measure only one damn thing, what would it be?
Don’t get me wrong: Of course you can have more complicated models for agile metrics, such as the Agilometer. But I was fascinated by the goal to measure one thing only. Therefore, this article investigates this goal. Agilometer. But I find the thought experiment of only wanting to measure one KPI quite exciting. Therefore, this article pursues this goal.
We use the following agile metrics as a basis.
Agile KPI | Explanation | Based on | Correlation to future value | Easiness to measure | Output |
|---|---|---|---|---|---|
Sprint Burndown chart | Shows the progress within the sprint towards reaching the sprint goal | Subjectively estimated effort | Low 🛑 | Partly manual ⚠️ | Productivity ⛹🏾♀️ |
Velocity | Indication of how much work has been done during a sprint | Sum of subjectively estimated storypoints (completely delivered user stories) | Low 🛑 | Partly manual ⚠️ | Productivity ⛹🏾♀️ |
Epic and Release Burndown | Track progress over a large work body towards an epic | Subjectively estimated storypoints | Medium ⚠️ | Partly manual ⚠️ | Productivity ⛹🏾♀️ |
Control Chart | Time duration from the “in progress” to “complete” status of tasks | Objective time measurement | Low 🛑 | Automated ✅ | Productivity ⛹🏾♀️ |
Cumulative Flow Diagram | Number of issues/tasks/work items in your backlog compared to time (left) | Objective Number of issues/tasks/work items & time | Medium ⚠️ | Automated ✅ | Productivity ⛹🏾♀️ |
Lead Time | Time period between making a request for delivering a product and the actual delivery of the product | Objective timeline | Medium ⚠️ | Automated ✅ | Productivity ⛹🏾♀️ |
Value Delivered | Customer value of a requirement in $ or points. | Subjectively estimated and assigned by a product manager / owner | Medium ⚠️ | Partly manual ⚠️ | Value 🍭 |
Net Promoter Score | Willingness to recommend your product | Subjectively by customer | High ✅ | Partly manual & tools needed 🛑 | Value 🍭 |
Work Item Age | Indication for time that passes between the start and completion of the current task | Objectively by a tool | Low 🛑 | Automated ✅ | Productivity ⛹🏾♀️ |
Throughput | Average tasks processed in a time unit | Objectively by a tool | Medium ⚠️ | Automated ✅ | Productivity ⛹🏾♀️ |
Blocked Time | Number and duration of tasks that cannot be proceeded for dependency reasons | Objectively by a tool | Medium ⚠️ | Automated ✅ | Predictability 🎲 |
Escaped Defects | Indication for number of bugs when a release enters production | Objectively by a tool | Medium ⚠️ | Automated ✅ | Quality 🏆 |
Failed Deployments | Number of overall deployments | Objectively by a tool | Medium ⚠️ | Automated ✅ | Quality 🏆 |
Code Coverage | Degree to which the source code of a program is executed (indicates software quality) | Objectively by a tool | Medium ⚠️ | Automated ✅ | Quality 🏆 |
Quality Intelligence | Helps to identify recent code changes (indicates software quality) | Objectively by a tool | Medium ⚠️ | Automated ✅ | Quality 🏆 |
Cycle time | Work in progress divided with the average completion rate of a task (indicates workflow health) | Objectively by a tool | Medium ⚠️ | Automated ✅ | Productivity ⛹🏾♀️ |
Customer satisfaction | Satisfaction of the customer / user with your product or service | Subjectively by customer | High ✅ | Partly manual & tools needed 🛑 | Value 🍭 |
Planned-to-done-ratio | Ratio between planned vs. delivered user stories | Objectively by a tool | Medium ⚠️ | Automated ✅ | Predictability 🎲 |
Usage index | Shows what features customers are using & their intensity | Objectively by a tool | High ✅ | Tools needed ⚠️ | Value 🍭 |
Innovation rate | Capacity of a team to build value-driven features versus non-discretionary work like detect fixes and application support | Objectively by a tool | Medium ⚠️ | Partly manual & tools needed 🛑 | Quality 🏆 |
Business value | The value of your business | Objectively or subjectively by your stakeholders | Medium ⚠️ | Tools needed ⚠️ | Value 🍭 |
Psychological safety | Probability of your employees to speak up freely | Subjectively through a (survey) tool | High ✅ | Partly manual & tools needed 🛑 | Culture 👯♀️ |
Purpose | A bold affirmation of your companies (or departments etc.) reason for being in business | Subjectively through a (survey) tool | High ✅ | Partly manual & tools needed 🛑 | Culture 👯♀️ |
Vision | An emotional image of a better version of your company (or department etc.) in the future | Subjectively through a (survey) tool | High ✅ | Partly manual & tools needed 🛑 | Culture 👯♀️ |
Employee satisfaction / happiness | Employees satisfaction regarding their work | Subjectively through a (survey) tool | High ✅ | Partly manual & tools needed 🛑 | Culture 👯♀️ |
Do we even want agile metrics and measurements?
Before we go deeper, one thing that I regularly hear or read on LinkedIn when I talk to agile coaches, scrum masters or scaled agile frameworks consultants: Do you even want to measure agile?
“There are lies, there are damn lies and then there are statistics.”
Mark Twain
Mark Twain is a little dramatic about it. But he has a point. A point that Albert Einstein put in a Nutshell.
Not everything that counts can be counted, and not everything that can be counted counts.
Albert Einstein
What is "KPIs" in agile - Velocity, Burndown charts, number of failed deployments? Are these metrics key for agile success? I doubt that.
But abandon all metrics? That would also be a mistake.
In today’s world, economical circumstances change fast. In good times, decision makers start thinking about an agile transformation because they have the resources and safety to do so. And in bad times?
In bad times, senior executives will fall back into traditional thinking. They will move to old behavioral patterns: e.g., top down decisions that are not compatible with modern agile thinking.
If you would accept not having agile metrics in bad times, you shouldn’t even start an agile transformation. Because at that point, any progress made within that transformation would be destroyed.
Therefore, the only chance that scaled agile frameworks survive difficult times is by beating the system with its own weapons: by providing metrics. Metrics that help to manage in times of uncertainty.
And I am sure about it: there are agile metrics that matter.
“In retrospect, I consider it one of my biggest mistakes to have always categorically rejected metrics for the agile transformation.“
Marcus Raitner
Einstein also says it in his citation: There are things that count. Those are the ones we are looking for in this article.
Agile metrics and measurements - What makes a good metric?
Let’s say, the question you ask in your daily stand-up typically is: What did you achieve today? This is, therefore, how you “measure” progress in your team.
Good question, right? No, not right. This question makes you want to show you are busy. It pressures you to get your “To do list” done, so that you can proudly update on your metric: Yes, I was very busy in the last 24 hours!
Getting your “To do list” done, is good, is it not? Well, it depends. Far more important is something else: reaching your goal. This is typically - in case of agile teams - delivering value to your customers.
Agile KPIs - how to measure agile success
Therefore, a better question (or metric) within your Daily Stand-up would be: “How did I help my team or organization achieve our (sprint) goal in the last 24 hours?”
Change the question (or the metric), change how people think and act: effective first, efficient second - to use the words of Peter Drucker. Peter Drucker accept.
Change the metric, change how people think and act.
To use Einstein’s wording, we have to find the one thing (if we want to measure one thing) that can be counted - and that really counts.
What does really count in an agile transformation?
How agile metrics and measurements should be viewed
The goal of your agile transformation is definitely not an agile transformation. Why do you do the agile transformation? Let’s try a “3-why’s technique” to understand that.
So the core reason for your agile transformation: you don’t want to end up like Blockbuster, ignoring industry trends and customer needs, unopen to change and finally running out of money.
You want to end up like Netflix, constantly evolving your business model around the core needs of your customer. More on the Blockbuster vs. Netflix here.
How can you translate that into measuring one thing in an agile transformation? , Well that’s tough.
What companies do instead: measuring parameters because they are easy to track. Because tools out there throw them at you. The probability that these are the right metrics is pretty low.
Because for example, it is not about improving sprint velocity. That is a common mistake: monitoring and measuring effort, or efficiency. Instead, it is about solving customer needs.
Agile metrics that matter - a key takeaway
Having laid down all of this, if we want to measure one thing in agile, we have to rate the validity of this metric against one thing.
We will have to measure outputs, not outcomes. We have to measure our agile transformation by how much it helps us achieve our goal. We have to measure people not by their effort, but in terms of their contribution to a shared vision or goal - we have to measure by the value created for the customer!
If we want to measure by value, we have to deeply understand the customer’s needs.
For example, a railroad company has to understand that it is not in the railroad business. It has to understand that it is in the transportation business. Customers do not care if they are transported by train or by plane.
One other thing about agile metrics that matter - timing
One other thought is important here. At best, metrics depend on the timing and/or at what stage you are in your agile transformation.
Let’s imagine that you already know that transforming to scaled agile frameworks or other agile working practices is the right move for you (That is by the way the first thing that many companies forget to check).
In that case, at the beginning of your agile transformation your focus should be on one thing: The agile mindset of the leadership team.
Is the leadership team really ready to change? Does it understand the implications of a transformation? Is it ready to be the first team of the company to implement agile methods by heart - e.g., Kanban & transparency, agile retrospectives & continuous self reflection?
Theoretically, an initial agile metric should focus on the "readiness of the leadership team". By the way, in this blog article my colleague Jean has 7 tips about the role of the leadership team in agile transformations.
The next most important question to ask in your transformation: Do we have the right processes in place to understand and monitor our customers’ needs continuously? This is where the next agile metric could focus on.
But stop. This does not help us reach the goal of this text - to measure one thing. a Thing to measure.
No. It serves to inspire you for your agile transformation.
So let us now have a look at what typical agile metrics there are and how much they correlate to the most important output your agile transformation is about: Value.
Agile metrics that matter - a ranking
You can find a table above with the agile metrics that matterwithin scaled agile frameworks and agile transformations. I ordered them by five areas, five goals that you generally want to reach through your agile transformation:
- Customer value: Are you meeting the customer’s needs?
- Predictability: Are you delivering on time with smooth processes?
- Productivity: Are you getting more and more done at the same time and with the same resources?
- Quality: Do you deliver a product free of bugs and other issues?
- Culture: Are the people in your organization satisfied, continuously learning and do they feel free to innovate, making sure you can maintain your pace?
Agile KPIs - how to measure agile success
The table above also gives an indication of
- how easy it is to measure the metric (based on my personal experience)
- how much that metric is correlated to our core goal with a long term perspective: future customer value (based on my experience).
So which agile metric from the table above is the one?
Interesting to note is that there does not seem to be an agile metric that is easy to measure and provides the right kind of value. Damn it.
I would say that the “Easiness to measure” is not as important as “Correlation to future value”. Therefore, the most valid metrics seem to be “Usage Index”, “Customer Satisfaction” or the “Net Promoter Score”.
Scaled agile framework - Measure one thing!
It does not matter if you use the Scaled Agile Framework (SAFe®) to measure one thing, or a different framework to do your agile transformation.
Given these three metrics, what correlates most with future customer value is probably customer satisfaction. Or, to be more specific, top-box customer satisfaction. You can find more on that topic following this link.
Customer satisfaction probably also has the highest correlation of all of these metrics with the ROI of your agile transformation. It shows you whether to pivot, persevere or perish.
Scaled agile framework - Really measure one thing?
But stop. By focussing only on customer satisfaction, how do you make sure you are staying ahead of your competitors long term ? How do you make it possible that innovative and disruptive ideas can flourish in your company? Which is your long term goal...
Given these questions, I think we are getting back to the basics: to continuously provide value and to innovate around it, you need two other things: smooth agile processes and a great company culture.
Your company culture makes sure that employees feel psychologically safe , are open to fail, raise their voice and share their ideas. And your agile processes make sure you implement these ideas faster than your competitors.
Agile, like a leader, aims at people-first approach; putting people above things.
Vikram Verma
The following graphic illustrates this in an easy way. If value is your long term goal, then our input is your working methods times culture.
I once read here that “agile transformation requires a cultural change, not a process change.”
I disagree with that. It requires both.
Scaled agile framework - Measure one thing? Measure three things!
In conclusion, if you only want to measure one thing in the Scaled Agile Frameworks (SAFe®) or other frameworks, it would be customer satisfaction. It is the agile KPI to measure agile success. But I cannot recommend measuring only one thing - sorry for disappointing you.
If you really want to keep it as simple as possible with your metrics, I recommend that you measure a minimum of three things that will help you thrive long term:
- Measure customer value - through customer satisfaction.
- Measure corporate culture - through psychological safety as an indicator for learning and innovation.
- Measure agile effectiveness - through planned-to-done-ratio as an indicator for how well you are able to incrementally deliver value.
After measuring, build. Then learn. And then iterate on your metrics...
That’s so easy, right? Nope. Are you serious about implementing agile frameworks throughout your department or company?
What is a good KPI in agile and can I do about it?
We are currently interviewing dozens of experts - release train engineers, agile coaches, Scaled Agile Framework consultants - around metrics in scaled agile frameworks and agile methods in general.
Based on these interviews, we will develop a quick and simple online video training that will help you implement agile metrics the right way.
Sign up for the free "Scaled Agile" video-training
- Expert interviews & experience reports
- Receive regular updates in your mailbox
- Focus on applicable tips for practitioners
So, if you are serious about your agile transformation, I recommend you fill in the form below so we can let you know about the online training as soon as we have it ready. See you then!
