Compendium / Development Process
Overview
Metrics are an essential feedback loop for development process, but they should be employed with great care. Metrics are easily misused and often misinterpreted, and there are decades of corporate dysfunction that incline us toward those unproductive patterns.
Goals
Healthy and effective development process metrics serve two primary goals:
- Help teams cultivate an understanding of their capacity for delivery.
- Serve as one point of information for leadership to see which teams need help; the type of help will be determined through conversation with a team to understand the context of their challenges.
Core Metrics
I I have found myself able to accomplish the goals of development process metrics with only two metrics across all development teams, Throughput_ and Projected-to-Completed Ratio.
Tracking these two metrics for a team can be quite lightweight. I have used a spreadsheet similar to the following for over a decade, even in organizations with hundreds of team Tracking these two metrics for a team can be quite lightweight. I have used a spreadsheet similar to the following for over a decade, even in organizations with hundreds of team members and dozens of teams. For folks who may be new to tracking these metrics, I have created a Google Sheets version of it titled Team Metrics Tracking – Core Metrics to help them get started.
Throughput
Throughput answers the question “On average, what volume of schedulable work have we delivered in the last several cycles?”
Throughput is intended to be an average of the volume of work completed in a number of the most recent cycles. Using a rolling six-cycle average is quite common, and seems to be enough of a sample to have a meaningful average where the impact of a given cycle is neither too great nor too small.
A team tracks throughput using the same unit of measure employed to estimate their queue or backlog of schedulable work items. Using the Story Points scale popularized with User Stories is a common approach. This allows the team to use Throughput to give stakeholders and dependent teams an idea of when a particular outcome is projected for delivery based on its current position in their queue.
Be aware that some flavors of Batch development process like Scrum use the term “velocity” to refer to Throughput.
Below is an example chart of Throughput being tracked for a development team.
Interpreting Throughput
Healthy applications of Throughput convey the following information:
- Teams in times of low volatility in their work queue and few changes in team composition have relatively stable Throughput with normal levels of variation.
- Teams undergoing periods of volatility, whether in team composition or in their work queue, will have destabilized and more variable Throughput.
- Throughput can appear stable while teams are experiencing considerable volatility. Reviewing a team’s Development Logs alongside their Throughput can provide more context.
Projected-to-Completed Ratio
Projected-to-Completed Ratio answers the question “On average, how much work did we think we could deliver, and how much did we deliver?”
I began using this ratio after seeing a pattern where some of my past teams were chronically overcommitting during Cycle Planning, and we didn’t seem to be learning from it. We would talk about it in the Retrospective, but we often had excuses or explanations for why it happened. By the time planning came around the next day, we would get ourselves into the same predicament.
When I began tracking this metric, we were able to see as a team that we were consistently projecting more than one and a half times the amount of work we would complete. Seeing it charted over time was effective in helping us to admit that it was not just a series of one-offs, but that it was a regular pattern. Admitting the pattern then allowed us to dialog about why we were overcommitting, and identify experiments to improve.
Interpreting Projected-to-Completed Ratio
Healthy applications of Projected-to-Completed Ratio convey the following information:
- Teams that have come to understand their capacity will generally average within a tolerable variation of 1, completing roughly what they projected at Cycle Planning.
- Teams that chronically project more during Cycle Planning than they complete at the end of a cycle will average well above 1.
- Teams that consistently commit to far less during Cycle Planning than they end up completing at the end of a cycle will average well below 1.
A Cautionary Note on Metrics
I strongly recommend avoiding the creation of metrics for team development process beyond what is outlined above. Leadership should take a judicious approach in the adoption and interpretation of metrics.
The key mistake I see in development process metrics is treating the available data as being far more meaningful than it actually is. Because the measurement, analysis, and visualization of data is such an established practice in business, creating metrics on that data takes very little effort. It seems logical that we would create all manner of graphs and interpretations of that data.
However, the accuracy of story points and task hours is deliberately constrained, because it models the degree of accuracy in software development estimation, which is directional at best. Applying overly-granular statistical analyses to this type of data leads to false conclusions and interpretations, not only failing to add value, but actually subtracting value by creating overhead and providing misinformation.
Never lose sight of the intentionally minimalist nature of effective development process. The goal is to create just enough structure to enable teams to be effective, and to not go further than that. The rest of the emotional and intellectual bandwidth of teams should be kept free to focus on creating the best possible outcomes for the products we deliver.