Metrics are an essential feedback loop to systems like development process, but they should be employed with great care. Metrics are easily misused and often misinterpreted.
Healthy and effective development process metrics serve two primary goals:
- Help teams cultivate an understanding of their capacity for delivery.
- Help leadership see which teams need help; the type of help will be determined through conversation with a team to understand the context of their challenges.
Within the context of development process for a team, I have found myself only needing two metrics across all development teams, Throughput and Projected-to-Completed Ratio.
Tracking these two metrics for a team can be quite lightweight. I have used a spreadsheet similar to the following for over a decade, even in organizations with hundreds of team members and dozens of teams.
Throughput answers the question "On average, what volume of schedulable work have we as a team delivered on the last several cycles?"
Throughput is intended to be an average of the volume of work completed in a number of the most recent cycles. Using a rolling six-cycle average is quite common, and seems to be enough of a sample to have a meaningful average where the impact of a given cycle is neither too great nor too small.
A team tracks throughput using the same unit of measure employed to estimate their queue or backlog of schedulable work items. Using the Story Points scale popularized with User Stories in Agile Software Development is a common approach. This allows the team to use Throughput to give stakeholders and dependent teams an idea of when a particular outcome is projected for delivery based on its current position in their queue.
Be aware that some flavors of Batch development process like Scrum use the term "velocity" to refer to Throughput.
Below is an example chart of the Throughput being tracked for a development team.
Healthy applications of Throughput convey the following information:
- Established teams in times of relative stability have relatively stable Throughput with normal levels of variation.
- Teams undergoing periods of volatility, whether in team composition or in their work queue, will have destabilized and highly-variable Throughput.
Projected-to-Completed Ratio answers the question "On average, how much work did we think we could deliver, and how much did we deliver?"
I coined the term Committed-to-Completed Ratio after I had been using it for some time. With a number of teams, I was having a problem where we were chronically overcommitting during Cycle Planning, and we didn't seem to be learning from it. We would talk about it in the Retrospective, but we often had excuses or explanations for why it happened. By the time planning came around the next day, we would get ourselves into the same predicament.