Skip to content

Empirical Process Control: Why Scrum Works

Being in my ninth year of applying the processes and technical practices of Agile and Lean software development, people are sometimes surprised to hear me say that Scrum is still my preferred process. Let me explain.

Scrum was designed to enable empirical process control for software development. Of all the things that are important to understand about Scrum in particular and Agile/Lean concepts at large, this is the one fewest people seem to have heard. Why is this important?

Scrum’s foundation in empirical process control is important because its model that fits the realities of creating software. Consider the following succinct definition from Wikipedia’s short entry on the topic:

The empirical model of process control provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs. – Wikipedia – Empirical process (process control model)

Let that compound sentence sink in for a moment. How effectively does that summarize your experience as a professional involved in the development and delivery of software?

Using empirical process control requires three basic elements: transparency, inspection, and adaptation. Transparency ensures that all the elements a process (I often define that as everything involved in going from concept to customer) is openly observable. Inspection is the activity of taking that observation enabled by transparency and critically evaluating how work flows through the process (in software, our cross-functional team). Adaptation takes the insights gleaned from that inspection as a basis for making incremental ongoing improvements to the process.

Scrum’s transparency comes in the form of an openly viewable Product Backlog, highly-visible information emitters in the form of Task Boards and Burndown Charts, a Daily Standup, Sprint Reviews, Retrospectives – all of that exists to clearly convey the flow of work through a cross-functional team. Scrum’s inspection comes in the ongoing daily observations and interactions of the team in moving work across the Task Board, and culminates in the Retrospective with the open, healthy discussion of what is working well and what is not working well. Scrum’s adaptation begins in the Retrospective, where the team identifies a few things to change in the next Sprint, then continues forward as those changes are implemented.

There is an indirect benefit to this application of empirical process control that, in my estimation, outweighs its immediate value. Through this process, people across functional designations are interacting with one another more than they ever had before. As they interact, they not only do the work, they think and dialog about how they do the work. Built into the cycle of transparency, inspection, and adaptation is this ongoing mental prompt for the group to ask itself, “Now why is it we are doing things this way?” Year after year in many different settings I have seen this result in a healthier group of people who are steadily improving how they work together.

Ken Schwaber emphasized the centrality of empirical process control quite a bit in the book Agile Software Development with Scrum, but it’s in the second half of the book. People rarely get all the way through books these days. So, I’m not exactly shocked when I bring up empirical process control when talking about Scrum and Agile and people tell me it’s the first they’ve heard about it.

Do not take these words to imply that Scrum alone is sufficient in creating a healthy Agile/Lean team. Mind you, it’s always accompanied by the other things that make for a holistic Agile/Lean environment: User Stories, testing (preferably Test-Driven Development, but not to the point of religious zeal), effective use of version control systems for change isolation and integration via branching, merging, etc., continuous integration, and, most importantly, a culture that is healthy enough to give all these things a fighting chance at taking root.

Barry Hawkins of All Things Computed coaches groups to successfully apply the processes and technical disciplines of Agile/Lean Software Development. In addition to coaching, Barry practices what he preaches when he develops software on contract as work for hire.

5 thoughts on “Empirical Process Control: Why Scrum Works”

  1. The transitioning towards empiricism is just part of a greater trend, as I argued in

    Whether Scrum should be preferable or not over other agile/lean approaches can be debatable; there are other process that create just as efficient transparent feedback loops, with opportunities for inspection and adoption. For instance, the (correct) application of Theory of Constraints will lead to similar result. What process to prefer is really dependent on the circumstance, and above anything else on leadership and organizational culture.

    1. Thanks for reading, The Chronologist. I think your parenthetical modifier “correct” to “application of Theory of Constraints” is telling.

      I have seen far fewer correct applications of the Theory of Constraints than I have Scrum. In my experience, that’s based on Scrum’s pragmatism and having been designed by software practitioners versus the esoterica of Theory of Constraints and the other complex adaptive systems that smack of management consulting and management science rebranded, where yet another wave of “experts” are telling practitioners how to approach their craft.

  2. Barry, You’re absolutely right. Scrum has to its advantage that it is based on “simple rules” that anybody can understand. It is so pragmatic, even “managers” seem to understand them…

    TOC, on the other hand, is even simpler than Scrum. (See “five focusing steps,” etc.), The hardest part of TOC is that it requires practitioners to… you know… think by themselves.

    The point I stress is that you cannot know a-priori which approach works in a given organization. Some will pick up Scrum and find it good. Others will fly with Kanban. Others with TOC. Others with Lean. The unifying thing is: the foundation in empiricism that underly all these approaches.

    The title of your post “Why Scrum Works” is spot on. It works because it is based on empiricism, feedback loops, inspection, self-reflective improvement, etc. It is the same for the other approaches mentioned too.

    1. Careful, I think your elitism is showing:

      “[Scrum] is so pragmatic, even “managers” seem to understand them…”

      “The hardest part of TOC is that it requires practitioners to… you know… think by themselves.”

      I see far too much of that type of condescension among proponents of TOC and the method called Kanban (not the original kanban the visual control system from Lean Manufacturing). When the message “these are all the same” is accompanied by that sort of thing, it translates to “we say these are all equal, but really we think the others are rubbish and ours is best because we are sophisticated and elite.” It’s rather polarizing and unproductive, and symptomatic of an unhealthy culture.

      1. Sorry! I maybe missed a smiley or two. You get them here:

        🙂 🙂

        Anyway: If you are truly driven by the empirical approach, then you should be open to experimenting approaches, and pick the one that works for *you*.

        One size fits all (whether you call it Scrum, TOC or whatever), does never work. You have to adapt the process to the organization, and not vice versa.

        Many Scrum practitioners see Scrum as a panacea that cures everything. There is a lot of elitism in Scrum, too. (Note: This is not directed at you. It is just an observation. In general, *any* school of thought that has produced any result sometime, somewhere, will generate elitism in one way or another. Especially when it gains a following.)

        Take any suggested approach – whether Scrum, TOC, or other – with a grain of salt.

        So how do you know what works? You don’t.

        Instead: make an experiment and find out. Maybe it turns out to be Scrum. Or TOC. Or something else.

        Some approaches are more, or less, prescriptive. Those that are less prescriptive, truly require you to sort things out (i.e. think) by yourself. With this respect, Scrum is definitively more prescriptive than TOC or Software Kanban. That is one reason why Scrum is easier.

Comments are closed.