Being in my ninth year of applying the processes and technical practices of Agile and Lean software development, people are sometimes surprised to hear me say that Scrum is still my preferred process. Let me explain.
Scrum was designed to enable empirical process control for software development. Of all the things that are important to understand about Scrum in particular and Agile/Lean concepts at large, this is the one fewest people seem to have heard. Why is this important?
Scrum's foundation in empirical process control is important because its model that fits the realities of creating software. Consider the following succinct definition from Wikipedia's short entry on the topic:
The empirical model of process control provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs. - Wikipedia - Empirical process (process control model)
Let that compound sentence sink in for a moment. How effectively does that summarize your experience as a professional involved in the development and delivery of software?
Using empirical process control requires three basic elements: transparency, inspection, and adaptation. Transparency ensures that all the elements a process (I often define that as everything involved in going from concept to customer) is openly observable. Inspection is the activity of taking that observation enabled by transparency and critically evaluating how work flows through the process (in software, our cross-functional team). Adaptation takes the insights gleaned from that inspection as a basis for making incremental ongoing improvements to the process.
Scrum's transparency comes in the form of an openly viewable Product Backlog, highly-visible information emitters in the form of Task Boards and Burndown Charts, a Daily Standup, Sprint Reviews, Retrospectives - all of that exists to clearly convey the flow of work through a cross-functional team. Scrum's inspection comes in the ongoing daily observations and interactions of the team in moving work across the Task Board, and culminates in the Retrospective with the open, healthy discussion of what is working well and what is not working well. Scrum's adaptation begins in the Retrospective, where the team identifies a few things to change in the next Sprint, then continues forward as those changes are implemented.
There is an indirect benefit to this application of empirical process control that, in my estimation, outweighs its immediate value. Through this process, people across functional designations are interacting with one another more than they ever had before. As they interact, they not only do the work, they think and dialog about how they do the work. Built into the cycle of transparency, inspection, and adaptation is this ongoing mental prompt for the group to ask itself, "Now why is it we are doing things this way?" Year after year in many different settings I have seen this result in a healthier group of people who are steadily improving how they work together.
Ken Schwaber emphasized the centrality of empirical process control quite a bit in the book Agile Software Development with Scrum, but it's in the second half of the book. People rarely get all the way through books these days. So, I'm not exactly shocked when I bring up empirical process control when talking about Scrum and Agile and people tell me it's the first they've heard about it.
Do not take these words to imply that Scrum alone is sufficient in creating a healthy Agile/Lean team. Mind you, it's always accompanied by the other things that make for a holistic Agile/Lean environment: User Stories, testing (preferably Test-Driven Development, but not to the point of religious zeal), effective use of version control systems for change isolation and integration via branching, merging, etc., continuous integration, and, most importantly, a culture that is healthy enough to give all these things a fighting chance at taking root.