IBM developerWorks Podcast: Barry Hawkins on agile software development

Earlier this year, my friend Andy Glover interviewed me for season 4 of his IBM developerWorks Java technical series Podcast (RSS, iTunes, MP3 download) on the topic of Agile software development. The episode is titled "Barry Hawkins on agile software development" and runs 41:03 in length.

This was an enjoyable interview; I've known Andy for years, and his style as an interviewer is quite comfortable and flows conversationally. What I really enjoyed about the venue is that it afforded me the chance to talk about software development, the Agile ecosystem, and the primacy of a healthy company culture in ways that I typically do in one-on-one or small group discussions, or talks at conferences that don't get recorded.

Here is a rough outline of the discussion with timing marks should anyone wish to jump straight to a given topic:

  • 00:55 - How I got here: my journey into software and then Agile
  • 05:50 - Things weren't always waterfall; Agile is a rediscovery of things lost
  • 11:12 - It turns out Agile is hard, but not because the old way was better
  • 12:25 - The underlying cultural elements of a healthy Agile environment
  • 20:35 - None of the Agile elements stand well on their own; they work best as complements to one another
  • 26:05 - As a newcomer to Agile, Lean, XP, and all this stuff, where should I start?
  • 29:25 - No matter what: Get better at interacting with the other groups in your organization
  • 31:30 - How to keep cross-functional interaction from becoming death by meetings
  • 36:35 - The scarcity of good ideas, and why they get expressed over and over

I hope you enjoy it as much as I enjoyed recording it. A big thank you to Andy and the IBM developerWorks folks.

Empirical Process Control: Why Scrum Works

Being in my ninth year of applying the processes and technical practices of Agile and Lean software development, people are sometimes surprised to hear me say that Scrum is still my preferred process. Let me explain.

Scrum was designed to enable empirical process control for software development. Of all the things that are important to understand about Scrum in particular and Agile/Lean concepts at large, this is the one fewest people seem to have heard. Why is this important?

Scrum's foundation in empirical process control is important because its model that fits the realities of creating software. Consider the following succinct definition from Wikipedia's short entry on the topic:

The empirical model of process control provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs. - Wikipedia - Empirical process (process control model)

Let that compound sentence sink in for a moment. How effectively does that summarize your experience as a professional involved in the development and delivery of software?

Using empirical process control requires three basic elements: transparency, inspection, and adaptation. Transparency ensures that all the elements a process (I often define that as everything involved in going from concept to customer) is openly observable. Inspection is the activity of taking that observation enabled by transparency and critically evaluating how work flows through the process (in software, our cross-functional team). Adaptation takes the insights gleaned from that inspection as a basis for making incremental ongoing improvements to the process.

Scrum's transparency comes in the form of an openly viewable Product Backlog, highly-visible information emitters in the form of Task Boards and Burndown Charts, a Daily Standup, Sprint Reviews, Retrospectives - all of that exists to clearly convey the flow of work through a cross-functional team. Scrum's inspection comes in the ongoing daily observations and interactions of the team in moving work across the Task Board, and culminates in the Retrospective with the open, healthy discussion of what is working well and what is not working well. Scrum's adaptation begins in the Retrospective, where the team identifies a few things to change in the next Sprint, then continues forward as those changes are implemented.

There is an indirect benefit to this application of empirical process control that, in my estimation, outweighs its immediate value. Through this process, people across functional designations are interacting with one another more than they ever had before. As they interact, they not only do the work, they think and dialog about how they do the work. Built into the cycle of transparency, inspection, and adaptation is this ongoing mental prompt for the group to ask itself, "Now why is it we are doing things this way?" Year after year in many different settings I have seen this result in a healthier group of people who are steadily improving how they work together.

Ken Schwaber emphasized the centrality of empirical process control quite a bit in the book Agile Software Development with Scrum, but it's in the second half of the book. People rarely get all the way through books these days. So, I'm not exactly shocked when I bring up empirical process control when talking about Scrum and Agile and people tell me it's the first they've heard about it.

Do not take these words to imply that Scrum alone is sufficient in creating a healthy Agile/Lean team. Mind you, it's always accompanied by the other things that make for a holistic Agile/Lean environment: User Stories, testing (preferably Test-Driven Development, but not to the point of religious zeal), effective use of version control systems for change isolation and integration via branching, merging, etc., continuous integration, and, most importantly, a culture that is healthy enough to give all these things a fighting chance at taking root.

Barry Hawkins of All Things Computed coaches groups to successfully apply the processes and technical disciplines of Agile/Lean Software Development. In addition to coaching, Barry practices what he preaches when he develops software on contract as work for hire.

The Discipline Deficit

I consult with all sorts of companies looking to adopt elements of process and practice from the Agile/Lean offerings. Whether it's Scrum, Test-Driven Development, User Stories, Test Automation, Continuous Integration, Agile Estimation and Planning, Sprint Planning, or Sprint Retrospective facilitation, one challenge pervades across most scenarios. I have come to call it The Discipline Deficit.

A disheartening number of groups who take up a given Lean or Agile practice or process underestimate the amount of discipline required to master and sustain it. As a result, they either put forth a half-hearted effort and the attempt flops, or they get off to a good start only to abandon it. The unfortunate outcome of either of these scenarios is that the people who interact with said group are left with a rather bitter taste in their mouth, and as a result even the mention of the word Lean/Agile/Scrum/etc. becomes a trigger for painful memories and a general sense that whomever uses these terms is someone to be avoided.

I suppose this should not be a surprise; humans have a fairly poor track record when it comes to resisting promises of gaining something for nothing. There are more than enough opportunists out there selling Agile/Lean as a silver bullet of sorts that turns even the muddiest sow's ear into a gorgeous silk purse while requiring little to no effort beyond mimicking several steps and leaving the existing culture intact, no matter how dysfunctional it may be.

So, once again let me declare that Agile/Lean practices and processes are neither substitute nor remedy for hard work, but rather effective vehicles for structuring and managing hard work. They are also wonderful tools for thinking about the way a group works and its culture, providing information that can be used to improve it - once more through hard work. If you are going to give any of these things a try, consider the cost both in effort and cultural impact. There is no shame in deciding not to apply them; in some cases it actually reflects wisdom and maturity.

Is Agile/Lean as good as it gets?

I was recently catching up on episodes of The Java Posse, and came to the one that features an Open Spaces topic I convened at last year's Java Posse Roundup, Should We Shoot Agile in the Head? It was a vibrant conversation with plenty of input from experienced people. At the 49:50 in the recording, D. J. Hagberg posed an excellent question:

"Is Agile[/Lean] as good as it gets?"

I added Lean, since the question really applies to both, their overlap being so great.

The short answer is no.

What D. J. is asking in the context of that conversation is something I wish more Lean and Agile practitioners were asking themselves. Is the full-fledged adoption of a given process as canonically defined by some authority the goal? If so, to what end? What does that buy us? Is that a guarantee of success?

I will tell you what is "as good as it gets".

Working as part of a group of talented, disciplined, and motivated people to build excellent products in an environment where trust and collaboration cross the boundaries of functional roles with just enough structure to hold things together is as good as it gets.

Agile/Lean processes, tools, and practices can greatly facilitate those things, but they will not create it. At the end of the day, yes, you are in fact going to have to get better at working with others. If you are applying anything Agile or Lean in your teams, ensure that it is with the aim of fostering trust and collaboration, executing tactically with strategic vision in mind. And yes, that is not easy; excellence has never been easy.

Agile: Your Mileage *Will* Vary

There is one question I can guarantee someone will ask as I begin an engagement with a new client. It usually comes in the second half of the first major session I have, called the Agile Orientation. By that point in the initial training, we have covered the roles and components of Scrum, using it as the primary (and my favored) option for an effective Agile process in most businesses. The looks on at least a few faces at that point let me know that people are starting to process just how disparate their current work life is from what I am describing. One of the more outspoken folks usually asks something like this:

If Agile requires all these things, and we can only implement part of it, is it still worth doing?

The answer is yes, with one caveat. A partial adoption results in partial benefits, so expectations should be adjusted accordingly.

I think that's one thing that most groups adopting Agile miss. They acknowledge at the onset that they cannot or are not willing to implement several (often crucial) elements of Scrum, but they sill expect to be "just like the stories" they hear at conferences. But, of course, it doesn't end up being the utopia that either they envisioned or some hand-wavy, hands-off coach sold them.

The real foolishness comes when practitioners blame the process they never even managed to implement for their less-than-satisfactory results. I am seeing this disturbingly often of late. People are speaking ill of Scrum, and when you manage to get a bit of context out of them, it turns out they don't understand it, and their attempts to apply it verge on abominable. If all a group managed to do was have a daily status meeting where some or all folks were standing and have people commit to work in variable-length sprints, can they really be that surprised to not see much benefit?

Yes, Agile is still worth doing, even if you cannot do everything at once. Any improvement is worthwhile, even if the full potential is not realized. Just be sure that expectations are adjusted. Consider the following comparison from another area of life:

If you signed up for a 10k race, and the morning of the race you decided to walk rather than run it, should you really be surprised or disappointed if it takes you 6 times as long as the first 100 runners across the finish line?

There's a direct relationship between how fully you adopt Agile process and practices and the amount of impact or improvement within your group. When adopting Agile, it's not that your mileage may vary, it's that it will vary. Accept that, and you'll spare yourself a fair amount of self-inflicted disappointment.

Is it just my company that has a hard time with Agile?

After an Agile adoption is underway and the feedback mechanisms of most Agile processes begin to function, one or more team members usually ask me something like the following:

Is it just my company that has a hard time with Agile?

No, it's not just your company. However, don't take comfort in that.

I shared something on Twitter that I think sums up why companies have a hard time with Agile:

Agile practices and processes serve motivated, self-directed teams who work hard. It does not create them. @barryhawkins

When most groups decide to start embracing any Agile process or practice, they usually begin by attempting one or more of the discretely observable activities. This is almost always undertaken with no understanding of how that one piece fits into the cohesive whole of the given process or what underlying cultural elements are assumed to be present in order for it to work as expected. This is how you end up with a team having a "daily standup" (even better if it's called a scrum) where most folks sit and talk about their day while staring at the wall, floor, or someone's pen, waiting for the magic to happen. (Spoiler: It doesn't.)

Why would a group even think this would work? Most likely because some overzealous (and typically unseasoned) practitioner or charlatan "coach" sold them on "Agile" (the fact that it was used as a noun is often your first red flag) as yet another silver bullet. You simply begin mimicking these few things that someone heard about in a talk (no need to read up on them; it sounded really simple), and sooner or later the excellence will kick in. The rhetoric usually reads like an infomercial for commoditized excellence in software development.

If you have a hard-working, self-directed team, an Agile process will most likely add some structure to your operation and show you areas to keep improving. If your team exists within an unhealthy culture, that is where it will begin to encounter its greatest difficulties. Effective cross-functional collaboration does not come about as a result of declaring a group "Agile". The team will have to really push their ability to communicate and forge relationships with cross-functional groups. Be ready to adapt as you find out who is willing to work with you in this way and who is not. You can start a decent-sized cultural upheaval before you know it; count the cost of such a thing and decide if it's worth it.

If you have an average team in a company where the technology is an afterthought, that process will probably be somewhat like salt on an open wound. Have you ever heard of a person who sent to rehab or for whom an intervention was staged, but they were nowhere near ready to admit they have a problem, much less deal with said problem? The outcome is seldom desirable, typically driving a wedge between those desiring to help and the one in need of help. Bringing an Agile process into a place that sees the current situation as perfectly acceptable is much like that.

If your team and the company it works for do not have a compatible culture for fully embracing a process like Scrum (that is my preferred one), prepare yourself mentally and emotionally for the fact that it will not "look like it does in the book". That doesn't mean you should not try. Almost any place there would be benefit from increased communication and accountability.

Agile processes and practices are hard for a reason; they are vehicles for pushing forward and honing your craft. Activities that drive toward mastery are always difficult; do not be surprised by this.

Ask an Agile Coach: What is an Agile Coach?

This installment of the Ask an Agile Coach series is a question normally asked by persons outside my field, but lately I have been asking it myself:

What is an Agile Coach?

Good question. I am not sure anymore.

The more I encounter others who call themselves Agile Coaches, the more I think I may not be up-to-date on what that is supposed to mean.

As I have historically interpreted it, an Agile Coach is someone who first actually established a proven track record of successfully leading a series of Agile adoptions as part of the actual teams. The insight, perspective, and expertise gained through that effort are what enable them to coach others. Multiple Agile teams in varied settings challenges what a person "knows to be true" and (hopefully) beats some humility into and the dogma out of them.

I have been under the impression that an Agile Coach also understands software development and the technical practices that support an Agile team. One must understand why testing, automation, code review, and pair programming exist. One must know what it feels like to work both with and without these things, know why professionals should adopt these practices, as well as when one or more of them are not a fit for the culture. It must be very strange to have management bring in a coach who tells you to do these things but cannot show you how.

I also thought that coaches should understand how to apply an Agile process in a way that stresses cross-functional participation. If what a coach is doing isn't helping the company to function more effectively as a whole, I fail to see the point. Agile Software Development is neither a defense mechanism for engineers nor a tool for more effective exploitation by management.

It feels weird to have to point this out, but I believe a coach should understand Agile estimation and planning techniques, especially the ones laid out in Mike Cohn's books. Estimating and planning are among the most painful parts of software development, and one of the most fulfilling parts of coaching for me has been mentoring groups in Agile estimating and planning.

I also think an Agile Coach is someone who can say no. Agile Software Development is a challenge at best, and a cultural nuke at worst. There are places where the introduction of Agile process and practices would wreak havoc. I suppose the toughest part of this is that it requires a willingness to turn down paying work.

I am seeing things these days that make me think my idea of what an Agile Coach is might be incorrect. If it is, I might need to stop using that word. It doesn't mean what I think it means.

Ask an Agile Coach: What Team Members can be shared across multiple Scrum teams?

Welcome to another installment of Ask an Agile Coach. Today's question centers around how to structure teams in light of constrained skill sets in an organization:

What technical roles, other than QA, can be successfully shared across scrum teams? DBA, UX, etc.? - @jakejgordon

Strive to have people in dedicated teams; only share in order to get work from concept to customer.

Any software application of substance requires a collection of specialized skills. A healthy, cross-functional team will have a pool of diverse talent: programming (in at least one language, but often more), testing (QA), database administrators (DBA), business intelligence (BI), user experience (UX), operations/infrastructure (OPS/INFRA), and others. However, some of these skills are not used enough to occupy a single person's work from week to week on a single team, so persons with those skills are allocated across multiple teams.

Recall that, with Scrum (and other Agile processes as well, though Scrum is most explicit about it), the process is designed to provide empirical process control in software development. The reliability of velocity as a measure of throughput depends upon the stability of the team. Therefore, an organization should try to structure its teams to consist of people dedicated to the workloads they serve. When team members are allocated across multiple teams, their available hours for each team can vary from sprint to sprint. Since these people are usually specialists like BI or UX, this variation is happening with the most constrained set of hours available for Sprint Planning. Any user story in a sprint that requires these hours is at risk of carryover if the team planned against hours that turned out to be unavailable.

The undesirable result of this variation is an unstable velocity and increased carryover of user stories in sprints. This in turn erodes the reliability of planning and estimation activities. The team shows itself to be unreliable and unable to deliver on commitments, eroding confidence and trust with stakeholders and management.

The primary risk with sharing team members is that Scrum Masters and management often do a poor job of accounting for how those peoples' time is allocated and managed, which inflames the symptoms described above. Bear the following points in mind when sharing team members:

Include all your cross-functional team members in user story writing, estimation, and Sprint Planning sessions. User stories that account for all the effort needed to go from concept to customer are more reliable and not as prone to surprises during delivery in a sprint. Prefer the bad news of resource constraints over the surprise of planning against naive user stories.

Avoid sharing a team member across more than two teams. Once a person has to divide his or her effort across three or more teams, estimates and available hours become unreliable. Furthermore, the more teams a person has to participate in, the more their time is wasted in switching their mind between contexts. They also end up in recurring events like Sprint Planning, Sprint Reviews, Backlog Grooming, and Retrospectives for each team, leaving a meager number of hours to do actual work. If a person has to be shared across more than two teams, that is a sign that another person with those skills needs to be hired.

Use the valuable information from Sprint Planning and tracking to inform the budgeting and hiring process. Is QA/BI/UX constantly a shortage in Sprint Planning? That is the gentle, loving tap of the clue bat on the collective skull of management to stop hiring more software developers and allocate the salary budget in a manner more appropriate to the actual work being done by the group. Hidden costs of effort are being uncovered as a result of properly applying the transparent, reliable process that is Scrum, and that is a good thing.

Barry Hawkins of All Things Computed provides coaching and mentoring in how to successfully apply the process and technical disciplines of Agile Software Development.

Ask an Agile Coach: Should I count partially complete user stories in my velocity?

Hello readers, apologies for the lack of posts lately; lots of work at hand. Today's Ask an Agile Coach question comes from a comment on the last installment, which is rephrased here. In one form or another, I have gotten this question for years now:

Should I count partially complete user stories in my velocity? For example, if I have a 100-point story that's almost done, should I add 80 points to my delivered points total for this sprint?

The answer is no, you should not.

Remember that Scrum was designed to enable empirical process control for software development. The team is committing to work in fixed intervals, with the intent being that a team becomes quite proficient at knowing how much work they can move from concept to customer within that fixed interval. To that end, the points that go into velocity are points delivered. Delivered points come from stories that are completed. When stories are incomplete, they are carried over, and velocity is lower than expected for the Sprint. Hopefully, these points plus the additional work pulled in will cause the next Sprint to have a higher number of delivered points and the two values will level one another out.

If a team or Scrum Master insists on counting part of a story's points in velocity, they are masking a symptom and thus choosing to ignore the underlying causes. This short-circuits the process and the valuable feedback it provides.

If a team finds itself with carryover continually affecting velocity in a negative manner, it is usually an indication that the team needs to improve their ability to plan and commit to work within a Sprint.

One area to target is honing the ability to break work down into thinner vertical slices of user-oriented functionality. If stories are perpetually 50-80% done at the end of a Sprint, revisit the team's approach to user story writing and trying to carve smaller functional slices (but not so small that you violate the INVEST heuristic from Bill Wake; an alarming number of practitioners have fallen into hyper-granularity in their stories).

Another area to address is the cross-functional team's (and your Scrum team is cross-functional, right?) understanding of what it takes to go from concept to customer. If stories seem to invariably look smaller or the team seems to perpetually overcommit, it may be that some of the more vocal team members are expressing estimates that reflect a myopic focus on simply the writing of code instead of all the tasks involved in completing a given story. Encourage the other cross-functional members to speak up in User Story writing, Sprint Planning, and Backlog Grooming about what they need to do to complete the stories in question. This has always resulted in more accurate story sizes for me over the years.

Barry Hawkins of All Things Computed provides coaching and mentoring in how to successfully apply the process and technical disciplines of Agile Software Development.

Agile with External Clients: Testing Is Not Optional

Today's installment in the Agile with External Clients series covers the topic of testing. A decade after The Agile Manifesto and over 16 years since Scrum and XP came on the scene, I still encounter a large number of teams where the use of testing is lip service at best and non-existent all too often. In this context, testing means the use of frameworks like xUnit et al to create of a suite of unit, integration, and functional tests that exercise a body of code by executing it and making assertions about the outcome of that execution at multiple levels of focus and granularity. Of all the practices of Agile software development, both process and technical, testing is the one people most readily acknowledge the value of while at the same time avoiding it altogether. So, let me make this quite plain:

Testing is not optional.

Testing does not slow you down; it allows you to speed up. It is the fundamental feedback loop in the creation of software. Testing is one of the strongest means of minimizing and eliminating waste by virtue of how it allows you to catch defects as early as possible. This is true both for errors introduced directly in the code being created and indirectly through breakage of existing code relies upon or integrates with what is being made. Manually exercising your code by running it locally and following along in your debugger is not an adequate substitute, but rather a companion to the judicious use of testing.

While this lesson applies to any Agile group, it is especially critical if you want to run projects for external clients using an Agile approach. Recall that for many clients, you may be their first encounter with Agile software development principles. Their historical context views "testing" as an afterthought, a separate process that does its best to verify that what was built approximates what was documented. That makes testing a separate value proposition and one to cut if cost or timeline become an issue.

Given the disappointment with phased approaches to software development, it should come as no surprise that the testing "phase" has a hard time living up to its potential. At that point, there's already an entire project of sunken cost in analysis and development. Feedback on things that aren't working as expected are taken as bad news, since some of the software was developed months if not over a year ago. If you read reports from testing groups that have gone over waterfall applications at the final testing phase, they all seem to fall under the theme of "Things That Would Have Been Great to Know at the Time I Could Have Done Something about Them".

An Agile approach that includes the technical practice of unit, integration, and functional testing restores what we have been missing for decades in classical approaches; we now have feedback and information at the time when we are best able to act upon it. It gives us a flexibility and confidence to move swiftly in implementing new features and responding to inevitable change without recklessness or irrational optimism.

This is particularly true in the case of working with external clients. If a team is effectively applying testing, the lack of mid-to-late-project "surprises" is refreshing. Functionality whose implementation can spin on a dime without tearing the ship apart astounds clients who've become accustomed to changing their mind or acting upon new information being a quite painful and costly activity. If User Stories are the most astounding process practice, then Testing is definitely the counterpart as the most profoundly beneficial technical practice.

Having testing be an inseparable thread in the fabric of how you deliver software can separate you from the rest of the pack who claim to be the "Agile gurus" in your market. However, the absence of testing doesn't merely deprive you of that benefit. As changes are introduced to the system and allowed to pass for days if not weeks unnoticed, your team invariably runs aground on one or more of those issues. You find yourselves in uncomfortable conference calls explaining how this iteration is going to have only half its stories delivered due to things that have surfaced on stories that had already been declared complete in previous iterations. Unchallenged assumptions arise late in a project to bite the team on the ankle, making the project less flexible should the client require nontrivial changes to the architecture.

As clients observe this behavior (and believe me, they notice), they come to the conclusion that maybe this Agile business is no different from everything they've seen before. Your group is just another one claiming to have "the secret", while their impression is that paying a premium for experts doesn't really deliver what it promises. And if that's the case, maybe they should have just gone with the lowball bid.