Category Archives: Process

IBM developerWorks Podcast: Barry Hawkins on agile software development

Earlier this year, my friend Andy Glover interviewed me for season 4 of his IBM developerWorks Java technical series Podcast (RSS, iTunes, MP3 download) on the topic of Agile software development. The episode is titled “Barry Hawkins on agile software development” and runs 41:03 in length.

This was an enjoyable interview; I’ve known Andy for years, and his style as an interviewer is quite comfortable and flows conversationally. What I really enjoyed about the venue is that it afforded me the chance to talk about software development, the Agile ecosystem, and the primacy of a healthy company culture in ways that I typically do in one-on-one or small group discussions, or talks at conferences that don’t get recorded.

Here is a rough outline of the discussion with timing marks should anyone wish to jump straight to a given topic:

  • 00:55 – How I got here: my journey into software and then Agile
  • 05:50 – Things weren’t always waterfall; Agile is a rediscovery of things lost
  • 11:12 – It turns out Agile is hard, but not because the old way was better
  • 12:25 – The underlying cultural elements of a healthy Agile environment
  • 20:35 – None of the Agile elements stand well on their own; they work best as complements to one another
  • 26:05 – As a newcomer to Agile, Lean, XP, and all this stuff, where should I start?
  • 29:25 – No matter what: Get better at interacting with the other groups in your organization
  • 31:30 – How to keep cross-functional interaction from becoming death by meetings
  • 36:35 – The scarcity of good ideas, and why they get expressed over and over

I hope you enjoy it as much as I enjoyed recording it. A big thank you to Andy and the IBM developerWorks folks.

Empirical Process Control: Why Scrum Works

Being in my ninth year of applying the processes and technical practices of Agile and Lean software development, people are sometimes surprised to hear me say that Scrum is still my preferred process. Let me explain.

Scrum was designed to enable empirical process control for software development. Of all the things that are important to understand about Scrum in particular and Agile/Lean concepts at large, this is the one fewest people seem to have heard. Why is this important?

Scrum’s foundation in empirical process control is important because its model that fits the realities of creating software. Consider the following succinct definition from Wikipedia’s short entry on the topic:

The empirical model of process control provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs. – Wikipedia – Empirical process (process control model)

Let that compound sentence sink in for a moment. How effectively does that summarize your experience as a professional involved in the development and delivery of software?

Using empirical process control requires three basic elements: transparency, inspection, and adaptation. Transparency ensures that all the elements a process (I often define that as everything involved in going from concept to customer) is openly observable. Inspection is the activity of taking that observation enabled by transparency and critically evaluating how work flows through the process (in software, our cross-functional team). Adaptation takes the insights gleaned from that inspection as a basis for making incremental ongoing improvements to the process.

Scrum’s transparency comes in the form of an openly viewable Product Backlog, highly-visible information emitters in the form of Task Boards and Burndown Charts, a Daily Standup, Sprint Reviews, Retrospectives – all of that exists to clearly convey the flow of work through a cross-functional team. Scrum’s inspection comes in the ongoing daily observations and interactions of the team in moving work across the Task Board, and culminates in the Retrospective with the open, healthy discussion of what is working well and what is not working well. Scrum’s adaptation begins in the Retrospective, where the team identifies a few things to change in the next Sprint, then continues forward as those changes are implemented.

There is an indirect benefit to this application of empirical process control that, in my estimation, outweighs its immediate value. Through this process, people across functional designations are interacting with one another more than they ever had before. As they interact, they not only do the work, they think and dialog about how they do the work. Built into the cycle of transparency, inspection, and adaptation is this ongoing mental prompt for the group to ask itself, “Now why is it we are doing things this way?” Year after year in many different settings I have seen this result in a healthier group of people who are steadily improving how they work together.

Ken Schwaber emphasized the centrality of empirical process control quite a bit in the book Agile Software Development with Scrum, but it’s in the second half of the book. People rarely get all the way through books these days. So, I’m not exactly shocked when I bring up empirical process control when talking about Scrum and Agile and people tell me it’s the first they’ve heard about it.

Do not take these words to imply that Scrum alone is sufficient in creating a healthy Agile/Lean team. Mind you, it’s always accompanied by the other things that make for a holistic Agile/Lean environment: User Stories, testing (preferably Test-Driven Development, but not to the point of religious zeal), effective use of version control systems for change isolation and integration via branching, merging, etc., continuous integration, and, most importantly, a culture that is healthy enough to give all these things a fighting chance at taking root.

Barry Hawkins of All Things Computed coaches groups to successfully apply the processes and technical disciplines of Agile/Lean Software Development. In addition to coaching, Barry practices what he preaches when he develops software on contract as work for hire.

The Discipline Deficit

I consult with all sorts of companies looking to adopt elements of process and practice from the Agile/Lean offerings. Whether it’s Scrum, Test-Driven Development, User Stories, Test Automation, Continuous Integration, Agile Estimation and Planning, Sprint Planning, or Sprint Retrospective facilitation, one challenge pervades across most scenarios. I have come to call it The Discipline Deficit.

A disheartening number of groups who take up a given Lean or Agile practice or process underestimate the amount of discipline required to master and sustain it. As a result, they either put forth a half-hearted effort and the attempt flops, or they get off to a good start only to abandon it. The unfortunate outcome of either of these scenarios is that the people who interact with said group are left with a rather bitter taste in their mouth, and as a result even the mention of the word Lean/Agile/Scrum/etc. becomes a trigger for painful memories and a general sense that whomever uses these terms is someone to be avoided.

I suppose this should not be a surprise; humans have a fairly poor track record when it comes to resisting promises of gaining something for nothing. There are more than enough opportunists out there selling Agile/Lean as a silver bullet of sorts that turns even the muddiest sow’s ear into a gorgeous silk purse while requiring little to no effort beyond mimicking several steps and leaving the existing culture intact, no matter how dysfunctional it may be.

So, once again let me declare that Agile/Lean practices and processes are neither substitute nor remedy for hard work, but rather effective vehicles for structuring and managing hard work. They are also wonderful tools for thinking about the way a group works and its culture, providing information that can be used to improve it – once more through hard work. If you are going to give any of these things a try, consider the cost both in effort and cultural impact. There is no shame in deciding not to apply them; in some cases it actually reflects wisdom and maturity.

Is Agile/Lean as good as it gets?

I was recently catching up on episodes of The Java Posse, and came to the one that features an Open Spaces topic I convened at last year’s Java Posse Roundup, Should We Shoot Agile in the Head? It was a vibrant conversation with plenty of input from experienced people. At the 49:50 in the recording, D. J. Hagberg posed an excellent question:

“Is Agile[/Lean] as good as it gets?”

I added Lean, since the question really applies to both, their overlap being so great.

The short answer is no.

What D. J. is asking in the context of that conversation is something I wish more Lean and Agile practitioners were asking themselves. Is the full-fledged adoption of a given process as canonically defined by some authority the goal? If so, to what end? What does that buy us? Is that a guarantee of success?

I will tell you what is “as good as it gets”.

Working as part of a group of talented, disciplined, and motivated people to build excellent products in an environment where trust and collaboration cross the boundaries of functional roles with just enough structure to hold things together is as good as it gets.

Agile/Lean processes, tools, and practices can greatly facilitate those things, but they will not create it. At the end of the day, yes, you are in fact going to have to get better at working with others. If you are applying anything Agile or Lean in your teams, ensure that it is with the aim of fostering trust and collaboration, executing tactically with strategic vision in mind. And yes, that is not easy; excellence has never been easy.

Agile: Your Mileage *Will* Vary

There is one question I can guarantee someone will ask as I begin an engagement with a new client. It usually comes in the second half of the first major session I have, called the Agile Orientation. By that point in the initial training, we have covered the roles and components of Scrum, using it as the primary (and my favored) option for an effective Agile process in most businesses. The looks on at least a few faces at that point let me know that people are starting to process just how disparate their current work life is from what I am describing. One of the more outspoken folks usually asks something like this:

If Agile requires all these things, and we can only implement part of it, is it still worth doing?

The answer is yes, with one caveat. A partial adoption results in partial benefits, so expectations should be adjusted accordingly.

I think that’s one thing that most groups adopting Agile miss. They acknowledge at the onset that they cannot or are not willing to implement several (often crucial) elements of Scrum, but they sill expect to be “just like the stories” they hear at conferences. But, of course, it doesn’t end up being the utopia that either they envisioned or some hand-wavy, hands-off coach sold them.

The real foolishness comes when practitioners blame the process they never even managed to implement for their less-than-satisfactory results. I am seeing this disturbingly often of late. People are speaking ill of Scrum, and when you manage to get a bit of context out of them, it turns out they don’t understand it, and their attempts to apply it verge on abominable. If all a group managed to do was have a daily status meeting where some or all folks were standing and have people commit to work in variable-length sprints, can they really be that surprised to not see much benefit?

Yes, Agile is still worth doing, even if you cannot do everything at once. Any improvement is worthwhile, even if the full potential is not realized. Just be sure that expectations are adjusted. Consider the following comparison from another area of life:

If you signed up for a 10k race, and the morning of the race you decided to walk rather than run it, should you really be surprised or disappointed if it takes you 6 times as long as the first 100 runners across the finish line?

There’s a direct relationship between how fully you adopt Agile process and practices and the amount of impact or improvement within your group. When adopting Agile, it’s not that your mileage may vary, it’s that it will vary. Accept that, and you’ll spare yourself a fair amount of self-inflicted disappointment.

Agile with External Clients: To Run It Agile, Sell It Agile

This installation of the Agile with External Clients series shows the relationship between how an engagement is sold and how that affects a group’s attempt to execute the project using an Agile approach. Let’s start with a maxim for every member of a professional services group or consulting practice to burn into their mind:

The Sold Destiny Principle

The way something is packaged and sold frames the expectations of the customer.

One of my first questions when engaging with a new client is “How is your product/service sold?” This is an important question because of The Sold Destiny Principle. If services are being sold in a way that does not dovetail seamlessly into how those services are delivered, the team builds friction into the way their work is done, heaping frustration upon all parties involved, including the customers.

Many professional services and consulting engagements are sold through a fixed-bid approach. A prospective client drafts a Request for Proposal (RFP) which includes a writeup of what they (think they) want, and firms then take the RFP and respond with a proposal document. Unless a group’s focus is competing on a commodity level, this is a horrible approach. This drives the value proposition straight to price with marginal consideration for skill and quality of the solution delivered. A consulting group can look at an RFP, frame what they believe it would take to deliver a quality result, and the conversation stops there. The customer is then left to decide from a collection of proposals whose differences amount to who is cheapest, which proposal is the most aesthetically appealing, and which groups the customer has worked with in the past. (There is a fourth differentiator, which is how much schmoozing the salesperson has done. This is a horrible and costly way to compete, and generally conveys a lack of confidence in one’s product. But, I digress. Let’s save that for another post.)

The tragedy here is that it appears to the customer that every option carries all the uncertainty and false precision of phased approaches to software development. This is not the case. The salesperson whose delivery team uses an Agile, iterative approach has a keen competitive advantage. If you want to run your projects in an Agile manner, equip your salespeople to press this advantage.

The Agile practice that has not reached out to their salespeople must shoulder most of the blame for the friction between sales and delivery. After all, the salesperson is going on what they have known for their whole career and the approach to which the industry at large defaults. The cross-functional focus of Agile in general and Scrum in particular exists for a reason. Cross-functional participation in the Agile process enables a company to embrace The Rule of Concept to Customer, which most certainly includes your salespeople.

Due to the rigid, transactional nature of a fixed bid process, the contents of an RFP give a false impression. It’s a concrete description based upon a collective conscious from a group that has never seen the very thing they are attempting to describe. The requirements all appear to be must-haves, and are based upon having seen nothing. The truth is that it is a collection of must-haves and might-wants, liberally dressed with a not-so-sure vinaigrette.

An Agile engagement allows the salesperson to say something like this:

“Look, I know this RFP probably has things you know you need, plus some things you aren’t so sure about. Instead of you guys locking yourself into this 6-month heap of work, what if we broke it down into smaller blocks of investment, say 2 months in size? We’ll break these requirements down into the must-haves and might-wants, and get started on the must-haves.”

“Every 2 weeks we’ll show you the working software we have developed so far, and then we can talk about what you want to put into the next 2 weeks. You can switch up the priority on the items, and if you want to push some of them out and pull an equal amount work forward from the lower-priority work, that is not a probelm. It’s your 2-month block to spend as you want in 2-week increments.”

“As we close in on the 2-month mark, we’ll take a look at this together and see if you want to invest in the next 2-month block. If you don’t that is OK. You take the working software you have so far and run with it. If you want to continue, we’re more than happy to keep delivering new working functionality in 2-week slices for another 2 months. We’ll repeat that evaluation and decision process for the third block, and by the end you have a product that you have been able to shape as we go based upon seeing the working solution evolve and deciding what you want to put into it.”

I have seen that approach work very well, and it generates a loyalty among customers and a reputation for being a breed apart among consulting companies. Looking for the best way to run consulting engagements with Agile? Sell them with Agile.

Agile with External Clients

I speak at conferences around the country each year, most often on the topic of real-world, full-blown application of Agile Software Development using Scrum. I always appreciate when folks come to me after the end of a conference talk to ask questions about the topic of adopting or applying Agile in various scenarios. I usually end up having at least a couple new challenges to my assumptions, which helps me further inspect and adapt my approach to coaching and mentoring.

I will almost always have at least one person who presents what they think is the tough question, the one area where Agile and Scrum simply cannot work. The question, in the form an assertion, usually goes something like this:

“This stuff is all well and good when you are working inside a company and your clients are the people that work with you in the same company, but our clients are external customers, and we have fixed-bid projects, which is why we cannot use anything like this.”

The embedded question in statements like the one above is, “Can Agile, and Scrum in particular, be used when you do professional services work for external clients, even with fixed-bid projects?”

Yes, you can.

The person who presents me with a statement like that is usually dumbfounded when I respond that at least a third of my clients are consulting practices and professional services firms that do exactly what was just described. Not only can you do it, there are many scenarios where you should do it. Using an Agile, iterative approach to professional services engagements can be a formidable sales tool and competitive advantage.

In the coming days I will be sharing some insight on the nuances of adopting Agility with external customers based on my years of experience working with clients who have done that very thing.