Pushing clouds with chopsticks

Perhaps if I hadn't had a previous career outside of software it wouldn't bug me so much, but I find the nebulous, amorphous task of software development thoroughly frustrating. It was back around 2003 that I was having a conversation about a project with my project manager at the time (who was old-school yet awesome), and I told her, "It's really like trying to push a big cloud and all I have are two chopsticks." I'm closing out my 13th or 14th year in software development, and I still feel this way. Even with all the insight, feedback, and visibility into a project that Agile Software Development provides me (and yes, real Agile, before anyone bothers to say that I feel this way only because I'm not doing full Agile Software Development), I'm still frustrated by how unwieldy the whole practice can be.

My "Why is Agile hard?" conversation is up on the Java Posse

The Why is Agile hard? conversation I convened at this year's Java Posse Roundup made its way into the Java Posse podcast feed. That was an enjoyable session with discussion of the challenges groups face when adopting Agile.

NOTE: The astute reader will notice that this entry has changed. My blog was compromised and the perpetrator substituted the original post with links to questionable material from faraway lands, and I don't mean overripe strawberries from the west coast.

Keep a long-term perspective; know your "sales cycle"

(This is part two in a series I have titled "Soft Skills in Software", which came out of points I came up with for a panel discussion at CodeMash 2008.)

When introducing change in a technology practice, you can save yourself some frustration by being aware of the "sales cycle" for new ideas within your group and maintaining a perspective that always bears the long-term in mind.

As technologists, we often have great enthusiasm for new ideas that we perceive to hold value in terms of their (potential) positive effects on our work. The value is so apparent to us that we often see no reason not to put this new idea to work right away. However, the speed with which a new idea can be adopted is affected by a number of factors. Let's start with these:

  • The organization's general disposition toward change
  • The change agent's standing as a thought leader within the technology practice
  • The resultant impact of adopting the idea in question
  • The temporal context in which the new idea is introduced

Depending up on the value of each of those variables (and more), the length of time it takes for a change to be adopted can be anywhere from a few weeks to months to years -- or a special category I have, "not for the foreseeable future". I use that category instead of "never", because there have been times when I thought there was no way a given thing would ever happen, and suddenly a merger or regime change or some other major external change suddenly opened the door for the previously-infeasible.

I use the term "sales cycle" because it was in a previous career of mine where I first learned this lesson. I used to sell industrial packaging and marking systems, and when I first started, I was greatly frustrated at the seemingly low number of sales that I would close. I would follow up with some clients to the point of irritating them, convinced that "something should have happened by now". I eventually learned that the different types of systems that I sold had different sales cycles. Because of a given system's cost and the role it played in the manufacturing process at large, there was a minimum amount of time it took to get approval, move through a cusotmer's procurement process, and finally issue me a purchase order. Some systems took 4-6 weeks to sell, while others were 6, 12, and even 24 months. In general, the longer the sales cycle, the bigger the sale. Once I understood this, I found myself far less frustrated, and there were fewer of those awkward interactions between my customers and I.

Introducing new ideas to your development group is quite similar. There is a direct relationship between the level of change and the time it takes for a team to agree and then adopt the idea. I have witnessed more than a few programmers who had good ideas that they presented and expected everyone to just approve right away. Their frustration with what they perceived as stalling and/or rejection typically resulted in undesireable outcomes. Even the most-valued technical person is only indulged a few temper tantrums before being labeled as "damaged goods". Recovery from this sort of thing within an organization is discouragingly rare. The person is usually avoided in discussions about strategic direction, and their ideas are usually dismissed or stifled. In general, if letting person Y talk about changes has resulted in discomfort in the past, the typical solution is to just not let that person talk about changes. It's warped, dysfunctional, and counterproductive, but that's mainstream corporate culture for you.

One area where technical change differs from sales is the possibility of incremental change. It's not always possible to sell only a piece of a large system; changes in software development, on the other hand, are well-suited to incremental change. Being able to adjust one's expectation from demanding a single big win to being content with a series of methodical, steady, small victories is a valuable skill in software. For some big changes, that can be the only way to introduce them. For example, a company may not be open to switching to Agile Software Development overnight, but having daily stand-up meetings and developing in 2-week iterations certainly seems benign enough.

Having said all that, there have been times where something with a 12 or 24-month sales cycle was adopted immediately for me. Urgency, crisis, severe failure with the current situation, any number of things can result in exceptions to the rule. The art is knowing when to spot these factors and acting appropriately to leverage them. If I could sum all that up in a blog post, I'd probably be in a different line of work ;-).

Believe in what you're selling

(This is part one in a series I have titled "Soft Skills in Software", which came out of points I came up with for a panel discussion at CodeMash 2008.)

Let's get right to the point, then we'll unpack it. If you do not believe in what your company is doing, seriously consider finding work that you can believe in.

This was the first point in a series of observations I wrote down in preparation for a panel discussion on the topic of "talking technology to humans". (The astute reader will have noticed that this recommendation does not directly relate with how to convey one's thoughts on technology to others with a less-technical bent. My hope is that those shining stars will read on before posting a comment or sending me flame mail.) I put this recommendation first because introducing change in technology practices is a taxing process. Since introducing change requires so much of your energy, I'd only recommend it if you believe that your organization's work is worth what you will expend of your emotional energy.

If you don't think that a lack of belief in your company's work is affecting you, I'd challenge you to try a few things:

  • Ask your spouse or significant other if they think your mood and attitude toward things in general are affected by your job.
  • Ask your spouse, significant other, or children if they feel like they are walking on egg shells when you get home from work because they don't want to "set you off". If they look uncomfortable and say "no", that means yes, and it's so bad they aren't even comfortable admitting the problem to you for fear that you may erupt. I speak from experience here.
  • Pay attention to how you answer the question "and what do you do for work?" the next time your are in a social context where that comes up; is your explanation apologetic or do you use a tone of sarcasm that indirectly lets the person know that you do not identify strongly with your company and/or the services it offers?
  • Pay attention to how you feel on the average day as you commute to work; are there more positive days than negative days?
  • Does reading this blog post incense you for reasons you can't fully explain, and you already have a litany of reasons who people shouldn't expect to be able to feel good about what their company does? :-)

Here's the thing; life is short. Too short to while away most of the hours of years of your life contributing to something you don't even agree with. Too short to have your family members' memories of their years with you be colored with how negatively your work affected you. You don't need the hypertension, and your loved ones don't need the unjustified bile.

Am I saying that finding work you believe in will eliminate stress and make you some fount of perpetual positive emotional energy? Why, of course not. Work is going to take something out of you, but it is going to take even more out of you if you're doing it against your conviction, without the benefit of an additional resultant gain.

Oh, and one last thing. There may be some who say, "well my company does sell anything." Sure they do. Everyone is selling something, a service in exchange for a form of compensation. The consumer may not be the payee, but something's being provided. Don't let that semantic keep you from examining whether or not your employer's work resonates with you.

Mike Cohn on transitioning to Agile Software Development at Agile Atlanta

I found out yesterday that Mike Cohn was going to be speaking at Agile Atlanta. Given that our AJUG meeting had fallen through, I pulled some schedule gymnastics in order to make the meeting. Don't pass up a chance to hear Mike. Succeeding With Agile: A Guide To Transitioning And Improving will be his new book that fleshes out the principles of the talk he gave tonight; I scribbled down some notes below.

Agile Transition Planning

  • Have goals in a transition backlog, to help you figure out how you're doing on those goals and what you can do to assist those goals.
  • Set quarterly transition goals,having those be like releases for software
  • Have monthly iterations where you talk about what you can do
  • Have weekly meetings talking about how you can do it
  • Establish a "guiding coalition" that includes a sponsor and area managers who can help ensure success

Establish transition teams, they're pursuing the tactical elements of some of your goals; some will be organic, some will have been formally established. Organic tends to be preferred; these will have the internal motivation necessary to get the most out of Agile.

Transition Team Members

Who to include
  • Be aware of power distribution
  • Know where expertise and credibility lies; passion alone does not a team member make
  • Know who benefits and who loses from this
  • Know what groups might have a tendency to resist this and/or may align to blockade this as an allied force; strategic inclusion can defuse this
Who to exclude
  • Avoid "snipers"
  • Avoid big egos
  • Avoid snakes who poison relationships
  • Avoid conscripts

Leading an Agile Transition

Transition team and other formal leaders must lead the transition, but not in the way they may have in the past.

Prerequisites of Self-Organization
  • Container - boundaries along which people align
  • Differences - enough difference to provide a vibrance
  • Transforming Exchanges - interaction meaningful enough to provide value to participants

NOTE: I have had to tweak this via changes to agile team organization

Patterns of Agile Adoption
  • Technical Practices First - high impact, does not scale well, partial adoption is risky
  • Iterative First - not as controversial, but teams can get very complacent, scales relatively well
  • Requirements First - if you're there, OK, but meh; wait for the right project?
  • Start Small - has been de facto, but less so these days, cheaper, good for those on the fence as to whether to commit, it's slow
  • All In - it's over quickly, no organizational dissonance of having two systems at once, risky, costly, usually requires a reorganization
  • Stealth Mode - no additional pressure, no one knows until you tell, no one can say no, no organizational support
  • Public Display of Agility - over the top, can reduce resistance, obvious need for organizational support
  • Impending Doom - can shock team out of complacency, admitting pending disaster can free team to experiment, can help overcome resistance, transition can be quick, not always an option, big change in time of trouble can increase stress
Patterns of Agile Expansion
  • Split and Seed - split team up, add in newcomers
  • Grow and Split - get team big then split
  • Internal Coaching - develop team, give team members additional responsibilities to coach other teams
Addressing Resistance
  • Sell the problem, not the solution
  • Understand the types of resistance you are encountering and address them in appropriate ways

Mark Ramm, friend and exemplar

My post about a Python web frameworks OpenSpace convened at CodeMash drew a bit of attention, and looking over it again, I can see how it can send messages I did not intend. In order to break my trend of sitting on posts for weeks and months, I pushed myself to get this one out even though I wanted to finesse it more. The main person who could have taken this the wrong way would be Mark Ramm, who convened the talk and is also currently heading up my favorite Python web framework.

Instead of launching some missive or flaming the comments of my post, Mark took the time to further explain the situation for TurboGears both past and future. He even apologized if he may have unintentionally spoken ill of the JVM (which he did not). Mind you, I am no defender of the JVM, but I can see how I might seem like it from that post. Mark also sent an email which explained some of the discussion items and after hearing that, I was far less concerned about where things are heading with that project.

I have known Mark for about a year now. We have had dinner together and shared many funny stories (he has many, should you ever have occasion to hear them). I consider him a friend, and his reaction to that post is an example of what I have come to enjoy about the Python community. After reading Zed Shaw's "Rails is a Ghetto" tonight, I am even more thankful for it. Man, even if you filter through the bile and profanity, it still sounds like this dude received some rough treatment for trying to do the right thing at times. And if some of it is true, then I need to rethink some of the folks whose work I endorse.

Curb Your (Python Web Framework) Enthusiasm

One OpenSpace topic convened at CodeMash was "Python Web Frameworks". TurboGears seemed to be the most-represented citizen of the Python web world, but some asserted that they use Django. I had skipped Brian Goetz's "Java Performance Myths" talk in favor of this, even though I am sure that would have given me more ammo to smack down the crusty turds in the Java world who are always denigrating the work of others for its lack of their arcane incantations that are allegedly "critical to performance". I have to say I was pretty dismayed after attending the OpenSpace, but maybe that's not a loss after all. Perhaps my Python web enthusiasm was due to be curbed. Also, maybe I shouldn't let those crusty Java turds get to me so much, even if I do feel they are a key agent in keeping the Java community from being more inviting.

Coming away from the discussion, my impression is that the Python web framework community is perfectly content with the dilution of effort and momentum that is caused by the proliferation of web frameworks. We talked about the number of projects out there, laughed a little, and then folks seemed to be ready to move on. One participant fortunately asked the question "How are TurboGears and Django different, because they look the same to me?" That's a paraphrase, but the gist of his whole question was, as the two leading Python frameworks, how are these two serving different needs, and if they're not, why are there two? What followed was some dialog that choice was good, the usual stuff you'll hear in Open Source discussions about duplication and fragmentation of effort.

I followed up on his question commenting that the degree of fragmentation has seriously hurt the credibility and perceived viability of Python web frameworks in larger shops that are entertaining dynamic web frameworks as possible alternatives to doing everything in Java or .Net. I added that the proliferation of choices in Python has really only served to dilute the attention and effort for those that are emerging as the leading ones. I also shared that those outside the Python community are perceiving its web offerings as momentary fascinations of the brilliant, yet quirky enthusiasts upon which nothing long-term should be based. This didn't seem to be well-received by the group, but I may be wrong. I think most folks present may have been up for a Python love feast instead of some serious introspection on where we are and how we got here.

Another thing we had was Python fans without significant Java experience continually emitting FUD about the JVM, which blows my mind. I have been following Python pretty closely for over a year now, and the amount of ignorance about Java in the Python community still surprises me. The lack of hands-on experience with Java does not seem to deter the constant, half-formulated criticisms of it. Some even seemed to be implying that Python's threading situation was better off than Java. The platform of Java quite frankly kicks the hell out of most all others in that arena; this is widely-known.

There is an unsettling meme that seems to be running through the Python web framework community of "doing what's cool", which is unsettling. Cool and pragmatic seldom keep close company. One of the most heartening things about Rails 2.0 has been the steps taken to pay down technical debt and bolster the soundness of the framework. Hearing that TurboGears is pursuing X because Django has it and that's cool disturbs me. "Cooler than Y" doesn't get me a framework I am comfortable building upon; robust and reasonably-documented does.

I viewed the merged effort with the Pylons project very positively when I learned of it. Collaboration and true sharing are all-too-absent in F/OSS these days. However, the more I learn about the culture and mantra of Pylons, the more I am inclined to think it may have been an ungainly partnership. I am wondering if TurboGears' original "compose a mega-framework of best-of-breed" may have subtly morphed its perception of best-of-breed to be "coolest". And cool, like beauty, is in the eye of the beholder.

The need for a degree of stability in a web framework is not unlike the dilemma faced by technical authors and producers. Something I have heard from multiple technical writers is that one must resist the temptation to be forever revising the work in hopes of keeping pace with the changes occurring in the technology being covered. In software that rate of evolution is such that it is a Sisyphean undertaking. One must pick a point of time and stick to it. The hope is that said point is an acceptable level of stability so that the book is not useless weeks after it hits the shelves. (This begs the question of whether or not many technical books in print today should not have been published, but that's a separate topic.)

My chief concern for TurboGears is that the apparent direction toward "cool" is in direct conflict with the need for stability and documentation. The quality of Open Source projects these days is such that few will tolerate a poorly-documented project that poses an unnecessary learning curve primarily brought on by the need to figure things out for yourself. Unless the documentation is shored up and the rate of significant component change slows, I think TurboGears will be challenged to gain more traction in the wider web framework world.

The reason I am even bothering to write about this is that I care about the TurboGears project. I chose it for my framework to explore at Bruce's Dynamic Web Frameworks Jam, and was blown away by how much had been accomplished in so little time with the project. It had things I have been wanting for years in Java web development, and the learning curve was so low that I found myself running ahead of my own understanding of Python at times. The DRY-ness of TurboGears is another strong draw for me. Implementing your own components when superior, highly-tested ones exist is just silly these days, unless considerable mitigating factors are present. While I think it holds lots of promise, I can't say that I am comfortable basing any significant projects on TurboGears right now.

My definition of "Enterprise"

Bruce and I were talking yesterday about the nebulous term "enterprise" as used in software, and as I pondered that over my Rooibus tea with a queezy stomach, my definition of it materialized:

enterprise - The illusion of highly-mitigated risk in exchange for your money

Sidelined at CodeMash

The night before CodeMash, I was up all night with my daughter who had a horrible 24-hour virus that kept her throwing up throughout the night. It made it really hard to leave her the next day. That's the first time she's thrown up, and it upset her about as much as it upsets me to throw up.

Well, the next day I made it through the keynote and most of the first two sessions, then bam! I made it back to the room shortly before losing breakfast and maybe some of dinner. I haven't been that sick in a long time. I think I got out of bed maybe 6 times in 24 hours. As a result, I missed many of the sessions I had come to hear.

Despite that, I had some great face-to-face time with colleagues and met some new people from the Ruby and .Net worlds. Try doing that at JavaOne :-).