Tuesday, June 09, 2009

Managing WIP isnt the same as Limiting WIP: Part 2

In Part 1 of this article I talked about how the evidence from the feature sets of popular Agile project management tools makes it evident that first generation Agile methods did not provide any guidance on limiting WIP or implementing a pull system. In part 2 I’d like to look at how Agile methods manage WIP and how this relates to Lean.

Again, some folks have argued that early Agile methods implement pull systems. They point to iteration planning as evidence for this. There is an argument that suggests that a strictly prioritized product backlog is “pulled” into an iteration in a batch transfer. The batch size is based on a velocity metric. The velocity for an iteration suggests the capacity of the team.

The problem I have with this definition and why I don’t believe it represents a true pull system in the Lean sense is that the batch transfer size - some number of story points or stories - is generally based on historical data for throughput (velocity measured at the iteration boundary level) and not on a fixed quantity. Yesterday’s weather does not represent a fixed quantity. A rolling average of 3 recent iterations doesn’t represent a fixed quantity. Some estimate of ideal hours does not represent a fixed quantity.

However, it could be argued that pull systems don’t need a fixed quantity what they require is a balanced quantity.

Again, it’s hard to justify any of the Agile estimation & planning techniques against this measure. There is just too much empirical evidence that too many Agile teams miss their iteration/sprint commitments. There are too many tails of heroic efforts in the last few days of a sprint to insure the team made it. There are too many stories of product owners of business representatives negotiating over estimates and sizing. The Agile ideal that the team estimate the stories is all very well but how many teams work in a world where they get to decide this in isolation?

Then there are more fine-grained issues. How many teams actually have a properly prioritized backlog? And what happens when something new and more important comes up? Do teams truly keep that Sprint commitment and refuse to take on new work? What happens to the backlog? How many teams actually have the dysfunction of 3 sprints worth of backlog in progress because one is in analysis fleshing out the stories, while yet another is in development, while another is in system test? While this is not the prescribed way to Scrum or XP it is often a reality.

But for me the most compelling reason why first generation Agile methods are not pull systems, is quite simply that the level of granularity is the sprint and that the method of development is in essence craft without hand-offs. While I have no problem with the notion of craft development, methods such as XP said little to nothing about how requirements were fleshed out or how to handle system integration testing and other forms of (often required) validation. A true WIP system limits the quantity of customer valued work at each stage in the value-stream. In the Lean sense of pull making an attempt to manage the size of a single batch transfer does not constitute a pull system.

On the other hand, I do feel that Agile methods manage WIP and some do it better than others. Some leave the method of managing WIP as an exercise for the implementing team and this leads to variability in results. It is for example, quite reasonable that an XP team agrees that each pair of developers will work on only one story at a time. If they have a sprint backlog of 10 stories and 3 pairs then 3 stories would be in progress at a time - at least until one becomes blocked. Setting these kind of policies or guidelines is clearly managing WIP. Iteration planning and estimation using historical data for velocity and trying hard not to over-commit to an unreasonable plan is clearly managing WIP. Agile methods did focus on WIP and encourage the managing of it. This was a revolution. Traditional project management pays no attention to quantities of WIP. However, the early Agile guidance failed to recognize the value of managing WIP. No Agile methods recommended tracking the quantity of WIP as a metric. It wasn’t until 2002 that this was even suggested and many years later before it was an accepted practice in the general community.

Managing WIP is not limiting WIP. An Agile team can be managing WIP but still find itself with a lot of WIP both within sprints and across sprints. The behavior on a team doing Kanban, proactively limiting WIP, is notably different. The WIP limit causes them to “stop the line” and pursue root cause analysis and elimination. Kanban teams suffer stop/go behavior and ragged flow much more than Agile teams until the number of impediments is reduced. The WIP limit creates a reaction outside the immediate development team and has been seen to modify behavior way beyond the sphere of control of the typical Agile manager. Business groups produce better articulated, clearer, better prioritized requirements. The organization gets good at escalation and issue resolution. Flow and throughput become the focus rather than commitments and heroics. Limiting WIP creates desirable behavior across the organization and leads to accelerated high maturity, higher productivity, shorter cycle times and higher quality as a general observation.

It’s clearly a thin line between small batch tranfer push and small batch transfer pull. There are clearly some Agile teams that do have a sufficiently trusting relationship with their upstream partners and they are pulling small batches of work into iterations. However, to be using a full pull system, pull signaling should be working at the individual piece level - at the story level - within the sprint. Some Agile teams may achieve this but that’s because they evolved it through their own knowledge, learning, experience and reaction to their circumstances, not because the textbook told them to. They evolved to pull because they are good. Good teams will always be improving and trying new things.

Learning to manage WIP through the early part of this decade brought me to the understand why it was a good and useful technique. Limiting WIP throughout the lifecycle (or valuestream) starting in 2004 with the XIT team at Microsoft was the next logical step. For me it was a logical progression. Managing WIP led to limiting WIP. Limiting WIP was simpler and easier from a management perspective but much harder to achieve from an organizational maturity and collaboration perspective. I wish others could evolve to this understanding. Recognize that they learned to manage WIP and learned its value and that they now see the value in going further and limiting WIP and implementing a full pull system. What is the value in getting defensive and trying to rewrite history? Technorati tag: David+Anderson, Agile+Management, Agile, Lean, Kanban, Software+Engineering, Project+Management, Lean