I've been reevaluating my view on refactoring!

It's interesting how your viewpoint can color your view of something. My viewpoint in to the agile community and its practices has always been rooted in my experience with FDD/The Coad Method. When I spoke at USC back in 2004, I suggested that though FDD may never be seen as the most agile of Agile methods, it was probably the most Lean. I stand by this comment three years later. FDD is very Lean. All the waste is trimmed out. The Coad Method techniques deliver a very precise definition of a domain, that is loosely coupled and highly cohesive, while the Feature definition technique delivers fine-grained customer-valued units of work that exhibit very low degrees of variation. The batching technique of Chief Programmer Work Packages is very efficient and minimizes transaction costs associated with work-in-process. I could go on, but you get the point that FDD is Lean. The modeling, planning and batching for design and build result in almost no need to refactor code on FDD projects. As a result, my book classified refactoring as rework and labeled it as waste.

From my FDD viewpoint that might have been a fair assessment. However, now that I'm running a software engineering organization where I inherited a waterfall process, that runs through a series of narrow and specialized departments such as Business Analysis, Systems Analysis, Development and Test, I've changed my opinion. From a new viewpoint refactoring is clearly a very valuable process.

I believe that Alistair Cockburn's paper from the ICAM 2005 International Conference on Agility will come to seen as a seminal paper in Lean Software Engineering (ironic as it was inspired by the Theory of Constraints). In this paper, Cockburn explains that asking a non-bottleneck resource to do extra work to rework something does not cost anything extra and can create a desirable effect because it allows progress to be made and demonstrated earlier. Ergo, rework is not waste when performed by a non-bottleneck resource.

There have been many versions of the idea that perfect is the enemy of good enough. The original is attributed to Voltaire. And it is this concept that refactoring (and Cockburn's paper) embrace. It is argued that it is better to make progress with imperfect information and refactor later when better information is available than to wait for better information before progressing.

Specifically, I am thinking that it is better for developers to start coding with imperfect analysis than to wait for a systems analyst to produce a "perfect" specification. The developer can then refactor the developed code when the analyst makes a final version of the specification available. My reasoning is simple. The developer would otherwise be idle. [Not truly idle, there is plenty of busy work available, grooming environments, training on new languages and APIs and so forth but idle in the sense that they are not adding value to the deliverable.] By definition a resource (or station) with idle time is a not a bottleneck resource. It is, therefore, OK to ask the developer to perform the refactoring. The refactoring cannot be classified as waste in this case.

We can think about this decision using real option analysis. The option we are buying is to deliver the working code earlier. The cost of the option is the cost of having the developer start work before a final specification is ready. The risk (or uncertainty) attached to the option is the risk that the early imperfect specification will be significantly different from the final specification and that any rework will take longer than waiting to start coding on delivery of the final specification. Note that the rework may absorb all of the slack in the non-bottleneck resource turning it in to a bottleneck and delaying the whole project. This gives us a framework to decide whether starting early and refactoring is the correct decision, or whether waiting and coding for "right first time" is the correct decision.

[It would nice if someone reading this and perhaps actively studying for a masters or doctorate in the field were to develop this concept and publish it complete with equations and data from sample projects. ;-)] Technorati tag: Agile, David+Anderson, Real+Option+Analysis, FDD, Coad+Method, Theory+Constraints, Lean, Software+Engineering, Refactoring