Monday, February 16, 2004

Embrace Dark Matter

Hal Macomber is interviewing the authors of Embracing Uncertainty this coming Thursday.

This reminded me of a term I coined in my book, “dark matter” - something which we know exists in the universe but we can’t see it. Project dark matter is unrecognized client-valued functionality. Call it missing analysis if you prefer. Dark matter is features which you didn’t have on the feature list but they were there all along. You discover them whilst doing the detailed design mid-project. Dark matter causes the feature count on projects to increase. To upper management it looks like scope creep but the customer will deny having asked for any changes.

Dark matter is always there - embrace it!

It is too easy to say, “next time we will do better analysis” or “next we will spend more time on analysis” - wrong! Embrace dark matter - it’s out there!

The real question to be asking is “how certain are we about this domain?” and “how comfortable are we that we caught all the detail whilst doing the domain modeling and client-valued functionality identification?” Use this to calculate a scope buffer. On an FDD Feature List I enter this as empty line items and comment them as “dark matter”. During the project as new features are discovered through more detailed analysis, I replace an empty line with the new feature.

On a recent project iteration, I asked for a 100% buffer for dark matter due to domain uncertainty. The request was denied. The project ran with a 50% buffer. During the project 61 features grew to 117 without any change requests from the customer. That’s a 92% increase due to dark matter. Luckily we made the date by surprising ourselves and outperforming on both production rate and quality.

Why was it better to make a probabilistic guess about the scope and move forward rather than wait to get the analysis perfect?

Simply put, it would have taken weeks to resolve the analysis issues and gain a high level of certainty that everything had been captured. OTOH, in 4 weeks 117 features were coded and complete with absolutely certainty. By embracing dark matter, it was possible to move forward earlier and deliver earlier. In this example, the deliverable was achieved earlier than analysis would have completed had total confidence been a gating prerequisite to moving forward.

Lose the deterministic project management paradigm. Embrace uncertainty using a probabilistic approach to project management. Analysis can never be perfect. When you can accept that, and accept the probabilistic paradigm, then you can move forward earlier and complete sooner.