David Greenfield, in comments to yesterday's post, asks me to comment on modeling and quality and on whether it is possible to "over model".

Quality Assurance

Firstly, let's look at modeling and quality. I've talked a lot about this recently at the Chicago and Scottish agile developer's groups. You might like to view my slides from these talks and then apply these words to them.

Modeling in FDD and specifically Coad Method color UML domain modeling, is the key to quality assurance because it is the key underpinning of communication in the project. The domain model acts as the project glossary - all communication should happen in the context of the domain model and its class names, features (method names) and attributes. All team working assignments are based off the domain model and all further design activity is based off and ought to be congruent with the domain model. The domain model is the map of what is to be done, the constraint on what is being done and the foundation on how it is being done.

When I use the term "quality assurance" I use it in the Edwards Deming form, meaning forward loop built-in quality rather than return loop quality control (or testing). Quality assurance is the art of the craftsman, the engineer who takes pride in his/her work. The domain model in FDD is the foundation for that quality assurance.

FDD has several quality assurance mechanisms built into it. Firstly, there is team working. As I said before, "there is no I in FDD." So work is collaborative and that improves quality. Design work is also done using the domain model as a constraint. A design review should include the domain model and the requirements as the standard against which the design is reviewed. If the design modifies or lies out with the domain model then it should be rejected or an exception recorded and a meeting of the senior members of the team held to discuss whether the model needs to change. The design which is by this time congruent with the domain model is then used along with the coding guidelines as the constraints on a code review. The code produced should match with the design and be within the coding guidelines. Again, if the code is not congruent with these constraints then it should be rejected or an exception recorded.

Exceptions recorded at code reviews are reviewed at a monthly meeting of the chief programmers, architects and development manager where the existing coding guidelines are reviewed and potentially extended. The new document is then circulated to the whole team with any new inclusions highlighted. The new document becomes the new standard (or constraint) for all code reviews. And hence, learning - even at the coding level - is built into FDD.

Over-modeling

Over-modeling and/or analysis paralysis are dangers in the initial FDD process - Build a Domain Model. Analysis paralysis is most likely to happen with novice or apprentice modelers who get into arguments about "is this green or yellow?" and "do we use a blue or create an inheritance hierarchy?" The master or expert will know what to do and move on. Over-modeling in my experience happens with individuals who have had a heavy and often high quality exposure to RUP in their careers - they have been trained to model deep and thrash out every detail.

In much of my writing about the difference between agile and older heavyweight or waterfall approaches, I have highlighted the what I see as the key difference - agile embraces uncertainty whilst traditional methods seek to drive out uncertainty and strive for perfect information at every stage. I feel that RUP attempted to achieve this through the five architectural views - get five views on something just in case you missed something in only one perspective. The Coad Method says - one perspective is good enough and let's move on.

Avoiding over-modeling or achieving optimal modeling is a skill which has to be learned. It may be the case, that simply sticking names onto classes in a DNC pattern is under-modeling. Coad, Palmer et al advise that you validate the model by trying a few key features on it to show that it provides the structure to answer the questions. The real skill is in knowing how many features to use. The answer is "less than you think but more than none". My advice is use the law of diminishing returns. If you find that you design a feature and nothing on the model changed and this happens two or at most three times, then move on to another section of the model.

[Blogging is a bit like working out in the gym - when you get out of the habit, it is so hard to get back into it. So here goes with my first day back on the blog-cycle. dja]

Here is a meme for an agile architecture control board - no it's not a committee - board is used literally. During recent FDD projects, we've taken to using a white board with the object model / functional architecture as a visual control and communication mechanism. Figure 1 shows such a board from a recent real project building telecom grade infrastructure.

Figure 1

The board shows the entire domain object model for the seven week iteration. Note that the project is being built with a Coad Method architecture where the domain model directly reflects the classes being built. There is no concept of an "analysis model" and a "design model", they are both the same. The dotted lines on the board show package level separation and there are small arrows showing the package direction dependency. The detailed design causes small changes to the model to resolve two way dependencies and occasionally to refactor the original model as more understanding was gathered in the detailed design stage. The names of class owners are also written on the board. Any team member can come by at any time and see what represents the definitive architecture for the iteration and see who they should be working with to complete any given feature or set of features.

Note the screen in front of the board. This is deliberate. It's a pair programming station which also serves other purposes. The morning standup meeting is held in front of the board to aid communication. The CP running the meeting directly enters issues and feature status updates into the KMS accessed via the computer. This way project status is electronically captured immediately in the standup meeting.

The machine is also used for refactoring work where the agreed changes have been modeled on the whiteboard first. Typically refactoring is at the package level, i.e. a group of classes are being assigned into a package. We practice a technique of assigning packages and component boundaries at the last responsible moment. At the beginning of an iteration all the classes are in one package often simply called "domain". As the project continues and the interfaces between classes are firmed up with detailed design, then component boundaries are fixed and classes are repackaged. I'll be talking more about this "bottom-up" style of distributed architecture at Borcon 2004 in September.

Figure 2 shows a print out from Together-J with the domain model from a previous iteration. This printed model together with the whiteboard model represent and the working architecture for the current project. There is only a one class overlap between the two diagrams - the leftmost pink class in Figure 2 is the 3rd pink from the left on the top row of Figure 1.

Figure 2

Figure 3 shows an adjoining whiteboard which contains the map of package/component dependencies for the entire project. All of these design diagrams are available in the project control room and anyone in the team can come and look at them and hold team discussions to modify and update them. A general rule exists that no one team member is allowed to make a change. Changes to the architecture require a consensus of chief programmers and other senior members of the team.

Figure 3

Now it might be easy to say - gee we could do all of this electronically - and yes you could! In fact a tool like Together can generate the HTML documentation for all of this directly from the source code and publish it to an intranet site such as an FDD KMS but...

The electronic version is not so accessible. A group can't so easily gather around a screen as they can a whiteboard. Changes are not so easily made on the screen and require the use of the tool and the use of version control software. The electronic version is buried inside a web site and not every team member will visit it and look at it, when it is on the wall everyone sees it, at least once per day. The version on the board is considered definitive. If someone brings a design or code for review which is not congruent with the board then the review should be rejected. If the board is out-of-date then that must be fixed and clearly communicated to the whole team before the review material is passed.

Sketching versus Design

In the product design world, they clearly understand the concept of a sketch - an idea, a work in progress - from a design - a plan for something to be manufactured. Designers use subtle clues to separate sketches from designs such as elongated lines, rough approximate lines, sketch strokes and other noise in the image to communicate "This is not finished! It is just an idea".

The architecture control board model counts as a design. It is not supposed to be reworked constantly - in fact, rework and changes happen very seldom. The team uses a separate war room for sketching - for dialoguing about new models or changes to a model. The sketching area often has what Peter Coad calls "model fragments" - models which map only a small piece of functionality. Several model fragments may be joined in the design model on the main control board. The model in Figure 1 is very carefully drawn to signify that this is a design artifact and not a sketch - note the careful horizontal and vertical lines, note the attention to layout which minimizes crossing lines, note the spacing of the classes and the alignment of related elements - anyone skilled in color modeling will be able to pick out the sequence of pink <<moment-interval>> classes in the middle of Figure 1 which usually indicates a business process or value chain in the domain. By taking the time to draw a clean, well thought out model, derived from the model fragments agreed in the sketching area, everyone on the team realizes intuitively that this is design and definitive and not merely a suggestion of what they are supposed to be doing.

Following up my post from last week on using FDD with legacy code bases, I wanted to relate some details of the only (partially) non-OO project I've done with FDD. It took place in Ireland in 1999 and had some really good people on the team including Brian O'Byrne, now of Statesoft and the horseman of the telepocalypse, Martin Geddes. The business logic had to be written in Oracle stored procedures using PL/SQL - this was a political decision which suited Oracle as they supplied the consultants to do the work.

However, I modeled the system, along with an Oracle data modeler - Jenny Hay - using the color modeling technique. The color classes mapped almost one-to-one to the data model with the exception of role classes which often disappeared into the (green) Person|Place|Thing class table. This model took a week to produce with only two of us working on it - so not quite the usual FDD team process. I then read the requirements document very carefully and marked it up as Features. I counted 153 business logic (PD) Features. These would be mapped to 153 stored procedures. I then categorized the Feature Complexity on a scale of 1 thru 5 based on the number of classes touched by a Feature and any other knowledge of algorithmic complexity. Then without any knowledge of PL/SQL coding I made a wild ass guess and invented by power law table of 0.5 days, 1 day, 2 days, 4 days and 8 days. I used this to estimate the total time for the business logic as 10 weeks for 7 developers and told the project manager. She went off and hired the people to do the work.

The project ran incredibly smoothly from their and we tracked it in the normal FDD fashion. This was also the project where we invented the UI solution using Statecharts where each State and each Event is a UI Feature.

I feel that any project which has an underlying persistent storage data model can be run using FDD even if the Features are to be implemented in a procedural language. The color modeling technique lends itself to normalized data model development and most modern databases work well with normalized models. On the Irish project we brought in a very experienced Oracle tuning guru for 2 days to optimize small parts of our data model. This was done without impacting the functional development and did not invalidate the initial modeling or feature planning.

[I'll be traveling for a while with no Internet access so no post until the last week of July]

America is fascinated with Lance Armstrong's attempt to win a sixth Tour De France - and why not, what an achievement that would be! Even our local Komo 4 news in Seattle gives us a daily update while the Outdoor Life Network (OLN) channel on cable TV has virtual 24 hour coverage repeating the race coverage 4 times daily. So here are some topical thoughts on the Theory of Constraints and the Tour De France.

Yesterday July 7th saw Stage 4 of the race which was a 64 km team time trial Cambrai to Arras. In this stage, each team rides with up to 9 riders as a group but each team rides separately against the clock. Naturally, the fastest team wins. The time is set by the 5th rider to cross the line. The usual principle in team time trials is to keep the team together in a long line. One rider fights the wind and the other 8 hide in the slipstream. The rider at the front rotates every few hundred metres and thus the top speed is maintained and each rider doesn't get too tired. Hiding in the slipstream can save up to 40% of a rider's energy.

With around 45 kms to go Lance Armstrong's team dropped one team member Benjamin Noval. Why? Another American rider and contender for the overall win is Tyler Hamilton. In the closing 20 kms Hamilton's Phonak team dropped 4 riders and finished with only 5 men together. Why? Surely in both cases, it made more sense to keep the team together. In Hamilton's case, earlier in the race he had waited for a rider who had a mechanical problem. So why change his mind later?

Quite simply, the constraint had moved!

When the constraint on top speed is the wind, then in TOC language the constraint is outside of the system and the system - the team of riders - must exploit it and subordinate to those exploitation decisions. The tactics for this are the standard tactic a team time trial - ride together in a long line, rotate the rider at the front as soon as the speed begins to drop off, conserve energy, keep the top speed as high as possible. However, sometimes the constraint will move inside the system. As soon as one or more of the riders in the team cannot maintain the pace of the rest, then that rider becomes the constraint. The team can only move at the speed of the slowest rider. The constraint is inside the system.

In both cases, Hamilton and Armstrong realized that they would go faster if they dropped the constraining rider(s) from their group. They made an on-the-fly decision to change tactic because they realized that the constraint on their productivity had moved. Noval who had been injured the day before in a crash, was evidently unable to maintain the pace with the rest of the US Postal team. Armstrong dropped him and left him to cycle almost 45 kms on his own at a speed around 8 kph slower than the rest of his team. Sometimes it is tough to be the leader or the manager but that's what managers and leaders are paid for - to make the tough decisions. The US Postal team won the race and Armstrong assumed the over all lead in the Tour De France. He couldn't have done so without understanding his constraint and making the correct exploitation and subordination decisions. He couldn't have done so without being a strong leader. And he couldn't have done so unless everyone on the team understood and believed in the goal - Lance Armstrong must finish as the over all winner in Paris after stage 20.

I've been asked by a few readers to comment on how to use FDD with a legacy code base under maintenance. [There is an assumption in this post that the legacy code is still OO in nature such as a monolithic 1999 vintage EJB system]. This topic has come up at Jeff De Luca's web site in the past - Maintenance & FDD. Daniel Vacanti and I have had a couple of experiences with FDD on legacy code bases over the last 5 years and my general advice to anyone undertaking a maintenance process and wanting to gain the benefits of a method like FDD, is that they should try to make as few changes to the FDD process as possible. FDD definitely works with maintenace projects. So how do you go about it?

First off - make a consensus decision that you want to move forward with FDD and give people time to mentally adjust to what that means. Adopt a norm that "We will adapt what we are doing to the FDD process."

Next, consider the new maintenance requirements and work through the 5 steps - build a domain model (in color preferably), then create a Feature List and plan the project by the groupoings of those features. This is the basic Coad Method as described in Peter's 1995 and 1997 books Object Models - strategies, patterns and applications. This should not change. The domain model provides the foundation for the communication mechanism in FDD. It provides the foundation for the quality assurance mechanisms such as design and code reviews.

So how do you reconcile the new model with the existing code? This seems to be the mental hurdle which causes many architects and developers to recoil and believe that FDD can't be made to work for them. There are two strategies for this.

(a) Legacy Wrapper Facade

Take the legacy system and effectively end-of-life it by wrapping it in a facade of service calls, i.e. make an API for the legacy system. This isn't always appropriate. It really only works when the new maintenance work is on the periphery of the existing code base. The new FDD system will talk to the existing system through an SI (System Interface) layer. All the features for the legacy wrapper and the SI classes in the new system would be listed on the feature list as SI features.

(b) Refactor the Model

Using a tool such as Borland Together generate a model for the existing code. Take that model and print it out - however, large! Then with a small team examine it and try to identify class archetypes and color them. Take the model for the new iteration and map it against the existing model and identify areas for refactoring such that the new classes will "fit" with the existing legacy. Write a feature list for the refactoring.

In both cases, there is a bucket of features allocated on the feature list for reworking the legacy. There is clearly a cost to this. Why is it worth making this investment?

It is assumed that the new code built with a Coad style domain model will be more robust, easier to maintain, more accepting of change, more flexible and ultimately the enabler of sustained momentum and higher project velocity in future iterations. You are paying now for improved velocity later. By surfacing the refactoring or system interface features, you are providing transparency into the process. It is then possible to make an informed and objective decision about whether the cost is worth paying.

[Martin Fowler discusses the re-write versus maintain and refactor approach in Strangler Application which appears to offer us some design strategies which would be compatible with color modeling. Fowler's Events are Coad's Moment-Interval archetypes whilst Fowler's Assets are Coad's Party|Place|Thing (green) archetypes.]

So after almost a month of posts about uncertainty and variation what is the point of all this Wheeler and Shewhart stuff and how does it relate to the Theory of Constraints?

Quite simply, understanding variation is necessary in order to follow TOC's Steps 2 (Exploit the Constraint) and 3 (Subordinate everything else in the system to the decisions in step 2). All systems involving humans tend to be probabilistic and empirical in nature. There is natural variation. In order to protect a constraint, where it is a work unit, a function, a person, or some other resource, it is necessary to understand the variation of the constraint and variation of the material running through it - whether that material is knowledge or plastic explosive (some people may recall my dad worked in an explosives factory - fascinating stuff similar process to the manufacture of sausages at your local butcher shop, I digress) - in order that you can adequately buffer for protection. If you are going to subordinate other parts of your system or process to the constraint's exploitation measures, then you need to understand the variation in other parts of the system so that something else doesn't accidentally become the constraint simply because you didn't leave enough slack in it to absorb its natural variation.

And so we have neatly tied TOC to Shewhart, Wheeler, Deming, the understanding of variation and its derivatives such as Six Sigma!

Lean and Six Sigma

How do Lean and Six Sigma interact with TOC? It is a question asked often. According to Eli Goldratt, TOC is the focusing mechanism for Lean and Six Sigma. In turn, they provide the mechanism for exploiting the constraint (TOC Step 2) and subordinating to the exploitation decision in step 2 (TOC Step 3).

For example, when the constraint is in the market, i.e. you can't sell enough of what you make, then the exploitation requirement is that you achieve

1. Excellent Due Date Performance
2. Short Lead Time
3. Satisfactory Quality (not excessive quality)

and do all three within the consumer tolerance. What better mechanisms for this than Lean and Six Sigma with their roots in understanding variation, Six Sigma's focus on reduction of variation, and Lean's focus on elimination of waste and shortening of lead time?

Yesterday's post should have gotten a few readers wondering about the constraints in a value chain. For example, if the first process in the value chain is in the Chaos state - say Requirements, because the customer doesn't know what they want - then how could it be possible for any other process to be in any other state than Chaos. Surely, if the input is chaotic then there can be no improvement at the output?

And yes, this would be true if the definition of conformant quality remained the same along the whole value chain. However, it rarely does and often the self-imposed definition of conformant quality is tighter than the customer tolerance. There is an argument which suggests that there is waste if the definition of conformant quality is tighter than the market demands. Over performance costs money and it sets a customer expectation which may erode trust in the vendor brand if it ever slips. So over delivery actually leaves no buffer. There is no concept of buffering for conformant quality.

So as a management tool, we must be prepared to rigorously determine what represents conformant quality in the marketplace and what is an acceptable tolerance in that quality. These metrics give us the definition and position of the Y axis in our Wheeler matrix. The tolerance in conformant quality defines the definition of the columns. Shewhart used the term "specification limits" as opposed to "control limits" or "natural process limits" which determine the X axis and the definition of the rows on the Wheeler matrix.

Using this approach and starting at the consumer, we can work our way back through the value chain and determine what represents conformant quality and tolerance such that the consumer's requirements are met appropriately.

What we achieve by doing this is a congruent alignment of departments (or suppliers) in the value chain. We also gain a tool for defining where change is needed and its impact when the consumer redefines their concept of conformant quality - or we (or a competitor) redefine it for them by using quality as a competitive weapon. When change happens we will see each process in the value chain move from Ideal State to the Threshold State (or some other state if special causes appear as a result of process changes) because the process cannot deliver with the "specification limits" of conformant quality. Each manager must then work to bring their process back to the Ideal state based on their new local definition of conformant quality and tolerance.