A good thread at Joel Spolsky's Ask Joel forum, is discussing, offshoring. What's really interesting about this is the amount of opinion without much in the way of objective data. The guys who mention cost as the motive are on the right track. The real problem is that the industry isn't measuring the right things and making an objective assessment of offshoring is difficult when there is a lack of information in the (software development) economy. What's missing is data on lead time and client-valued functionality produced.

Quoting from the 3rd sentence in my book, "Senior executives, perplexed by the spiraling costs of software development and depressed by poor results, poor quality, poor service, and a lack of transparency are simply shrugging their shoulders and saying, 'if the only way this can be done is badly, then let me do it badly at a fraction of the cost.'"

Rather than argue, that design can't be separated from coding - because no one who isn't in the software business will believe you - and even some that are won't believe you either - [Different parts of a design have different levels of uncertainty, not all parts of a system are truly unknown until the code is tested. Ref: Donald Reinertsen, Managing the Design Factory and some other pieces at this site including Separation of Concerns.] - instead developers must embrace transparency.

When executives have all the metrics - the throughput (or production rate),  the work-in-process inventory, which is directly proportional to, the lead time (the process flow time to complete working code) which is a direct input to, the rate of depreciation of the design/ideas inventory, and leads to a depreciated net present value for projected revenue, from dollar values associated with sales made/(or lost) through faster/(slower) delivery, together with the working capital required to fund less/(or more) work-in-process - only then can they make informed decisions about offshoring. While executives only have cost metrics, they will continue to make sub-optimal decisions. It's up to development organizations to learn to measure the right things and to report them transparently to the top of the organization. This will allow offshore facilities to be measured by the same yardstick. It isn't enough to subjectively argue that a subject matter expert in Chicago and a developer in Bangalore will take longer to produce a product with lower quality than when the two of them are in the same room together in New York. You have to supply the objective metrics to back the argument.

Quoting the 5th and 6th sentences from my book, "Software development has to cost less and produce better results, more reliably, with better customer service, and more transparency. This book will teach the agile manager how to achieve that."

|

This old but good thread at the FDD community site discusses how effective class ownership can be. I particularly liked this comment from Jeff De Luca.

If you do not use class ownership, when you assign a feature to someone then they must touch four classes (for example) in the system. They must have knowledge of four classes in the system. Think of very high use classes - such as LoanApplication in a Lending system. There would be a class with dozens of methods and attributes each written by a different person at a different date and time!!! (if not using class ownership)

Encapsulation is one of the founding principles of objects and is about a class being able to hide how it does what it does, and vary how it does what it does as long as it preserves its interface to all other objects. Well, class ownership naturally fits this fundamental requirement of objects. With class ownership you get better consistency of implementation and better consistency of interface.

|

Here's a suggestion for a new meme in FDD style projects - the Package Owner.

A Package Owner would be a Chief Programmer. The responsibility of package ownership is to take responsibility for the API/SPI of the package. To full understand its responsibilities as a larger-grained component and to hold the domain knowledge of its public interface. No one else on the team is allowed to change the public interface of a package, whether class or method, without the package owner's approval.

Package ownership, just like class ownership is all about maintaining internal quality and integrity. It's about developing coherent libraries which are maintained overtime and continue to make sense. Its about good OO design.

Can we take this meme up another level and declare distributed component owners? or component owners? Chief Programmers or Architects who take responsibility for the service-oriented interface externalized from a distributed component such as a SessionBean? More on this another time...

|

It's been written in several places how the Chief Programmer and Feature Team model in FDD were inspired by the surgical team model of Harlan Mills and advocated by Fred Brooks in The Mythical Man Month. It is worth a few lines to explain why the word "inspired" is so carefully chosen.

In Mills' surgical team model, the chief programmer is the surgeon and does all the programming. The rest of the team support the surgeon and keep him (or her) productive. This is very much a military style model - the 1:6 frontline troop:supply chain support troops ratio where it takes 6 soldiers to keep a single one fighting at the front. The argument for Mills' model is that one great developer is probably a 10x performer and hence it is worth spending resources for 5 or 6 people to support that.

In FDD, the Chief Programmer plays a different role. It is a role of communication but also leadership and design expertise. A Chief Programmer is a chief communicator. Communication happens within Feature Teams and is channeled through the Chief Programmer between Feature Teams and other people on a project. FDD is self-organizing at the Chief Programmer level because only the Chief Programmers are doing the project-wide communication. The other developers are Class Owners. They focus only on their classes and collaborate and communicate (generally) only with owners of other classes affected by the same Features in their current Work Package.

FDD seeks to get a performance improvement by elevating the individual developers by keeping them busy and uninterrupted. Chief Programmers absorb that communication on their behalf. It's a different formula than that of Mills' but its intent is the same - higher productivity.

|

As I've stated before, class ownership creates a constraint. The decision to use class ownership is a specific choice by a manager who wishes to achieve a higher degree of internal quality and accepts that the resultant constraint needs to be managed by subordinating all other decisions to the choice of constraint.

When assigning the classes to owners, some thought is required. Something we learned on the first FDD project in Singapore was that <<Moment-Interval>> classes, particularly key domain classes, must never be assigned to a chief programmer. For example, in a bank lending system, the class called ApplicationForLoan becomes a bottleneck. In a blogging application, the class called BlogEntry becomes a bottleneck. And so on...

For such classes, where you know there will be a high demand, you must choose a good strong developer who is preferably one of your fastest developers and most diligent with external and internal quality. However, you should not choose a developer who has communication responsibilities such as the chief programmers in FDD. Time spent communicating is time not spent coding. A key bottleneck class must be protected and the decision to assign an owner must be subordinated to other decisions. Hence, a chief programmer cannot be the owner of any of the most important classes in a system.

|

It is often (and naturally) assumed that the constraint on software development is the ability to program code. If only code production was faster then everything would be fine - right? So all we need is more programmers - right? or as I discussed yesterday, better tools which partially or fully automate coding - right? Maybe! The reality is that coding isn't as much of a constraint as people believe and that automating it will not deliver the significant gains that are anticipated.

Over the last 3 years or so, I've observed several projects and built up anecdotal evidence about how constraints move in a software organization. I haven't formalized it yet so I haven't written a paper about it. However, here is the anecdotal report [Note - as I suggest in the book, I am normalizing quality metrics using the Function Point concept]...

First the coding is the constraint and often this is because the developers are poorly organized and development is chaotic. As a result quality is often poor. Let's suggest that 3 bugs per Function Point would be typical - perhaps worse. [Note: Capers Jones reports that 5 bugs per function point is about as bad as it gets in America. Around 1 bug per Function Point is CMM Level 2 quality falling to 0.15 bugs per Function Point at CMM Level 5]. When you introduce any method - production rate goes up - often by several fold. Suddenly, development is not the constraint but rather the quality assurance function. QA can't produce full test coverage but bugs are found and development is once again the constraint whilst bugs are fixed. If you introduce more rigorous quality methods - typical of an agile method - then development once again improves and QA is once again the bottleneck. The answer is to increase the number of QA people. Strangely as quality improves, the number of QA people has to increase. With a very high quality development team, you may need a 1:1 ratio of testers to developers. As the QA function is elevated, test coverage increases and bugs found increases until eventually it levels out with full test coverage and quality better than 1 bug per Function Point.

At this point the developers start to starve for work to do. The constraint has moved forward in the process. It may have moved to architecture or analysis or UI design. The ability to feed the developers with input material has to be elevated. One way is to bring the business owners to the developers - as all agile methods advocate - and to reduce the formality of analysis work and increase the tactic knowledge or direct knowledge capture e.g. modeling requirements straight into UML as is typical with an FDD project. This may leave the constraint in the UI design group. This is where I prefer it.

However, what happens if you elevate UI design? Well the constraint tends to move to the other end of the process - the deployment or field operations unit, or down the value chain to consultants or systems integrators deploying the product. Elevating this may require better documentation, so more technical writers, or more adaptable code with a more modular architecture that is more customizable, or maybe quality is still insufficient with many bugs being found in the field through lack of proper production testing. However, if you fix all of these, where does the constraint move next?

It can go to the front or the very back of the process. At the front, the development organization is going too fast for marketing people to respond with a new product vision and new requirements - writing those MRDs or PRDs takes time. Alternatively, the constraint may move to the customer. The customer is simply not prepared to take new releases at the pace you are delivering them. This is a real problem because it means that the customer does not value your releases or your delivery lead time. Loss of value results which affects top line sales.

In Lean Production, the goal is a balanced flow where all the elements in a value chain move at the same pace. The TOC community differs from this view slightly suggesting that there must always be a single "drum" - a single capacity constrained resource (CCR) - in the system. Hence, it is "almost" balanced. However, the tools I mentioned yesterday disregard this concept. They seek to make one small piece of the value chain - coding - very fast. This will produce a limited benefit. Perhaps only cutting lead time by 20%-30% and resulting in less than a 100% improvement in throughput with only a small drop in work-in-process inventory. Only if almost (or full) automation is realized will there be a significant reduction in costs. Hence, if

Net Profit (NP) = Throughput (T) - Operating Expense (OE)
                        Inventory (I)

then automated coding tools will have a limited effect...

Net Profit (NP) = T*1.3 - OE*0.7
                        I*0.7

an approximate improvement approaching 100% might be a good guess.

On the other hand, end-to-end agile without fancy tools but rather a focus on people issues and discipline is likely to produce a 400% to 1000%improvement because agile methods allow the manager to deal appropriately with the moving constraint whilst a single point solution such as automation tool merely elevates a single resource which may or may not be capacity constrained.

|

Following up from yesterday's post, former Microsoft executive Charles Simonyi has clearly had his publicist working overtime. Not only did he get his picture in Business Week but an interview with him appears at News.Com (via Loosely Coupled). All this noise about his new company Intentional Software Inc. but apparently no product ready for another 2 years. Hmmm.

Simonyi claims that programming is the bottleneck (Ah ha! constraint language) and that this [bottleneck] needs to be eliminated through automation. The details here are somewhat nebulous but Simonyi appears to be talking about the notion of domain specific languages. Martin Fowler had something to say on this topic and how the OMG's MDA might enable it.

The crux of this ties closely to the notion that the design is the information in an information system. The code is merely a translation or representation of that information. In Lean terminology coding is "muda" (waste), if only the design could be directly implemented. If you can have the domain or subject matter experts express their design in their own language then why not automate it from there? No need for human intervention. No signal to noise attenuation. No implementation level bugs - just design or domain bugs. Simonyi claims that 7 sigma would be possible with such a technique. Hmmm (again).

It's a new paradigm! A new approach to software systems. It will require a new way of managing the lifecycle. From the wisdom of Eli Schragenheim writing in the Foreword of my book, "Once the limitation is vastly reduced, people should replace the old rules with new ones that take full advantage of the removal of the limitation. If this does not happen, then there is no added value...". A new paradigm would require a new way of managing, a new way of working - perhaps even a new workforce.

I remain skeptical of purely technology based solutions. My recent experience has taught me that it doesn't take a huge improvement in programming productivity to move the constraint somewhere else in the information systems value chain. At the point where the constraint moves, further improvement in the previous constraint has little overall effect - though it may reduce costs and improve ROI, it does not further improve throughput or shorten lead time.

[I'll write more about how and where the constraint moves tomorrow.]

|

In an interesting quirk of editorial timing the new print editions of The Economist and Business Week both have stories about outsourcing high-tech services jobs from America. The Economist points out in an editorial that this is nothing new and was to be expected and Adam Smith would approve, whilst there is a more in-depth piece against the backdrop of the US Presidential election in 2004 in the America section.

I'd love to link to the cover story of Business Week but they have hidden it behind a 65 cents per issue subscription login. What I love about this article is its accuracy and insight and understanding that the author clearly developed whilst writing it. So here are some poignant quotes from the hard copy...

"programmers ... must undergo the career equivalent of an extreme makeover. Traditionally, the profession has attracted brainy introverts who are content to code away in isolation. With so much of that work going overseas, though, the most successful American programmers will be those who master people skills."

Right on! As I pointed out a while ago on Duncan Smeed's weblog - it is way past time that a course in communication was part of a degree in computer science.

"Architects A few thousand tech visionaries sketch out entire systems to handle complex jobs... Outlook Outsourcing a non-issue."
"Project Managers Crucial cogs in global software factories. They coordinate the work of teams in different countries and time zones and provide dependable products on schedule. Outlook Good managers can write their own tickets. Pay has jumped 14.3% in the past two years."

The writing is on the wall - move up the value pyramid or suffer the economic consequences. [Note to self - discuss that 14.3% number with my boss]

In a related commentary...

"it's especially important that corporations keep innovating in software development. One technique that was born in the late 1990's is called extreme programming. In this approach, programmers work directly with business people breaking projects into pieces that can be written and tested fast. Pairs of programmers literally write code together simultaneously - creating real-time checks and quashing bugs as they go along. The technique is not yet widely employed, but it should be."

Hang on there! Has Business Week just declared in an editorial commentary that American jobs can be saved if agile software development methods are more widely adopted? Wow!

Meanwhile, a doff of my cap to Kent Beck who has clearly built a very strong brand - no mention of "agile" in the article only "extreme" and Kent is actually quoted in the piece.

|

via Steve Norrie, Watts Humphrey's people principles in programming...

|

Hal Macomber is interviewing the authors of Embracing Uncertainty this coming Thursday.

This reminded me of a term I coined in my book, "dark matter" - something which we know exists in the universe but we can't see it. Project dark matter is unrecognized client-valued functionality. Call it missing analysis if you prefer. Dark matter is features which you didn't have on the feature list but they were there all along. You discover them whilst doing the detailed design mid-project. Dark matter causes the feature count on projects to increase. To upper management it looks like scope creep but the customer will deny having asked for any changes.

Dark matter is always there - embrace it!

It is too easy to say, "next time we will do better analysis" or "next we will spend more time on analysis" - wrong! Embrace dark matter - it's out there!

The real question to be asking is "how certain are we about this domain?" and "how comfortable are we that we caught all the detail whilst doing the domain modeling and client-valued functionality identification?" Use this to calculate a scope buffer. On an FDD Feature List I enter this as empty line items and comment them as "dark matter". During the project as new features are discovered through more detailed analysis, I replace an empty line with the new feature.

On a recent project iteration, I asked for a 100% buffer for dark matter due to domain uncertainty. The request was denied. The project ran with a 50% buffer. During the project 61 features grew to 117 without any change requests from the customer. That's a 92% increase due to dark matter. Luckily we made the date by surprising ourselves and outperforming on both production rate and quality.

Why was it better to make a probabilistic guess about the scope and move forward rather than wait to get the analysis perfect?

Simply put, it would have taken weeks to resolve the analysis issues and gain a high level of certainty that everything had been captured. OTOH, in 4 weeks 117 features were coded and complete with absolutely certainty. By embracing dark matter, it was possible to move forward earlier and deliver earlier. In this example, the deliverable was achieved earlier than analysis would have completed had total confidence been a gating prerequisite to moving forward.

Lose the deterministic project management paradigm. Embrace uncertainty using a probabilistic approach to project management. Analysis can never be perfect. When you can accept that, and accept the probabilistic paradigm, then you can move forward earlier and complete sooner.

|

I was acutely reminded of work this evening whilst performing my domestic duties - making my daughter's pre-bedtime snack!

The customer was busy munching on a plate of cucumber slices which were left over from dinner. She had placed an order for "apple". As our house is semi-Japanese we serve Fuji apples. The point here is to realize that they are large. I knew that my customer was in capable of consuming a whole apple and my agility with the sharp knife was not up to sushi chef standards so there was also no way that I could peel, core and slice a whole apple before the customer would be demanding delivery. So I cut out a quarter and proceeded to core and slice it. Peeling the slices individually [I have no idea if this is optimal as I have never run a time-in-motion study on apple preparation].

With half of the quarter apple sliced and peeled, suddenly a voice proclaimed, "Finished!" and a translucent orange plate appeared above the sofa gripped by a small hand. Realizing that this is a prelude to the plate succumbing to the laws of gravity, I stopped what I was doing and went to retrieve it. I returned to finished the apple preparation.

Within 30 seconds the small hands were now prying the fridge door open to a chant of "Pear! Pear!" As I finished preparing the apples I offered the plate with a conciliatory, "I'll get you some pear. Why don't you eat these apples?" The plate was taken from me and duly launched across the kitchen floor. My diversion to multi-task and bus the empty plate had delayed delivery of the apples beyond the customer's threshold of tolerance. The customer had consequently changed her mind. Apple was out! Pear was in!

So here is what I should have done...

When collecting the empty plate, I should have delivered the partial batch of apple. My batch size had been small but not small enough. This would have bought me enough time to finish the whole batch and the customer would have been distracted enjoying yummie apple and not left to think about alternatives. Having delivered the second smaller batch of apple, I could then have moved on to pears, if indeed they were even required.

So how did this reflect my current reality...

It was only last Friday when I heard a colleague say, in response to yet another impending and impossible deadline, "When the date comes, we will deliver what we have working. We'll only build the most important features first. This will allow the customer to start testing and buy us time to complete the others. When they log a few bugs, we'll slip all the extra features into the bug fix build and hopefully no one will be any wiser".

|

I've been so busy recently, I didn't get a chance to mention Frank Patrick's excellent series on estimates and buffer management in Critical Chain. Read all 5 parts: Part 1; Part 2; Part 3; Part 4; Part 5.

|

After a lengthy break, Stephen Palmer is back at The Coad Letter. This new article looks at internal quality as a means of achieving the Lean principle of "build integrity in". Recently, I pointed out class ownership as one way that FDD achieves this. Steve goes further and hints at many FDD processes which combine to insure integrity is built-in to all systems developed with FDD.

|

My former colleague and fellow founding FDD team member, Stephen Palmer, has a nice solution for the problems of class ownership and continuous integration that I mentioned last week. He calls them Feature Team Areas.

The idea is simple. A Feature Team does not check-out code to their individual machines but instead to a communal network drive reserved for the Feature Team to use. They check-out all the classes involved in the Chief Programmer Work Package and modify them in the shared location. The classes can be tested in the shared location. Later, all the classes are checked-in together. This means that the build is never broken. The use of Feature Team areas facilitates continuous integration with FDD.

|

Slack! or "room for maneuver". You simply cannot work without it. If you want to mature, if you want to improve, if you want to perform better or out-perform a competitor, you simply must leave slack in your organizational loading. If you load your organization 100% you will never improve. More likely, things will gradually get worse. If you overload it, by say 150% then things will get worse, never better. Don't fool yourself by pretending that to get 100%, you have to ask for 150%. [Amazingly, a senior VP of a large company I used to work for, believed in this 150% rule. He actually ran the company that way. Somewhat unexplainably, I actually liked and respected him - but I never agreed with his policy for strategic planning].

|

I started this thread on the FDD website to discuss the problematic nature of class owenrship and the requirement to draw a sequence diagram as part of Step 4 - Design By Feature, whilst using the Together Control Center product in its code-diagram synchronous mode, i.e. the code is the model and the model is the code - there are no model files, just .java files.

First I ask the Feature Team to draw the sequence diagram for a Feature on a white board. I then ask the diagram creator to make the method signatures in all the classes - breaking the class owner constraint for a few brief minutes. I then have the classes checked-in. The diagram creator then draws the diagram with all the method signatures in place. This allows Together to prompt for the method call between classes on the diagram which makes drawing the diagram easier and faster.

In the event that a class in a sequence is currently checked-out by its owner due to work-in-progress, I ask the diagram creator to comment the method arrow on the sequence diagram rather than designate an "operation". This looks the same on the diagram but makes no changes to the code. This means that the diagram creator did not have to check-out the class but that the code and diagram are not synchronized.

Then the design review occurs. The diagram is checked-in.

In the original FDD project, Together did not keep sequence diagrams and code synchronized so that the act of drawing a sequence diagram did not create a need to lock class files. I would like to hear how others are dealing with this now and what is considered to be best practice.

|

There has been some considerable debate in the agile community about the percentage of good people required to make agile methods work. Most recently this thread at the FDD site. The topic was also a common running theme in Boehm and Turner's Balancing Agility and Discipline where on page 185 (for example) they postulate that FDD requires a lot of good people in order to scale. Schwaber and Beedle's Agile Software Development with Scrum claims on page 121 that it needs 50% experienced people to work. As I state in the thread at the FDD site, my experience with FDD is that 1 in 6 is about sufficient. Why?

Simply put, I believe that this endeavor to define how many good people or how many experienced people are required for any given method, is a red herring. The real issue is one of leadership versus uncertainty. Uncertainty comes in many forms - change (see Satir's change model, see also Peopleware 2nd ed page 199), scope ambiguity, domain ambiguity, schedule uncertainty, job description and positional ambiguity, fear, difficulty, novelty of process, technique, tools and lack of experience therein, then there is variance both common and special cause. The greater the uncertainty in any project, the more leadership is needed (at all levels) in order to compensate for it. By loading too much uncertainty into a project, without sufficient leadership, there will be the chaos that Satir identifies but rather than recovering from the slump to show an improvement - the J-Curve effect - there is simply a slump and things remain worse than they were before.

FDD gets its leadership at many levels - the Chief Programmer (the shop floor foreman) is the first level of leadership, then there is the Development Manager and Chief Architect. All these roles must provide leadership on a day-to-day basis in order to overcome the uncertainty. When a project is happening in a new domain, with uncertain scope and schedule, and FDD is being introduced for the first time, along with color modeling and perhaps new middleware tools or development tools such as a JDO implementation and the Togethersoft Control Center, then everything is in flux. In such a situation, a team needs every CP, the Dev Man and the Chief Architect all to be great leaders, mentors and teachers. In simpler situations, where there is less ambiguity then less leadership is required.

In order to be a leader, an individual must be both good at software development and experienced in the software lifecycle (with a particular method). Hence, "good" and "experienced" are actually emergent properties of "leadership".

The amount of leadership required on a project is situational. Trying to measure it by method and define a scale of agile methods versus percentage required is futile!

|