For those of you not reading this weblog in Scottish, "Happy New Year's Eve". As ever the best party is in Edinburgh and as I write this around noon PST, it ought to be in full swing back home. The Scots enjoy Hogmanay and New Year's Day so much - typically staying up until the wee hours have started elementary school - that 2nd of January is a holiday. So I'm guessing that people back home don't get back to work until Tuesday. Lucky them!

If your New Year's resolution is to "go agile in 2004" then I would urge you to remember that management is a necessary and vital part of that process. When looking for a management approach for agile development, I urge you to consider the wisdom and humor of Mike Myers and remember that "If it is not Scottish it's crap" :-) Happy New Year!

McKinsey Quarterly argues for a more sober future and a more rational approach to assessment of management. This article by Ian Davis argues that longer time windows are required for assessing the performance of an organization against its goal. The short-termism of the 1980's and 1990's has left a terrible legacy and resulted in many bad management decisions, the reward of bad managers and the punishment of many good ones.

If I have an issue with article it is the "hopefulness" of it. Sure it would be better if we redefined success over a longer time window, if we nurtured talent better and assessed its development over longer periods and if we looked to a wider role for business in society - more than just making profits and returning value to stockholders. In fact this 3rd theme is a key lesson from Built to Last by Collins and Porras. However, some others such as Warren Buffet argue that the only role for in the incorporated enterprise is to return value to the stockholders in the most economically efficient manner possible within the law. Society will then regulate in other ways what is or is not ethical. This is an argument for a central control of ethics and values rather than a distributed one. Davis seems to be arguing that a more distributed approach where companies self-regulate their own behavior for the longer term will produce better results.

So if we feel that a self-regulating world would be better and we would like companies to behave more soberly in 2004 and beyond, how do we go from here to there? That is the real question and it remains unanswered by McKinsey Quarterly and Mr. Davis.

I found this really appropriate quotation from the English philosopher Sir Francis Bacon. I wish I'd found this before the book went to print as it would have been a really good quotation for the start of Chapter 4 Dealing with Uncertainty or Chapter 7 Agile Project Management.

If a man will begin with certainties, he shall end in doubts; but if he be content to begin with doubts, he shall end in certainties. Bacon, Francis, Advancement of Learning, 1

Developers and managers alike often laugh at the suggestion that 4 fold improvements in programmer productivity are possible with changes in working practices. Numbers like 10 fold seem like fantasy. Most people I speak to think I'm crazy when I tell them that I believe that 40 fold improvements will be possible within 15 years and that history will look back on the first 40 years for software engineering as a craft era.

There is a split opinion on the usefulness of history. Henry Ford declared that it was "Bunk!" Whilst we are often reminded that "those who do not learn the lessons from history are condemned to repeat it". For those of you who align with that second sentiment then you may enjoy this article, A Great Yarn from The Economist Christmas Special.

The article charts the history of cotton. What has that got to do with software development? In my opinion - a lot! Why?

The industrial revolution has a lot in common with the information revolution through which we are living. Landed gentry farmers from England used their wealth to expand into plantations in the colonies. This meant running banana, sugar and cotton plantations in the Americas staffed by slaves from Africa. The raw material was brought back to England for added-value processing and then sold throughout the rapidly expanding British Empire.

Processes like spinning cotton were the high technology of their day. Investment banking was basically invented during his period to facilitate the flow of capital from the gentry farmers with raw material wealth to those with ideas for spinning jennies and steam engines and locomotive power and steam powered looms and so on and so on. It was the venture capital industry of its day. A virtuous cycle was started where wealthy people invested in new ideas which generated yet more wealth. The Economist does a good job of explaining this for cotton - a key element in the industrial revolution and the creation (eventually) of untold new wealth and higher standards of living for all.

Note how closely the now unfashionable use of imported slave labor reflects the use today of the H1B and (even more so) L1 Visa in Silicon Valley. Rapid expansion fueled by new technology creates labor shortages. Migrant workers fill that demand.

The most important details in this article for me are the statistics. Over a 70 year period cotton production got 200 times better. Not only did this not destroy jobs but instead it created yet greater demand for the product and generated yet more wealth.

I firmly believe that a confluence of two things - management science and knowledge of best working practices, together with improved tools - is creating the beginnings of the "spinning jenny effect" for software development. OMG's MDA or Steve Mellor's "executable UML" or OASIS ebXML or BPM may not be the right tools but they are going the right way. Combine the right answer in tools with the knowledge we are gaining about agile software engineering and it's a sure thing that we are on the verge of a paradigm shifting change for the better. The craft era is ending and 70 years from now an article in The Economist Christmas Special will look back at the changes in software and knowledge work and reflect that there has been a huge improvement in productivity (of at least 40 fold).

I was watching the DVD of Rudolph the Red Nose Reindeer with my daughter last night. I was amused to note that the constraint in the elf toy factory is the paint shop operated by an elf called Herbie - remember the walk in the hills example from The Goal where the slowest boy is also Herbie. How do we know Herbie is the constraint? Because the foreman sees the inventory building up in front of him. Why is Herbie the constraint? Because he is not motivated in his job and his throughput is reduced through lack of proper motivation. Herbie wants to give up being the factory constraint and become a dentist instead. As I've pointed out before, dentists really understand how to be the constraint and how to manage around it.

I see a parable in the life of Herbie and the elf toy factory. I'm not the first person to observe that software engineers need to be properly motivated to be productive and I have often talked of the role of leadership and management in creating a properly motivating environment and jobs for software engineers. However, I may have been the first person to point out that lack of motivation can cause developers to become the system constraint. I spend a lot of effort in the book making this clear. The techniques in "Agile Management..." help a manager to identify the constraint. In the case of a lack of motivation, I delegate to others, the problem of elevating the constraint. There has been mush written by the likes of Jerry Weinberg, Tom De Marco, Tim Lister, and Larry Constantine on how to motivate software engineers. I didn't need to repeat their advice.

In the movie, the foreman elf "motivates" Herbie by telling him that he should shut up and put up, and that his life is that of a toy factory elf - forget becoming a dentist. The result is that Herbie drops out and becomes a "misfit". I'm sad to reflect that I have seen many similar attempts at motivation over this last 2 years during the recession in IT. All too often, management will tell developers that they are "lucky to have a job" and to "stop asking for help" and to "just get busy coding".

My prediction then for 2004 is that as the economy improves (even if this is temporary) it will unleash a backlog of pent up frustration and we will see excessive staff churn as geeks move on in search of a better boss, a better, more motivating environment and new challenge.

[Updated 12/28/03 1330 PST]

This thread of discussion at Alan Francis ' blog has prompted me to re-evaluate my affiliation with the "agile community" (to use my own phrase).

First, let's define "agile community" as the Agile Alliance and its members, those who have signed the Agile Manifesto or those who have written a paper, spoken at a conference or written a book on the topic of agile software development.

I often refer to the "agile community" as if I were an observer of it rather than a part of it. Why should this be so? Firstly and foremostly it is about objectivity. It is important to be able to hold a neutral mindset and try to evaluate "agile" as others see it - those others are many of my colleagues in management positions in Fortune 1000 companies. So a standoffish objectivity is important for me because it is important to my audience.

However, there are other reasons why I don't feel part of the "agile community". One is that I don't work as a consultant and I don't make a profession out of attending conferences and hanging out in hotel lobbies and bars with the other "agilists". Fair enough! But there is something more subtle - I worry that the agile community gets sucked into perpetual naval gazing. A state where it is more important to worry about whether something is "agile" or not, or whether something is "more agile" than something else. To me these are not important questions. The important question is whether software development is aligned with the stockholders interests and whether software development can be performed in such a manner as to improve the return to the stockholders. It's all about better software, faster! Good, fast, cheap - pick 3!

Hence, I have no time for discussion about "but is this self-organizing" or not. I'm not interested in the observed attributes of some "agile" methods - only in the results.

As such I will continue to pursue the study of management science and software engineering, as I see it happening in the Fortune 1000 on medium and large scale projects, for continual improvement of shareholder aligned performance. What is the goal of the organization? How best can the day-to-day activities of software engineers be organized to align with that?

I am not interested in being a member of a club for the sake of membership, nor am I interested in political association for the benefits of that association (though I recognize that political association is a key part to being a successful manager). I am not interested in aligning with a movement because I have something to sell. I don't work as a consultant. I don't sell anything. I may make some pocket money from selling a few books but it's small reward for the hours spent developing the material (including the on-going effort of this site).

As long as the "agile community" continues to focus on the right questions, "How can we govern software engineering better? How can we align our day-to-day activities with the stockholders interests? How can we improve the return to the stockholders through software development? How can we report our progress transparently and in a customer valued manner? and How can we continue to build better software, faster and cheaper?" then I will want to part of that community. It is these principles that were at the heart of Feature Driven Development and why I am proud to be part of it. If the "agile community" degenerates into discussion of "my method is more self-organizing than yours" or "my method has faster feedback than yours" or "my method is more fun than yours" then I'm not interested. As Jeff De Luca has said (paraphrasing) at the start of many projects, "You're not here to have fun. You're not here to make your resume look good. You're not here to use cool technology. All those things may be important but they are secondary. You are here first and foremost to make a profit for the stockholders. Now let's get started."

Chapter 16 of "Agile Management..." looks at Product Management. It's an area that arguably doesn't get enough coverage in the book. In fact, I point out that there is a whole other book to be written on the topic. In the Foreword, Eli Schragenheim tries to compensate for this shortcoming by giving some examples of how to better understand and define the value associated with the features in a product.

This post on Venchar points out how valuable the role of usability engineering and both product and interface design are to realizing value and its associated attributes such as market share acquisition. I've commented elsewhere in online posts that if I did another book it would look at this "marketing" aspect of software development management and focus on increasing Throughput through better value acquisition. I observed that such a book would need to merge the best of product management with the best of usability. Initial feedback showed that few people could see a role for such a combination of topics.

I intend to explore this subject more - product management, design and usability - over at my uidesign.net blog. If you've got opinions on how usability and design can be used to enhance product management and ultimately enhance ROI through improved realization of Throughput, then I'd love to hear from you.

A couple of recent posts have reminded me of one of my rules of thumb - developers who spend too much time on the debugger are not good developers and should be rolled off my team.

Keith Ray writes in Cost of Debugging quoting Christian Sepulveda,

One half of each day was completely wasted. The developer couldn't really doing anything productive during the four-minute debug cycle because he had to navigate to the correct place in the application and execute the right steps so he could observe the results of the change he just made; he was forced into tedium. In other words, half the development budget was being spent on performing repeated, monotonous tasks. This isn't a very effective use of resources or talent.

It was my former mentor Duncan Smeed who pointed at Robert C. Martin's Debuggers are a Wasteful Timesink, which took me back to experiences as a contractor working at IBM's PC Company and my earlier roots as a machine code programmer in the games industry in the early 1980's. [Aside: In fact if you ever owned a Sinclair/Timex Spectrum there is a strong chance you owned and played one of my games.]

In a world where programmers are measured by activity and dedication to "activity" then they get rewarded for spending hours on the debugger. I can specifically recall one developer who was earning almost twice as much as me to sit and play with the debugger until 9pm every night when I was going home at 3.36pm (the earliest available moment under the flexitime rules). Why did I go home so early? I had finished my code and it was tested and working.

In the 1980's, we didn't really have debuggers and when we did they were useless because we were testing real-time performance and so much was running off timers and interrupt driven. It just wasn't possible to test with a debugger. It was time wasting and useless. I learned to code by putting logging and instrumentation into my code. I learned to develop by small increments fully unit tested. And the biggest lesson I ever learned when working as a team is - you agree the interface (the method signatures) at design time and you b****y well stick to them. With assembly language, the concept of "breaking the build" blows the machine up - and in those days we didn't have multi-process spaces to protect the memory and keep the machine alive.

The smell of an over-revving debugger resembles the teen spirit which encourages idle youth to spin tires with the parking break on. It makes a lot of noise, it creates smoke, it provides an illusion of activity and intelligence, but ultimately the velocity is zero.

[Updated 12/21 1300hrs PST]

Blogging has been light this month and it will get lighter as Christmas approaches. My dad has been staying with me for the holidays. It's his first trip outside Europe. I've been trying to spend as much time with him as possible. I've had a couple of chances to do that this year.

Back in September, he was reminiscing about his short spell as a line manager. He worked for 30 years at the Nobel's Explosive Company. Few people remember that Alfred Nobel - famous for the Peace Prize (and others) - was the inventor of Nitroglycerin and later Dynamite. After scouring Europe looking for a manufacturing site, he was granted permission to build a factory on 7 square miles of sand dunes on the north bank of the river Irvine in Scotland at Ardeer where the British Dynamite Company opened in 1871. Over 100 years later, the plant was still operating, mostly producing detonators for the mining industry.

For 1 year of his career, he managed 300 women employed to hand assemble these miniature explosive devices. After only a year in the job, he was moved because he was "too nice for the job" and instead became an instructor in the factory's apprentice school. During his tenure, he build a good rapport with the staff. He averted 3 strikes and increased the productivity whilst avoiding lost production due to threatened industrial action. In Britain in the 1970's, strikes were common and the unions strong. Managers who were good at industrial relations were few and far between.

It's so important that managers are measured objectively using metrics which are aligned with the stock holders interest. Objective measurement should lead to rational decisions about good and bad managers and good and bad management techniques. It is all too easy to fall into the arrogance of a "them" and "us" attitude - to think that being a manager makes you (somehow) better than other people. It's counterintuitive in a world where people arrogantly think they are better than their staff that the right way to manage people is to treat them with respect and empower them with the ability to perform better. To see things the right way, objective, rational measurement is required. This insures that managers who produce more by being "nice" to their staff will be rewarded not punished.

Jason Marshall points out why it is so important (more than ever) to treat staff fairly and to focus managers on keeping them well motivated - to balance the "emotional capital". Knowledge workers have a lot of power to undermine without appearing to be doing anything other than their job. When a development team is poorly motivated by bad management, the results will show up in the productivity figures.

If an organization is not measuring the flow and delivery of client-valued functionality then managers can not be judged objectively and the trend in staff productivity cannot be monitored.

Chapter 6 revisits a theme from Chapter 4, the main variables of scope, budget and schedule, and examines how traditional software engineering methods choose a different constraint than Agile or RAD methods. Traditional SDLC methods fix the scope and allow the budget and schedule to vary. They define "quality" as conformance to scope delivery. This choice is rooted in the project management methods of the PMI and is indelibly fixed into the SEI SW-CMM (capability maturity model) and the definition of ISO 9000-13 software development quality assurance. Agile and RAD methods fix the delivery date and allow the budget and scope to vary.

The Agile choice is the better choice and Chapter 4 explained why - the scope has much greater uncertainty than either the budget or a date. In fact as sure as the Sun rises, any date chosen will arrive. Dates are absolutely certain. What is unknown is how much functionality can be delivered by that date. The Traditional method of fixing the scope means that the variable with the greatest uncertainty is set as the mark by which quality is measured. This inevitably means that any estimates of delivery date and budget based on a fixed scope have a high margin of error. Error on the high side is failure under ISO 9000 standard and CMM quality definitions.

Next, Chapter 6 looks at the effect of cost accounting and its focus on tracking effort expended and value-added. This leads to the false security of recording work-in-progress as an asset. This is somewhat of a historical perspective as, at least in the United States, it is no longer normal practice to capitalize intangibles such as software development in-progress. However, even as recently as 3 years ago, this was common - particularly in high technology companies which were losing money. Recording WIP on the balance sheet takes it out of the P&L and makes the loss look lower.

Throughput Accounting assigns all costs incurred in software development to OE (operating expense) whilst only recording value-added on completion and delivery. Hence, value is only recorded at two points - the input and the output. There is no value-added for partially completed work.

When you start to measure things this way, it naturally leads to a different way of tracking projects - tracking the flow of value. This is a concept from Lean Production and really it isn't a commonly accepted project management concept at all. The notion of using it for Agile project management was floated academically by Koskela and Howell in the paper, The Underlying Theory of Project Management is Obsolete published in 2002 by the Project Management Institute.

I recognized this proposal as reflecting how FDD projects are managed. With a slight tweaking of the FDD Feature Complete graph, to use the Lean Production reporting tool of Cumulative Flow Diagrams (CFD), as I described at in this article at the FDD website, it can be shown that Agile software methods were already employing the technique advocated by Koskela and Howell. CFDs had been advocated by Donald Reinertsen in his 1997 book Managing the Design Factory as the ideal method for tracking the flow of added information during a design process. As software development is an information discovery, knowledge work endeavor, it seemed only natural that CFDs were the correct choice.

I've mentioned Chapter 5 - Software Production Metrics on this site before but not yet in the weblog. The whole of Chapter 5 is available as a PDF from the book section here. The content of Chapter 5 comes up repeatedly on the site, for example, measuring non-functional requirements as inventory which has value which can be tracked independently from Dec 7th.

The essence of Software Production Metrics is establishing tracking and reporting for software engineering which is aligned with the financial metrics described in Chapter 2 - Management Accounting for Systems. Ultimately, this is about alignment - alignment of the daily activities of software engineers with the financial goals of the business and the best interests of the stockholders. Chapter 5 is about good governance at grass roots level. Everything else in the book is about implementing this good governance through alignment with ever increasing performance.

The ideal metrics are - Production Rate (often called velocity in the Agile community) of client-valued functionality, Inventory of client-valued functionality in progress, and its derivative, Lead Time (time to complete a single unit of client valued functionality. Finally to help with cost estimation, it is useful to calculate Average OE per unit of inventory.

Quality is not mentioned in Chapter 5 because quality is really a derivative. If quality is poor then Production Rate will fall and Inventory will rise. Quality measured in bugs per client-valued functionis described in Chapter 9.

I've been reading Built to Last by Jim Collins and Jerry Porras. The authors conducted painstaking research into many large companies which have lasted a long time - more than 50 years - and extracted out in an abstract form the common themes which these businesses have used to be successful. I'll talk more about the many themes in the book at a later date but tonight I'd like to talk about just one - "The Tyranny of the OR".

In software engineering, we have been brought up with the notion that there are three main constraints - quality (good), time to market or schedule (fast), or the cost - resources allocated to the project (cheap). It has been widely written that a business must choose 2 - "Good, Fast, Cheap - Pick Two" - and sacrifice the other to gain the chosen two e.g. good quality, low cost (small resources), with a long, slow schedule. This is manifestation of what Collins and Porras have termed "The Tyranny of the OR". That is to say that you can have good and fast, OR good and cheap, OR fast and cheap, but critically not all three.

Collins and Porras point out that companies which are built to last do not accept the "The Tyranny of the OR" but instead embrace "The Genius of the AND". These business simply refuse to accept that it is not possible to do it all.

The Agile software community and the Agile Manifesto declare a belief in "The Genius of the AND". Agile software development is all about having it all - good quality through rigorous testing, reviewing, and learning - fast speed through face-to-face communication, less bureaucracy and more tacit knowledge - low cost through small teams of empowered generalist developers.

50 years from now, the companies that were built to last in this decade will have demonstrated that they refused to be intimidated by "The Tyranny of the OR", but rather believed in "The Genius of the AND" and by doing so embraced Agile software development as a key strategy in having it all.

I'd like to revisit Joel's December 1st article, Craftsmanship, and examine the problem he describes for his import file feature...

The user chooses a file using a standard dialog box, and the program copies that file into the CityDesk database. This turned out to be a great example of one of those places where "the last 1% of the code takes 90% of the time."

In "Agile Management..." I describe tracking the flow of value, where value is defined as "small pieces of client-valued functionality". In this case Joel wants to import a file from disk into the CityDesk database. [Disclosure: agilemanagement.net is published using CityDesk]. However, as Joel points out delivering the feature produced deceptive tracking information, "the last 1% of the code takes 90% of the time". Why?

Simply put this feature was dominated by non-functional requirements such as usability rather than its functional requirement of importing a file.

To deal with situations like this, "Agile Management..." recommends that the client-valued non-functionality is also broken out and tracked separately. This would produce a usability requirement that says for  long operations to bring up a progress bar of some sort, another functional requirement of cancel a file import in progress and a final architectural requirement of run long import operations (requiring a progress bar) in a separate thread and permit the application to continue in use without error whilst the import is in progress.

Once we have done this, we now have 4 features instead of 1.

The benefit of this breakout for the developer, development organization and the customer is immense. Each of these 4 features can now be tracked separately and values can be assigned to them separately. Using Joel's example, the first 99% of the code would show that feature 1 was completed. Then the other 3 features would allow that "last 1%" which takes "99% of the time" to be tracked separately. The ROI for these additional features could then be estimated separately and the management would be able to make a tradeoff - how much is it worth spending in order to delight customers with these additional 3 features?

Now, I'm not suggesting that FogCreek Software go and change how they do things. No sir! FogCreek is a small owner managed operation with the owner still coding himself. However, had CityDesk been a product of a larger company then the "last 1%" taking "90% of the time" would have resulted in a broken promise. Broken promises are bad for business and they cause a loss of trust. A loss of trust causes a systemic effect which means that estimates are not believed in future which means that the work is not balanced with the capacity which causes overloading of the organization, which causes stress, resulting in a tendency to abandon quality, which results in long lead times whilst bug fixes are made, causing further stress on QA, further slippage and more missed schedules, which leads to a further loss of trust and even less belief in the ability to estimate. A vicious downward spiral!

In larger organizations, it is vital that client-valued non-functionality is broken out and tracked separately. When the business is playing with all of the facts then it can make objective decisions and tradeoffs between desirable product behavior and the costs incurred to create it. As Joel says, craftsmanship is too expensive and I would add too obscure. Large companies need objectivity. You can get this through analysis which reveals all teh value then tracking it separately.

Paul Glen re-examines the annual planning ritual in his monthly column in Computerworld. He thinks its a chance to look at the psychology and leadership not just the budget and management. Here is a snippet...

Leadership. Leadership is particularly important because it has the ability to transform all the other facets of group dynamics. Good leadership offers the possibility of positive change rather than stagnation or chaos.

 

Although a very complex subject, there are a few questions to ask about the quality of your leadership team. How strong are our relationships with our clients and peer organizations? How do the staffers feel about their managers and one another? Do we have coherent and generally accepted processes and goals?

Human brains are just as good on any continent as long as they receive adequate education. The mainstream American press are beginning to realize this and talk about the potential impact on the economy. When you see an article like this one, The Rise of India, in Business Week then you know it is mainstream.

Still, this deep source of low-cost, high-IQ, English-speaking brainpower may soon have a more far-reaching impact on the U.S. than China. Manufacturing -- China's strength -- accounts for just 14% of U.S. output and 11% of jobs. India's forte is services -- which make up 60% of the U.S. economy and employ two-thirds of its workers. And Indian knowledge workers are making their way up the New Economy food chain, mastering tasks requiring analysis, marketing acumen, and creativity.

No one seems quite sure whether this is good or bad. In principle, all economists will tell you it is good. The work should move to where it can be done most cost-effectively. This will return the most money to the U.S. economy. However, the short term and localized impact on a section of the population may become a political problem.

If the US feels compelled to protect its steel industry with tariffs and barriers what is the likely response to its current economic crown jewels - the software industry?

Joel Spolsky shows that he really gets it, with this piece on Craftsmanship and why it is too expensive as a method for commercial software development. Joel prefers the Design approach described by Donald Reinertsen. As Don points out in various books and articles, knowledge work is all about creation of information. Once you have the right and complete information, design stops and production begins.

Crosstalk, the Journal of Defense Software Engineering, has a special on management this month. It includes pieces by management bloggers, Johanna Rothman and Esther Derby, as well as a book advertising piece by Boehm and Turner. My eye was caught by "Back to Basics: Measurement and Metrics". As an exercise for you the reader, you might like to compare the approach discussed here with the simplicity of the model in Chapter 5 of "Agile Management..." - WIP Inventory (number of client-valued functions in process), Lead Time (time to code a client-valued function), average cost per client-valued function and quality (number of bugs per client-valued function). One linked reference looks interesting, "Goal Driven Software Measurement - A Guidebook". I haven't had time to read this yet. However, it caught my eye - having just written a book about goal-driven software engineering. However, compare the length of this piece on metrics at 189 pages with Section 1 of "Agile Management" which is about the whole lifecycle process including marketing, at a mere 150 pages.