Chaos is Bunk (Allegedly)

Sometimes those queasy feelings of something not being at all right are entirely justified. Only, you don’t find out until years afterwards. According to J. Laurenz Eveleens and Chris Verhoef writing in the Jan/Feb 2010 issue of IEEE Software, the much-quoted Chaos reports are seriously (for which read fatally) flawed.

Back in 1994, Standish created a sensation by claiming that only a “shocking” 16% of projects were successful, 53% were “challenged”, and an astounding 31% failed outright.

Further, Standish presented a small table of figures which purported to demonstrate beyond doubt that the overwhelming reasons for project failure were down to poor project management, and in particular to lack of proper attention to requirements.

The figures arrived at just the right moment for would-be vendors of Requirements Management tools to set out their wares. They quoted Standish to “prove” that SOMETHING HAD TO BE DONE to improve requirements, and that something was their tool. It was far too convenient to question.

Requirements textbook authors and publishers, too, leapt onto the bandwagon. Page 3 of Writing Better Requirements, for example, quotes the Standish “Reasons for project failure”, with “Incomplete Requirements” at 13.1%, “Didn’t involve users” at 12.4% and so on. Mind you, the same book also mentioned a study in IEE Review which found that only 22% of project managers identified requirements as an important cause of project difficulties.

In any case, there was an odd mismatch with experience. Everybody had seen projects which were running a bit late, a bit over budget, or which had shifting requirements – but out-and-out disasters were not especially common, even allowing for the fact that people tended not to shout about them.

Doubts about Standish were raised by Robert Glass, writing in IEEE Software (2005) and elsewhere; and by M. Jørgensen, writing in Information and Software Technology (2006). They suggested, say Eveleens and Verhoef, that “the only way to assess the Chaos results’ credibility is to use Standish’s data and reiterate their analyses.” But “there’s another way: obtain your own data and reproduce Standish’s research to assess its validity.”

And that’s what they did. They looked at no fewer than 5,457 forecasts applied on 1,211 industrial projects. Guess what they found? “Our research shows that the Standish definitions of successful and challenged projects have four major problems: they’re misleading, one-sided, pervert the estimation practice, and result in meaningless figures”, they wrote. So much for feather-bedded academic pussyfooting: RQ hopes they’ve got good lawyers.

The trouble with Standish, explain Eveleens and Verhoef, is that success is defined as forecast/actual (f/a) cost and time are greater than or equal to 1, while f/a functionality is less than or equal to 1. Challenged is defined similarly, but using strict “greater than”. They quote Jørgensen to show that a project that is, for example, within budget and time does not fall into ANY of the Standish categories! They arbitrarily assigned such projects to challenged, which may be what Standish did also.

They point out that projects that are, say, 25% over budget or time are often perfectly acceptable. This seems obvious to RQ: work may have been novel and obviously risky; overruns may have been allowed for in contingency; and such projects may have led to software that brought in many times the money invested.

The Standish definitions also do not allow for underruns. Eveleens and Verhoef show by modelling their own data that estimation approaches such as Barry Boehm’s or Tom DeMarco’s rightly expect some estimates to be high and some low; but Standish does not. In one case, a large financial services provider made 667 careful estimates on 140 software projects. The forecasts were assessed independently by a metrics group at the time they were made, and shown to be “best-in-class”.

Those projects were, in other words, exceptionally well managed.

But under the Standish definitions, working with the initial forecasts, the projects were (calculated Eveleens and Verhoef) only 59% successful. It wasn’t the projects that were at fault, or the organization’s management: it was the Standish methodology.

In another example, Eveleens and Verhoef compare Standish “success” to “real estimation accuracy”. Organization X seems on the Standish approach to be far more successful than others; but that is because X “overestimates from tenfold to a hundredfold”.

The authors communicated their findings to the Standish Group, “and Chairman [Jim] Johnson replied: ‘All data and information in the Chaos reports and all Standish reports should be considered Standish opinion and the reader bears all risk in the use of this opinion.’”

Eveleens and Verhoef conclude “We fully support this disclaimer, which to our knowledge was never stated in the Chaos reports.”

There are good reasons for doing requirements well. But dodgy statistics isn’t one of them.