Tom Gilb Q6

Q: Where and when do you think it all went wrong in our industry?  I'm particularly interested in the historic misunderstanding that caused the waterfall model to spread.  What made it so "sticky?" compared to incremental / evolutionary development?

(I have asked Craig if i can put this on my website).

Well, It is a long story! Once Upon A Time, we had small hardware memories, and small programs, and smart people. So projects were short, simple and worked pretty well.
Then memory started getting much bigger.  Software got much bigger, and complexity of the software got horrendously much bigger. Things got late and buggy.

The obvious solution seemed to use good 'engineering methods'. So, based on a misunderstanding of Winston Royce, who pointed out that a Waterfall Process would only work for small systems, and that iteration/feedback/learning was necessary for complex systems, the US DoD, encouraged by Barry Boehm (a TRW Colleague of Royce), adopted the Waterfall Standard (DoD 2167 and 2167a). Well, it was not intended to be a strict Waterfall Process, the author personally told me, but it got misinterpreted that way. A little bit of requirements and design before coding is not such a bad idea. But the DoD, whose project size was exploding during this time, discovered that half of their $40 Billion annual software projects were total failures anyway. So in 1994 thy officially cancelled the 2167a Standard, and officially instituted the Mil Standard 498 Evolutionary Standard. It turns out that smart rocket scientists, who were not forced to do Waterfall, had always used Iterative/Feedback methods (http://clublet.com/examples/docs/IIDHistoryByLarman.pdf).  I campaigned for Evolutionary methods, mostly for deaf ears  practicing in Sixties, and publishing in Seventies. But the real awakening came when the Agile Boys read Principles of Software Engineering management (1988) and made iteration a fundamental practice. Too bad they did not read the entire book and realize that to control the iteration and learn, you also need to quantify the top level critical improvement objectives of the system. Some of my Agile friends, like Ryan Shriver, have recognized this failing and are actively practicing iterative control as I would advise. With focus on delivering measurable value to stakeholders, and not just stories to customers.

My underlying theory is that the Waterfall Method allowed Software Companies to earn a lot o money without actually delivering any or much value. So the real culprit is not unenlightened programmers, they are just foot soldiers. The real problem is the political leaders (top managers) who persist in paying so much for so little to so few.

http://www.gilb.com/community/tiki-download_file.php?fileId=38

So there was a historic misunderstanding (of Royce's message) - but the real problem, as I see it was lack of responsible management - to pay for value , not pay for useless programming work.