Tuesday 23 December 2014

Shane Warden - Q6

Q6.  How did you two work together when writing?  Are you geographically close?  Oh, and how did the two of you meet?  And how and when did you decide to write a book together?

We both live in the Portland, Oregon metro area, somewhere between 25 and 40 minutes by car, depending on traffic.  It was easy to get together in person if we needed to, but usually we ended up in the same place for other reasons (hanging out with the guys on Thursday nights, for example).

We spent a couple of weeks coming up with a really great outline as part of the book proposal process, and then did it again when it was clear that the book needed to change dramatically from our original vision.  That gave us several small sections for each chapter, and we divided those between us equally, where Jim would write the first draft and then I'd edit it, and vice versa.  Occasionally we spent an afternoon in person making notes and plans for more complex sessions.  For example, we took a couple of days to spread notecards around my living room to figure out the third section of the book when we realized again that our original approach wasn't quite right.

We originally met... well, Jim's girlfriend at the time (now wife) called my roommate (at the time) to invite him to Jim's 30th birthday party a few years ago.  I tagged along.  Somehow he found out that I was a programmer and I found out that he was a consultant and programmer and that was a few months before I started working on the XP Pocket Guide.  He seemed like a perfect candidate to review it.
A couple of years after that, he had the idea of revising the pocket guide.  I resisted for a while, but it eventually made sense, at least until we looked at the outline and said "This isn't a pocket guide anymore."

Q6 and Q7 - Johanna Rothman

Q6) What's the most fun you have in your job?

I love all of the consulting, teaching, and writing that's part of my job now. I love meeting all these people all over the world. I love doing assessments, and seeing how people are actually working. Writing about that so the leadership can choose to change--that's a challenge and I do enjoy it. I get a kick out of teaching and helping people learn new approaches. I really enjoy all the other consulting too: facilitating a retrospective or strategic planning or portfolio gathering and evaluation--helping people see what to do and helping them be successful--is such a blast. Writing is a different kind of fun. Most often, writing is solitary, so I enjoy it, but sometimes I wrestle with my brain to get the words out :-) When I do succeed, I feel great.

Q7) What's the least enjoyable part of your job?

It's sometimes difficult to help the accounting people at my clients live up to the terms we agreed upon at the beginning of the work.

Q5 - Johanna Rothman - The Hudson Bay Story

Q5) I loved the Hudson Bay story in your book ... can you share that here?

Sure. Here's the quote from _Manage it!_ :"The Hudson Bay Start approach was originated by the Hudson Bay Company in the 1600--1700s in north eastern Canada. The Hudson Bay Company outfitted fur traders. To make sure the traders hadn't forgotten anything they needed, they left Hudson Bay and camped just a few miles away. By making camp just a few miles away, the traders ensured they hadn't forgotten any tools or supplies---before they abandoned civilization. With just a short start to their journey, they had a better idea about their ability to endure the winter.

On one notable project, the management team mandated a new computer, which necessitated a new compiler, a new build system, a new product with a new GUI, and most of a new team. The team had no idea how to make anything happen, so I suggested we use a Hudson Bay Start. I suggested we do "Hello world" just to see if we could. Of course, everyone laughed at me (sometimes that's the role of the PM), and it took us an entire week to write Hello world and nothing else to the screen. We celebrated at the end of the week, and now we had enough data to estimate what it might *really* take to do the work. Sure, we went a little faster as people learned how to make the infrastructure work, but the product was difficult to imagine and implement. Our estimates were surprisingly close for several months. That's the value of a short iteration. Learn by doing and use that data to predict the next iteration.

Q4 - Johanna Rothman

imageQ4) Where would you send a seasoned PM who wants to know more about agile project management and software development?

 

First, I would suggest that the PM read _Manage It!_, because some seasoned PMs have funny ideas about Agile. Agile is the most disciplined approach to software, because the discipline arises from the team (with obstacle removal by the PM). But some seasoned PMs don't realize that the team's discipline is more important than the PM's discipline. With a disciplined team, the team can accomplish great things. But with just a disciplined PM, everyone is frustrated, and only the PM can solve problems. Not a scalable solution. Then, it depends on how the PM learns and the PM's experiences. A seasoned PM with an open mind might benefit from an Agile conference or a Scrum class. But in my experience, the PM will need to learn collaboration and team skills. The best place to do that is at the AYE conference (yes, I'm one of the hosts).

Q3 - Johanna Rothman

Q3: Of your books, which one is your personal favourite (for whatever reason)?

Come on! That's like asking which of your children you love more :-)

I'm very proud of Hiring the Best, because I learned how to write while writing that book. I'm thrilled with the pairing I did with Esther on Behind Closed Doors. And, I love Manage It! because although I prefer agile or incremental approaches, I can see when it makes sense to use other approaches, and I hope my readers will see that too.

Q2 - Johanna Rothman

Q2: Could you tell us a little about each of your books?

  • Hiring the Best Knowledge Workers,Techies & Nerds: The Secrets and Science of Hiring Technical People is about how to hire, from developing a hiring strategy to the first day and what to do if you can't find the right person. I wrote it because I've been fortunate enough to hire 100 or so people when I was a manager (and live with them in the organization). I also wrote it because I realized when I did assessments that a lot of the problems I saw in organizations was that nice people were in the wrong jobs.
  • Behind Closed Doors: Secrets of Great Management I pair-wrote with Esther Derby. We saw managers failing at their jobs--not even completely failing, but missing significant portions of their jobs. If managers don't develop personal and trusting relationships with their staff (and their peers), they can't effectively manage. What a waste. So, we wrote the book.
  • Manage It! Your Guide to Modern, Pragmatic Project Management is the book I've been writing for 10 or so years :-) I started writing articles about project management in 1997, and had been teaching project management since the late 80's, so it made sense to finally write the book. It's funny--some people have said, "It's a big book" but I couldn't write it any smaller. I'm very proud of this book, because I didn't include anything I haven't actually done on a project.

My next book is about managing the project portfolio. It turns out that a number of schedule games (I named 15 in Manage It!) arise from not managing the project portfolio, so I decided it was time to tackle that subject. I've been managing the portfolio in organizations and helping managers determine what they need to do as a consultant since the late 80's, so I have a lot of knowledge here too :-)

Q1 - Johanna Rothman

Today I kick off a series of questions and answers with another big-hitting agile thinker Johanna Rothman.   Johanna has recently published the excellent Manage It! and she'll be giving away a copy as we near the end of the questions.  She also writes one of my favourite blogs.

Q1: Hi Johanna, I know you mostly through your (excellent) blog and your other writing.  Could you please tell us a little more about yourself - both personally and professionally?

Okay, this is a tougher question than it should be :-)

Personally, I'm married to a wonderful guy who can actually fix stuff around the house. When I "help," he always asks me if I've read step 2, because I tend to gloss over some of the early details in my impatience to get to the meat of the problem. We have 2 daughters, 19 and 15. I no longer know anything about small children, but I know a lot about teenagers :-)

Professionally, I have a BS in Computer Science and an MS in Systems Engineering. For you BAs, that MS is about how to develop systems, not how to write requirements. I started working professionally in 1977 as a developer. About 8 years later, I became a tester for a couple of  years, and then focused on project management, program management, and people management. I write code for fun still, but it's definitely not production quality code. I started my consulting business Labor Day of 1994. (For my non-US readers, that's early September, 1994.)

Q1 - Johanna Rothman

Today I kick off a series of questions and answers with another big-hitting agile thinker Johanna Rothman.   Johanna has recently published the excellent Manage It! and she'll be giving away a copy as we near the end of the questions.  She also writes one of my favourite blogs.

Q1: Hi Johanna, I know you mostly through your (excellent) blog and your other writing.  Could you please tell us a little more about yourself - both personally and professionally?

Okay, this is a tougher question than it should be :-)

Personally, I'm married to a wonderful guy who can actually fix stuff around the house. When I "help," he always asks me if I've read step 2, because I tend to gloss over some of the early details in my impatience to get to the meat of the problem. We have 2 daughters, 19 and 15. I no longer know anything about small children, but I know a lot about teenagers :-)

Professionally, I have a BS in Computer Science and an MS in Systems Engineering. For you BAs, that MS is about how to develop systems, not how to write requirements. I started working professionally in 1977 as a developer. About 8 years later, I became a tester for a couple of  years, and then focused on project management, program management, and people management. I write code for fun still, but it's definitely not production quality code. I started my consulting business Labor Day of 1994. (For my non-US readers, that's early September, 1994.)

Shane Warden - Q5

Q5: Can you tell me something odd, unexpected or just plain fascinating?  Something your friends know ...

Jim knows this, but not everyone does.  My undergraduate work was in music, specifically performance, with a smattering of history and almost zero mathematics or science.  It's not that I hate either math or science (I don't), but I decided against studying engineering the year before college and went the other direction to the arts and things you can't prove with the empirical method.

Once in a while I do wish that I'd taken a compilers class though.

Jim Shore - Q4

Q4 - What has been your least satisfying Agile experience?

Well, just as when it's great fun to have everything firing on all cylinders, it's very frustrating when the team struggles, particularly when you've had the opposite experience.  For me, that's usually because people dismiss an idea, like pair programming, without trying it.  They just decide that it can't possibly work, and that's all there is to it.  Also, because I'm a consultant, some people form this instant dislike of me before I even walk in the door.  You've seen the Dilbert cartoons: the consul"tick".  That's me.  It can be disheartening.

One of the worst moments for me was during the .com crash, shortly after I went independent.  Work dried up, so I found a job through a headhunting firm and ended up as an "in-the-trenches" programmer on a web app in bug-fix crunch mode.  I ended up getting them to try agile development (after about six months of soul-destroying effort), but it didn't work out.  That project taught me the importance of making sure a company is ready for agile before convincing them to try it.  In retrospect, I was pretty arrogant in my approach, too.  (What can I say--excessive humility has never been my problem.)

I wrote about that experience in my "Change Diary"--on my website at http://jamesshore.com/Change-Diary/.  It's one of the most popular pieces on my site.

(There's a silver lining to it all.  That company has now switched over to Scrum+XP, at least in part, and they're apparently having success with it.)

Jim Shore - Q3

Q3 - What has been your most satisfying Agile experience?

My favorite aspect of agile development is the collaboration and camaraderie that builds up over time.  The really great teams have this intense energy to them.  People laugh and joke and really have a lot of fun.  The work fires on all cylinders and there's this feeling of immense productivity.

I love that feeling.

I first experienced a hyper-productive team with my second XP project.  I first tried XP out on a small project (three developers) and had moderate success.  The manager asked me to come coach a larger dev team and I convinced them to give XP a try.  We worked together for nearly a year, and then I moved out of state, set up a satellite team, and we continued working on the same codebase for another six months.

That project was probably the most fun I've ever had at work.  We all got along very well.  Most of the team would go out for lunch together every day.  We pair programmed, wisecracked, and produced some of the best code I've seen.

The code didn't actually start out all that well.  It's not like we were super-geniuses or anything.  A lot of it was my fault: I didn't totally trust XP's ideas of evolutionary design, so I worked with one of the leads to create an up-front architecture.  It was terrible!  We didn't know that, of course.  But we gradually improved it, and by the end of the first year we had a nice, elegant abstraction.  That project taught me that code can actually get easier to modify over time, not harder, so long as you commit to constantly improving it.

(I wrote a paper about the experience of constantly improving design for the first XP Universe, here: http://www.xpuniverse.com/2001/pdfs/EP203.pdf.  That was seven years ago... I'd probably be embarrassed to read it now.)

Shane Warden Q4

Q4 - What has been your least satisfying Agile experience?

My last full-time software development job before I wrote the first book was underwhelming.  I had a couple of great co-workers, but our development process was... ambitious and underwhelming at the same time.  We used a mixture of Scrum and XP where we dutifully broke up all of our work into stories and engineering tasks, estimated the necessary work, let our tests drive our design and refactoring, and did pretty well at all of the development practices.  (I wish we'd paired more, but....)

The problem was that it was a very small consulting shop with lots of different small customers to satisfy, so we'd work on multiple projects every week. We did have a single grand project to work on, but only when we had had enough billable work for the month for us to spend time on a new product. Mostly we lacked a grand cohesive vision for our work, at least one that could predict what we should work on at a high-level on a weekly basis.

It wasn't an unpleasant experience -- I do remember favorably the experience of customizing a F/OSS point of sale system to the point where our largest customer could use it successfully in a month's time -- but it was unsatisfying because we had to spend so much time just staying in business that we didn't have as many opportunities to succeed at our business. Despite that, our software quality improved dramatically and we were able to meet the needs of our customers much better, thanks in part to much better development practices.

Shane Warden Q3

Q3 - What has been your most satisfying Agile experience?

I think this story's in the book, near the end.  A good friend of mine has been skeptical about agile development and testing and pair programming for many years.  She's actually a great reviewer of drafts and manuscripts because she's honest and analytical and asks good questions.

We sat down a couple of years ago to try pair programming for the first time. She had said "I don't think I'm ever going to understand this deeply unless I do this with someone who already knows how."  We spent a couple of hours ping-ponging back and forth, where I'd write a couple of lines of a test case and she'd write a couple of lines of code to make the test pass, then talk
about any refactorings, and then she'd write a test and I'd write some code. She didn't care for that at all.  She kept saying that it felt like we weren't making any progress, like we were taking baby steps and doing screechingly obvious things.

I won't say that that project was much of a success; we managed to get in about 90 minutes of pairing and could have gone another hour or two to start seeing really great results, but when she went back to work the next week, a handful of other people suddenly asked her to show them how pairing worked, and she had some experience to draw on.

Maybe you want a more successful experience.  I have to say that every time someone asks me to write some code and hands me a failing test, I feel plenty proud.

Shane Warden Q2

Q2 - One of the things which annoys me most in the agile community is when I hear people saying things like "That's not pure agile" when what they really mean is "That's not XP, like I read about in a book".  For me Agile is about getting better and XP is just one of the  ways to do so - albeit, a very good way!  I recall that you received a bit of flak while we were reviewing your book because the focus was on XP, but the book title indicated it was about Agile in general.  In retrospect I think you made the right choice and you've explained your reasoning very well in the book.  Could you share your reasoning here?

My very high level goal is to have better software and a better environment for people involved in creating and using software.  When I work on a project, I want a pleasant work experience with high morale, fulfilling work, and personal and professional satisfaction.  I think agile development helps achieve that, and I think XP is the most effective and cohesive approach within the agile universe, but if projects achieve organizational and personal success without strictly adhering to what one book or website or consultant or another says, I count that as success for the team.

What we're trying to do is achieve organizational success as well as personal success.  Jim and I tried to explain what we think is the best way for a team fairly new to agile development to start achieving that success.  Ultimately we believe that teams who follow our advice and eventually start changing the process to meet their own environments more effectively will have mastered agility, even if they end up doing certain things very differently from how they started.

We did (and do) feel very strongly that we have a responsibility to give the best advice we have, though, and we tried to do that even if it wasn't always fun or pleasant.  For example, a lot of people hate the idea of pair programming, and so I occasionally see a lot of agile advocates back away from recommending pair programming.  That's a mistake; it's a powerful practice that has many benefits you can't get in gestalt anywhere else, not to the same degree.

With that said, there are ways to make pair programming more difficult and ways to make it less difficult.  I believe we explored the discussion honestly, but our recommendation is that no matter how awkward it seems for the first few days, if you can, you should try it for at least a month and measure carefully how it affects your work.

Shane Warden Q1

Shane Warden is co-author of "The Art of Agile Development" and author of the Extreme Programming Pocket Guide (and others).

Q1- My first question is the same as I asked Jim,  I'm really enjoying your book.  I honestly think it is the best agile book I've read - and I've read a  good few of them.  Good job!  Can you tell me a bit about yourself - both personally and professionally?  What got you to the point where you could produce such a wonderful book?

Like Jim, I started programming when I was very young.  I must have been six or seven years old when I saw my first minicomputer, and I sat down, opened the manual, and typed the first program I came to.  (For some reason I ignored the line numbers in the book and typed the program directly, which didn't exactly work.  I suppose some of us learn to debug early.)

For a long time, nothing happened.  My undergraduate degree is in music, but I realized that I wanted something with more stable job prospects, so I returned to technology after I graduated.

During a stint as a system administrator as the dot com era heated up, I rediscovered programming and spent a lot of time reading the source code of free software projects, trying to improve my skills and discovering that successful projects had development processes very different from those I was seeing in our corporate IT group.  When I read the first XP book, I realized that there were important commonalities I'd seen in both, as well as practices that were missing (in particular, refactoring and test-driven development).

Through the next phase of my career (consultant, writer, software developer) I continued to explore the ideas of agility especially as they related to software quality.  I wrote a lot of tests, test frameworks, test tutorials, and testing advice, and I started to see how adopting even just one or two practices improved software development.  I also saw first-hand how adopting just one or two practices without considering their context made it difficult to succeed in software development.  (In particular, the lack of stakeholder support can doom almost any project.)

I wrote the XP Pocket Guide for O'Reilly, and in the year of research and writing (and admittedly arguing with Jim occasionally about how strict or lenient to be and how people were likely to pick and choose what they wanted to do) I really understood how everything fit together.  If you look at the Art of Agile Development, you'll notice that almost every practice we discuss has one or more contraindications.  Sometimes you really can't practice the practice precisely the way we recommend, and that's fine -- but you need something besides a tremendous amount of luck to fill in the gaps left behind.

Jim Shore - Q2

Q2 - One of the things which annoys me most in the agile community is when I hear people saying things like "That's not pure agile" when what they really mean is "That's not XP, like I read about in a book".  For me Agile is about getting better and XP is just one of the  ways to do so - albeit, a very good way!  I recall that you received a bit of flak while we were reviewing your book because the focus was on XP, but the book title indicated it was about Agile in general.  In retrospect I think you made the right choice and you've explained your reasoning very well in the book.  Could you share your reasoning here?

Sure.  We wanted to make a book that was very practical and useful, but we didn't want to fall into the trap of watering down agile development.  In my consulting work, I've seen that when people pick and choose agile practices, they usually pick the ones they like (iterations, mostly, and sometimes test-driven development) and ignore the ones they don't like (pair programming, sitting together, cross-functional teams, evolutionary design).

That's not really that surprising, is it?  People only choose practices they like--big insight!  But the problem is that they're leaving out essential practices without realizing what they're missing.  It's okay to leave a practice out, but you have to replace it with something, and they're not.  Then they have trouble with technical debt and making reliable plans, but don't know why.  After all, what do pair programming and on-site customers have to do with estimating?  Everything, as it turns out.

Now, we agree that each team needs a customized approach to agile development.  The problem is that teams new to agile make poor customization choices.  So we decided to describe a single method, with a fairly limited set of choices, that will work well for most teams. (And we describe when it won't work well.)  The idea is that the team can follow the method we describe, practice it a bunch, talk about their experiences, learn from it, and then use that experience to make good customization decisions.  One of my favourite aspects of the book is our inclusion of a third part, designed to be read months after the rest of the book, that talks about customizing your agile method.

Jim Shore - Q1

Jim Shore is my first guest on the AgileThinkers blog.  He's just co-authored a wonderful book "The Art of Agile Development" along with Shane Warden.  Shane's also going to answer the questions but I'll be posting them separately.  We're going to be giving a way a copy of the book shortly.

Q1.  Hi Jim,  I'm really enjoying your book.  I honestly think it is the best agile book I've read - and I've read a good few of them.  Good job!  Can you tell me a bit about yourself - both personally and professionally?  What got you to the point where you could produce such a wonderful book?

Thank you!  I should start by recognizing my coauthor, Shane Warden.  Shane and I collaborated closely on the book, so that's part of the reason it turned out so well.  Another reason would be the sheer amount of time and effort we put into it--I put nearly a year of full-time work into it myself, and that's not counting Shane's effort or the effort of all the people at O'Reilly.  I think our open review process helped a lot, too.  We put nearly every section of the book online and we got tons of feedback--over 1200 messages were posted to the reviewer list. Our best work started as drafts that got brutal beatings.

So: blood, sweat, and tears.  You know, the usual.

As for how I personally got here... you know, it's hard to say.  I've always liked reading and writing, even though I was born a hardcore geek.  (I placed in my first programming contest at age ten or eleven.  I think I would have come in first if my program had actually loaded off the tape drive.)  I actually did better on the English portion of the SATs than the Math portion.  I also wasted a huge amount of time on Fidonet and Usenet, talking about programming and (in retrospect) being a general pain in the ass.  I could spend hours composing careful explanations of why OS/2's Worksplace Shell was better than the Windows desktop.  I expended far more effort than the topics were worth, except for two things: I had fun, and I had a lot of practice writing and seeing how my writing was received.

So that's how I learned to write.  On the software side, the really core experience was when I had an AppleSoft BASIC program dissolve on me.  Applesoft BASIC was old-school: line numbers, two-letter variables, GOTO's, the works.  When I was in high school, the complexity of one of the programs I was writing overwhelmed my capacity to keep track of it all.  The program just fell apart in my brain.  Ever since, I've been fascinated with the human side of programming: if programming is partly an exercise in making the unbelievably complex comprehensible, how do we do that?  That led me to structured programming, then OOP, then software engineering texts, then Ward's Wiki, then XP and agile development, and now to questioning what makes software successful.  Along the way, I did the same thing I did with writing: I spent hours every day just playing with the ideas, writing programs, writing about programming, and so on.  (As well as my professional experience.)

So really, the answer is that, for most of my life, I've had a pretty single-minded dedication to writing, programming, and software development.  (The word you're looking for: Nerd.)  It's amazing I ever married, let alone reproduced.

Q1 - The Poppendiecks

Q1.  Hi Tom and Mary,  I suppose I should have started this blog with you two since it was your book (actually, an early draft of your book) that got me into this business, but I didn't, so please forgive me :) It must be about 5 years now since you started writing the book, but can you step back even further than that and tell me about your lives before that?  Several people have emailed me to complain that there's too much techie stuff in this blog and not enough romance, so how'd you two meet?  Was it love at first sight?  I'd also like to hear more about your "pre-LSD" lives - correct me if I'm wrong, but your resumes don't hint strongly that you were both headed towards gurudom ...

Mary replied:

Life before Lean Software Development

When the professor was calling role the first day of our college physics class, he asked some guy in the front:  “Are you the one who won the high school science fair a couple of years ago?”  At the end of the class he asked us to choose a lab partner, and since I didn’t know anyone in the class, I made a bee line toward the guy in front who was smart enough to enter and win a science fair. Meanwhile, that guy was looking around for a female lab partner, and so we met. It was certainly not love at first sight, it was all about getting through physics class with a decent lab partner. 

But by the end of the semester, we were serious about one another, and I had finally learned to spell “Poppendieck”.  The professor had graded all of our lab reports and put them in a pile for us to claim.  As Tom and I dug through the pile to retrieve our lab reports, time and again Tom would laugh and say – look at how you spelled Poppendieck this time!  Yes, there are many ways to spell the name, and when our kids were young, they discovered that there are also a great number of nicknames that can be derived from Poppendieck.  When they complained, I would tell them “Talk to your father.  He grew up with the name – ask him how he handled it.” 

I proceeded to get a masters degree in mathematics (before computer science programs were widely available), while Tom supported us by teaching physics in high school.  Then he went on to get a PhD in physics, while I supported us by working as a programmer in the physics department at the University of Wisconsin.  We lived in married student housing, where the apartments were so small that we refused to share our meager space with a TV.  Parents traded babysitting script for evenings out, and Tom took our daughter to a day care center on the back of his bike.   

My job was to teach one of those new things called mini-computers to digitize bubble chamber film and send a preliminary analysis to tape for later number crunching on the big Univac computer on campus.  Tom got his PhD by analyzing the (100) surface layer of silicon in ultra-high vacuum.  Eventually Tom ended up getting a job as assistant professor at Hamline University in St. Paul.  I asked my local DEC salesman in Madison:  “Who’s buying DEC mini-computers in the Minneapolis area?” and I quickly found a job in the engineering research department of 3M, exploring options for mini-computer control of processes.

After a few years as a physics professor, Tom found himself bringing the first computers into Hamline University.  He remembers crawling through the heating tunnels stringing cables, and eventually helping the school establish the policy that all students should have access to a word processor.  I was glad that Tom was a professor, because I was traveling quite a bit starting up process control systems in 3M plants.  Professors have free time to take kids to the doctor and dentist, and they can even stay home most of the day when kids are sick. 

But then our jobs changed.  Tom’s university decided to formalize the job he was doing and call it “Director of Academic Computing”.  He was asked to apply for the job.  I suggested that if they were going to have him apply for his own job, he might as well send out some resumes and see if anyone else would want to hire him.  Within two weeks, Tom had a job working for Honeywell designing a product data management system for a division which developed inertial laser gyroscope based navigation systems for commercial airplanes. 

At about the same time, I was offered a job as IT Manager in a nearby 3M plant where I had just overseen the installed a process monitoring system.  This was my first management job, and my first encounter with the quality movement and with just-in-time.  Tom encountered the total quality movement, activity based management, and just-in-time at Honeywell at about the same time.

After a few years, I returned to 3M headquarters and worked in new product development, while Tom went on to become a software consultant.  When 3M was spinning off Imation, I was offered an early retirement package I couldn’t refuse.  So in the late 90’s I abandoned Minneapolis for a year and moved to California, working for a start-up company specializing in high purity polymer that I had taken an equity position in while I was at 3M.  Meanwhile, Tom traveled extensively, trying to help companies as far away as Brazil use IBM’s “SanFrancisco” framework.  We ended up in the same city every weekend – sometimes home, sometimes California, sometimes Las Vegas, or wherever. We had two tandem bikes – one in California and one home in Minneapolis – and we usually went on a long bike ride together every weekend.  Our vacations were almost always week-long bike rides across some state or other.

Back home in Minneapolis, I was looking for some way to make a bit of money for a few years, after which my pension would kick in and pay the bills.  Tom said there was a great need for project managers, and I figured that I could get back into software development easily enough, having plenty of background in programming and IT management. I started up our business and got a job gathering requirements (using JAD) for a state government project.  JAD I understood, I had used it before.  This thing called “Waterfall,” however, was new to me; I couldn’t figure out how that could possibly work – and actually it didn’t.  I ended up as the third project manager of that project, trying to rescue it when it got into trouble, with mixed results.  When it was over I decided to write a book about how ideas from the quality movement and lean manufacturing offer a better way to develop software.

Part way through the book, an economic downturn caused Tom’s consulting firm to falter and thus Tom joined me as author of the book and as a partner in the business.  At first we worked separately, but one November I was invited to give a talk at XP Days in London.  Airfares to London were so cheap that I thought it would be nice if Tom joined me.  That trip turned into a whirlwind tour of Edinburgh and Malmo, and as we were struggling with the luggage and the logistics I said to myself:  “I was going to come here alone?  How dumb was THAT?”  And we have never traveled separately overseas again.

We have found that we love to travel and see new places, we love to meet people and talk about lean software development, and we are very fortunate that people want us to share our insights with them and give us more insights to share with others.

Tom Gilb Q7

Q: Can you give us a simple example (or two) of how to quantify requirements?  I recall one interesting example you gave when you did training for us at AgileScotland that involved (I think) making online survey forms easier to use ...

Well, the longer answer is available:
http://www.gilb.com/community/tiki-download_file.php?fileId=124  <- Quantifying Quality Paper.

But the main idea is that for a critical improvement (usually 'Quality') attribute - we need to articulate the idea with Numbers.

This involves two steps:

1. Define  scale of measure
2. Define some useful numeric points on that scale.

So instead of 'better user friendliness"

we might write:

Usability:
Scale: The average time needed for an average user to correctly perform representative tasks.
Past 10 hours.
Goal [2009] 1 hour.

Most people can quickly pick up on doing this.
When you get stuck there are lots of available Scale patterns to use, and to tailor to your specific purposes.

My favorite teaching example is quantifying 'Love' !
A friend pointed out the Bible, Corintians 1, had the solution! - as I pointed out earlier.

Google them!  For example:  "Intuitiveness software metric"  , if all else fails try   Intuitiveness software metric gilb  :)

Tom Gilb Q6

Q: Where and when do you think it all went wrong in our industry?  I'm particularly interested in the historic misunderstanding that caused the waterfall model to spread.  What made it so "sticky?" compared to incremental / evolutionary development?

(I have asked Craig if i can put this on my website).

Well, It is a long story! Once Upon A Time, we had small hardware memories, and small programs, and smart people. So projects were short, simple and worked pretty well.
Then memory started getting much bigger.  Software got much bigger, and complexity of the software got horrendously much bigger. Things got late and buggy.

The obvious solution seemed to use good 'engineering methods'. So, based on a misunderstanding of Winston Royce, who pointed out that a Waterfall Process would only work for small systems, and that iteration/feedback/learning was necessary for complex systems, the US DoD, encouraged by Barry Boehm (a TRW Colleague of Royce), adopted the Waterfall Standard (DoD 2167 and 2167a). Well, it was not intended to be a strict Waterfall Process, the author personally told me, but it got misinterpreted that way. A little bit of requirements and design before coding is not such a bad idea. But the DoD, whose project size was exploding during this time, discovered that half of their $40 Billion annual software projects were total failures anyway. So in 1994 thy officially cancelled the 2167a Standard, and officially instituted the Mil Standard 498 Evolutionary Standard. It turns out that smart rocket scientists, who were not forced to do Waterfall, had always used Iterative/Feedback methods (http://clublet.com/examples/docs/IIDHistoryByLarman.pdf).  I campaigned for Evolutionary methods, mostly for deaf ears  practicing in Sixties, and publishing in Seventies. But the real awakening came when the Agile Boys read Principles of Software Engineering management (1988) and made iteration a fundamental practice. Too bad they did not read the entire book and realize that to control the iteration and learn, you also need to quantify the top level critical improvement objectives of the system. Some of my Agile friends, like Ryan Shriver, have recognized this failing and are actively practicing iterative control as I would advise. With focus on delivering measurable value to stakeholders, and not just stories to customers.

My underlying theory is that the Waterfall Method allowed Software Companies to earn a lot o money without actually delivering any or much value. So the real culprit is not unenlightened programmers, they are just foot soldiers. The real problem is the political leaders (top managers) who persist in paying so much for so little to so few.

http://www.gilb.com/community/tiki-download_file.php?fileId=38

So there was a historic misunderstanding (of Royce's message) - but the real problem, as I see it was lack of responsible management - to pay for value , not pay for useless programming work.

Tom Gilb Q5

Q: Is it true that you are related to Santa?

I love giving gifts, especially knowledge, and I have many helper elves who help spread my gifts. I live nearer the North Pole than you do.

And my Summer cabin is right next to Drøbak - home of Santa Claus

http://www.emb-norway.ca/facts/Traditions/Christmas/julenissen.htm

But, I am not allowed to admit any more.

Tom Gilb - an example from Jens ...

Jens Egil Evensen has been using Tom and Kai Gilb's EVO approach with Scrum.  Here's a few questions and some very interesting answers with Jens.  I strong recommend you read on ...

A very

Q: Hi Jens, Tom said that you've been using Scrum combined with EVO.  Can you tell me a little background about the project and what problem EVO solved that Scrum didn't?

I’ve been studying and using Evo for the past four years. First as an employee in the Norwegian Product Register (governmental) after attending a course with Tom Gilb,, and later as a consultant in one of Norways largest IT corporations (EDB ASA (Avenir)). Avenir has a rather complex and “heavy” framework for managing large waterfall like projects called AvePro (“homemade” PMI inspired) that they have been working on and used for 20 years.

One of my first assignments in Avenir (in December 2006) was when Microsoft asked us to be an implementation partner for a Norwegian telecommunications company that had decided to implement Microsoft CRM 3.0 as their standard CRM platform. The requirement specifications that where presented to us where purely functional, and there where no references to what goals or business values they wanted to achieve by implement the system. I quickly concluded that our traditional methods of project management would not cope with such a project. One of the reasons where that I suspected that if we just developed a system based on the specification that we had received, the changes in requirements during the course of the project would turn the project into one giant contract nightmare, and that delivery of the system would be infinitely delayed. It would simply join the majority of all other software development projects in the basket of failure.

We agreed to do a one month test period with one consultant (me), kind of no cure no pay, to qualify us and the system for their needs. I told the customer that with the method I intended to use, we were quite sure to give at least some stakeholder some value, and that this value would be far beyond the cost of the initial implementation of the project.

To do this I had two simple requests for the customer. I wanted one workshop with the sales managers ad the director of sales, and one workshop with the employees from their internal IT department that where suppose to participate in the project.

I started the workshop with the management by asking two simple questions:
1: What is this company's vision?
2: If you don’t think about IT or software at all, what are your greatest problem today, that prevents you from fulfilling this vision or goal?

(BTW, I’m not of the very formal types, so I attended this meeting with the management in my usual work outfit; T-shirt and jeans – in Norway that is almost perfectly okay)

At question #1, they answered a bit staggering, and after a short while, they said with a little laugh that is was their vision to be a leading telecommunications company in Norway. Laughing as if they where embarrassed by this vision (I call this Norwegian modesty). I thought that was an excellent vision, and a good starting point for our project. No point in setting the goal to low. If you’re aiming for the moon you are more likely to reach the sky than if you only aim for the roof of the nearest building.

At question #2, the director of sales had a clearer view. He said, and I quote: “We are loosing xx million Norwegian kroner each year on contracts that are ending and lost, without even trying to win them back, or prolonging them. Nobody knows they even exist”.

I answered: Okay, so if I implement a function in your system that alerts the account manager of a customer, let’s say, 2 month’s in advance that a contract is ending, you will potentially earn xx million kroner more?”. He answered yes.

This was actually a function that they could have implemented in their existing system at any time, but the management of sales had never discussed this business related problem with the IT department, and the IT department where probably to busy surviving bugs in the existing systems to listen to the real stakeholders everyday problems.

I told them that I could implement such a function for them in only 14 days and where met with a short silence... Then the director of marketing said: I never thought I would hear such a thing a IT consultant”.

In the workshop with the resources from the IT department, we discussed all the practical tasks necessary to install and configure the CRM database and server, and some other technical stuff. Just enough for me to gain their respect, and to make them know that I really knew what I was talking about when it came to programming and technical issues.

I also told them that to be real heroes to the management, they had to deliver solutions that really made some increased business value for important stakeholders in the management. Cool features might be impressive, but they are short lived and soon forgotten. Business value is sustainable. They where very skilled resources, and had many years of experience, but they had never thought of it in that way.

I told them: When your boss ask you to do something for him, you must ask why. Only then will you know what his real goal is, and only then will you be able to really help him. The answer to his question is never yes or no, it is rather one or more design ideas with an explanation, a cost estimate, and an impact estimate on his goal and other relevant impacts (good or bad). This information will make your boss able to make a smart decision on whether the solution should be implemented or not, and it will make the IT guys real heroes in the eyes of the management.

After working in this way for a month, creating business value they really needed, and showing the potential of the method, they prolonged the contract, and wanted more resources from our company. The total number of hours for this project became about 5,300. In Norway, that is considered a mid size to large project. We kept delivering business value to the customer in small weekly (!) increments for 15 months.

When the number of people working on the project increased, organizing the work became challenging. Mostly because of the rapid increments in the project, and  that not all of the developers working on the project where experienced, and needed more follow up. Evo is a great method to break down top level goals into business value and end state requirements, but contains no mechanism to organize the developers, and how they work. Although design specifications where made in detail, we had trouble with developers that where developing outside the scope of the project, and struggling for days with problems that weren't important to solve at all, without telling anybody (also called thrashing?).

To get the necessary control of the production environment Scrum where partly implemented with one team, daily stand-up and a product backlog. That product backlog where populated with design specifications from the Evo cycles. This, I found, was an ideal combination. Now I had total control of the most important requirements, expressed as the business value they wanted to achieve. A breakdown into end state stakeholder requirements. Mapping between business value and designs that where implemented. The stakeholders received business value for their money (bang for bucks), and the programmers no longer wrote code; they created business value!! Their work had suddenly got meaning, and they loved it.

As far as my knowledge goes, no single method widely used today covers all aspects of a project, from Vision to values, values to requirements, requirements to design, and design to working systems.

All methods seems to cover the “project layer” where the author of the method works, and maybe one layer up and one layer down. To cover all layers you will have to use more than one method, and select methods that have compatible interfaces, such as Evo and Scrum (Evo design specifications into the Scrum product backlog).

Avenir has now added this methodology (in a modified form) as one of our preferred agile project delivery methods (Avegility). It’s built up as three parallel cycles named “Goals and requirements”, “Production (ie Scrum)” and “Delivery and measurement”.

Q: How did you learn enough about EVO to get started?  Was that easy?

As mentioned earlier, I first learned about Evo when I attended a course held by Tom Gilb, and I must admit that the first two days of the course, I did not understand where he was going with this at all. After the course I tried it in a project at work, and I was truly amazed with the result. Lifting my eyes from the code and the software, and focusing on the required results opened up a whole new world of possible solution for me. As a consultant for a software company I must admit that this sometimes is a problem; when looking at clients requirements and goals today, I far too often sees that the best solution seldom is buying or developing new software, but to begin with the existing software, or the way they work.

If you got that business gene, and a true passion for the results you create, living the Evo is easy. If code, technicalities and gadgets are all you live for, you will probably not get the point at all.
Some people never realize that the best system is the one delivering the best results for the business, and not the fastest, prettiest and the one with the perfect architecture (although we have to care about those things as well, but then as expressed quality and performance requirements related to business values and goals)

Q:  Did you have to change Scrum much?

I suppose I get a lot of people angry now, but in this particular project, because of the short cycles, we removed the Sprint term, and worked continuously with developing and delivering items from the back log, meeting every day in the daily stand-up. In the formal method Avegility Scrum is unchanged but for one thing: It’s not the product owner that is responsible for handling and prioritizing the product backlog, it’s the Evo cycle’s output in the form of design specifications. Cost/benefit ratio decides the priorities.

Tom Gilb - Q4

Q4.  I first met you at XPDay2004.  You were one of the keynote speakers.  I loved your talk because it resonated with my own concerns about "Agile".  Can you elaborate on those concerns? Do you still have them?

You bet!

For readers who were not there, the slides are at

 Paper: http://www.gilb.com/community/tiki-download_file.php?fileId=50

 Slides http://www.gilb.com/community/tiki-download_file.php?fileId=170

 

The good news is that some people who have practiced Agile thoroughly, now see my point - and realize that if we are to deliver value to our customers, we need to add to the conventional Agile vocabulary. We need to be far more explicit about what value improvements our stakeholder need, and how we are going to deliver them.

 

'Stories' and 'Use Cases', Features and Functions - are nowhere near what we need to deal with. Quantification of Values (Qualities) is essential. The Agile boys got the message about small iterations OK, but they completely ignored the message about quantified quality requirements. I put that down to immaturity. We are all immature at some stage. It has to be put right in time. Rapid delivery of the wrong values is still wrong.

 

A great example of a Scrum Master who 'gets it' is a friend I met at the Agile Business Conference in London, Ryan Shriver:

 Ryan's home page www.dominiondigital.com

 Slides… http://www.gilb.com/community/tiki-download_file.php?fileId=148

 Goalkeeper Tool  http://goalkeeper.dominiondigital.com

 

I have a very simple message to all those failed projects for which we are so  famous, and will be more infamous:

 

1. Quantify the critical stakeholder values      (current Agile culture does not understand 4 of 5 of those words)

2. deliver those values early and frequently.     (delivering value to stakeholders is NOT the same as delivering code or a story)

 

The survivors will get it :)

 

My favourite theory of how to change our world: "no cure no pay"

 No Cure Paper http://www.gilb.com/community/tiki-download_file.php?fileId=38

 

 Slides No Cure http://www.gilb.com/community/tiki-download_file.php?fileId=85

 

I cannot believe there are so many managers, 99.999% of them, who prefer to pay for effort and failed results, when common sense says they should only pay for good results.

People called 'programmers' sometimes crowning themselves 'software engineers' , have learned to fleece these rich managers thoroughly. A fool and their money will soon be parted. Hang in there guys, it may soon be illegal!

 

Well, I'd better quit before I insult too many people, after all they have a right to use their company's money any way they want to?

 

Warm Wishes to You All

 

Tom

Tom Gilb - Q3

Q3.  Can you tell us a little about your personal life?

Yes.

 

Ohhh you mean will I detail it here now?

 

Well I can tell you it has been fun - and to protect innocent people I can't tell you all the details!

 

The essence starts when I was 17 and met this wonderful Norwegian girl of 18 in London. She is sitting next to me in my living room as I write this. True Love = Endureth Long! (Corinthians 1 again). I was in my first year of 6th Form (boring). So of course I got engaged, moved to Norway and got a job at IBM just by asking! All really great decisions. :)

I mean who knew that Norway was going to be the richest country with the best quality of life in the world?

 

I continued my studies at University of Oslo for about 10 years, on the side of full time job, and a family from 1964. Sociology, Economics, Political Science, Philosophy. What fun - and I did it for fun, not for a job. Yes it is all useful for me today. I realized that I would not be a good bureaucrat at a University - and also realized that 95% of University people dislike me or my ideas. That is probably a good sign. The key is who the other 5% who do!

 

Thank goodness I never had to do a Thesis approved by the 95% who dislike my ideas! So I decided my books would be my 'Thesis". My ideas are evaluated by the readers - and I have full respect for that - even from those who feel they do not like my ideas. At least they had a look at them!

 

We have 4 boys and 5 grandchildren. We live just outside Oslo in a new apartment, half hour away is our amazing beach-front cabin on the Oslo fjord, and we have a "town cabin" in Covent Garden - which is useful as we love to go out in London, and visit my London Family.

 

In January 2008 the Norwegian Government decided to pay me a nice sum (which I have apparently been forced to save for 50 working years) every month for the rest of my life. I still continue working with my son Kai - but I am even less focussed on earning money, and more focussed on fun, travel for me and my wife, playing grandfather, and writing creating and spreading the word. I like the model of my friend W. Edwards Deming (Statistical Process Control) who held his last lecture at 93. I like to tease my listeners that I will teach their grandchildren when they are retired.

 

I enjoy perfect health, which I know is a gift at my age. No, I never smoked.

 

I am Norwegian Chairman of the Board for The Art of Living Norway (artofliving.org) which has made available to  me a lot of wisdom, and contact with a younger generation.

I think I believe, but cannot measure or prove, that we had past lives and will have future lives. This explains some things to me - but I might be wrong - so I am doing the best I can in this life. If I can get back to you afterwards and confirm my hypothesis, I will. Watch this website. Failing that I hope we can discuss this 'afterwards'.

 

One consequence of this hypothesis is that maybe my ideas are not as 'original' as I might like to believe: maybe I either 'remember' ideas, or are fed them by spirit guides? A least I get to keep the income and royalties. And spirits don't sue for plagiarism! Hey with a bit of luck I will either remember, or find an old copy of, my books, when I am young in my next life, and can follow Newton's idea of standing on the shoulders of giants. Well, even if I can't - you can!

My philosophy, from 17 years old:       do what you love,        do it well,       don't let money dictate what you do -    you will survive.

 

Tom Gilb - Q2

Q2: How did your life change after you published the "Principles" book?

Not much. My working life has been similar for almost 50 years, teaching, lecturing, consulting, writing, reading, networking, inventing, travelling - running my own company. So much fun - who would want to change it!

Tom Gilb - Q1

Tom Gilb is one of my Agile heroes.  He's been doing and teach agile for a looooong time, but many Agile newbies may not recognise his way as the agile way.

Q1: Hi Tom. I started University in 1988 ... which is the same year that your classic book "Prinicples of Software Engineering Management" was published.  According to my calculations that means your book has now been around for 20 years.  I only came across it a few years ago but it has influenced my thinking more than any other "software development" book.  I know of several of the better known Agile guru's who say similar things.  Can you tell us a little (or a lot) about what lead you to write the book and where your ideas came from?

Yes PoSEM is now in 20th Printing - I was asked by my publisher to make an 'evergreen' - and apparently it is!

 

I like to think my 2005 book, Competitive Engineering (CE), is a worthy successor. It treats the same subjects, but more deeply. It is not such an easy read, but we have other books to serve that purpose including Kai Gilb' s "Evo" book          http://www.gilb.com/community/tiki-download_file.php?fileId=27

It was important to me that the CE, which officially defines Planguage, was reasonably rigorous. Time will tell, but it is in Second Printing and is well reviewed, so there is hope for the future!

What led me to write the book?

Well, let me remind you of the book's predecessors:

1976 Software Metrics (coining the term, and the official foundation for IBMs CMM Level 4   [I don't take all blame for CMM!]). In USA 1977.]}

1974-5 Reliable EDP Application Design, and Data Engineering (neither in USA, but both in Sweden and UK).

These were the real beginnings - PoSEM was second generation!

 

You might say the centre of my technical universe is 'quality quantification'.

The driver there is -  the Lord.

 

 

OK!       Lord .....   Kelvin.

 

 

About 1965 the following appeared in a Norwegian newspaper: (I joined IBM Norway 1958, at 17)

"I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it;

but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind;..."

http://zapatopi.net/kelvin/quotes.html,  

 

 

 

That got me thinking about whether we could quantify critical quality properties like portability and security. I had no internet to search, and my international personal network from conferences was a good Who's Who (Dijkstra, Langefors, Naur, NygÃ¥rd, Dahl)  - so I tried common sense and invented and published some ideas on how to quantify them. I also quickly found out that they worked in practice.

 

As I worked as a consultant, I got challenged with extending the vocabulary, for example extensive work with ICL UK in the early 1980's led to defining Adaptability, and from 1980 with IBM led to defining Usability. As late at 1995 at Ericsson (Erieye airplane) we extended Usability with Intuitiveness and Intelligibility. And these and others became stable, and became patterns for tailoring local variations. More recently my friend Lawrence Day of Boeing pointed out to me that my method is in the Bible, 1st Corinthians, defining Love by decomposition into several quantifiable factors ('endureth long'). And I realized that Rene Descartes was on to 'my' method centuries ago.

 

Another early inspiration was a UK Professor Christopher Strachey, Oxford University Computing Labs, in the 1969 NATO Science Council "Software Engineering Techniques"  Proceedings page 65)  http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1969.PDF

 Christopher Strachey (Oxford) : “Computing science has been under some attack on the grounds that it isn’t software engineering. I propose to attack it on different grounds. I think we should seriously ask ourselves the question: is computing science? 

Recently I did a small survey as to whether computing is suitable as an undergraduate subject in an English university. I did this by grading all the topics I could think of under the headings of relevance and state of development. The premise is that it is clearly wrong to teach undergraduates the state of the art; one should teach them  things which will still be valid in 20 years time: the fundamental concepts and underlying principles. Anything else is dishonest.

The gradings for relevance ran from “clearly relevant and essential” to “part of another subject” (like numerical analysis) and those for state of development from “well developed with theorems, laws and text books” to “a fruitful field for research”. Note, incidentally, the importance of text books. They are designed to be taught from; they are quite different from treatises and even further from research papers. Now, it turned out that all those subjects which score highly for relevance score very low on state of development and vice versa. 

Until we have a sufficient body of topics which are important, relevant and well developed we cannot call the subject a “science”. I am quite convinced that in fact computing will become a very important science. But at the moment we are in a very primitive state of development; we don’t know the basic principles yet and we

must learn them first. If universities spend their time teaching the state of the art, they will not discover these principles and that, surely, is what academics should be doing. 

I do not for a moment underestimate the importance of the state of the art in engineering. Clearly it is essential and furthermore from engineering practice we must get our experience and material from which we develop theory. But, before teaching students we must get our basic principles right." [Strachey 1969]

Well, I found he was right. There were not many principles published or available (Langefors (Theoretical Analysis of Information Systems) and Naur had a handful).

 

So, I drafted a number of principles, based on my 'engineering experience'. About 124 principles were published in PoSEM (1988) and at least 100 Principles in CE (2005).

I like to think they "still be valid in 20 years time". In fact we can now test the 1988 Principles - would anyone like to argue they are not valid today?

 

My own arguments for  subjects  that will "still be valid in 20 years time". is in my paper Undergraduate basics http://www.gilb.com/community/tiki-download_file.php?fileId=98

The paper argues that principles, concepts and measures are primary tools to learn about.

 

A few academics have understood this, but not many, as far as I can see. I also see the consequent helplessness and waste in software projects that seem to me to be the consequence of the lack of fundamentals. We cannot seem to get decent quantified requirements at the top level for large projects - ever. We never seem to get past step One.

Jerry Weinberg - Q2

Q2.  You've probably written more books than I've read in my lifetime ... and, as far as I know, none of them have been specifically about testing. Why this book, now?  Does the software development world really need another book on testing?

The software development world definitely needs THIS book on testing. I've written it now because I've grown almost fatally tired of dealing with clients who have all sorts of cuckoo ideas about testing. Reviews have already ordered multiple copies of the book to give to their managers, customers, developers, and testers. It's short and hopefully easy to read by non-techies. It's time we cleared up these recurrent fallacies if we want to our profession to progress. That's why I wrote the book now.

And, BTW, I've written about testing in every software book I've published, but people have been asking me to bring lots of material together in one place. This book is not a compendium of what I've already written, but more of a deeper analysis behind all of those things written by me and a bunch of wise commentators. I intend it as a must-read for everyone connected with computing in any way.

Jerry Weinberg - Q1

Perf Q1.  Hi Jerry, I'm intrigued by the title of your new book "Perfect Software and other illusions about testing", which will be published later this year.  Are you saying that "perfect software" is impossible but a lot of people believe it is?  Can you elaborate on this?

Perfect software is indeed impossible, and even if we had it, we'd have no way of knowing. The Laws of Thermodynamics tell us why it's impossible, but that doesn't prevent lots of people from believing it's possible--just like many people believe perpetual motion machines are possible.

The book elaborates on this and many other illusions about testing--why people continue to believe them, what kinds of trouble they cause, and what can be done to mitigate that trouble.

(BTW, the book will be available in July 2008, any day now.)

[Clarke: I'll post an entry here when it's out]

Q6 - Israel Gat

Q6: What should question 6 be?  And what's it's answer?

I would suggest the following question: Can one apply Agile beyond software?

While I have not really done any systemic primary research on the subject, my hunch is that the answer is a resounding “Yes!” I offer three data points to support my hypothesis, as follows:

  1. Jim Highsmith has been doing Agile with an architecture firm in Italy.
  2. Paul Beavers and Becky Strauss are currently working on introducing Agile in BMC’s Customer Support organization.
  3. My own experience with my staff.

Of these three data points, I will elaborate on the deep transformation we went through as the staff of BMC’s Distributed System Management business unit for the simple reason that it offers special insight on the nature of Agile principles.

As mentioned in previous postings, we started doing Scrum in 2004. By 2006 we had mastered many of the aspects of Scrum. Furthermore, the business unit was “Scrummed” in entirety at this point in time.

During this year (2006) a product line was moved into my business unit. While the leaders of this product line were generally competent, their assimilation in the business unit was not working. It was clear they were struggling. We, as staff, were struggling to figure out why they were struggling…

The answer to the question why they were struggling eluded us for quite sometime. Examination of the “usual suspects” – strategy, market, execution, structure and time (for integration/assimilation) yielded no insight. There existed, of course, all kind of very legitimate differences in thinking and approach between the old hands and the new product line folks, but nothing to write home about, nothing that could shed light why the assimilation was so problematic.

Sean Duclaux, who was the director of product management for the business unit, came up with the insight we needed. Sean grasped that doing Agile transformed me and my staff with respect to our mode of operation. Unbeknown to us we became so-very-agile in the way we functioned. Actions were taken quickly; decisions were postponed to the latest responsible time; something we were working on in a certain staff meeting would be dropped in favor of a more important item in the next meeting; etc. None of these was conscious – we did not set ourselves to run staff using Agile principles. These things were just happening.

Checking with the struggling product line team, Sean’s hypothesis was validated. From their perspective we were an unbelievably chaotic bunch. We were not even managing an “organized chaos” – in their eyes we were increasing chaos. They found it next to impossible to operate within the fluidly dynamic environment that de facto prevailed in my staff.

The problem actually proved very hard to solve. While we had any number of staff members and staff members of staff members who could immediately help with just about any implementation aspect of Scrum, we really had no expertise on the transformation in our own functioning as decision makers. I am out of my depth here, but I tend to think we had gone through a prolonged evolution period in which Agile principles, in the most general sense, sank into us and became part of our standard operating procedures in any domain we focused on. This evolution period was missing altogether for the new folks on my staff. Expecting them to adapt quickly, we were sort of implicitly asking them to skip a developmental phase that one has to go through whether one is doing Agile in the R&D lab or performing in an agile manner in staff.

I find it fascinating that we had not realized Agile was transforming us as staff while we were transforming software engineering practices at BMC through Agile. There is an intriguing symmetry here that speaks straight to the heart of the software craftsman that I am.

[Clarke: My thanks to Israel.  If you have a question for Israel then leave a comment and I'll pass it on.]

Q5 - Israel Gat

q5) Has the conversion stuck and what are you doing to keep it sticky?

 

The conversion has more than stuck. In terms of scope of the practice, it flourished to the level of close to 1,000 Scrum users/seats at BMC nowadays. In terms of quantifiable accomplishments, the joint SQMA/Cutter Consortium/ BMC study has concluded:

"Nearly three times faster time to market than industry average; 20-50% improvement in individual team productivity; and, one quarter the expected number of defects based on team sizes and schedules...”

Furthermore:

“In their 20 years of benchmarking software development projects, the QSM Associates team has not seen such a high performing team [BMC’s] distributed across such a broad geography."

Suffice it to say I believe transforming BMC to achieve this level of excellence in Agile/Scum is probably the greatest accomplishment of my whole career.

To reach even higher levels of effectiveness and efficiency in software development, I believe we need to focus on four elements: methodical change, system development,operational innovation and organizational adaptation.

The initiatives in these four areas I am promoting/pushing these days are as follows:

  1. Methodical change: We are more and more moving these towards Lean Agile.
  2. System development: Agile, to me, is not “just” a R&D “thing”. Rather, one reaps much greater benefits by end-to-end Agile, instilling it all the way from the R&D lab to the customer shop. For example, Innovation Game® from Enthiosys is a tool I am considering using to refine our Agile/Scrum system and process.
  3. Operational Innovation: My recent work on Agile-Based-Market-Of-One (ABMOO) is a good example (even if I have to say so…) of innovative exploitation of the methodology. The power of ABMOO is in transforming the business design to fully capitalize on the methodical capabilities.
  4. Organizational adaptation: With so much programming work carried out through outsourcing, the boundary between where the corporation ends and the outsourcer begins becomes very fuzzy. My contention is that the next big frontier for Agile is in developing joint Agile infrastructure which will be used by both the corporation and its strategic outsourcers.

It is a little premature to assess which of the initiatives listed above will really take off big time. Until we know the answer, I am so very happy to be part of the Agile drama. This is not a drama we passively watch in the theatre. Rather, you, me, the developer in the cubicle adjacent to yours, the tester next to him, etc. are active players in this drama. What else could one ask for?!

Q4 - Israel Gat

q4) When did you start seeing results?

I will have to give a complex answer. Some results were discernible very quickly. Others took quite a long time to materialize.

Three discernible results became very clear within three months:

  1. Our testers became much more knowledgeable on the product(s) almost overnight. Working “real time” with product management and development from the very early stages of product conception, design and implementation immensely enriched their understanding, experience and knowledge.

  2. It soon felt “As if a new sun had arisen”1. Various things were not working well for the business unit before we started Scrum, and numerous team members were quite worried. The introduction of Scrum shifted the mindset from “The world hates us” to “Gee, this stuff [Scrum] is cool!”

  3. In three months we shipped the first Scrum release. It was not a great release, but it was working code at the hands of our customers. Walter Bodwell, the director in charge of the BMC Performance Manager (BPM) summarized the effect very astutely:

If we used Waterfall on BPM, we would still be in development. We would likely be cutting features right and left to try to bring the date back in. Changes requested along the way by the solutions teams would have been pushed back on rather than embraced.”

In contrast, it really took us one to two years to master various important aspects of Agile. Examples are uniformity in story cards, accurate assessment of velocity and effective Scrum of Scrums. All in all, I would characterize the phases of our methodical progress as follows:

  1. First year – we were learning how to do Scrum on a fairly large scale basis.

  2. Second year – we were starting to be proficient in Scrum.

  3. Third year – we rocked.

Various practitioners with whom I discussed our Scrum evolution felt that the three phase learning curve described above is too slow for executive management to buy into Agile. As this observation indeed is quite true sometimes, whenever I discuss Agile with executives who seem to have a very short time focus, I add a stern warning:

Don’t agile the Agile!”

1 This phrase had originally been coined to describe the reign of Edward III (1327-1377)  in England.

Q3 - Israel Gat

q3) What were the (say) 3 biggest obstacles to conversion and how did you overcome them?

  1. Is Agile another good-for-nothing management consulting fad? We did not start from square zero trying to implement Agile. Rather, having been exposed in the past to various methodical and management consulting experiments which were carried out half-heartedly, people in the trenches wondered at the beginning whether Agile was another one of “this too shall pass” initiatives. Will Israel and his staff walk the talk was an open question when we started rolling.

The defining moment for our sincerity and commitment happened at the conclusion of one of the release planning days. Paul Beaver, the senior director in charge of PATROL and the BMC Performance Manager, asked for a thumb-of-five vote1. One of the teams came back with an average of 2 as their confidence level. Upon enquiring, the team members sort of said “Well, we do not really believe we can deliver, but we thought this is the plan management wanted…”

Paul, to his great credit, sent the team back to the drawing board, held another release planning day at a later date, and ultimately accepted a revised plan for which the team confidence level was 3.5. Since this episode, nobody ever doubted our total commitment to Agile.

  1. Always-releasable: I can’t convey in words how difficult this intuitively simple principle was to implement. There is nothing extraordinary that I can point out to as the reason for always-releasable being so difficult to attain. The grand total of “infinite” number of trivial deltas/incompatibilities between components of the product suite, their containers (i.e. the SCM’s), and the engineering practices across five development sites proved to be almost insurmountable. We overcame the challenge by simply putting in an incredible amount of hard and tedious work, eliminating the obstacles one by one over a period of time that felt like ages.

Once we reached the always- releasable state, the improvement in productivity, velocity, quality and predictability was dramatic. The code became reasonably stable on an on-going basis rather than rarely stable till the very end of the release. The effects of various bugs in our code were not accumulated nor compounded. Rather, most of the bugs got knocked out “real time.”

While in general I believe Agile practices should be reasonably flexible – i.e. tailor the practices to the situation at hand – always-releasable is for me a non-negotiable.

I feel so strongly about it I would go as far as saying one does not really practice Agile until the always-releasable state has been achieved.

  1. When the magic fails to work: Towards the delivery of a major release I had a layover in NYC on the way back from Tel Aviv, Israel to Austin. TX. I called my good friend Roy Ritthaler who was in charge of product management for the business unit. To my friendly question “How are you doing, Roy?” I got the authentic, maybe too authentic, answer “Better than you, Israel”…. Turned out the release was in fairly deep sneakers. The number one reason for the difficult situation we were facing was me being overly aggressive on the release, betting that our velocity would rise to the level required to produce a very complicated suite of products. The fact of the matter was our velocity at this point in time was not yet high enough (thought it became incredibly fast later). I lost in the critical race between Scrum methodical progress and the need to introduce a “big bang” product suite in our market.

The crisis around the release being in trouble revealed I had not done a good enough job educating Marketing, Sales, Software Consultants and Customer Support on the nature of Agile. Things that are standard operating procedure for Agile in R&D - e.g. ship the features you can and catch up on the features you missed in the next release – became highly controversial. In particular, the spirit of Agile, as distinct from the mechanics, was extremely difficult to explain amidst the crisis.

Bill Miller, who was my boss at the time, saved the day. He grasped that the fundamental problem had nothing to do with Agile – we would have faced the very same crisis had we been playing Tin-ker-toy instead of Scrumming. The furor we were facing was about our go-to-market machine being thrown out of balance as a result of the release problems. The challenge was to align R&D with the business, not about Agile.

Painful that the crisis was at the time, it was the trigger for our transition from Waterfall optimized organization to Agile organization. This transition eventually led to our pioneering work in the area of Agile-Based-Market-Of-One. While it certainly did not feel so at the time, the release crisis actually was a blessing in disguise – it forced us to thoroughly revisi our approach to Agile. We ultimately gained much deeper mastery of the subtle issues of end-to-end Agile within the corporate fabric.

 

1 A 1-5 confidence voting scale based on the number of fingers one displays: 1=this project is heading towards total disaster and I am updating my resume as soon as the meeting ends; 5=we will deliver even if a hydrogen bomb explodes in 10431 Morado Circle, Austin, TX (one of the BMC site where we were Scrumming).