Mon 31 Mar 2008

Software Development Motivation

I've re-read a few books about successful and unsuccessful software and other large projects over the past couple of weeks. Some rambling about the why of this later, but first, a quick summary of what I've read and why those books above others.

I continue to be impressed with Dreaming In Code, which I haven't reviewed here, but a lot of my peer group will have heard of. Initially I thought it was quite a tough read because it is a chronicle of stuff that didn't quite work and I know some of the people involved (and have heard of most of the names). So it's a bit too much of a real life tragedy at times. But Scott Rosenberg does quite a good job of periodically pulling out of the story he is telling about the Open Source Application Foundation (OSAF) and addressing the reader's concerns along those lines. He makes a point of identifying that many people will have been saying "they're doing it all wrong" and "I wouldn't have done that" and making you realise that might not be true and 20/20 hindsight is both easier and of less practical usefulness than foresight. All in all, the book is a chronicle of what happens if you try the ultimate "we'll work it out as we go along" and then aren't tough on yourselves about actually working to a conclusion on the mini-issues. There are software management lessons galore there, both positive and negative. Rosenberg, and particularly the individuals who worked for (in some cases, still work for) OSAF and contributed recollections and quotes for the book, have added something useful to IT history by writing down what happened during the early Chandler years. It's not a happy story, but it's motivating and it is worth knowing.

It's very hard to contrast Dreaming In Code with many other books on that theme, because it is recording something that is happening right now and started only a few years ago (I remember the initial announcement about it). Tracy Kidder's Soul Of The New Machine is close and I read that again over the weekend. The big difference is that Soul Of The New Machine was published after the project was complete and it (the project) was a measured success and easier to measure as a success in the sense that a specified piece of hardware was produced for sale. Open Source software is actively never "done", so unless a project fails to the extent it is abandoned, you can't quite write the story of a piece of Open Source software after the event.

Two more closely aligned books that are opposite sides of the "success" coin and also on the recently read list are Fred Moody's I Sing The Body Electronic and Andy Hertzfeld's Revolution In The Valley. Moody's book covers an ultimately unsuccessful project within Microsoft in the early 1990's to develop a sort of "Encarta for Children" product. Hertzfeld's justifiably lauded book is the story of the creation of the Apple Mac. Although told in different ways and with quite different moods and results, these two books are the stories of products developed within hierarchical, highly managed organisations and that's why I've separated them from Soul Of The New Machine and Dreaming In Code. Yes, the Macintosh guys did some funky stuff and were real hackers to get the thing done, but they still worked largely within the Apple system and got sign-off and rejections at times from the higher-ups (Steve Jobs, in particular). Moody tells a story that could largely be turned into a fortune cookie form as "over management and waterfall planning stifles productivity, kills product", but that hides a lot of the subtlety and skips a number of details.

By the way, I prefer I Sing The Body Electronic over G. Pascal Zachary's Showstopper, a story about the creation of Windows NT. Both paint similar pictures of life inside Microsoft, both good and bad. However, Moody's book seems to get in closer with the team involved (admittedly, it's a much smaller team and a smaller product) and since there are less obnoxious people involved in his story, you (or, at least, I) feel more able to relate to the story of the team developing the software. However, that's more possibly a function of my mood than the books themselves. If you want to alternately read about an interesting and awesome piece of computer history and shake your fist at Microsoft's bumbling and arrogance, both I Sing The Body Electronic and ShowStopper! are good reads, if you can find them in print anywhere.

All four of these books, even the two that end with less successful outcomes, provide a lot of food for thought. They also describe a lot of stuff I can identify with concerning motivation, the ebbs and flow of individual people's enthusiasm and that of a team, and the various things that are done to try and change direction. Other books like Scott Berkun's project management book, Christopher Duncan's The Career Programmer or J. Hank Rainwater's Herding Cats — three of my personal references for software development and technical management in commercial situations — cover much of the same territory in more of a reference fashion. Thus they are easier to learn from if you are wanting to apply hindsight to the present, but they aren't nearly as easy to identify with. You think of it more as happening to "other people" and the emotional impact and retention is less. This is both good and bad, of course, since removing the subjective to drill down to the objective points is what makes a good reference and checklist book. But it's not as inspiring.

By contrast, Tom De Marco's The Deadline is entertaining and has some important lessons in amongst the laughs, but it's not nearly realistic enough to empathise with. Again, a good reference. A short read that introduces some interesting techniques and possible results with quite some humour, but not something that helps me sort out what's going on in my own headspace. Still, this was the book that made me remember "it's not the stuff you don't know that hurts you; it's the stuff you think you know that is wrong." When I first read the book some years ago, that stuck with me and a couple of friends I've lent the book to have commented on that portion as well. It's become a bit of a mantra when I'm designing stuff, so I can't say the lesson was lost or the presentation was bad.

Anyway, thus endeth the book review/recommendation section of today's post.

I've been doing all this reading in an attempt to get out of a bit of a funk about writing code and designing things. It's not too serious, but I'm grumpy a lot and missing the feeling of being enthusiastic about stuff and have been spending a lot of time trying to work out why this is the case. Reading about other experiences — and this was why I chose books that were at least somewhat contemporaneous with the single project they were documenting, rather than those that tried to explain broader conclusions for the future — has helped sort out some of this. I want to work on something that matters, something that has a completion point and will be useful. Contributing to projects like Django are still fun, obviously, but that's mostly doing little things to help out others and sometimes I'm not as invested in it as I should be. I think I'm missing the fun of working on something that is hard but realistic and which can be said at some point to be done. I'm also worried that the lack of periodic work with teams is not giving me any outlet for ideas or source of problems. I'm hardly a hermit, but I sometimes miss working closely with other people and succeeding together on something that is non-trivial. Self-education is fun and something I do a lot, but the real world tends to throw up a lot of interesting problems that you don't see in the books.

Okay, some people I know who read this blog will possibly misinterpret or over-emphasise that last paragraph. Please don't. This isn't a whine. I'm writing it down as a record to look back on. It's my problem to solve, I'm addressing it and sometimes my blog is for me. Right at this moment, it's career evaluation time and this is part of that. I don't want anybody to think I've lost faith in Open Source development or am considering cutting back my involvement there. In fact, I'm trying to work out how to be more effective in my participation in that arena. It's one of the fastest moving areas of the industry and periodic reassessment is always worthwhile. That's why I've been refreshing my memory on what's worked and what hasn't in other places. There are a few other posts I want to write based on recent conversations I've had with some other people that essentially come down to "Open Source code seems to be of a significantly higher quality". I don't want to give up that advantage by only writing in a closed room. Hanging code out there for people to see periodically and participating in the feedback cycle keeps me honest and responsible. In addition to other things, I need to pick something to work on that is a little more concrete than merely "do useful stuff". I'm being a bit more active in looking around for ideas lately. The trick is finding a balance between interesting, useful and wanted. The latter sort of work is the type that people pay for and that helps fund work on other stuff on weekends and evenings.

Topics: books, software/design, life