Mon 1 Dec 2008

Thoughts About Two Releases

In the past week, two software releases have caught my attention for various reasons: Wordpress 2.6.5 and MySQL 5.1 GA.

Thus, a couple of observations about some aspects of the two releases. Nothing that hasn't been noted elsewhere here, but I wanted to write down what I'd been thinking about.

Wordpress

"You are handed a large PHP codebase. Do you choose to run away and use practically anything else, or stay and fight on behalf of humankind?"

Given its popularity and feature base, it's probably not too surprising that security holes are found in Wordpress with reasonable frequency. Every package has bugs. Web frontends to databases tend to have bugs that have large consequences.

My experiences from reading the codebase to make small customisations or fix bugs for friends and clients has been that it's reasonably easy to read and logically laid out. I haven't seen a lot of evidence of any symptomatic negligence in the way it's written. There are a few inconsistencies, but, again, every codebase has those, because code is developed over time. End of the day, practical experience shows that the product works, is easy to use (hence popular) and relatively well-supported, all things considered.

The Good

An interesting security problem was found. Interesting because it showed how server-side cache poisoning can be used, in conjunction with a controllable HTTP header, to inject malicious content into a served-up page. The Wordpress announcement doesn't reference it (possibly because it was written first), but there's a full description of the problem in the BugTraq archives.

Having somebody write up a clear description of the problem that is being solved is helpful for people trying to avoid the same problem in other circumstances.

Wordpress also took notice of the fact that there had been a malicious release of a package calling itself Wordpress 2.6.4 and skipped that version number, drawing attention to the reason in the announcement. Nicely proactive.

The Bad

Releasing tarballs with no way to verify the validity of the package is really unforgivable these days. Wordpress have been bitten by intentionally modified tarballs in the past and they still haven't taken mitigating steps. It's kind of basic stuff to generate a hash of the tarball and sign it with the key of somebody who make efforts to have their key signed by other widely trusted people.

Secondly, the security release did not just contain the security fix. They smuggled in some unrelated changes as well, calling them bug fixes. Again, this is fairly traditional behaviour for Wordpress, but it's not really defensible. Every change introduces the chance of introducing another bug (or even, heaven forbid, a security hole). Security releases really should fix just the problem at hand. That way, upgrading to the fixed version doesn't include the risk of rolling out extra, unrelated code. Okay, we might mostly trust the Wordpress committers' diligence and attempts to produce bug-free code. But mistakes happen and increasing the risk by shipping "bonus" code in a security fix seems like bad practice to me. I realise the Linux kernel has recently gone with the explicit public policy of not differentiating (in announcements) between "normal" bug fixes and security fixes, but I don't think that attitude needs to filter down the userspace packages that are installed from scratch by a much broader and more varied userbase.

(In the plus column, there was a clear link to all the code changes between 2.6.3 and 2.6.5 for people to review. Fortunately, this diff wasn't too long, so it was possible to review it. That's just lucky this time.)

I really wish Wordpress would get their stuff together in the release department. It requires just a bit more effort than necessary to work with them (in the sense of using their product as a choice for clients) in the current situation.

MySQL

Wow. MySQL made a 5.1 "final" (in some kind of way) release. It's been a while. That was kind of news in itself, although I was still in a "wait and see" stupor regarding whether to bother looking at what made the final release. Things got really interesting when Monty Widenius made a blog posting putting the boot into the release. A bit of a "insert cat amongst pigeons. Stand well back" moment.

What is interesting here is not whether the release is particularly good or bad. It's going to have bugs and, yet, it will still be mostly reasonable. Whether it's worth an upgrade remains to be seen for me, but if you're hanging out for all the promised new features, caution may be required.

No, interesting was when one of the project founders and genuinely invested (I mean in the emotional, caring sense, not the $$$ version) feels the need, and thankfully feels he has the freedom, to post something like this. It's hardly the first time Monty has let his personal feelings be known, which is nice. His commentary on the Drizzle announcement is a nice case. But this time wasn't really sugar-coated.

I might quibble somewhat with the start of Monty's post, as I think he overstates the severity of some of the immediate problems. But I suspect I shouldn't. From the blog post, it sounds like a few of these "critical" bugs are more or less contained to a couple of broad areas, such as, but not exclusively, row-level replication. So you put down that some particular feature areas don't work particularly well. However, Monty is on the inside and he's seen the process slipping away. So whilst I don't know where the objective balance lies between his and my observations, I'll defer to his feelings, as they've kept MySQL going along nicely since the mid-90's. Bugs in code tend to become personal embarrassments. You don't want particular problem to be released unresolved. You hope or suspect people might understand that time ran out and it's not forgotten, but you know, deep down, that it's a broken facet and something that could be better. My sympathies are with the developers and maintainers on that one.

It's towards the end of the post that the real value lies. Monty identifies real problems that he sees and lays them out in simple English. No hiding behind alternate interpretations. Even if the post is mostly a rant against other people in the release process, it's a big deal to identify the flaws in your own project so clearly.

I can't really say I disagree with any of them, either. Nor that the problems are unique to MySQL -- inside or outside the Open Source world. Hopefully, as the furore dies down, people will keep that list of problems, the causes and the effects, in mind as they try to predict the future. They're really restatements of reasonably "well-known" saws like the fact that you can't just drop extra programmers into a project and expect it to go faster.

Similarly, you shouldn't abuse the term "RC". Release candidates should be just that: candidates for release. Not attempts at garnering a wider testing base by sucking in your users with false promises of a candidate for release existing. In MySQL's case, that ship sailed long ago and there wasn't really much they could do. People don't trust the RC nomenclature from mysql.org any longer.

I should add that I find MySQL's versioning process to be fairly odd and not really helpful. The final, allegedly "production ready" release isn't given the normal name of "final". It's called "General Availability" (the GA bit). Which apparently differentiates it from the previous 5.1 releases that were also generally available. Further, this very first "production ready" release is called 5.1.30. Not 5.1.0, so that you tell it's the first in the new serious. It's number 30 in the series. Now, I realise this is fairly traditional practice in the MySQL release cycle. But it's so contrary to popular practice that it's both confusing and deceptive. 5.1.30 sounds like a well-established product. Not the very first one that you're willing to claim is ready for use in real environments. It's marketing-speak again, just like the RC nonsense.

Fortunately, I think there's fairly broad acceptance that the 5.1 release cycle didn't go well. There was no graceful exit, so something had to be done. I don't know enough about the specifics to know if calling for a December release date of 5.1 GA was responsible or not. Similarly, if the quotes that MySQL is focusing on timelines, not quality, for releases are really accurate. I mostly trust Monty, but he's letting off steam in that post and there's another side to that story. Still, something had to be done and this might have been the way out of the hole they just kept digging deeper for themselves. Time will tell.

I don't know what to make of the MySQL release at the moment. It's too early. Nothing I use it for needs the new features they're pushing, so I think staying clear for a bit is worthwhile. However, I also can't see anything in the known bugs list that would prevent me from upgrading if I had to (after suitable testing). I simply don't need the bleeding edge of functionality for that particular database server. But it's a nice project and one I've used for a very long time. I remember seeing a mailing list announcement for the initial Windows release, which was apparently in early 1998, and I'd been playing with it on Linux before then. I hope that Sun can work out how to repair the damage with the community (not all of it caused by Sun, this time, but they're in it now) and that the key developers and community contributors can manage to refocus, or stay focused and keep working on whatever the products might be.

In conclusion, I have nothing but kind thoughts for people who are heavily involved at the upper levels of MySQL development and release. Whatever side of the many debates they may sit on. As I've mentioned, I think it's a nice piece of software. There's no solution that's going to make all those problems go away (including the problem of having most people agreeing on the problems), so it's going to be a long process. I think there have been some positive signs over the past year, as well as some more visible cracks appearing.

Topics: software