Sat 1 Mar 2008

Debugging In My Sleep

Everybody has had experiences where they solved a problem or were able to think something through whilst sleeping. Happens to me quite a lot since my day-to-day life involves a lot of "working in my head", whether that's computer programming, or playing chess, or working on a mathematics problem. The number of times I've played a chess game and then analysed some related position in a dream (after finally falling asleep) is enormous. Or been trying to solve some geometry problem and realised how to redraw the diagram whilst resting.

Last night and today was particularly surreal, though.

I was up most of the night. Not by design, but I was on a roll and then the problem took longer than I thought. I was filling in some holes in a branch of Django code I'm working on and I'm down to a phase I'll call "mostly only hard problems left" and have trying to knock them off one at a time.

Anyway, after working out a design and thinking about the problem cases, I sat down and wrote some code. Everything was going reasonably smoothly and I think I avoided most of the traps that were lying in wait. Finally, I was writing down a list of all the permutations of some variables — updating one or multiple tables, filtering on the main table, another or a combination of both, etc — and I realised there was a problem. More hacking and I had something that worked. Checked it again, nothing obviously wrong and the major use-cases all worked. So checked it in. At this point, it was about 6:00 this morning, so I went out for an early breakfast then came home and went to sleep. We've all done this. Extended hacking session, satisfaction at the end, then catching up on sleep. I'm getting too old for this stuff, but it still happens sometimes. But now things got weird...

There was some code I'd written that ended up look like this:

count = self.count_active_tables()
if not self.related_updates and count == 1:
    return
main_alias = query.tables[0]
if count != 1:
    query.unref_alias(main_alias)
#... (etc)

Nothing too terrible about this. Particularly not in isolation like this. We bail out early in one case and then carry on. The code and what follows is trying to remove any unneeded tables from the front of a sequence of SQL table joins, in this case as part of a nested select in an UPDATE statement (filtering the rows that are to be updated).

At the time, I was a little worried by the asymmetry here. Actually, "a little worried" is too strong. I had noted the asymmetry and wanted to revisit it later. Good code looks "nice", whatever that means, and edge-cases tend to be symmetric (if you exclude two possibilities in one place, you have to handle them both later on), or at least repetitive. This was, compared to other code that constructed SQL queries, quite isolated in the way it worked. It was also a little worrying that the number of active tables — the number of tables involved in the SQL join — required any special casing at this level. Although it's clear that removing the last table in an SQL statement will lead to tears (what are you selecting from?), it's annoying that this didn't fall out naturally. Why was the case with multiple tables leading to an extra reference count on the first table?. That was as far as I got at the time. The code felt like it could be better, but it was a tricky area, it seemed to work correctly in everything I tested, including some quite twisted cases, and I was tired.

Thus, to the dream sequence. My brain was apparently still working on this problem and decided to do away with the usual communicating via the usual metaphors of eight fat cows and eight thin cows, one of whom is playing a trombone. It needed to send a message! So I dreamt of being in a lecture theatre whilst some guy was discussing this code. He pointed out that special cases like count==1 need special justification, but he (the speaker) couldn't see any reason. He also said — and this was one of those really clear dreams — that the code was doing something similar in another place via a different method, which means both approaches will be buggy, since they don't generalise to fit the other case. Wow! Good point.

Sometimes you know you're in a dream whilst you're having it. In this case, I didn't want to leave early because it looked important. But I was telling myself "you have to remember this when you wake up. Take notes. Wait.. it's a dream, you can't bring things out of the room. Okay, remember this!" Maybe you had to be there. Fortunately, the whole experience of having my code dissected in front of a large, albeit completely imaginary, audience was emotional enough to mean I remembered the dream and woke up afterwards enough to make a note. The note says, and I quote, "Incswrercb cod. Jun the trmhjg." I'm not a good note-taker when I've just woken up. I think it was trying to say "inconsistent code. Join the trimming" but the guy writing it had a dead arm from lying on it.

Turns out, unsurprisingly, that the code above does lead to a buggy case that wouldn't bite most people, but a bug is a bug. Sadly, the dream lecturer didn't tell me how to fix this problem, so I had to solve that myself. Having gone for a long walk (my standard way approach to thinking), I have a solution in mind that should work and possibly generalises to the other case. I'm not sure my subconscious was right about the two cases being completely similar, since they're called in different ways, but it's worth thinking about.

Anyway, this has undeniably damaged my credibility as a non-crazy guy beyond all repair, but it was so strange. The human mind really does work in wonderful ways. My brain is smarter than me and apparently has been having some difficulty getting the message across in the past. So it sent me back to school.

Topics: software/debugging, thinking