Using the long weekend to get some intensive hacking done, I'm converting Django's internals to be more transparently unicode aware. All this character encoding twiddling has me thinking about performance, so I've been writing lots of little test programs to time features.

One unusual result that popped up this afternoon concerned reading a UTF-8-encoded file. Contrary to my intuition, this version:

data = open(filename).read()
data.decode('utf-8')

was consistently a little bit faster than this version:

data = codecs.open(filename, 'r', 'utf-8').read()

Admittedly the differences were generally (much) less than 5%, in favour of the first version, but I was a little surprised there was any real difference at all. I'm not worried by this result, but I would have guessed incorrectly.

In both cases, I'm reading in the data and converting it a unicode string. I was running it against some examples I had lying around from Markus Kuhn. The results were consistent if I changed the order of the tests or intermixed them. Aliasing codecs.open to a global variable sped up the second method very slightly, but not enough to catch up. I was careful to pre-fill the disk buffer cache and run each test enough times in a loop for any noise on a single run to be absorbed.

Turns out, the results are closest (essentially identical speed) for files that have mostly one byte per character (pure ASCII files being the fastest) and diverged the most for more complex characters. The runic poem, with lots of three byte characters, and Greek text, which is entirely two byte characters were the most divergent.

Topics: software/django, software/python

[#]

This let me start the day with a smile...

"One thing that Django does very well is its debug mode error pages. I saw more of those then I probably should have. ;-)"

Django: helping users maintain good humour in the face of periodic failures since 2005.

(From Speno's Python Avocado via Planet Python )

Topics: software/django

[#]

Been a while since I added to this series. I've come across a couple of repeated questions lately, so it's time to give back to the knowledge pool again.

This time: using iterators to customise the options presented via the choices attribute on a model field.

(Read more...)

Topics: software/django/tips

[#]

Apropos of nothing in particular (I was looking for something totally different and typed in a lazy Google query), I stumbled across a quote Guido van Rossum made last year:

"This is an illustration of the dilemma of maintaining a popular language: Everybody hates change (me too!) but everybody also has one thing that's bothering them so much they absolutely want it to be changed. If you were to implement all those personal pet peeves, you'd get a language that's more different from Python than Python is from Fortran." — from the python-dev archives

Too true, and it strikes me that this doesn't just apply to Python. It is also very true for any large system with a userbase. This isn't an argument against change, but it's a rationale for why caution is a good idea.

Topics: software/django, software/python

[#]

Been enjoying attending linux.conf.au this week, in between a few other commitments.

As I wrote about previously, the conference organisers made a few changes to the traditional schedule this year, moving some of the keynotes to odd days and putting the tutorials in the midst of the talks. By and large, the changes have worked, although for any organiser reading this, I have some feedback to send in tomorrow.

Gave my Django tutorial today. It was not the greatest tutorial I have ever given, but it wasn't too bad in the end. I would have preferred more than just 90 minutes, since giving a real tutorial (rather than just a quick overview) can take some time. Feedback from the audience afterwards (just chatting to people in the afternoon) was positive and it seems like I've encouraged a few more people to take the plunge and see what we're doing in Django land.

For those who attended the talk, I have, as promised, put up the slides and example code. Those slides probably won't make a lot of sense to people not at the talk; I have grown tired of putting everything on the slide and prefer to be able to talk and wave my hands around a bit to get the message across. The tutorial was recorded to DVD, although there was no microphone at my podium, so I'm not sure if the sound carried to the camera microphone. If the video does turn out to be reasonable, I'll post a link here. Nothing really special was presented, since this was a beginner/introductory tutorial, but we had some fun and nobody threw fruit.

Been going mad taking photographs. Some of the more acceptable efforts are now in a Flickr set.

For other photos from the conference, look for the lca2007 tag on Flickr. More blog postings from attendees at Planet linux.conf.au.

Topics: software/django, conferences/linux.conf.au

[#]

Thanks to some inefficiencies in airline booking systems, I'm going to be stranded in visiting London for three days in early February. If anybody wants to meet up for dinner or some beers or coffee sometime between Friday evening (2 Feb) and Sunday evening (I fly out first thing Monday, 5 Feb), drop me a note at malcolm@pointy-stick.com. I'm most likely staying somewhere around the Kensington / Earls Court area (still waiting on hotel booking confirmation), so pretty much easy access to everywhere.

Topics: software/django, travel

[#]

I received some pleasant email from the organisers of linux.conf.au yesterday: they have accepted my tutorial about Django for the 2007 conference (January next year). I love speaking at that conference because the audience is generally very appreciative, it's very well organised and the talks are always interesting. No conference is perfect, but linux.conf.au is very, very good.

Coincidentally, this keeps up my 100% success rate with having talks accepted in that conference since 2001, although I had to miss the Brisbane (2002, although I accidentally had a talk accepted before I knew I wouldn't be able to attend) and New Zealand (2005) conferences due to other commitments.

Not sure who else is talking yet, since the acceptance emails are no doubt still being replied to, but Rusty has admitted to having a couple of talks. His technical presentations are always compulsory viewing for me.

Topics: software/django, conferences/linux.conf.au

[#]

After travelling and being generally busy for the three weeks, I got back to Australia last Saturday and immediately came down with a heavy cold or the flu or something. So most of this week was spent swimming through a haze and generally not being a human being. Slowly started catching up on some urgent outstanding items on Thursday and Friday, in between taking naps, but today is the first day I actually feel like moving around a lot. So it was time to tackle all the unread email I had let accumulate.

Most lists were disposed of pretty quickly, since I read them for information, rather than participation. Django was another story, though, since I like to play a bit in that space.

What with working strange hours whilst away and various things, I had let about 200 django-developers emails pile up unread and 700 django-users. Essentially two to four weeks of only very partial skimming and saving everything else. These aren't just emails I can breeze through without thinking, either, particularly since there was a serious discussion about forms and manipulators that seems to have lead somewhere useful, although petering out a bit now and a discussion on some parallel SQLAlchemy integration (nothing really special in the thread beyond the initial accouncement, but that's something to watch). There were a few other threads that had something to do or note in them, so I've ended up with a couple of dozen bullet points of things to investigate or fix. Now, if those 30 hour days I ordered would just arrive.

Only about a third of the way through the django-users unread pile so far, so apologies if anybody was expecting a response from me on something. I'll get through them all by the end of today.

One thing this did bring home to me: we need to be really understanding of people who may miss something that was posted on one of these lists. It is a non-trivial task to keep up to date if you aren't putting in a chunk of time each day to read and digest all the emails. I'm not sure if a weekly mailing list summary would help or not, but it might be worth a shot. There's a lot of good information on those two lists and the signal to noise ratio is surprisingly high most of the time. The constant understanding from everybody and "treat no question as too silly" attitude is very good, compared to many lists I've been on for other projects. It is a lot like drinking from the fire hose, though, and I felt really out of touch with development trends after only a few weeks away (this either makes me a concerned citizen or a control freak).

So, hopefully I've learnt the right thing from that and will be able to post repeats and references to older threads with sufficient good humour.

And, yes, I will get back to writing some more Django tips articles in the near future.

Topics: software/django

[#]

I'm going to be in New York City on August 31 and September 1. Are there any Django users (or other random readers of this blog) who would like to meet up for dinner on one of those nights? Drop me an email if you're interested and I'll try to pick a popular evening.

I'll be staying in Manhattan somewhere, so if people also have suggestions for a good place to meet, let's hear them.

Topics: software/django

[#]

I've been spending a fair amount of time over the past couple of weeks thinking about features and guidelines and things that might work and things that shouldn't. Whilst wandering the web on Monday night, I came across an link I had made to an article on the Creating Passionate Users site. In particular, this graphic:

(Image used under Creative Commons license, from here )

This strikes a chord with me. It also helped me pull together what I had been thinking about. It did not help me write it down coherently, but let's see how this goes. Let's go for a (long, rambling) wander from the world of commercial software services to the world of open source project development. With references to writing from people with far more clues than me.

[Aside, or advance warning, at least: I'm not quite sure who the audience for this post really is, but if you find yourself thinking "why aren't they listening to me?" when trying to contribute to software development (whether as a developer, a writer, a bug triager, whatever), it may provide some food for thought. At a minimum, it should clarify some of my prejudices.]

(Read more...)

Topics: software/django, software/GNOME

[#]