Sun 30 Nov 2008

Removing The Model-View-Controller Straitjacket

Here's a summary (no, seriously!) of a collection of notes I've been making for quite a long time about the infamous model-view-controller architectural style. I keep intending to post it during some quiet period when there isn't a parallel "MVC furore" debate going on somewhere, because it isn't a reaction to anything. However, such lulls do not exist, it seems, so time to publish.

It seems to be traditional to add a note that if you disagree with me, that's fine. In this case, though, if you disagree with me, you're claiming that there's no ambiguity or misunderstanding or misuse of the term and pattern MVC. Since there's lots of evidence to the contrary, I would suggest that if you disagree with me, you need to do more reading. I'm not taking sides (here) on which of the many alternatives might be correct; I'm identifying the problem. Pretending that an issue doesn't have two sides just because you don't agree with one of the sides is, I'm sure, one of those styles of debate that has a fancy Latin name and all the validity of "proof by aggressive hand waving".

This is a fairly long piece with a lot of offsite linkage. For those with short attention spans or things that you want to get done today, here's the capsule summary:

• We have this term that everybody likes the throw around. Unfortunately, the only clear definition (the historically documented one) is not how people commonly understand the term MVC. In fact, there appears to be little common understanding.
• There are reasons we give labels to patterns.
• MVC is not the all-encompassing pattern you think it is.
• Using the term MVC doesn't really make things easier or clearer.
• Solve problems and understand solutions in context. Don't pigeon-hole.

Even more directly, stop saying "thing X is (or is not) MVC". You're almost certainly fudging at the edges and certainly not making things any clearer.

What Does Model-View-Controller (MVC) Mean?

Defining the terms up front to avoid confusion seems like a good start. Except that there's an almost immediate problem. Nobody agrees on the precise scope and definition.

Historically, the development of the term MVC is easy to trace. Like so many other great computing concepts that are treated as commonplace these days, the origins of this name and pattern (and the two are independent, since a small-p pattern can exist without a name and naming something attempts to distinguish it, but isn't guaranteed to succeed) can be traced back to Xerox PARC in the 1970's. There's not really any doubt as to where and when the term originated, as it was contemporaneously documented. The original papers have been made available online by their author, Trygve Reenskaug. The concept was well developed by May, 1979 and given the current name by December 1979.

If you read through those papers and the other presentations available on that page, you'll notice how the MVC pattern was very much tied into GUI systems. In particular, active feedback from the model (application unit) to the view, as a way of signalling that something had changed, was an important component of the system. Thus if you were looking at a chart of some data and also changed the data in a spreadsheet-like view somewhere else, the chart would update to present the correct view at all times.

Once the World Wide Web was invented -- and although the Web is, in many ways, an evolutionary step beyond networked systems that already existed, I think it's fair to say that the concept itself is self-contained and well-defined enough to speak of Tim Berners-Lee as having invented it -- and scriptable client-server applications became possible to produce without too much strain, the term MVC was also used in reference to those sorts of systems. Web applications are predominantly asynchronous. The view you get on the web page is a snapshot of the system at the time the page was produced. Realtime updates are still not the norm, for a number of valid reasons. So the web-based version of MVC drops one of the significant components: notification update. I've seen references to people calling that "passive MVC". This seems to be the term of choice on Wikipedia's MVC page, for example. I've always thought of it as "asynchronous MVC", since, after all, why invent a new term (passive) when there's an existing technical term to describe the situation accurately: the updates happen on a delayed schedule.

Recently, with the rise of AJAX-like systems and what we are now calling Comet, active representations are again possible, although not always feasible (not all networks are fast or provide effectively unlimited traffic levels).

Of course, the other term for the web usage would be "fake MVC" (or, even more plainly, "not MVC"), since it's clearly not MVC as it misses one of the key components and the original description of the term was hardly ambiguous. But that's where the arguments as claims are made that the term changes over time and it leads to the point of this essay: the term "MVC" is highly ambiguous and the difference in definitions is relevant to the usage. Since the web-based usage is less than 15 years old (the Web was invented in 1992 and commonplace scripting was a couple of years after that) and the original term was in use by late 1979, I personally feel the usage and definition that is more than twice as old as the new-fangled Web-based one has clear precedence, but I won't argue the point. My goal isn't to convince people that my preference is correct.

As we'll see in the next section, the synchronous/asynchronous (active/passive) difference isn't the only unspecific variation that has crept into the usage here, either. When person A says "MVC" there's really very little you can assume about what they mean if the discussion you're having depends on the details.

For those interested in the history, there's a somewhat brief and fragmented history of the traditional (errk ... sorry .. fell into old habits. Let's call it the original) usage at the c2.com wiki (the original wiki): http://c2.com/cgi/wiki?ModelViewControllerHistory. There's also Martin Fowler's historical survey, but I'll be using that further in the next section, so we can postpone the linkage for a moment.

I don't think the wikipedia page is a particularly strong reference for learnign anything much about this topic, since it focuses on examples (many of which, arguably, aren't MVC at all) and stumbles on the various ways to define MVC. My personal feeling is that if you want to get an understanding of the complexities and layers behind the terminology, the ModelViewController page at c2.com and following the links from there is a good motivator. Note that many of the people writing on those latter pages were actively involved in the historical usage and development of the pattern, or worked with people who were directly involved.

Where Are The Difficulties?

As I read more and more articles, online and off, about MVC-styles, I think there's a consistent set of fuzzy areas that cause problems. Again, I can't really say that any viewpoint is wrong, but where there's inconsistency, there's confusion. It also often comes down to a lack of motivation driving a particular term or style: the original MVC roles were strongly motivated by the GUI interfaces they were modelling and the processes involved there.

The boundaries between the various roles are not always clear. What's model, what's controller, what's view. The original Smalltalk papers bound controller and view together fairly tightly (not to the point of conflating them, but they seemed fairly co-dependent in some ways). Today's frameworks tend to blur the model boundaries a lot, not always separating presentation independent and dependent features (possibly deliberately, which is fine, but if you're going to use the pattern, you have to follow the plan).

What Is Gained From Having A Term Like MVC?

Naming things is a useful shortcut to aid communication and retention. So long as we all have the same understanding of the meaning of a term, it can be used where the definition would otherwise be inserted and we can move along a lot faster. That hypothesis is important, though. This utility value is well understood, I feel, and hardly particular to computer science disciplines.

However, there's another side to terminology like this that isn't as widely known. That motivation is the genesis of various pattern languages. We give a name to a particular form of a solution so that we can reuse it later. Typically (I hesitate to say "traditionally", given the problems I've noted in the previous section about the respect paid to traditional legacies), named patterns are in some way useful or even preferred solutions. Some people (yeah, weasel words, I know) go as far as to use the phrase anti-pattern to describe a bad solution pattern (not a bad pattern, a bad solution that can wrapped up in the pattern language). I kind of like the pattern/anti-pattern split, but the split between good and bad isn't always clear.

MVC can be viewed as an architectural style (difference definitions lead to different architectural styles, but the claim is still valid) and can be viewed as a combination of patterns. Opinions seem divided as to exactly which patterns are involved, but that shows that the edges of patterns are sometimes a bit fuzzy. The talk page for the wikipedia entry on MVC gives some examples of this confusion.

All this is a slow way of saying that having a vocabulary of architectural styles and patterns can help us to write solid software. Firstly, it provides a catalog of ideas In theory, good ideas. Periodically glancing over a list of patterns is a good way to remind oneself that we might have gotten into a habit of thinking in a particular rut and it's time to consider other options.

Secondly, the existence of such a pattern provides a prompt as to something worth thinking about. MVC is a reminder and a description regarding splitting up responsibilities. I like Martin Fowler's term of "separated presentation" for one of the big drivers here. We'll pick this up in a minute, but first a momentary aside, just so I can get this out of the way...

Martin has documented a whole pattern called Separated Presentation and here's where we bump into some of my problems with patterns. Or maybe just with Martin's particular method of describing patterns. That page is very proscriptive in the examples and it's heavily example based. I am in complete agreement with the nature of the pattern, but I think Martin's particular penchant for separation of responsibilities, particularly driven by the Java-style of most of his examples, sometimes feels like it's emphasising "form over function". Where the method/function barriers lie is an implementation detail and patterns have multiple implementations. Still, that's a fairly consistent trait of Martin Fowler's writing (e.g. his refactoring book does it as well) and I think it's forgivable both because the advice isn't actually stupid and it does make the examples clean. I'm only claiming it's not compulsory and alternative implementations are valid, too. Which, incidentally, I don't think Martin is not allowing. Also, there's an entirely valid argument that the particular spaces Martin Fowler and his Thoughtworks teams work in is quite different from my experiences and domains. What works for me and my colleagues is just an non-universally applicable as anybody else's advice.

I'm not merely pointing out a somewhat inconsequential difference of opinion here. The difference between pattern and implementation is very relevant to our story of using patterns to write, understand and maintain code.

Now, back to the meandering story...

Understand The Problem Before Naming The Solution

I've spent more than a few paragraphs arguing somewhat loosely that MVC has ambiguities. But the problem it originally and continues to try and solve is still valid and definitely common: How do we write maintainable, lasting code that covers multiple responsibilities, including business logic, user interaction and data retrieval?

At this point, I don't have to write a lot more. I can point you to Martin Fowler's article on GUI Architectures. Note the title carefully. He's talking about GUI's in general, not websites in particular. Many of the problems, solutions and generalised patterns carry across, but there's still the synchronous/asynchronous differentiator to add to the web-based case (in many instances). A careful study of that article should help remove any cobwebs that have accumulated as to whether MVC is the solution to a problem. I'm not claiming the article is definitive or even 100% correct (I'm not claiming otherwise, either, since most of the parts I know about and understand independently gel with Martin's description). However, the mere fact of laying out a number of use-cases and showing how responsibilities can be split in different ways, leading to justifiably different patterns should encourage people to see that not every implementation of business logic and presentation has to be called MVC. In fact, most of them shouldn't be.

A Concrete Example: Building On Top Of Django

I want to look at a specific example of applying patterns to implementation. In particular, how one might classify or design the various pieces of an application that uses Django to supply some of the common web and database integration functionality.

It really gets on my nerves when people write that Django is an "MVC framework". It's as valid as saying it's a "circus support mechanism", since the statement is both true, in some contexts, and false in others (you can definitely use Django-based code to help run your circus; stop looking so sceptical).

Fortunately for them, those people are going to be relatively safe from retribution when I start ruling the world, because I'll be too busy dealing appropriately with the people who try to definitively attach MVC-like labels to various pieces of Django! I'm not going to single anybody out for special attention here, since these are statements I've seen from multiple sources. I claim they are simply incorrect:

"In Django, templates are the view, view functions are the controllers and Django models are the models."

Um ... no. Unless, you've gone out of your way to do that. In which case, you're hiding a large portion of logic in custom template tags -- at which point you're using more than the template component -- or your application has very simple presentation functionality. This isn't a mischaracterisation that's unique to Django, either, but I'm using that as an example here, since, in the past few years, it's the particular set of libraries that I've spent the most time seeing people use in a wide variety of situations and experience levels.

There are variations on this statement, but they all seem to fall victim to being cases of the same over-generalisations. I also don't think this is at all specific to Django.

Django Models

In the MVC pattern, the Model is the application object. It contains all the presentation-agnostic, data-centric logic, which is often labelled "the business logic". In Django, small-m models are representations of persistent objects. They are the things that are saved to the database (or other persistent store of choice). There are often methods on models that act on that data, but it's rare to see all of the business logic on a single model.

That's not too surprising. A particular "business application" piece could easily consist of more than one persistent object type. Permitted interactions between those different types and even within instances of one type, constitute business logic as well. So if the separation between MVC-Model and other pieces of code was clearly indicated somehow, I wouldn't be at all surprised to see some pure functions (not methods on an object) as part of the Model side of things.

Django Views

Django's template "language" is intentionally simple. Out of the box, it's not even Turing complete (which is why I put language in quotes, although adding the missing bits with custom template tags is trivial and has been done in a few extensions and alternative versions).

Turning presentation-agnostic information into a presentable version requires some processing. It may require a lot of processing, depending upon what the output format might be. Hence, claiming that or attempting to implement all presentation logic in templates is going to lead to tears, leaky abstractions, tightly coupled pieces of code or some combination of those events. For most things, it's going to be quite normal to have some Python and associated libraries involved.

For example, I might pull back a bunch of data from the database and want to group them in some particular way for presentation. Writing code is simple enough for that and it's an MVC-view role. Happens to be done, typically in Django "view" functions as well. But then things might split because I could have multiple output formats, say, HTML web page and an XML version, such as an Atom Feed document.

Most Django view functions I have looked at (and an awful lot that I have written) combine multiple parts of the MVC pattern into the same Python function. We might have form validation, model saving, response redirection, external process triggering all in the same function. Since a lot of those are only one or two lines of code and it isn't really too confusing to mix them or necessarily clearer to split them up (if you have to jump all over the place to read a single flowing piece of operation, readability is reduced. If you have to use a special editor to avoid this problem, utility is reduced). This paragraph covered things that are Model, View and Controller in their responsibilities.

Understanding the responsibility of each piece of code is a good idea. This is why I was suggesting earlier that knowing a term like MVC is a useful mnemonic for remembering to think about responsibility assignment. But knowing that there are more terms and patterns than just MVC is more useful at the implementaiton layer.

I've written two reasonably large pieces of Django code where I made quite an effort to split up the responsibilities into different Python functions. This was because, in both cases, there were clearly multiple interfaces to the code and the potential to have to scale out the system was part of the design. So the Django "view" -- which is only a convenient notation for the function that is called directly from the URL resolving code (another piece of the Controller layer) -- did dispatching on method name, having worked out the data area that was required and the final presentation format. So it was pure MVC-Controller (although not the entire controller, since there's at least URL resolving and middleware involved as well). Retrieving data in a neutral format (MVC-Model), validating input data (MVC-View), creating presentations in the right format (HTML, XML, Json; MVC-View), were all separate functions. The path to get and then display the data was a bit verbose, but it did make creating different presentation formats easier.

The downside is that verboseness in code is typically a consequence of longer execution paths (more functions, more function calls, more data passing) and the corresponding speed cost. In these cases, it hasn't yet been a problem but some "denormalisation" at the code level in the future (at least on the project that is still going) wouldn't surprise me.

I wouldn't say that a strict splitting of responsibilities into functions is necessary on every project. I've already pointed out some of the upsides and downsides, so it's a matter of considering the circumstances.

Django Templates

The template component is the area where you might be safest making a generalisation. Or at least, a recommendation. If the template is not purely part of the "view" side of the equation, you are possibly trying to do too much in the template and could certainly benefit from pushing a lot of that information gathering back into the function that initially calls the template renderer.

Still not a hard and fast rule and the boundaries are blurry. Where does caching fit into this scheme? If a presentation representation of some piece of data is cached, knowing whether to redisplay the cached version can often depend upon its freshness -- determined by the business logic. Displaying and knowing there's a cached copy to display is a presentation-specific function. Caching isn't clearly in one camp or the other. The uses of caching, though, I think can be cleanly split into MVC-Model and MVC-View situations, but even that might be debatable.

If Not MVC, What Is It?

You can program in an MVC style using Django, but you can also use a number of other styles, certainly including most of the more web-appropriate ones mentioned in Martin Fowler's paper, linked above.

If you really need to put labels on Django's components, I think you'll struggle to do so if your labels are all about responsibility roles. The various phases are functionality roles, but that's not quite the same thing. Specifically, the ORM provides a persistence layer and a passage from storage to Python objects. The template layer provides a way to put a bag of data (a slightly souped-up dictionary) into, well, a template, of mostly unchanging pieces for presentation. The view functions are an interface that provides an entry and exit point for request/response processing. They are the main entry point (modulo middleware processing and URL capturing) for an incoming request and return something that can go back to the user.

Three different pieces of functionality: storage, final string output and "the rest" (Python processing). Since that final component is so huge (and since templates can call back to Python via custom tags), any attempt at pigeon-holing by sticking on responsibility-based labels that are unvarying is bound to fail.

Conclusion: Is There Anything Useful Here?

I've rambled a bit and alternately dismissed MVC and showed how it maps onto concrete components. But I do have a destination in mind; the conclusion of this journey through the various problems and options.

All of this is written as a silent scream of frustration. I genuinely feel that people are locking their thinking in, and causing unnecessary confusion, by getting hung up on "MVC practices". When, as I hope I've shown and linked to, the problem space is much more subtle than that and multiple solutions exist. Focus on the problem you are trying to solve. Keep in mind that there are many similar, but different solutions and think about the Django mapping example I did in the previous section. Hopefully you can see that it's ridiculously constraining (not to mention, simply incorrect) to assign labels to broad components in a framework when they work at a much finer-grained level.

If sticking to the MVC pattern somehow made coding significantly easier and less bug-prone on many levels, there wouldn't be a problem and I would be encouraging it. Sadly, the ambiguity in the terms and constant debates about what is or isn't "proper" MVC, means the net effect is only harmful. Don't get stuck in the straitjacket of thinking always in MVC terms. In fact, don't say MVC for a while. It's over-reliance on a poorly defined term these days and polarising a conversation more than helps. If you say "this breaks the MVC pattern" are you really identifying a problem? Probably not. Are you indicating a direct path to a solution? Doubtful.

Understanding the solution is the most significant thing you can do. Knowing where the separation of responsibilities lies, and even where one line of code might be taking on two roles, is the key. All our patterns and shortcuts and claims that "XYZ is not pattern ABC" are attempts to identify places where we might not understand the solution or where the solution might be difficult to maintain or pass onto somebody else. A solution is something that solves a problem. That's the primary measurement of success, although auxiliary items like future maintainability have a large role to play as well (in other words, any solution is better than none, but it's not an excuse to plump for the first thing you can think of).

Topics: software/design, software/django