Tuesday 3 April 2012

All models are wrong, but some are useful.

Economists are often mocked for the assumptions we make in our models. People accuse of us surging ahead with elegant computations, forgetting that our entire analysis rests on some wholly implausible assumptions. Well, yes, these assumptions are often totally ridiculous. And yes, given their general contribution to the financial crisis, the macroeconomists of the last 20 years have got a hell of a lot to answer for. But, a good model, humbly deployed, can still be useful despite it's implausible assumptions.


A model is just a system of hypothetical propositions. A model is to Economics what the word 'if' is to Philosophy. Imagine if every time a philosopher said 'if' they were shouted down by everyone else because their 'if' simply wasn't true. 'If' is a bloody useful concept. "All models are wrong, but some are useful", said the statistician George Box. We know the assumptions we make are frequently implausible, but that doesn't make the results of our models useless. The tedious stereotype implies we economists believe these assumptions to be true, or will assume anything to reach some predetermined conclusion. I would suggest the opposite, that economists deliberately propose false assumptions in order to try and prove their theories to be false.


Take the following statement:


"Grammar schools and selection on ability only benefits the children of the wealthiest families because ability is so highly correlated with SES (socioeconomic status)."


You might agree or disagree with this statement and God knows plenty of people have a view. But to actually know whether this is true requires the analysis of several phenomena at once. Grammar schools likely increase the access to quality schooling for high ability children in poor neighbourhoods. They also probably harm the children remaining in those poor neighbourhoods as their more able peers leave them behind. Then again, schools in poorer neighbourhoods might be able to better target their teaching at their remaining students, so this segregation by ability could help the less able and so on.


Proper, objective analysis of this issue becomes very complicated very quickly using language alone (or as an economist would say, both the first and second derivatives of complexity with respect to language used are positive). Each of the statements I made in the previous paragraph can be articulated more precisely with algebra. You can sit in the pub all day long and argue about which of the above effects is the strongest, but persuasive anecdotes skilfully delivered with elegant rhetoric can't deal with internal theoretical inconsistencies in the way maths can. For this reason, the evolutionary biologist J. B. S. Haldane said "an ounce of algebra is worth a ton of verbal argument". Words can paint a picture and give you the perspective of the artist, maths can build a 3D (or more) model that can be manipulated and examined from any angle we like.


With this example, a mathematical model could articulate the conditions that would have to be true for selection on ability to benefit children from poorer families. Say we assumed, for the sake of computational simplicity, that all parents were aware of all possible schools they could send their children to. Ignore, just for the moment, the likely scenario that high SES parents will be more informed of their choices. Now let us suppose show that even if this assumption were true, our model showed that allowing all schools to select on ability would cause such strong social segregation that the implied necessary improvement in teaching quality for less able pupils in poorer areas was implausibly large.


The initial implausible assumption was there to provide the ideal scenario for some hypothesis to be true, just as engineers initially simulate plane designs in frictionless skies. If, given that ‘ideal world’ assumption, that plane doesn’t fly, or the theoretical model generates implausible results, then we can probably ditch that design or rule out that hypothesis altogether. The model was wrong, but it was also useful.