In this essay I discuss how theoretical models in finance and economics are used in ways that make them “chameleons” and how chameleons devalue the intellectual currency and muddy policy debates. A model becomes a chameleon when it is built on assumptions with dubious connections to the real world but nevertheless has conclusions that are uncritically (or not critically enough) applied to understanding our economy. I discuss how chameleons are created and nurtured by the mistaken notion that one should not judge a model by its assumptions, by the unfounded argument that models should have equal standing until definitive empirical tests are conducted, and by misplaced appeals to “as-if” arguments, mathematical elegance, subtlety, references to assumptions that are “standard in the literature,” and the need for tractability.

An engineer, a physicist and an economist are stranded on a deserted island with nothing to eat. A crate containing many cans of soup washes ashore and the three ponder how to open the cans.

Engineer: Let’s climb that tree and drop the cans on the rocks.

Physicist: Let’s heat each can over our campfire until the increase in internal pressure causes it to open.

Economist: Let’s assume we have a can opener.

…

At one conference I attended not too long ago, I heard a discussant of a paper say that the paper’s author had done a very poor job trying to defend one of his model’s assumptions. The discussant then said that the author should “just make the assumption and move on.” The implication seemed to be that since the assumption in question was difficult to defend, it would be best not to call a reader’s attention to this by mounting a very weak defense. Perhaps this was “good” advice for building a bookshelf model and sneaking the paper past a referee, but I would characterize it as a strategy to make the paper’s model a chameleon, a model that is intended to circumvent the filter.

As I have argued above, although a model may be internally consistent, although it may be subtle and the analysis may be mathematically elegant, none of this carries any guarantee that it is applicable to the actual world. One might think that the applicability or “truth” of a theoretical model can always be established by formal empirical analysis that tests the model’s testable hypotheses, but this a bit of a fantasy. Formal empirical testing should, of course, be vigorously pursued, but lack of data and lack of natural experiments limit our ability in many cases to choose among competing models. In addition, even if we are able to formally test some hypotheses of these competing models, the results of these tests may only allow us to reject some of the models, leaving several survivors that have different implications on issues that we are not able to test. The real world filters will be critical in all these cases.

…

I have argued above that before applying theoretical economic models to the real world we need to critically assess them by seeing how well they pass through real world filters. It might be thought that this is not the practice in the hard sciences. In particular it might seem that the success of models in fields such as quantum mechanics shows that we should not judge models by the reasonableness of their assumptions or the model’s plausibility. After all, almost everything in quantum mechanics seems to defy common sense. Richard Feynman is reputed to have said, “If you think you understand quantum mechanics, you don’t understand quantum mechanics.” The models of quantum mechanics are well accepted because of the strength of their predictions, not the “reasonableness” of their assumptions.

There are, however, huge differences between the settings in which the models of quantum mechanics are applied and those in which we attempt to apply economic models. Figure 4 depicts how models are used in quantum mechanics. We don’t get information about what “motivates” electrons and photons to make “decisions,” but we do see the outcomes of those “decisions” in the paths they take and other observational evidence. Of course, it is silly to anthropomorphize electrons and photons and think of them as making decisions. The main point is that all of our observational evidence is based on interactions these particles have with other particles and we can produce voluminous data through observation and experiments. Models in quantum mechanics have been extraordinarily successful in making predictions, with these predictions being consistent with what is measured in experiments to many decimal places.

Most importantly, the models in quantum mechanics don’t blatantly contradict anything we know about electrons and photons. They are basically consistent with all that is observed. If a systematic inconsistency between what the models assume and what is observed were to emerge, the models would generally be viewed as incomplete or flawed and efforts would be made to make corrections.

Now consider models as they are often used in finance or economics. Figure 5 is, I believe, a fair depiction of what we face in many cases. We often have competing models (e.g., models A through F) that are all roughly consistent with some aspects of what we observe (e.g., certain patterns of capital structure decisions). The amount of data we have is often (but not always) quite limited and endogeneity problems generally make it challenging to test and discriminate among models. (We are justifiably suspicious of models that fit the data extremely well, since we suspect they may be over-fitted.) A major issue is that the models are often based on assumptions or have implications that contradict things we know about the economic decision makers or other economic phenomena we observe. As I have argued at length above, simply ignoring or dismissing these contradictions is not justified and is not a good practice for developing models and theories that are useful for advancing our understanding of our economy.

Economic agents (i.e., human beings) are much more complicated than electrons and photons and they interact in environments that are much more complex and constantly changing. It is even possible that the development of an economic theory or the publication of an academic paper in economics can change how these economic agents behave. The same cannot be said about electrons or photons changing their behavior based on something written by a physicist. This means that developing useful models in economics and finance involves challenges not found in quantum mechanics and other research areas in physics and the hard sciences, where much more data are generally available and, at least in many cases, controlled experiments are possible.

Because of these greater challenges, models in economics and finance will of necessity be simplifications and will abstract from much of the complexity of the domain they are designed to explain. This means that there will be tensions between what these models assume and things we know are true. The fact that this will be the case does not mean that we should ignore these tensions. When the major drivers of a model’s results cannot be connected to things we see in the real world, we are surely justified in questioning how useful the model is as an explanation of what we see. This is true even when the predictions of the model are supported by some empirical tests. When the model ignores some factors that we have good reason to believe are of first-order importance, we likewise are justified in questioning the model’s usefulness. Confronting these tensions head on is surely a better way of making progress than glossing over them.

Vir: Paul Pfleiderer, Stanford University