Have you heard of Goodhart’s Law? If not, I have come to believe it is a crucial piece in understanding why complex systems in which intelligent agents adapt their behavior can easily deteriorate over time: If agents are rewarded for optimizing a proxy, and they have limited resources to spend on improving their condition across the board, then rewarding them for performing well on a specific measure will reduce their performance outside of that measure.
Two pertinent examples that come to mind are intelligence (IQ) tests as a measure of a person’s ability to solve general complex problems, and politician popularity as a measure of their ability to create conditions in which political change can occur.
Under the conditions when IQ tests were developed, a high IQ was indeed predictive of a person’s general ability to deal with life’s complexity. Given that we have become somewhat obsessed with assessing intelligence using certain measures, however, I no longer believe that the people with the highest IQ are, in fact, the most capable of obtaining the highest quality of life.
There admittedly seems to be a limit at the bottom of the distribution. People lacking in certain cognitive capacities—the ones measured by IQ tests—might find it more difficult to perform well in many of the new occupations. But does that mean they have lower quality lives?
Looking at what a sole focus on optimizing IQ has exposed, I would suggest a lack of wisdom and embodied understanding of reality that goes beyond the propositional. Our entire economy has become very much aimed at improving efficiency, in other words the speed with which operations are executed. And as a consequence, we have chosen to focus on a person’s ability of easily manipulating complex, abstract information in working memory. Unfortunately, that isn’t the only aspect of mental functioning that supports good decision making…
On to the second example: our way of selecting people for political office has created an entire industry which during the campaign season kicks into high gear, producing candidates that appeal to our sense of what makes a person a capable leader. Applying Goodhart’s Law, it is then no wonder that the people we now select may score incredibly high on the measures we (currently) care about, while then turning out to be terrible leaders.
And that brings me to what I have dubbed the Goodhart-Sinclair Trap. Upton Sinclair wrote in his 1935 account of his run for California governor “I, Candidate for Governor“: “It is difficult to get a man to understand something when his salary depends on his not understanding it.”
These two forces—that of selection pressure on a measure that is a poor proxy for a desirable quality, which over time decorrelates and potentially even decreases the desired quality, paired with our mental inability to accept a truth if acceptance means lowering our status in life—keep us in a variety of traps.
It goes something like this: I have a quality I care about, and I want to improve it. I choose a certain indicator that tells me how well I am doing in achieving that quality. At some point, I have probably invested many hours in optimizing that measure. And once I get there, it then seems hard to imagine, let alone accept, that all that effort might have been in vein. Instead, I will simply insist that “my strategy is working”—no matter how much evidence there is to the contrary.
That’s roughly where I see our political discourse on a range of topics: to pretty much any external, uninvolved observer, it probably is self-evident that neither side is getting what they want, and that instead we have become grid-locked in petty arguments and infighting. To the actors and their supporters, however, it still seems to be the “winning strategy”.
And since there are still a lot of people benefiting from the status quo—take the Facebook whistleblower revelations, for instance—it is also no surprise that little effort goes into discovering how to actually improve matters. It will probably require for us to go through a period of unbearable pain before we are willing to consider real change. That’s the Goodhart-Sinclair Trap.