Stop Goodharting Yourself

On doing what matters

I.

If I had to choose an idea that most underlied my thinking throughout my freshman year of college, it would be Goodhart’s Law. It states that once a measure becomes a target, it ceases to be a good measure. Charles Dickens is a classic example: his publisher paid him by the word, so he chose to write wordier sentences, rather than more stories, to increase his pay. Word count may have been a good measure of past output, but by becoming a target for Dickens, it distorted his goal and output.

I would estimate that Goodhart’s Law came up in about 4% of my conversations over the past 9 months. This is a lot! Here are some places it came up:

1. The Honors College at Purdue tries to give its students an “interdisciplinary education.” In theory, I think this is a good idea: different disciplines have valuable insights, and many important discoveries of the past have come from cross-disciplinary discussions. However, rather than focusing on the valuable lessons and insights from across fields, my peers and I learned about how airports are “non-places”, “modern water” is disconnected from its cultural roots, and world heritage sites are places with “outstanding universal value.” This was highly interdisciplinary but not very meaningful.

2. Paul Graham calls learning to get good grades “the most damaging thing you learned in school.” Young startup founders often ask, “What do I have to do to receive funding and make money?” The answer is, of course, to create a good product that people actually want to buy. However, school has conditioned these founders from a young age to seek success on testing metrics, rather than on learning/doing the valuable things. In schools, students aren’t most directly rewarded for having a broad and deep understanding of a topic; they’re rewarded for studying the test questions the night before. This “test-hacking” mindset is fundamentally misaligned with trying to produce the thing that makes a startup succeed, and it leads young founders to worry too much about obvious metrics, like money raised.

3. Many of my friends send out connection requests to hundreds of random people on LinkedIn simply to gain the “500+ Connections” badge. I don’t know what they expect to gain from this. Connections are only as good as the possible opportunities they could yield for you, and the random person who clicked “Accept” is not going to give you any opportunities. Random connections also come with a neglected cost: they make it more annoying to find out whether or not you have a mutual friend with someone. This metric is useless at best, distracting at worst.

4. The more disconnected a hiring manager is from the costs/benefits of their hiring decisions, the easier it is for them to fall into one of Goodhart’s traps. I first heard about this idea in Scott Alexander’s seminal blog post, Meditations on Moloch. Here, the targeted metric is a beautiful resume full of leadership positions from a top university. In theory, this correlates with students who are passionate about doing things, and in practice, it can still identify some highly ambitious students. However, overreliance on this metric causes hiring managers to choose students who game this metric, the ones who did the minimum work required to have “internship” on their resume, instead of the passionate student who spends her time reading and researching the topic she cares about. A startup founder whose success relies on finding dedicated talent will likely avoid this failure, but a hiring manager at a large company will find a marker that lets them tell their boss, “But look at John’s resume! How was I supposed to know he would be a bad hire??” when the candidate turns out to be underwhelming.¹

5. Wages for Housework was a feminist movement from the 1970s. Social justice purposes aside, I’ve always found it funny that despite changing little about economic production, it would greatly increase GDP. Allowing your spouse to use your credit card does not add anything extra to the GDP, but paying them a wage to spend would. If a country only cared about maximizing this particular accounting identity, it might implement this.²

6. Effective Altruism university groups are built with the following idea in mind: you may spend your whole life working a highly impactful job and doing good for the world, but convincing another person to do the same doubles your counterfactual³ impact. However, club leaders can get distracted from this. Club size is a simple success metric, but overreliance on it can lead to bloated social clubs where nobody takes ideas seriously. In practice, 5 people who get effective careers will be much more impactful than 50 people who like to argue about politics, abstracted from any discussion of tractability.⁴

7. To some extent, congresspeople base their voting decisions on calls and emails from their constituents. They want to make decisions that maximize their chances of re-election, and to do this, they primarily need votes. Calls and emails about a particular topic can be a good proxy for how many votes they may receive for behaving a certain way. However, many of them are also aware of Goodhart’s Law, even if not by name. They might understand that even though they’re receiving dozens of calls urging them to oppose the Save Our Bacon Act, these calls come from a vocal minority who will not sway the outcome of their future elections. Animal welfare is not a voting issue for most people, and until that changes, successful political advocacy will be difficult.⁵

II.

Based on my seven examples listed above, you should be able to tell that I’m pretty good at thinking about Goodhart’s Law. So, you should listen to me when I say that you are probably Goodharting yourself, and you should stop.

At some point along the process of creating and working toward your goals, you have created metrics for progress that are misaligned with what you actually want to accomplish. This may have happened when you saw someone working toward a similar goal. Perhaps they came up with a good plan to achieve that goal in a particular way, and without understanding the pieces, you replicated their process. Perhaps they were Goodharting just as much as you are, and you mistakenly believed that they knew what they were doing. Or maybe your Goodharting was all your own doing, the result of getting distracted by a moving number. Either way, you’ve somehow become deadset on doing something that’s only 80% effective at doing what you want to do.

I don’t know who you are, so I can’t say which specific mistake you are making. However, when I look at my peers and past self, the most damaging place where I see Goodharting is how we try to prepare for future opportunities, such as college and careers.

Here’s what usually happens: Someone tells you that to get a good job, you need to have valuable skills from working at internships in the past. You hear, “to get a good job, I need to have ‘internship’ on my resume for as long as possible.” Or maybe someone tells you that to get a good job, people need to think of you when an opening comes up. You hear, “to get a good job, I need to show my resume to as many people as possible and make sure they recognize my personal brand^TM.” This strategy will get you hired by a risk-averse hiring manager for an unimportant role at Big Company, but if you really want to harness your ambition, you actually need the skills⁶ and connections to make things happen. I would go as far as to say that most advice on how to get a great career should boil down to “get good, be known,” and any plan that doesn’t has gotten distracted.

To explore the process of Goodharting in more depth, let’s talk about undergraduate chemistry majors. Most chem majors hear that they need lab experience to succeed, but rather than recognizing this advice as a recommendation to build skills, they behave as if they just need to have “lab” on their resume. They put in their 5-10 required hours per week, not taking advantage of the many other opportunities to improve their craft that appear with more dedication.

I have a friend who defies this pattern: in addition to spending a few years putting in the hours at labs to build skills, he also goes beyond what the curriculum requires of him, and he is much better for it. He goes to his professors and talks to them about the cool chemistry work that they’re doing. He doesn’t wait for a required class to teach him what he wants to know; he asks his professors for book recommendations so he can learn about the forefront of analytical chemistry and ask the important questions. Before his lab starts this summer, he’s reading three chemistry textbooks so that when he goes back to school in the fall, he can begin working on truly meaningful research problems.

I think my friend is doing what every ambitious undergraduate should be doing: looking for the things that will help them gain valuable knowledge and abilities, rather than wasting their time getting a leadership role in a club they don’t care about and where they’ll learn nothing. At the end of the day, professors and employers like finding the people who care about something and will do a good job with it. Even though understanding a few textbooks deeply isn’t visible on a resume, it’s obvious enough in conversations to give him better opportunities than he may have otherwise.

My friend isn’t the only person who has benefited from caring about a topic and trying to improve themself within it. Do you know how much professors love finding students who care about their disciplines and want to talk about them more? The best decision I made over winter break was studying the econometric methods used for causal inference, as it allowed me to understand a bunch of extra papers and have cooler conversations with professors. You can call this “networking” or “making connections,” but at the end of the day, I’m a guy who likes economics talking to other people who also like economics. This is where the exciting things happen!

And it pays off in ways you might not expect: aside from playing a significant role in getting me a prestigious summer internship offer⁷ (as a freshman!), it also got me invited to some cool lunches with guest speakers. I know of a decent number of people who have received job offers and other opportunities because they wrote some good and interesting blog posts about important topics.⁸

I can’t emphasize enough that you don’t need to wait for permission to do these things. Allow yourself to be passionate, study the things that matter, and improve yourself! Don’t worry about the obvious metrics so much. Those will come naturally.

Also, you should read this, this, and this.

III.

The most general way to avoid Goodharting yourself is to remember that you are rewarded for getting things done, not for putting in effort.⁹ In the immortal words of Cate Hall, “choose hard work if hard work is the desired result. If it’s not, and you can take a shortcut, take a fucking shortcut.”

Many musicians (or other people trying to improve at a skill) fall into a trap where they try to maximize time spent practicing, rather than the amount improved. To the musicians, I say unto you: your audience hears not the hours of practice, but the skill with which you play.

2023 me only somewhat understood this

Many readers fall into a trap where they think they need to read their non-fiction book from cover to cover. To the readers, I say unto you: wisdom comes not from reading every word, but from reading words of importance.

Many writers fall into a trap where they try to show their readers how much they’ve learned¹⁰, or tell them about the full journey they went through to write their work. To the writers, I say unto you: good writing comes not from writing more, but from writing that which is worth reading.

Spend your time doing what matters.

Image by Nano Banana 2

1. I suspect different fields struggle with Goodharting resumes to different degrees. Building an impressive engineering portfolio is likely more directly indicative of skills than being on the leadership council of one of your school’s nine consulting clubs. To some extent, other aspects of the hiring process can also resolve these issues. Letters of recommendation, interviews (especially ones that ask you for specific stories about things on your resume), tests, and competition wins all add valuable information that resolves much Goodharting. Still, each of these may be gamed by students in their own ways.

2. Actually, doing this would probably make GDP per capita a better proxy for gender equality. Goodhart defeated???

3. Counterfactual impact is measured as follows: compare the world where you perform an action to the world where you do not perform the action. The difference in goodness between those two worlds is the counterfactual impact of the action.

4. The trade-off between popularity and impact is also tricky. I’ve heard people argue that Effective Altruists should try to make their ideas seem as pretty as possible so as to reach the most people, but I suspect that this might make the ideas less appealing to the most impactful people or draw people away from the most impactful cause areas. I’m highly uncertain about where the benefit-maximizing point is, though.

5. But you should still call your representative! It’s still one of the highest EV ways to spend 5 minutes! Gestation crates are really terrible!

6. Intuition pump showing that increased productivity leads to higher wages: Imagine a company paid you significantly less than the marginal benefit they receive from hiring you. Another company that can utilize your skills equally well profits by poaching you and paying a higher wage. This process continues until the marginal benefit of hiring you equals the marginal cost (including the opportunity cost) of paying your wage. Because the marginal benefit of hiring you increases with your skill level, so does your wage. This process is less than perfectly efficient for a number of reasons (measuring productivity is hard, people face status quo bias, and wages are somewhat inflexible), but it is generally accurate, and the popular graph claiming otherwise is wrong because it excludes many of the employees who have experienced the most productivity gains in the productivity line.

Alternative way of thinking: if your goal is to get a job that does good in the world, you should get skilled because that does more good.

7. Which I, ironically, ended up declining for a better offer in a different field

8. This might be an EA-specific phenomenon, though

9. I’m rejecting the labor theory of value

10. One of many flaws I blame on high school English classes