A Scott Alexander post really resonated with me, when he talked about one "development milestone" (#4): the ability to understand and speak in terms of tradeoffs, rather than stubbornly try to insist that there are no downsides to your preferred course of action. Such "development milestones" indicate a certain maturity of one's thought process, and greatly change how you discuss ideas.
When a Hacker News discussed Alexander's post, I remembered that I had since started checking for this milestone whenever I engaged with someone's advocacy. I named my spot-check the "Scylla-Charybdis Heuristic", from the metaphor of having to steer between two dangers, either of which have their downsides. There are several ways to phrase the core idea (beyond the one in the title):
Any model that implies X is too low should also be capable of detecting when X is too high.
Or, from the metaphor,
Don't steer further from Scylla unless you know where -- and how bad -- Charybdis is.
It is, in my opinion, a remarkably powerful challenge to whether you are thinking about an issue correctly: are you modeling the downsides of this course of action? Would you be capable of noticing them? Does your worldview have a method for weighing advantages against downsides? (Note: that's not just utilitarian cost/benefit analysis ups and downs, but relative moral weight of doing one bad thing vs another.) And it neatly extends to any issue:
- If raising the minimum wage to $15/hour is a good idea, why not $100?
- If lifting restrictions on immigration is good, why not allow the entire Chinese army to cross the border?
- If there's nothing wrong with ever-increased punishments for convicts, why not the death penalty for everything?
One indicator that you have not reached the "tradeoff milestone" is that you will focus on the absurdity of the counter-proposal, or how you didn't advocate for it: "Hey, no one's advocating that." "That's not what we're talking about." "Well, that just seems so extreme." (Extra penalty points for calling such a challenge a "straw man".)
On the other hand, if you have reached this milestone, then your response will look more like, "Well, any time you increase X, you also end up increasing Y. That Y has the effect/implication of Z. With enough Z, the supposed benefits of X disappear. I advocate that X be moved to 3.6 because it's enough to help with Q, but not so much that it forces the effects of Z." (I emphasize again that this method does not assume a material, utilitarian, "tally up the costs" approach; all of those "effects" can include "violates moral code" type effects that don't directly correspond to a material cost.)
I've been surprised to catch some very intelligent, respected speakers failing this on their core
What about you? Do you make it a habit to identify the tradeoffs of the decisions you make? Do you recognize the costs and downsides of the policies you advocate? Do you have a mechanism for weighing the ups and downs?
