The conjunction fallacy & Grice
In “Here’s Why Automaticity Is Real Actually”, Scott Alexander describes the conjunction fallacy as “not only well-replicated, but easy to viscerally notice in your own reasoning.” Can we better understand what’s going on here?
It’s important to distinguish the various meanings that we attribute to the word “probable.” In fact, “probability” is a shaggy, irregular word which requires a lot of conceptual engineering for various specific domains, and it’s no coincidence that Carnap, who gave much stimulus to conceptual engineering, wrote a book that attempted to carve out probability (he distinguished probability1 and probability2, which I’m not going to touch on).
A cognitive scientist like Kahneman thinks of the word “probable” in a mathematical way, and identifies this statistical interpretation with the common notion of the word. Suppose we asked him the Linda question. His thought process would develop in this way:
“The experimenter is asking me which has a higher value, P(Linda is a bank teller) or P(Linda is a bank teller AND Linda is active in the feminist movement).” To him, the question is in the genre of a mathematics word problem, which is why he throws away all the extraneous, humanizing detail about Linda. You don’t have to be a Bayesian to think in this way; you can either ask in the frequentist way, “In a population of people with Linda’s broad characteristics, and I take samples, would I get more bank tellers or bank tellers AND active feminists?” A Bayesian would wonder, “What is the subjective likelihood (consistent with the laws of probability theory) would I assign to both P(A) and P(A and B)?”
Suppose, however, I drag Average Alice and Basic Bob into my laboratory and I present them with this exact question. Mind you, this does not preclude the possibility that Alice and Bob implement some sort of Bayesian-ish plausible reasoning in their brains (on that, see E. T. Jaynes’ paper here). For Alice and Bob, the word “probable” is essentially asking a social question. Of course a woman who is “deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations” is likely to be an active feminist! I’ll also point out that the first option, “Linda is a bank teller”, when contrasted with “Linda is a bank teller and is active in the feminist movement”, actually implicitly sounds like, “Linda is a bank teller and is not a feminist.” The average person, the average intelligent person who doesn’t stake their life on the genre of mathematical word problems is very reasonable in interpreting the options in this way for Gricean reasons. In fact, from a mathematical point of view their answers are quite reasonable, because the implied mathematical question is not “Which is larger, P(A and B) or P(A)?” but rather “Which is larger, P(A and B) or P(A and not B)?” You can use this to point out that it’s not the subjects who are wrong, but Kahneman.
In fact, I suspect that people would give a better answer if you made it into a matter of numbers. “Suppose I have a room of 100 women. How many of them are likely to be bank tellers, and how many of them are likely to be bank tellers and active feminists?”
The fact of Gricean norms comes up elsewhere, in Yishan Wong’s attempt to show that people don’t know how to understand counterfactuals. If you go up to your coworker and ask them, “If I were to go to France, where would I get a baguette?”, it’s a reasonable assumption that I am planning on going to France, because if not I would not be asking this question. (Unless, of course, you know that I’m asking this question to be insufferable, for which the appropriate answer is “Sod off.”)
A bigger divergence comes from Kahneman’s forecasting questions. Kahneman’s experiment involved asking two different groups of forecasters to rate the probability of two different statements, each group only being shown one statement.
“A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.”
“A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983.”
For Kahneman, a good cognitive scientist, of course all these disparate activities, the “probable” involved in the bank teller problem is the same as the probability involved here. Perhaps Kahneman (or perhaps a LessWronger, because they seem to be more keen in discussing counterfactual worlds) would reason this way: Imagine there are a hundred possible worlds in the future, and I tag off each world where the USA and the USSR suspend relations with a A, and I tag off each world where Russia invades Poland with a B, which is larger, P(A) or P(A and B)?”
Yudkowsky helpfully tells us this;
The scenario is not “The US and Soviet Union suddenly suspend diplomatic relations for no reason,” but “The US and Soviet Union suspend diplomatic relations for any reason.”
But this is not what the words imply; Yudkowsky and Kahneman might mean the second statement, but that is not how the first would be interpreted. In fact, I suspect that the forecasters involved would do better if 1) was simply “The US and Soviet Union suspend diplomatic relations for any reason.” without them knowing anything about Bayes’ theorem or the conjunction fallacy. A sentence is not a mathematical proposition but a narrative in nuce. The story presented by 1) is that there’s a sudden suspension of diplomatic relations. The story presented in 2) is more plausible, because again, these are narratives we are talking about, so of course you’d be more willing to press this. Moreover, it’s not clear that our forecasters are thinking in terms of “checking off” possible futures, but are thinking causally. It’s also possible that for sentence 2, they’re not thinking of P(A and B) but rather P(A|B): given that the Russians invade Poland, it’s quite likely that diplomatic relations will be suspended. These forecasters are probably intelligent people, people who are rather good at solving mathematics word problems, but even they have to be pushed to consider this like a word problem.