‘Do You See What I See?’: the conjunction fallacy & latent incentive structures in experimental settings

by RIPDCB

The conjunction fallacy — ever heard of it? One of the more controversial bias experiments of the past 40 years (RIP DK), the conjunction fallacy elicits the strongest responses from believers and non-believers alike. The initial findings were so offensive that Kahneman & Tversky were tasked with replicating their results again, and then again, and then again…concocting variation upon variation of the original experiment until they had managed to isolate every possible interpretation of ‘probable’ while also testing different social contexts (casuals & industry-specific folks alike). This is all to say: most of, if not all, of the pushbacks against Kahneman & Tversky’s findings have been responded to in some experimental form or another.

Here’s a helpful explanation of the conjunction fallacy, by way of Yudowsky:

In the conventional interpretation of the Linda experiment, subjects substitute judgment of representativeness for judgment of probability: Their feelings of similarity between each of the propositions and Linda’s description, determines how plausible it feels that each of the propositions is true of Linda.

Representativeness and probability get mistaken for each other by subjects who select the description that they think is most representative of the situation over the description that is, in fact, most probable. That this would happen is believable, especially for those who do not carry with them a mathematically rigorous conception of probability (experiments on those with backgrounds in statistics have also been run and the results are similar, for what it’s worth).

This representational fallacy is probably real. But why do we mistake representativeness for probability? For Yudowsky, Kahneman, Tversky, and Scott Alexander, the answer is cognitive bias. And while that very well may be the case — that in certain situations we instinctively go against our most rational selves — there might be an alternate explanation, or at least a complementary one, that deals with the structures of experimental settings.

When Pierre Bourdieu was first getting started, he went to study the Kabyle people in present-day Algeria. He wasn’t the first, though. Structuralist anthropologists had been working in the area for decades, speaking with informants in order to build genealogy charts. These charts would become quite influential in understanding the marriage rituals and structures of the Kabyle people, namely the relative prestiges of different arrangements (paternal cross cousin down to maternal same-side cousin). They deduced that your social rank would define which partnerships were available to you, and those partnerships would then reinforce your social rank. For the structuralist anthropologists, they believed that this hierarchy was culturally inscribed—i.e. that it was adhered to predominantly because of a shared belief system and community-wide buy-in.

Bourdieu found something else entirely, however: most marriage arrangements were not a function of ritual & social rank, but instead born out of very practical concerns. How people were paired often came down to capital (who had it and who needed it), protection (again, who had it and who needed it), and environmental factors (e.g. a hard-hitting drought might push someone to accept a marriage arrangement they otherwise might not). The disjunction between his findings and those of previous anthropologists came down to a matter of formalization. Bourdieu believed that the informants––in sitting down with social scientists who were trying to map a culture through a series of rigid, objective relationships––tended to reify the practical circumstances of life in order to fit the formalizing desires of their interlocutors. Sensitive to the power structures underlying the relationship between anthropologist and tribal informant, the latter wanted to meet the former at his level, ultimately providing a misrepresentation of the logic driving the informant’s culture. This misrepresentation was probably in good faith (they wanted to be helpful), but a misrepresentation nonetheless, one that the informants themselves may have even believed despite their experience of the material realities that influenced their lives.

Around the time Bourdieu was studying the Kabyle people, psychologist Martin T. Orne was growing frustrated with his scientific practice. Social experiments he would run on UPenn students kept coming up all-too in line with his hypotheses. What he kept butting up against was quite similar to what Bourdieu found in Kabyle: his subjects were providing him with the answers that they thought he wanted. Subjects were picking up on the expectations of the experiment and Orne’s expectations more generally, and were trying to anticipate the ‘right’ response, rather than provide an intuitive or rational response. Orne came up with the term demand characteristics to describe the structural features of experimental settings that might push subjects to give certain responses over others. Examples range from subject-experimenter relationships to foreknowledge of the experiment to the physical features of the lab itself. Incentives are latent yet legible in the structures of experiments, despite a scientist’s best intentions, and accidentally incentivized responses may very well muddy the validity of a given study’s findings.

This is all to suggest that the conjunction fallacy may be a bias circumscribed by the payouts subjects project onto experiments. In the Linda experiment, subjects may have been flexing their interpretive skills when they selected “bank teller + feminist” for the most probable proposition, emphasizing for Kahneman & Tversky that they were capable of making connections between the experiment’s frame and the possible answers. In the version of the experiment run on medical internists–where the internists were asked to assess the probability of different symptoms after a patient’s pulmonary embolism—the subjects overwhelmingly picked “dyspnea and hemiparesis” (the former is quite common to pulmonary embolisms, the latter not at all), and perhaps they did so because they wanted to prove that they had, in fact, studied their books. Parallel incentive structures can also be unearthed in the ‘Russia invades Poland’ example that Yudowsky talks about extensively.

Ultimately, I think it’s unclear the extent to which subjects are confusing representation for probability instead of selecting for representation over probability. I want to emphasize that this demand-characteristic reading of the conjunction fallacy does not preclude such a bias existing in an experimental setting. However, it puts the experiments in a different light, one that might help us better contextualize social scientific findings. If in fact the conjunction fallacy is in any part the product of subjects responding rationally to incentive structures unintentionally baked into an experiment, then the relevancy of cognitive biases in non-experimental settings might look quite different.