A famous instance of "irrationality" reconsidered

The usual presentation

There's a rather famous demonstration of human irrationality that goes like this. Suppose there are four index card on a table in front of you, with the following things drawn on them:

You're told that every card in fact has a letter or number on one side, and a square or a circle on the other. You're now asked to consider the statement "Every card with a letter on one side has a square on the other.". Obviously you can't tell whether it's true or not without more information than you can see on the table; so you're allowed to turn some cards over and see what's on their other sides. You should turn over as few cards as possible. Which ones do you turn over?

If you haven't seen this before, you will probably find it illuminating to consider it now. (Although, having been warned that people tend to get it wrong, you'll be on your guard.) The only real purpose of this paragraph is to make it slightly less likely that you'll see the answer before even thinking about the question; it would be a shame if you did that. If you're still reading then stop doing so, decide on your answer to the question, and then go on to the next paragraph. There's nothing interesting in this one. Honestly. Nothing at all. Go on, shoo. Thank you.

The usual response

OK. Almost everyone turns over card 3, with the letter A on it. Obviously that's correct. Many people also turn card 1, with the square on it. Very few turn cards 2 (circle) and 4 (number).

In fact, the two cards you need to turn to verify or refute the claim are 2 (circle) and 3 (letter). You don't need card 1; if you found a number on the other side of it, that wouldn't say anything about the truth of "Every card with a letter has a square". And you do need card 2; if you found a letter on the other side of it, that would kill the hypothesis at once.

Most people (about 90%, in fact) behave irrationally when presented with this test.

A different version

So far, so good. However, recently I was reading a book about irrationality which happened to give a slightly different version of the experiment. In this version, instead of shapes drawn on the cards, each card has

and the statement we're asked to consider is "People with delusions of persecution tend to draw faces with unusually large eyes". The rest of the setup is as before: so we have two cards showing psychological descriptions (one with delusions of persecution, one without) and two showing face drawings (one with large eyes, one without).

The book then described the usual findings, and made the usual remarks about irrationality: turning card 1 is irrational, not turning card 3 is irrational, etc.

What's wrong with this version

But, with this form of the experiment, all the cards are worth looking at! So, not turning card 2 is still irrational, but turning card 1 isn't irrational at all. So is not turning card 4!

Here's why. What does "tend to" mean here? Presumably that people with delusions of persecution are more likely to draw faces with large eyes than people without. To tell whether this claim is true or not, we need to know how likely both groups are to draw large-eyed faces.

An example

Think about it like this. Instead of having just one card of each type, you have piles containing 1000 of each of the four kinds of card. You're told that the cards came from a random sample of the population. (So we're obviously interpreting "delusions of persecution" rather broadly.) Suppose that, like a good rational person, you turn over the allegedly relevant cards. You find that

So, now we have: 500 delusions/large-eyed; 1000 delusions/small-eyed; 500 no-delusions/small-eyed. What do you conclude from this?

The answer, provided you're really rational, is that it depends on what the other cards turn up. Here are two possibilities.

  1. Of the "people without delusions" pile, all have drawn small-eyed faces. Of the "people who've drawn large-eyed faces" pile, all have delusions of persecution.
  2. Of the "people without delusions" pile, all have drawn large-eyed faces. Of the "people who've drawn large-eyed faces" pile, none have delusions of persecution.

In the first case, our tallies now come to: 1500 delusions/large-eyed; 1000 delusions/small-eyed; 0 no-delusions/large-eyed; 1500 no-delusions/small-eyed. So, large-eyed drawings are much more common among those with delusions of persecution (60% versus 0%); and if you encounter a large-eyed drawing then you can be absolutely certain that it came from someone with delusions of persecution. I'd certainly describe that as "people with d.o.p. tend to draw large-eyed faces".

In the second case, our tallies now come to: 500 delusions/large-eyed; 1000 delusions/small-eyed; 2000 no-delusions/large-eyed; 500 no-delusions/small-eyed. In this case, large-eyed drawings are much less common among those with delusions of persecution than among those without (33% versus 80%); delusions of persecution are much less common among those who draw large-eyed faces than among those who don't (20% versus 67%). I'd certainly never describe that as "people with d.o.p. tend to draw large-eyed faces".

In conclusion

The answers people usually give aren't any less irrational when applied to this modified version of the problem, but they're irrational in a different way. The only real conclusion one can draw is that the authors of the book I found this in (who didn't notice this, and made some very bogus claims as a result) aren't too rational themselves.