Given these premises, what can you infer?

1. If there is a king in the hand then there is an ace, or if there isn’t a king in the hand then there is an ace, but not both.
2. There is a king in the hand.

Practically everyone draws the conclusion “There is an ace in the hand.” But this is wrong: We’ve been told that one of the conditional assertions in the first premise is false, so it may be false that “If there is a king in the hand, then there is an ace.”

But almost no one sees this. Princeton psychologist Philip Johnson-Laird writes, “[Fabien] Savary and I, together with various colleagues, have observed it experimentally; we have observed it anecdotally — only one person among the many distinguished cognitive scientists to whom we have given the problem got the right answer; and we have observed it in public lectures — several hundred individuals from Stockholm to Seattle have drawn it, and no one has ever offered any other conclusion.” Johnson-Laird himself thought he’d made a programming error when he first discovered the illusion in 1995.

Why it happens is unclear; in puzzling out problems like this, we seem to focus on what’s true and neglect what might be false. Computers are much better at this than we are, which ironically might lead a competent computer to fail the Turing test. In order to pass as human, writes researcher Selmer Bringsjord, “the machine must be smart enough to appear dull.”

(Philip N. Johnson-Laird, “An End to the Controversy? A Reply to Rips,” Minds and Machines 7 [1997], 425-432.)

10/18/2016 UPDATE: Readers Andrew Patrick Turner and Jacob Bandes-Storch point out that if we take the first premise to mean material implication (and also allow double negation elimination), then not only can we not infer that there must be an ace, but we can in fact infer that there cannot be an ace in the hand — exactly the opposite of the conclusion that most people draw! Jacob offers this explanation (XOR means “or, but not both”, and ¬ means “not”):

I’ll use the shorthand “HasKing” to be a logical variable indicating whether there is a king in the hand.
Similarly, “HasAce” is a variable which indicates whether there is an ace in the hand.

We’re given two statements:

#1: (HasKing → HasAce) XOR ((¬HasKing) → HasAce).

#2: HasKing.

#2 has just told us that our “HasKing” variable has the value “true”.

So, we can fill this in to #1, which becomes “(true → HasAce) XOR (false → HasAce)”.

I’ll call the sub-clauses of #1 “1a” & “1b”, so #1 is “1a XOR 1b”.

1a: “(true → HasAce)” is a logical expression that’s equivalent to just “HasAce”.

1b: “(false → HasAce)” is always true — because the antecedent, “false”, can never be satisfied, the consequent is effectively disregarded.

Recall what statement #1 told us: (1a XOR 1b). We now know 1b is true, so 1a must be false. Thus “HasAce” is false: there is not an ace in the hand.

Jacob also offered this demonstration in Prolog. Many thanks for Andrew and Jacob for their patience in explaining this to me.