The Cognitive Reflection Test: why smart people fall for simple traps

Here is a puzzle that has been printed in hundreds of papers, books and classrooms:

A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?

Most people hear an answer almost before they finish reading: 10 cents. It feels obvious. It is also wrong. If the ball cost 10 cents, the bat would cost $1.10, and together they would cost $1.20. The correct answer is 5 cents: a 5-cent ball plus a $1.05 bat makes $1.10, and the bat is exactly $1.00 more.

Nothing about this problem is hard. Any adult who can do basic arithmetic can solve it. The interesting part is why so many capable people do not. That question is what the Cognitive Reflection Test was built to explore.

A quick note before you read on: if you plan to take our Cognitive Reflection Test, consider taking it first. As you will see below, knowing how these puzzles work changes how people score on them.

Where the test came from

The Cognitive Reflection Test (CRT) was introduced by the economist and psychologist Shane Frederick in a 2005 paper (Frederick, 2005). It has just three questions. Besides the bat and ball, there is a question about machines making widgets and one about a patch of lily pads that doubles in size every day.

What the three items share is their design. Each has an answer that springs to mind immediately, and that answer is wrong. Getting the right answer is easy once you notice something is off. The hard part is noticing at all.

Frederick described the test as measuring the ability or disposition to resist reporting the first response that comes to mind. He gave it to thousands of people, many of them students at well-known universities, and found that wrong answers were common even in highly educated groups. People’s mistakes were also not random: most wrong answers were the specific intuitive answer the question invites (Frederick, 2005).

Frederick also reported that CRT scores were related to how people made other kinds of decisions. In his data, higher scorers tended to be more patient in choices about money now versus later, and their choices about risk also differed from those of lower scorers (Frederick, 2005). This is what made a three-item puzzle set interesting to economists, not just to psychologists.

Fast thinking, slow thinking

The CRT is usually explained with a dual-process account of the mind. In the version Daniel Kahneman made famous, “System 1” works quickly and automatically, producing impressions and answers with little effort. “System 2” is slower and effortful, and it can check, and sometimes override, what System 1 produces (Kahneman, 2011). Kahneman uses the bat-and-ball problem as an example: System 1 offers “10 cents”, and many people’s System 2 accepts it without checking.

On this view, the CRT catches people at the moment when a quick intuition and a careful check conflict. A correct answer suggests the check happened. The tempting wrong answer suggests the first impression went through unexamined.

Criticisms of the two-system picture

This explanation is popular and useful, but it is not settled science, and it has serious critics.

Jonathan Evans and Keith Stanovich, two leading dual-process researchers, have acknowledged that the theory has been attacked from several directions. In their response, they argue that many features often attached to the two “systems” (fast versus slow, unconscious versus conscious, emotional versus logical) do not reliably go together, and should not be treated as defining. They prefer the terms “Type 1” and “Type 2” processing and focus on working memory as the key difference (Evans & Stanovich, 2013).

Arie Kruglanski and Gerd Gigerenzer go further. They argue that intuitive and deliberate judgments follow the same underlying rule-based principles, so there is no need to split the mind into two separate systems at all (Kruglanski & Gigerenzer, 2011).

For a test-taker, the practical point survives either way: some answers come quickly and feel right, and checking them sometimes changes the answer. What remains debated is whether that reflects two distinct mental systems or one system working under different conditions.

What CRT scores relate to

Because the test is so short, researchers have been surprised by how much it predicts.

• Thinking biases. Maggie Toplak, Richard West and Keith Stanovich gave people the CRT alongside a wide range of classic heuristics-and-biases tasks. The CRT predicted performance on these tasks, and it did so even after accounting for measures of cognitive ability, thinking dispositions and executive functioning (Toplak et al., 2011). In other words, it seemed to capture something that standard intelligence measures miss: a tendency to avoid lazy, low-effort thinking.
• Misinformation. Gordon Pennycook and David Rand asked people to judge the accuracy of real and fake news headlines. People with higher CRT scores were better at telling real headlines from fake ones, and this held whether or not the headlines matched their own political views (Pennycook & Rand, 2019). The authors argued that falling for fake news is better explained by not thinking carefully than by motivated, partisan reasoning.

Two cautions apply. First, these are correlations at the group level: a high score does not mean a particular person never falls for a bias or a false headline. Second, the size of these links varies across studies and samples, and part of what the CRT captures may be something other than reflection, which brings us to the next debate.

Is it reflection, or is it math?

All three of Frederick’s items, and many later ones, are arithmetic word problems. That raises an obvious objection: maybe the CRT mostly measures numeracy, the ability to work with numbers, rather than the tendency to stop and check.

Andrew Sinayev and Ellen Peters tested this directly. They separated the CRT into a “reflection” part and a “calculation” part, and also measured numeracy with a separate scale. In their studies, numeric ability explained the CRT’s power to predict better decisions, and reflection added little on its own (Sinayev & Peters, 2015).

Others disagree. Gordon Pennycook and Robert Ross reanalyzed related data and argued that, scored differently, cognitive reflection was the robust predictor and calculation was not (Pennycook & Ross, 2016). The debate is still open. A fair summary is that the classic CRT mixes reflection and numeracy, and it is hard to fully pull them apart with three math-based items.

This is one reason later researchers wrote CRT items that need little or no arithmetic. Keela Thomson and Daniel Oppenheimer developed an alternative set, the CRT-2, made up mostly of verbal puzzles. They found it relied less on numeracy than the original while still appearing to measure closely related abilities (Thomson & Oppenheimer, 2016).

The familiarity problem

The CRT has a weakness that most psychological tests do not share: once you know the trick, the trap stops working.

The bat-and-ball problem is now everywhere: in popular books, online quizzes, job interviews and lectures. Stefan Stieger and Ulf-Dietrich Reips studied a large community sample of 2,272 people and found that about 44% had seen the CRT or similar tasks before. Those people scored noticeably higher (Stieger & Reips, 2016). Prior experience, together with education, was among the strongest predictors of a high score. The authors recommended newer and longer versions with fresh items.

Thomson and Oppenheimer raised the same concern about online research participants who take many studies, and reported that far fewer of them had already seen the CRT-2 items than the originals (Thomson & Oppenheimer, 2016).

The lesson is simple. A high score on a famous puzzle may mean you reflected, or it may mean you remembered. That is also why we suggested taking our test before reading this article, and why our items are original rewordings rather than the well-known originals.

Practical habits for slowing down

You cannot switch off quick intuition, and you would not want to: most of the time it serves you well. The aim is to notice the moments when it deserves a second look. These habits are common-sense practices rather than proven training programs, but they target exactly what the CRT is about.

• Treat “obviously” as a signal. When an answer to a question that looks like a trick arrives instantly and feels certain, that is the moment to check it, not skip it.
• Plug the answer back in. For any number problem, test your answer against every condition in the question. The bat-and-ball error disappears the moment you add 10 cents and $1.10.
• Reread the exact wording. Many traps rely on a word you skimmed past: “more than”, “at least”, “in total”. Read the question once more, slowly, before you commit.
• Ask what base rate you are ignoring. When a vivid description points one way, ask how common each option is in the first place.
• Separate story from probability. A more detailed story can feel more convincing while being less likely, because every added detail is another thing that must be true.
• Pause before sharing. For headlines and claims online, a few seconds asking “is this actually accurate?” is the everyday version of the bat-and-ball check.

How our test works

Our Cognitive Reflection Test is a short, untimed self-reflection tool, not a clinical or diagnostic assessment. Here is exactly what it does.

• 10 questions. All are original rewordings of published item types from the CRT literature and classic heuristics-and-biases tasks. They are not the famous original items.
• Three kinds of trap. Four are number traps (you type a number), two are wording traps and four are probability traps (you pick an option).
• No time limit. We ask you not to use a calculator or search for answers.
• Reflection score out of 10. This is the number of questions you answered correctly.
• Intuitive answers. We also count how many times out of 10 you chose the specific tempting wrong answer each question invites, separately from other wrong answers.
• Breakdown by trap type, showing how you did on number, wording and probability traps.
• A band and a percentile. Scores of 9–10 are labeled Highly reflective, 7–8 Reflective, 4–6 Balanced and 0–3 Intuitive. The percentile uses provisional norms until enough people have taken this version, so treat it as a rough guide.

We do not reveal the answers here, for the reason the familiarity research makes clear: knowing them in advance would make your score less meaningful.

Frequently asked questions

What is the answer to the bat-and-ball problem?

The ball costs 5 cents and the bat costs $1.05. The intuitive answer, 10 cents, fails because it would make the bat only 90 cents more expensive than the ball, and the total $1.20.

Does a low CRT score mean I am not intelligent?

No. CRT scores are related to cognitive ability, but the test was designed to capture something different: whether you pause to check a quick answer (Frederick, 2005; Toplak et al., 2011). Plenty of very capable people miss these items. A low score on a short, ten-item test is a prompt for self-reflection, not a verdict on your mind.

Can I improve my score?

Your score will almost certainly go up if you have seen similar puzzles before, which is precisely why the original CRT has become less useful (Stieger & Reips, 2016). Whether practicing makes you more reflective in everyday decisions is a different, much less certain question. The habits above are a reasonable place to start.

Is fast, intuitive thinking bad?

Not at all. Quick judgments are how people handle most of daily life, and they are often right. The CRT uses questions specifically built so that the first answer is wrong. The useful skill is not distrusting intuition in general, but recognizing when a question deserves a second look.