When an AI gets a question wrong, it almost never does the sums badly. It does
the wrong sums, perfectly, and explains them in a calm and orderly voice
all the way to a confident wrong answer. Learning to catch that is an essential
AI-ready skill.
Practise it on work your robot student has just handed in.
Your robot is called Pip.
You have been teaching it, so give it a name you like.
Stays in this browser — nobody else sees it.
Some solutions are perfectly good work; others have exactly one step where the
reasoning breaks. Every calculation Pip did is
correct. The question is whether each calculation was the right thing to do at that
point in the problem.
Practice
Pip's solutions to
20 GSM8K problems
Read each problem and Pip's
solution. Decide whether it holds up, and if it doesn't, find where it goes wrong.
01
Kimberley, Ela, and Houston all are sent to collect firewood by their grammy. Kimberley collects ten pounds of firewood, and Houston collects 12 pounds of firewood. If the three of them managed to collect a total of 35 pounds of firewood, how many pounds were collected by Ela?
Pip's solution
Pip's answer: 3
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
✌️ The Double Dipper — Adds the same amount in twice and never notices.
Kimberley and Houston collected 10+12+10 = 32 pounds of firewood.
should be: Kimberley and Houston collected 10+12 = 22 pounds of firewood.
The amount 10 goes into that total twice, so it is counted once more than it exists. Every number is one the problem really mentions and the addition is
performed correctly, which is why re-checking the sum never finds it.
The real answer is 13.
02
Jerry has three times as many stickers as George. George has 6 fewer stickers than his brother Fred. If Fred has 18 stickers, how many stickers does Jerry have?
Pip's solution
Pip's answer: 36
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
36 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
03
Hannah is making banana bread. She needs to use 3 cups of flour for every cup of banana mush. It takes 4 bananas to make one cup of mush. If Hannah uses 20 bananas, how many cups of flour should she use?
Pip's solution
Pip's answer: 15
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
15 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
04
In seven years, Talia will be 20 years old. Talia's mom is currently three times as old as Talia is today. In three years, Talia's father will be the same age as Talia's mom is today. Currently, how many years old is Talia's father?
Pip's solution
Pip's answer: 78
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
📥 The Hoarder — Asked how many are left, and adds them on.
In seven years, Talia will be 20 years old, and therefore Talia is currently 20+7=27 years old.
should be: In seven years, Talia will be 20 years old, and therefore Talia is currently 20-7=13 years old.
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
36, not 78.
The habit to watch for: it should have used
− subtract there.
05
A hot dog stand sells 10 hot dogs every hour, each one selling for $2. How many hours does the stand need to run to make $200 in sales?
Pip's solution
Pip's answer: 10
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
10 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
06
James decides to try and collect rare artifacts in the field. He spends 6 months researching before going on a 2-year-long expedition for his first find. The second artifact takes 3 times as long for the research and discovery. How long did it take him to find both?
Pip's solution
Pip's answer: 10
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
🕳️ The Leaper — Produces a number out of nowhere and carries on confidently.
Read that step again: one of the numbers in it was never worked out
anywhere above. It happens to be correct, and you may well have filled
it in without noticing — but the page never shows where it comes from.
The missing step: It takes 2.5*3=7.5 years for the secondthat is where the number comes from
The final answer is still 10, so calling this one
right is fair. The habit worth keeping: when a number turns up without
working, work it out yourself before you trust it. Here it checks out;
from an AI, it might not.
07
James finds 3 bills in his pocket. They are all 20s. If he already had $75 in his wallet how much money doe he have now?
Pip's solution
Pip's answer: 98
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
➕ The Adder — Hears "three times as many" and reaches for plus.
He found 3+20=$23
should be: He found 3*20=$60
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
135, not 98.
The habit to watch for: it should have used
× multiply there.
08
Barbara Blackburn can type 212 words per minute. Due to Carpal tunnel syndrome, Barbara cannot use her left hand for a while so her typing speed is now 40 words less per minute. If she is supposed to type a document with 3440 words, how many minutes will it take her to finish typing the document?
Pip's solution
Pip's answer: 20
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
20 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
09
Milly and Floretta are having a water balloon fight in their garden. They use 3 packs of their own water balloons and also use 2 packs of a neighbor’s water balloons. Each pack of water balloons contains 6 balloons. Milly and Floretta fill all of the balloons then split them evenly, but Milly takes an extra 7 balloons when Floretta isn't looking. How many water balloons is Floretta left with?
Pip's solution
Pip's answer: 15
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
🎯 The Early Finisher — Gets to a real number, decides that will do, and stops.
Every step here is correct, and so is every sum. The problem is what
is missing: it stopped at
15, which is a real quantity,
just not the one the question asked for.
The step it never took: After Milly steals some balloons, Floretta is left with 15 – 7 = 8 water balloons.so the answer is 8, not 15
You could not have found this by checking any line. The only way is to
go back to the question and ask what it actually wanted.
10
Stacy was 50 inches tall last year. If she grew 6 inches more than her brother who grew 1 inch last year, how tall is Stacy now?
Pip's solution
Pip's answer: 55
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
➖ The Taker — Asked to put two amounts together, and takes one away.
Stacy grew 6-1=5 inches.
should be: Stacy grew 6+1=7 inches.
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
57, not 55.
The habit to watch for: it should have used
+ add there.
11
There are 30 players on a football team. The coach has 8 liters of water. She pours 200 milliliters of water for every player so they can get hydrated. Unfortunately, there was a water spill and 250ml of water was spilled. How much water was left over?
Pip's solution
Pip's answer: 1750
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
1750 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
12
Alfonso earns $6 each day walking his aunt’s dog. He is saving to buy a mountain bike helmet for $340. Currently, he already has $40. If he walks his aunt's dog 5 days a week, in how many weeks should Alfonso work to buy his mountain bike?
Pip's solution
Pip's answer: 10
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
🕳️ The Leaper — Produces a number out of nowhere and carries on confidently.
Read that step again: one of the numbers in it was never worked out
anywhere above. It happens to be correct, and you may well have filled
it in without noticing — but the page never shows where it comes from.
The missing step: Alfonso has to work for the $340 - $40 = $300.that is where the number comes from
The final answer is still 10, so calling this one
right is fair. The habit worth keeping: when a number turns up without
working, work it out yourself before you trust it. Here it checks out;
from an AI, it might not.
13
Of the 24 students in class, one-third are in the after-school chess program, and half of those students will be absent from school on Friday for the regional chess tournament. How many of the students in this class are going to that tournament?
Pip's solution
Pip's answer: 36
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
✖️ The Piler — Told to share it between four, and multiplies by four.
There are 24*3=72 students in the program.
should be: There are 24/3=8 students in the program.
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
4, not 36.
The habit to watch for: it should have used
÷ divide there.
14
Jillian had 80 oranges which she had bought to give to her friends. She divided each orange into ten equal pieces for each friend to get four pieces. How many friends were there?
Pip's solution
Pip's answer: 2
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
➗ The Splitter — Hears "times as many" and divides instead.
If she divided each orange into 10 pieces, she got 80/10 = 8 pieces of orange.
should be: If she divided each orange into 10 pieces, she got 80*10 = 800 pieces of orange.
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
200, not 2.
The habit to watch for: it should have used
× multiply there.
15
In four years, Suzy will be twice Mary's age then. If Suzy is 20 now, how old is Mary?
Pip's solution
Pip's answer: 44
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
✖️ The Piler — Told to share it between four, and multiplies by four.
Then Mary will be 24*2=48.
should be: Then Mary will be 24/2=12.
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
8, not 44.
The habit to watch for: it should have used
÷ divide there.
16
The city’s bus system carries 1,200,000 people each day. How many people does the bus system carry for 13 weeks?
Pip's solution
Pip's answer: 109200000
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
109200000 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
17
Carla needs to drive her car to do a bunch of errands. She needs to drive 8 miles to the grocery store, 6 miles to pick up her kids from school, 12 miles to drop her kids off at soccer practice, and twice the distance from the school to soccer practice to get everyone home again. If Carla's car gets 25 miles per gallon, and gas costs $2.50, how much will Carla have to spend on gas for her trip?
Pip's solution
Pip's answer: 2
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
🎯 The Early Finisher — Gets to a real number, decides that will do, and stops.
Every step here is correct, and so is every sum. The problem is what
is missing: it stopped at
2, which is a real quantity,
just not the one the question asked for.
The step it never took: Then multiple the gallons of gas by the price per gallon to find the total cost: 2 gallons * $2.50 = $5so the answer is 5, not 2
You could not have found this by checking any line. The only way is to
go back to the question and ask what it actually wanted.
18
Mark spent 220 dollars to fix his car. It needed two parts that cost 20 dollars each, and labor cost .5 dollars a minute. How many hours did the mechanic work?
Pip's solution
Pip's answer: 7
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
➗ The Splitter — Hears "times as many" and divides instead.
The cost for parts is 20/2=$10
should be: The cost for parts is 20*2=$40
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
6, not 7.
The habit to watch for: it should have used
× multiply there.
19
Mitchell has 30 pencils. He has 6 more pencils than Antonio. How many pencils does Mitchell and Antonio have together?
Pip's solution
Pip's answer: 66
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
📥 The Hoarder — Asked how many are left, and adds them on.
Antonio has 30+6 = 36 pencils
should be: Antonio has 30-6 = 24 pencils
Everything after it was then worked out correctly from the wrong number,
which is why the solution reads so smoothly. The real answer is
54, not 66.
The habit to watch for: it should have used
− subtract there.
20
Felix is chopping down trees in his backyard. For every 13 trees he chops down he needs to get his axe resharpened. It cost him $5 to sharpen his axe. If he spends $35 on axe sharpening, at least how many trees has he chopped down?
Pip's solution
Pip's answer: 91
Would you sign off on this?
What kind of mistake is it?
Click the step where it goes wrong.
What should that step have done instead?
Nothing to fix — the working is sound and the answer
91 is right. Checking isn't only about
catching mistakes: being able to say "yes, this one is right" is half the job.
Problems and worked solutions from GSM8K, released by OpenAI
under the MIT licence; the broken step in each one is ours. Source:
openai/grade-school-math.
This is not really about AI
Telling a confident answer from a correct one is an old skill, and chatbots only
made it matter more often. The critical thinking toolkit is the same move applied
to headlines and claims instead of worked solutions.