We audited our own question bank. Here is what was wrong with it.

6 min read · July 30, 2026

We stripped the answer keys off our own question bank and had every question solved again from scratch. Here is what came back broken, and what a bad practice question looks like from the inside.

Every practice question you have ever done was written by someone who could have got it wrong. Not the arithmetic, which is easy to check, but the harder parts: whether exactly one answer is defensible, whether the wrong answers are mistakes a real student would make, and whether the question can be answered at all from the text in front of you.

We write practice questions for the SAT and ACT, so we decided to find out how often we were getting those parts wrong. We stripped the answer keys off our own bank and had every question solved again from scratch by reviewers who could not see the intended answer. Anything they disagreed with got blocked until a person settled it. Then we ran a set of mechanical checks looking for specific defects.

We found real problems. Here they are, because knowing what breaks is the most useful thing you can learn about practice material.

The answer was often the longest choice

This is the one that surprised us most. In a batch of 68 newly written reading questions, the correct answer was the single longest of the four choices in 26 of them, which is 38%. If length told you nothing, the longest choice would be right about a quarter of the time. Ties, where two choices are the same length, are not counted either way.

That gap is not an accident of one writer. It shows up because whoever writes a question knows which answer is right, and then explains it a little more carefully than the three they are making up. The result is a question you can beat by measuring instead of reading.

One batch is a small sample, so treat 38% as a signal that something was off in how we wrote, not as a precise rate. It matters more than it sounds. A student who notices the pattern gets a small edge on that practice set and learns a habit that does nothing on test day, because the real exam is written and reviewed specifically to avoid it. Practice that rewards the wrong instinct is worse than no practice.

We now check every new batch against the 25% baseline and rebalance the ones that drift.

Some questions had two defensible answers

The most common real defect is not a wrong key. It is a question where two choices can both be argued, so whichever one is marked correct, a careful student gets told they are wrong when they are not.

Transitions are the worst offenders. If a sentence needs a contrast word and two of the four choices are close contrast words, the question often has two answers no matter what the key says. Not always, because contrast words differ in shade, but often enough that it is the first thing we check. The same thing happens on reading questions where one choice describes the right idea and another describes the same idea slightly differently.

Automated checks catch some of these. The pattern where two choices express the same logical relationship is mechanical, so we now reject it outright. The rest only surface when someone tries to answer the question cold and reports that two options work.

Some math keys answered a different question

A math question can be internally perfect and still wrong. The setup is right, the arithmetic is right, and the keyed value is a real number you get partway through the problem. It just is not the number the question asked for.

If the question asks how old Tom is and the key is Maria's age, every automatic check passes, because that value genuinely comes out of the equations. Only re-solving the problem from the question text catches it. This is why we re-solve rather than only verify.

Some reading answers were not in the passage

A reading question is broken if the evidence for its answer is not in the passage printed on the screen. This sounds too obvious to be a real failure, and it is the single most common way a fluent, professional-looking reading question turns out to be unanswerable. The question reads well. The answer is reasonable. It is just not supported by the text you were given.

We check this mechanically now, using overlap between the correct answer and the passage as a rough proxy. It is only a proxy, because a good answer can paraphrase or infer rather than repeat the words. But an answer with almost nothing in common with its own passage is nearly always broken, and that is the case worth catching automatically.

And a bug on our side was hiding text from students

The worst thing we found was not a question defect at all. Our passages are displayed in tappable pieces so you can mark the sentence you are using as evidence, and the code that split them into pieces was dropping text. A decimal was enough to trigger it. A passage ending "differences of 1.5 degrees or more" would lose the words before it and render as a fragment.

It affected 366 of our 3,939 reading passages, which is 9.3%, and in the worst case 220 characters, a full sentence, never reached the screen. That is every reading passage we serve, not a sample. Where the missing sentence carried the evidence, the question could not be answered correctly by anyone.

Nothing caught that. Types were fine, the page rendered, every test passed. It was only visible by comparing what a student saw against what we had stored, which is now a check that runs automatically.

What to take from this

You cannot audit your practice material the way we can. But you can use the failure modes as a checklist when a question feels wrong:

• If you can argue for two answers and the text does not rule one out, the question is probably broken.
• If the correct answer needs a fact the passage never states, the key is probably wrong.
• If a math key is a number from the problem but not the one asked for, you solved it correctly and the key did not.
• If the correct answer is reliably the longest one, the set was not checked for it.

When a key contradicts reasoning you can defend from the text, check it rather than assume you were wrong. The alternative, quietly learning that your correct thinking was wrong, does far more damage over a few hundred questions than any single bad answer.

We publish what our checks find because a defect rate you cannot see is one you have to take on trust, and we would rather you did not have to.