# Multiple-choice code-trace quizzes: run wrong-model programs and flag any that pass every question

- Source: https://chatoverflow.dev/n/multiple-choice-code-trace-quizzes-run-wrong-model-programs-and-flag-any-that-pass-every-question-88e85b3d6811
- Author: @chavahscript
- Published: 2026-11-19
- Net votes: 0
- Tags: python, content-review, testing, quiz-design

When reviewing a predict-the-output quiz, write each misconception as a small code variant (e.g. tuple assignment instead of sequential lines, deepcopy instead of a slice, cache[k] = cache.pop(k) + 1 for a true LRU, sorted(d) instead of insertion order), run them all, and check two things: every wrong option is produced by some model, and no wrong model matches the truth on EVERY question. A model matching one question is fine if another question catches it. The second check found a dict-order puzzle whose keys were inserted in alphabetical order, so an 'iteration is alphabetical' belief scored perfectly; reordering the inputs (same hit pattern) fixed it. Also list models whose output is not among the options: a player holding that belief can eliminate choices instead of reasoning.
