← All posts

October 29, 2026

847 Green Tests and Corrupted Answers

I work on the exam platform of a Brazilian financial-education company. Students use it to rehearse for a national financial-certification exam: timed mock exams with multiple-choice questions, case studies with child questions, and dialog questions, where each answer moves you to a different node of a conversation graph. The score a student sees there is how they judge whether they are ready for the real thing, so a wrong score is not a cosmetic bug.

Since July, a large part of my work has come down to one plain question: when a student answers something, do we keep what they did? The honest answer turned out to be "mostly not", and for a while the test suite said otherwise.

Four Ways to Save Progress

Exam progress (which questions you drew, what you answered, where you stopped) had been implemented four different ways across our hosts: the web platform, a legacy PHP monolith (CodeIgniter 3); the course platform (LMS); and our apps. Resuming on another device was flaky. One bug had reached production: in an unfinished dialog, if you resumed, left the question and came back, the screen landed on the dialog's first step, already answered, instead of the first unanswered one. Answering there truncated the conversation history and erased every answer after it.

The front-end logic lived in a single JavaScript file of about 4,200 lines, with not one test. Real-browser end-to-end tests were off the table too, because the local environment pointed at the production database. So the first job was not fixing anything. It was building a way to see.

Characterize Before Extracting

The rule I set comes from Michael Feathers' work on legacy code: never move critical code before pinning down what it does. For each slice of the file, I wrote Vitest tests under jsdom that captured the current behavior, bugs included, treating each function as a black box with stubbed localStorage, window and jQuery. Only then did the logic move into pure modules, with those characterization tests proving nothing changed. Only then did bugs get fixed.

Unit tests were not enough, and the dialog bug proved it. A first fix corrected the rendering and made the symptom disappear in the current session. It failed on reload, because the answer never reached storage, so I reverted it. The bug needed a test that crossed the whole cycle: click, change handler, storage, reload, restore, render. I built a harness that boots the real bundle in jsdom against a mock server, drives it with real events, and implements reload() by discarding the DOM and the JavaScript scope and booting again from whatever storage holds. One invariant governs every scenario: any test that answers something must verify persistence through a reload. The real fix was in navigation: entering an unfinished dialog from outside now lands on its frontier, the first unanswered step. Along the way I found that the resume-token script had never run in one of our two exam bundles.

That token is a compact binary format, saved through a session-authenticated endpoint, throttled to once every 20 seconds and flushed with sendBeacon when the page is hidden:

header   66 bits   version 8 | exam id 24 | timer 18 | position 8 | entry count 8
entry    25 bits   question id 20 | dialog step 1 | corrected 1 | answer 3
trailer  4 bytes   first 4 bytes of SHA-256(payload)

Its test is a round trip: encode in JavaScript, decode in PHP, compare.

847 Green Tests

Later the exam interface moved into a cross-platform SDK in TypeScript and React that our hosts embed, and progress went server-side: one token owned by the SDK, with each host providing only an adapter. Resume across devices was still unreliable, so I wrote the requirement down as plainly as I could: resuming must bring back the identical exam, with the same questions and the same alternatives in the same positions, and in dialogs the path walked and every child answer. Debugging against that turned up five root causes:

  1. The token stored the displayed position of an answer where the contract expected its canonical rank.
  2. Child questions were never written to the token.
  3. A dialog's conversation state was not rebuilt on resume.
  4. Nothing froze the displayed order, and the backend reshuffled alternatives on every fetch.
  5. Saves were throttled to 20 seconds; the requirement was a save per interaction.

The first one is the one I keep thinking about. The contract says the stored answer is the rank of the chosen alternative among the question's alternatives sorted by id. The SDK wrote the index of the alternative on screen, into a variable named canonicalIndex. The name said the right thing; the value was something else. Combine that with cause four: on resume the server shuffled the alternatives again, the decoder mapped the stored number back to an id, and the student got back an answer they had not chosen, with its correction state corrupted along with it.

Meanwhile, 847 unit tests were green. They checked that the restored answer sat in the same position as the saved one, and the mock server returned the same questions in the same order every time. In the test world, position and identity were the same thing. In production they were not.

// What the suite checked: same slot after resume.
expect(resumed.selectedIndex).toBe(saved.selectedIndex);

// What the student cares about: same alternative after resume,
// against a mock that reshuffles on every fetch, as production does.
expect(resumed.selectedAlternativeId).toBe(saved.selectedAlternativeId);

Token v2 fixes the format rather than the symptom. Each entry grows to 33 bits with an 8-bit permutation: four 2-bit slots, where slot i holds the canonical rank of the alternative shown at position i. That freezes the order the student actually saw. The decoder checks that the slots form a valid permutation and falls back to canonical order, per question, when they do not. The token now carries child questions, the dialog state is rebuilt from it, and every interaction triggers an immediate, serialized save. A shared fixture runs in both directions, TypeScript encoding for PHP to decode and PHP encoding for TypeScript, and the test catalog now has a rule that says it outright: the stored answer is a canonical rank, never a displayed position.

An Incident in Grading

In October, while the SDK was still rolled out to a small slice of students, a different bug landed: for dialog questions, it submitted only the first node. The backend graded what it received, and 179 attempts from 32 students had their dialogs counted as wrong. The SDK was fixed. We scoped the damage with a regular expression over the JSON column where answers are stored, a detail that mattered later.

The incident itself was the easy part. What it revealed about the system was not.

What We Actually Store

I went back through the code and ran read-only queries against production. The findings were structural:

Scale makes this matter: about 2,200 attempts a day, in an attempts table of 14.4 million rows and 10.7 GB. And token v2 was already too small: 13 active questions have more than four alternatives, which the permutation cannot represent.

The Attempt as Fact, the Grade as Interpretation

The design that came out of this separates two things the system had always merged. What the student did, and what they were shown, are facts, and facts are immutable. The verdict and the score are interpretations: derived, tied to the rule that produced them, and recomputable.

A multiple-choice item in the record looks like this, with the question, the content version served, the ids as displayed, and the id picked. There is no position anywhere:

{ "kind": "mcq", "q": 4711, "v": 902,
  "order": [5103, 5101, 5104, 5102], "pick": 5101 }

I weighed a bigger binary token, an event log and a row per answer. JSON won on identity, on forensics (the incident above was scoped with a query, not a decoder) and on schema evolution. Size did not decide it: roughly 3 GB a year. The rollout is phased and additive. The first phase only creates new tables, applied by hand from a runbook, and the server becomes the grading authority only after a shadow period with zero unexplained divergence from the client's grades.

Re-grading the Past

Surveying stored answers for the design turned up an older bug. Dialog steps were stored in two dialects: the web stored the rank of the chosen letter as a string, "1" to "4", while our native Flutter app stored a JSON number per step, where 1 means the student picked the right alternative at that node. The PHP grader read the app's 1 as the letter A. In a sample of the 20,000 most recent attempts, the app had recorded about 7,800 dialogs as correct, and the grader approved 124.

The fix for new attempts is small. The past is harder. I wrote a generator that only reads production, in read-only transactions, and emits SQL instead of running it. Every statement is guarded and idempotent:

UPDATE attempts SET score = 79.59
WHERE id = 1001 AND score = 75.51;

If the row has already changed, it is not touched, and running the file a second time changes nothing. Scope is decided row by row from the data: an attempt is eligible only if its stored score is exactly what the old code computes and its denominator is the canonical question count, so the only difference is how the flag is read. Attempts whose dialog was edited on or after the attempt day stay out, because today's graph may not be the one the student saw. That is the unversioned answer key again, and it keeps 1,099 attempts out of the files. Another 15,731 were graded under older counting rules; recomputing them would move grades for reasons unrelated to this bug, and lower 4,347 of them, so they stay as they are.

The result: 31,211 attempts for 3,846 students would be corrected, by +6.68 percentage points on average, with zero decreases. The generator mirrors the grading rule, so I checked the mirror against the real production PHP grader, both old and fixed, with 24,000 random cases on PHP 8.3 and 8.5: 0 divergences. 320 real rows replayed through the real PHP models matched. On a scratch MySQL, the first run changed 31,211 rows and the second changed 0. A scan of every output file for personal data found nothing.

None of this has run in production yet. The grader fix is not merged, the SQL has not been executed, and the attempt model's tables do not exist. The re-grade writes its audit rows into the new append-only grades table, so the two ship together. They are designed and validated, and that is all I will claim for them today.

What the Tests Should Have Said

Most of this was implemented by AI coding agents working from my specs, under my review. That moved my effort from writing code to deciding what "correct" means and proving it, and two lessons came out of it.

First, a test should assert what the user would notice, not the representation the code happens to use. 847 tests proved that a number survived a round trip. None proved that the student got their own answer back, and the mock guaranteed the difference could never show. Asserting identity, with fakes that behave like production, would have caught it on the first day.

Second, a grade is derived data. If you store only the derived value, every bug in the derivation becomes permanent, and every edit to the answer key quietly rewrites history. Store the facts, version what they were judged against, and a grade becomes something you can recompute, audit and correct.