A 500-Rule Catalog That Found an Auth Bypass
Every system I've worked on has rules nobody wrote down. Which requests skip authentication. What happens to an unfinished exam when the timer runs out. Whether a field on the profile form is something a user may change or something only an admin may touch. The rules exist, because the code enforces them every day, but they live as side effects scattered across files, and the only way to learn one is to read the code or break it.
Over the past two months, at a Brazilian financial-education company, I turned that implicit behavior into catalogs: numbered, reviewable rules, each one mapped to a test that proves it. I built three, for our exam SDK, for the rewrite of a classic financial calculator emulator, and for the course platform (LMS). The last one, 500 rules long, surfaced an authentication bypass in production. This post is about the method and what it found.
What Counts as a Rule
A rule is one verifiable assertion with exact values. Not "the checkout looks right", but the exact string, the exact status code, the exact number. Every rule has three parts: an ID, the assertion, and the authority it comes from. Here's the shape (the rows are illustrative):
| ID | Rule | Reference |
|-----------------|-------------------------------------------------------------|----------------------------------|
| R-E2E-LOGIN-04 | A protected route called without a session returns 401 | characterization |
| | and never reaches the handler | |
| R-LOGIC-CART-07 | The order total is rounded half-up to 2 places after | web app file:line; |
| | discounts, never per item. Decision: the web app wins; | mobile app file:line |
| | the mobile app rounds per item | |
| R-E2E-REPORT-06 | [DEFECT] Current: the exported total includes cancelled | export file:line; |
| | orders. Desired: cancelled orders are excluded | owner decision, 2026-10-05 |The ID encodes the kind of rule and its area: R-, then the layer (LOGIC for pure logic, COMP for component behavior, UI for visual values like markup and compiled CSS, E2E for complete flows), then a short area token, then a number that is never reused. The layer decides the tool: logic rules become plain unit tests, component rules become Testing Library tests, end-to-end rules run in a real browser. And every test cites the full ID in its title. That one convention is what makes the rest possible: the catalog and the test suite can be cross-checked by a script, 1:1.
Authority Is a Decision, Not a Default
The most important column is the last one. Each rule cites where it comes from: a file and line in a reference implementation, a page in a primary source, a dated decision, or "characterization", meaning the system's current behavior is being written down for the first time. What a rule can never treat as the definition of correct is "whatever the code does today".
When there are several references, authority is assigned per axis, never to one implementation globally. One codebase can be right about branding and wrong about mobile behavior. I write that down as an explicit table before any rule is extracted. When sources still disagree and the table doesn't settle it, the rule decides anyway and records its reasoning inline, in a line that starts with Decision:. Extraction never stops to ask. I review the decisions in batches and veto the ones I disagree with. Every recorded decision is a point someone can argue with, and that's the point.
There's a corollary I care about: cataloging is not endorsing. The catalog records what the system does, and sometimes what it does is wrong. When an extractor finds behavior that looks broken, it flags the rule "to confirm with the owner". Only a human promotes it to a defect, and a defect rule has two halves: Current, what the code does, and Desired, what it should do.
Defect Tests That Go Red When the Bug Is Fixed
A defect rule still needs a test, and that test asserts the desired behavior, so today it fails. It's marked as an expected failure (test.fail() in Playwright, it.failing in Jest), which keeps CI green while the bug exists:
// Rule tagged [DEFECT]: assert the DESIRED behavior, expect it to fail today.
test.fail('R-E2E-REPORT-06 — exported total excludes cancelled orders', async ({ page }) => {
await page.goto(reportUrl);
await expect(page.getByTestId('export-total')).toHaveText('R$ 900,00');
});The interesting moment is the fix. When someone corrects the code, the test starts passing, and an expected failure that passes is reported as a failure. The suite goes red because the bug is gone. Whoever made the fix has to turn it into a normal test and remove the defect tag from the rule, in the same pull request. No fix lands without updating the catalog and the test, and no catalogued defect gets quietly forgotten. The list of defect rules is the fix backlog, each item with its test already written and waiting.
A Gate That Fails in Both Directions
The coverage gate runs inside the normal test command and checks the catalog against the test titles both ways. A rule with no test fails. A test citing an ID that doesn't exist also fails, which catches typos, renumbered rules, and tests that outlived the behavior they described.
const defined = idsDefinedIn(catalogShards); // IDs that open a table row
const cited = idsCitedIn(testTitles); // IDs named in test titles
const phantom = [...cited].filter(id => !defined.has(id));
const uncovered = [...defined].filter(id => !cited.has(id) && isActiveArea(id));
if (phantom.length) fail('tests cite rules that do not exist', phantom);
if (uncovered.length) fail('rules without a test', uncovered);It starts lenient. A catalog comes before its tests by design, so areas are either active or pending: active areas require a test for every rule, pending ones are exempt until someone writes their tests. Phantom IDs fail in any mode. When extraction is over, the gate flips to strict. There's also a small tooling test that runs the extractor against the real catalog files and asserts the result isn't empty, because a regex that silently stops matching after a format change would turn the gate into a rubber stamp.
Shards, So Agents Can Work in Parallel
Most of the extraction is done by AI agents, several at a time. That only works if they never write to the same file. So the catalog is sharded by area: one file per area, one agent per area, and that agent is the exclusive owner of its shard while it works. I learned this the usual way, from a collision. A single monolithic catalog is only safe when writes are strictly sequential. Approval is incremental too: after an area is extracted I review it, and no test for that area gets written before I approve its rules.
Three Catalogs
The exam SDK. Our exam SDK replaced exam logic that had been duplicated across five codebases, so its catalog has five reference implementations, each authoritative on a different axis: the legacy web app on branding and production behavior, the Flutter app on mobile UX, the course platform on visual hierarchy and animation. It started at 104 rules and grew to 373 across 20 areas, and the gate fails the test command on any gap.
The calculator emulator. The existing app was about 4,100 lines of untested JavaScript. The function that solves for the number of periods was a stub that returned 0, and several functions printed on the keypad had no code behind them. The code could not be the authority, so the catalog was built from the official 220-page manual. Even that needed validating: the table of contents was laid out against slightly different pagination than the final PDF, so printed and physical page numbers drift apart, from 0 at the start to +7 by the appendices. I kept an offset table and cited physical pages, each one confirmed by opening it.
The result was 429 rules (342 logic, 29 component, 52 UI, 6 end-to-end) in 20 areas, with 76 recorded decisions. It found errata in the manual itself: an error condition printed without its minus sign while the four sibling keys in the same table carry it, and a worked example whose key sequence contradicts a definition two lines above it on the same page. It also found a real bug in the existing app. The manual shows the count of amortized payments accumulating, 12 after the first call and 24 after the second; the app subtracted instead. The new core is being built test-first against the catalog, with 199 rule IDs already cited in tests, and the legacy code only goes away after full coverage and a check in the browser.
The course platform. The LMS catalog is the largest: 20 shards, 500 rules, 52 of them tagged as defects. On October 26 the gate counted 332 of the 500 rules covered, backed by 50 Playwright specs, 47 component rule tests and 741 Jest tests. The end-to-end layer runs the whole chain for real: a browser, the React front end, the NestJS API, our SSO, the identity provider behind it, and the database.
What the LMS Catalog Surfaced
Writing rules forces exact answers to questions nobody asks out loud. "Which requests skip authentication?" has to become a list of paths and methods. For the LMS, the honest answer wasn't a list. Four findings stand out, and all four are fixed in production:
- Authentication bypass. The global auth guard decided whether a request was public by substring matching on the URL, and a query parameter could make any request look public. A protected route reached that way skipped authentication entirely. Public access is now granted only on an exact path and method.
- Password hash in responses. The edit-user handlers returned the full user row, password hash included. Both now strip it.
- Insecure direct object reference (IDOR). An endpoint returned any user's access-group row, including the password hash and national ID number, and the bypass above meant it didn't even need a login. It's now admin-only, and the sensitive fields are stripped.
- Mass assignment. The profile-edit endpoint spread the request body straight into the update, so a user could write any column on their own account. It now accepts a whitelist of profile fields; everything else stays on the admin-only route.
None of these looked dramatic in the code. They were small conveniences, a substring check instead of an equality, a spread instead of a pick, that only showed their real shape when someone had to write down the rule they implemented. The first and the third also compounded each other, which is the kind of thing you only see when the rules for both areas sit side by side.
The security fixes are on main. The catalog itself is still in progress on a branch: more areas need their tests before the gate goes strict. The 52 defect rules have their tests too, pinning known gaps until they're fixed.
Why This Works With AI Agents
Part of it is volume. Five hundred rules with exact values and citations is not something I would write by hand in a month alongside everything else. Agents extract in parallel and write tests in parallel, and the method is shaped around that: sharded files so they don't collide, IDs so their output can be checked by a script, a gate so "done" is a number instead of a feeling.
What the agents don't own is judgment. I own the authority table, the area list, the approvals, the recorded decisions, and the call on what counts as a defect. That split is the part I'd keep even without agents. The code is a witness, not a judge, and someone has to be accountable for what "correct" means.
After the third catalog I generalized the method into a reusable skill: the ID grammar, the authority table, sharding, recorded decisions, the lenient-then-strict gate, red-when-fixed tests. I validated it RED-GREEN on another repository, the same way I'd validate code: first the run that gets it wrong without the skill, then the run that gets it right with it. The next system starts from the contract instead of a blank page.
The auth bypass was found by a catalog demanding an exact answer to a simple question. That's what I keep coming back to. A lot of bugs live in the gap between what a system does and what anyone has said it should do, and writing the rules down closes that gap one row at a time.