How the measurement works, and where it went wrong first
Four engines, 1134 official test cases, one code path. Nobody gets a longer timeout, a retry, or a special case. Marlo's engine goes through the same door as the others.
Vendored, hashed, and never fetched at runtime
All 1134 test cases are committed with a SHA-256 each. Two reasons, and convenience is not one of them. CI has to be green with no network, which is also the only thing that makes the offline claim true rather than asserted. And a number that moved because somebody upstream edited a file is not a measurement, it is a reading.
pnpm corpus:verify fails on a changed byte, on a file the manifest
does not list, and on any drift in the documented totals. If a test case could be
edited, an inconvenient result could be fixed by changing the question rather
than the answer.
The first run was wrong, and both mistakes taught more than the code did
Every engine failed one rule in exactly the same way
All four, Marlo included, scored 0.67 precision on rule b5c3f8 and
graded as incorrect. Four independent engines failing identically is not four
coincidences.
That rule's two inapplicable examples are an SVG document and a MathML document. Writing either into an HTML document produces an HTML page containing that element, so all four engines correctly reported that the page's html element had no language. The finding was right about the document it was given. The document was wrong.
The harness now refuses to grade a document the renderer cannot represent, and those cases get their own column so a reader can see how many test cases each number rests on. A measurement that could not be taken honestly is reported as not taken. Never as a result.
Then the fix turned out to be worse than the bug
The first correction rejected any document whose root element was not
html. That skipped 444 of 524 cases and left every published number
resting on a sixth of the corpus.
Nobody questioned it, because the numbers improved. Marlo's precision went to 1.00 on several rules. An over-strict filter looks like caution and produces figures that mean nothing.
Only a foreign-namespace root is genuinely unrepresentable. An HTML fragment is fine, because a browser wraps that in html and body too. Sample size went from 80 back to 513.
The rule written down afterwards: a change that makes your own numbers better deserves more scrutiny than one that makes them worse.
Layout is where honesty gets expensive
A renderer declares what it has: dom, script,
layout, paint. A rule declares what it needs. A rule
whose needs are unmet reports unsupported, and unsupported is never a
pass anywhere in this codebase.
That one rule is what lets the default path run with no browser at all without
quietly claiming coverage it does not have. It is enforced structurally rather
than by review: the suite runs every rule twice, once with resolved styles and
once without, and fails if a rule that did not declare layout
changes its verdict.
axe-core arrived at the same conclusion independently, which was a pleasant surprise. Run over all 19 test cases for the minimum-contrast rule under the same Node DOM, it answered "cannot tell" every time and "failed" never, because the colours cannot be resolved without layout. Two engines declining for the same reason is a better argument than either one alone.