Where Marlo was wrong
still adding to itNot a postmortem. A design constraint. A project whose pitch is "we tell you when we are wrong" needs somewhere to write that down before it has anything to be wrong about.
The question each entry answers is not how bad it was. It is what reported success at the time. A loud crash is a changelog entry. A quiet wrong answer belongs here.
"If any engine reports a failure, Marlo may not report clean" was tested over every combination of engines and outcomes, about rules. Within a rule, the findings came from the routed engine alone, so a field a peer failed and the routed engine passed was dropped. On the demo checkout page that was the email input labelled only by a placeholder, which is the exact example the README's sample output used to show the invariant working.
It surfaced when the browser path first ran and the finding count fell while every rule-level assertion passed. Failures are now matched across engines by the element each one names, identical elements are counted rather than merged, and an element a peer failed is reported under that peer and listed first. The same comparison exposed a false positive in Marlo's own image rule, which the official test cases never exercise.
On real pages, a rule Marlo declined as unsupported was dropped from its grading, while the other three engines answered every page. Marlo was graded on a smaller, easier set, and its published recall there was higher than the error analysis said, while the error analysis claimed the two agreed.
A declined or crashed cell now counts as no detection for every engine. Marlo's real-page recall fell when that changed, and the two documents now reproduce each other with nothing filtered.
The sample output led with an email field labelled only by a placeholder, which Marlo failed and two peers passed, as the invariant at work. HTML-AAM accepts a placeholder as a name, so the peers were right. The rule was corrected, the demo field now has no name at all, and the recorded scan on the home page was regenerated rather than edited.
The README called pnpm calibrate:check the CI gate that fails when
a published number moves. No workflow job invoked it. The check the whole claim
rests on was run by hand, when somebody remembered.
It is a required job now, on both corpora, and the real-page figures fail on any movement in either direction.
The Node renderer shipped, in a first draft, with an option to disable script execution and a confident comment about not evaluating untrusted markup in-process. The option did not work. happy-dom runs inline scripts under every combination of its own switches, and its DOM parser runs them too, which the HTML specification says it must not.
An option named "do not run scripts" that runs scripts is worse than no option, because somebody would rely on it. It was removed. There is now a test asserting the real behaviour, so if a working switch ever appears the test fails and a human revisits the documentation, instead of the documentation going quietly stale.
marlo scan --json into a file truncated at exactly 65526 bytes.
Correct exit code. No warning. No error. What you got was a file that started
like a report and would not parse.
process.exit() immediately after writing. To a terminal that write
is synchronous; to a pipe it is not, and the process died before it drained. It
sets the exit code and returns now, and a test pipes the JSON through a
subprocess and parses it.
This is the failure mode the whole project is about, happening inside the project. Every line of logic was correct. The artifact was incomplete and looked whole.
Two defects, both on the method page in full. Every engine failed one rule identically because the harness misrepresented two test documents. Then the fix for that skipped 444 of 524 cases and left every number resting on a sixth of the corpus.
Neither was caught by a test. Both were caught by staring at a result that was suspiciously uniform. Which is the argument for publishing numbers at all: a number nobody reads is a number nobody checks.
The HTML CodeSniffer adapter treated a DOM node as a string. Thirteen rules threw at once, the first time the CLI met a real file. A crash is never a pass, so it was loud rather than silent, and that part is the design working.
Thirteen rules still went unmeasured, because no unit test exercised that adapter against markup with the shape that triggers it. The end to end suite now asserts zero crashes on the demo pages.
The first audit found serious violations on every page: 48 tap targets under 24 CSS pixels, a light mode contrast failure at 2.18:1, and four scrollable regions no keyboard could reach. All 26 site tests passed throughout, because contrast needs layout and they have none.
The audit had a defect of its own. It loaded pages over file://, so
the stylesheet never applied, and axe was measuring the browser's default link
colour on white. It serves over HTTP now and refuses to believe any result until
it has confirmed the CSS arrived.
Not defects. Things Marlo cannot do, written down so the gap is a decision.
Where a name would depend on CSS generated content or the box tree, Marlo answers "cannot tell" and returns a confidence rather than pretending. That caution is why its recall on the naming rules sits below its peers', and the table shows exactly what it costs.
With layout, Marlo walks the paint order behind each piece of text, compositing transparency and ancestor opacity. Over an image, a gradient, a shadow, or an element painted on top, it answers "cannot tell" and names the element responsible. Without layout, both contrast rules report as not evaluated.
Two language models rated every real page against every rule, and a third settled their disagreements. Every figure resting on that says so, and the raters' own accuracy is measured on the official cases. A human rating of a sample would test it, and the protocol for one is ready.
IDs are scoped to the tree they are written in, so folding shadow content into the page would invent duplicates. Marlo leaves it out, and on a page with shadow roots every element rule says "cannot tell" rather than calling the page clean.
Findings carry a DOM selector and say plainly that the source location arrives with the repair layer. A fabricated byte offset would look more finished and be worth less than nothing.
The demo page has a duplicated attribute on one input. Every HTML parser drops
the second one before any DOM exists, so a DOM-based check is blind to it by
construction. It is listed as undetected in
apps/demo/expected.json rather than removed from the page, because
an undetected defect nobody wrote down looks exactly like a defect that is not
there.
Was Marlo wrong about your code?
That is the most useful message this project can get, and reporting it is deliberately the shortest path in the repository. You do not need to know which ACT rule fired or which engine reported it. Every confirmed false positive becomes a fixture, so the same mistake fails the build afterwards.