This page existed before the product did

# Where Marlo was wrong

still adding to it

Not a postmortem. A design constraint. A project whose pitch is "we tell you when we are wrong" needs somewhere to write that down before it has anything to be wrong about.

The question each entry answers is not how bad it was. It is **what reported success at the time.** A loud crash is a changelog entry. A quiet wrong answer belongs here.

Reported success: an exhaustive test of the wrong property The invariant held for every rule and not for every element

"If any engine reports a failure, Marlo may not report clean" was tested over every combination of engines and outcomes, about rules. Within a rule, the findings came from the routed engine alone, so a field a peer failed and the routed engine passed was dropped. On the demo checkout page that was the email input labelled only by a placeholder, which is the exact example the README's sample output used to show the invariant working.

It surfaced when the browser path first ran and the finding count fell while every rule-level assertion passed. Failures are now matched across engines by the element each one names, identical elements are counted rather than merged, and an element a peer failed is reported under that peer and listed first. The same comparison exposed a false positive in Marlo's own image rule, which the official test cases never exercise.

Reported success: a consistency check that skipped the cells in question The real-page figures left out the cells Marlo declined

On real pages, a rule Marlo declined as unsupported was dropped from its grading, while the other three engines answered every page. Marlo was graded on a smaller, easier set, and its published recall there was higher than the error analysis said, while the error analysis claimed the two agreed.

A declined or crashed cell now counts as no detection for every engine. Marlo's real-page recall fell when that changed, and the two documents now reproduce each other with nothing filtered.

Reported success: golden files agreeing with each other The showcase disagreement was Marlo's own false positive

The sample output led with an email field labelled only by a placeholder, which Marlo failed and two peers passed, as the invariant at work. HTML-AAM accepts a placeholder as a name, so the peers were right. The rule was corrected, the demo field now has no name at all, and the recorded scan on the home page was regenerated rather than edited.

Reported success: every required check The calibration gate was described as CI, and nothing ran it

The README called `pnpm calibrate:check` the CI gate that fails when a published number moves. No workflow job invoked it. The check the whole claim rests on was run by hand, when somebody remembered.

It is a required job now, on both corpora, and the real-page figures fail on any movement in either direction.

Reported success: nothing did A security option that did nothing

The Node renderer shipped, in a first draft, with an option to disable script execution and a confident comment about not evaluating untrusted markup in-process. The option did not work. happy-dom runs inline scripts under every combination of its own switches, and its DOM parser runs them too, which the HTML specification says it must not.

An option named "do not run scripts" that runs scripts is worse than no option, because somebody would rely on it. It was removed. There is now a test asserting the real behaviour, so if a working switch ever appears the test fails and a human revisits the documentation, instead of the documentation going quietly stale.

Reported success: everything did A report file that looked complete and was not

`marlo scan --json` into a file truncated at exactly 65526 bytes. Correct exit code. No warning. No error. What you got was a file that started like a report and would not parse.

`process.exit()` immediately after writing. To a terminal that write is synchronous; to a pipe it is not, and the process died before it drained. It sets the exit code and returns now, and a test pipes the JSON through a subprocess and parses it.

This is the failure mode the whole project is about, happening inside the project. Every line of logic was correct. The artifact was incomplete and looked whole.

Reported success: the harness did The measurement was wrong before the code was

Two defects, both on the [method page](https://trymarlo.pages.dev/method/) in full. Every engine failed one rule identically because the harness misrepresented two test documents. Then the fix for that skipped 444 of 524 cases and left every number resting on a sixth of the corpus.

Neither was caught by a test. Both were caught by staring at a result that was suspiciously uniform. Which is the argument for publishing numbers at all: a number nobody reads is a number nobody checks.

Reported success: the architecture worked Thirteen rules crashed and it took a real page to notice

The HTML CodeSniffer adapter treated a DOM node as a string. Thirteen rules threw at once, the first time the CLI met a real file. A crash is never a pass, so it was loud rather than silent, and that part is the design working.

Thirteen rules still went unmeasured, because no unit test exercised that adapter against markup with the shape that triggers it. The end to end suite now asserts zero crashes on the demo pages.

Reported success: 26 passing tests This site failed Marlo's own bar

The first audit found serious violations on every page: 48 tap targets under 24 CSS pixels, a light mode contrast failure at 2.18:1, and four scrollable regions no keyboard could reach. All 26 site tests passed throughout, because contrast needs layout and they have none.

The audit had a defect of its own. It loaded pages over `file://`, so the stylesheet never applied, and axe was measuring the browser's default link colour on white. It serves over HTTP now and refuses to believe any result until it has confirmed the CSS arrived.

Standing limitations

## Not defects. Things Marlo cannot do, written down so the gap is a decision.

The accessible name computation is incomplete

Where a name would depend on CSS generated content or the box tree, Marlo answers "cannot tell" and returns a confidence rather than pretending. That caution is why its recall on the naming rules sits below its peers', and the table shows exactly what it costs.

Contrast is computed, and declines where one ratio would be a guess

With layout, Marlo walks the paint order behind each piece of text, compositing transparency and ancestor opacity. Over an image, a gradient, a shadow, or an element painted on top, it answers "cannot tell" and names the element responsible. Without layout, both contrast rules report as not evaluated.

Real-page ground truth comes from models, not people

Two language models rated every real page against every rule, and a third settled their disagreements. Every figure resting on that says so, and the raters' own accuracy is measured on the official cases. A human rating of a sample would test it, and the protocol for one is ready.

Shadow trees are not entered

IDs are scoped to the tree they are written in, so folding shadow content into the page would invent duplicates. Marlo leaves it out, and on a page with shadow roots every element rule says "cannot tell" rather than calling the page clean.

Source locations are not implemented

Findings carry a DOM selector and say plainly that the source location arrives with the repair layer. A fabricated byte offset would look more finished and be worth less than nothing.

One defect cannot be seen from a DOM at all

The demo page has a duplicated attribute on one input. Every HTML parser drops the second one before any DOM exists, so a DOM-based check is blind to it by construction. It is listed as undetected in `apps/demo/expected.json` rather than removed from the page, because an undetected defect nobody wrote down looks exactly like a defect that is not there.

## Was Marlo wrong about your code?

That is the most useful message this project can get, and reporting it is deliberately the shortest path in the repository. You do not need to know which ACT rule fired or which engine reported it. Every confirmed false positive becomes a fixture, so the same mistake fails the build afterwards.

[Report a false positive](https://github.com/KarthikSubramanian07/Marlo/issues/new?template=false-positive.yml) [The full HONESTY.md](https://github.com/KarthikSubramanian07/Marlo/blob/main/HONESTY.md)
