Regenerated in CI on every push

# The numbers, including the ones that are bad for us

One harness, one corpus, one code path, four engines. Marlo's engine is one of the four and gets no exemption, because a table where the author's own engine happened to win would be worth nothing.

Alfabest 1.5% false positives

axe-core2nd 1.9% false positives

Marlo3rd 4.4% false positives

HTML CodeSniffer4th 9.4% false positives

Per engine

## Marlo is third of four

Alfa and axe-core are both more precise and more sensitive. Which is why the router sends 18 rules to axe-core and only 22 to Marlo's own engine.

513 official test case outcomes, strict view, where cantTell counts as no detection.
| Engine | Rules | Precision | Recall | False positives | cantTell, miss / cautious |
| --- | --- | --- | --- | --- | --- |
| Alfa | 41 | 0.964 | 0.827 | 1.5% | 21 / 30 |
| axe-core | 39 | 0.952 | 0.768 | 1.9% | 24 / 23 |
| Marlo | 41 | 0.894 | 0.733 | 4.4% | 24 / 41 |
| HTML CodeSniffer | 12 | 0.684 | 0.366 | 9.4% | 0 / 0 |

!

**HTML CodeSniffer has the worst false positive rate of the four, 9.4%.** Its results are severities rather than outcomes, so its warnings and notices are read as "cannot tell", and the adapter reads silence on a rule it claims as a pass. That inference is the weakest step in the whole engines package, and the table is where its cost shows.

What the product reports

## Measured as a whole, and with the router held out

The router picks each rule's engine from the same grades the table publishes, so scoring the routed engine on those cases grades it on its own answers. Here every case is decided by the routing chosen without it, over the 41 rules `marlo scan` evaluates and 568 official cases. The ranges are 95% intervals.

Strict view: cantTell counts as no detection.
| Strategy | Precision | Recall | False positives |
| --- | --- | --- | --- |
| What marlo scan reports | 0.848 (0.790 to 0.892) | 0.853 (0.795 to 0.896) | 7.7% (5.4% to 10.8%) |
| Routed engine, leave one out | 0.939 (0.888 to 0.968) | 0.732 (0.664 to 0.789) | 2.4% (1.3% to 4.5%) |
| Routed engine, in sample | 1.000 (0.974 to 1.000) | 0.763 (0.698 to 0.818) | 0.0% (0.0% to 1.0%) |
| Majority of engines | 0.931 (0.878 to 0.962) | 0.716 (0.648 to 0.775) | 2.6% (1.4% to 4.8%) |
| Alfa alone | 0.960 (0.915 to 0.981) | 0.753 (0.687 to 0.809) | 1.6% (0.7% to 3.4%) |
| axe-core alone | 0.954 (0.904 to 0.979) | 0.658 (0.588 to 0.722) | 1.6% (0.7% to 3.4%) |
| Marlo alone | 0.894 (0.832 to 0.934) | 0.663 (0.593 to 0.727) | 4.0% (2.4% to 6.4%) |

The routed engine's perfect in-sample precision does not survive being held out, and holding one case out changes the router's choice on 16 of 719 cases across every routed rule. What the product reports is the union of its engines, because none may be silenced: the highest recall here, at the cost printed in the first row. Routing decides which engine's evidence a finding carries, not what is found.

The same corpus, inside Chromium

## Every engine, run inside a real browser page

The table above needs no browser, so anybody can reproduce it from a clean clone. The same harness also runs all four engines inside Chromium, which is the only way to evaluate the rules that need layout. These figures sit beside the headline rather than replacing it.

590 official test case outcomes, strict view, where cantTell counts as no detection.
| Engine | Rules | Precision | Recall | False positives | cantTell, miss / cautious |
| --- | --- | --- | --- | --- | --- |
| axe-core | 39 | 0.965 | 0.912 | 1.6% | 9 / 6 |
| Alfa | 41 | 0.950 | 0.863 | 2.1% | 21 / 26 |
| Marlo | 41 | 0.895 | 0.714 | 4.0% | 33 / 50 |
| HTML CodeSniffer | 12 | 0.622 | 0.394 | 13.0% | 0 / 0 |

Marlo's false positive rate in the browser is 4.0%, still third of four. On minimum contrast, which every static scan reports as not evaluated, Marlo computes the ratio from a paint-order walk: precision 1.000, recall 0.500. Every failing case it misses is one where it says it cannot tell rather than guess: text on an image or a gradient, or inside a shadow root it does not enter.

A second corpus

## Real pages, where the accuracy falls

the uncomfortable one. read this bit.

Every number above comes from hand-written test cases built to isolate one condition. 25 real pages from 12 kinds of site are vendored beside them and measured separately, never pooled.

Ground truth is **model-adjudicated, not human judgment**: two different language models rated each of the 1025 cells independently, blind to every engine, and a third settled the cells where they disagreed. Their agreement is a Cohen's kappa of 0.890, and how accurate they are is measured on the official cases, whose answers people wrote, in [RATER-VALIDITY.md](https://github.com/KarthikSubramanian07/Marlo/blob/main/corpus/real/adjudication/RATER-VALIDITY.md).

On those pages, what `marlo scan` reports has a precision of 0.696 and a recall of 0.591, against 0.848 and 0.853 on the official cases. The routing chosen on the official corpus finds far fewer violations there and is no more precise, so routing is not an accuracy mechanism. Every cell Marlo's own engine gets wrong there has a named root cause in [ERROR-ANALYSIS.md](https://github.com/KarthikSubramanian07/Marlo/blob/main/calibration/ERROR-ANALYSIS.md).

Even without ground truth the official corpus is plainly not what people scan. On 19 of the 25 pages, some engine fails a rule the routed engine passes.

Findings per page on the real pages. A count, not accuracy.
| Engine | Pages finished | Findings | Per page |
| --- | --- | --- | --- |
| Marlo | 25 | 82 | 3.3 |
| axe-core | 25 | 50 | 2 |
| Alfa | 25 | 2511 | 100.4 |
| HTML CodeSniffer | 20 | 15 | 0.8 |

A third corpus

## Held-out pages, from sites nothing was tuned on

Marlo's errors on the first 25 real pages were analysed cell by cell, so a rule corrected after reading that analysis would score better there whether or not the correction holds anywhere else. 25 more pages, from sites the first set does not use, were snapshotted, rated by the same raters and committed before any such correction was measured on them. They are the out-of-sample check.

The raters agreed on the held-out cells with a Cohen's kappa of 0.931. Every engine is graded the same way on both sets: a cell an engine declined or crashed on counts as no detection, and each engine is pooled over the rules it claims, so the rule counts differ. Ranges are 95% intervals.

Strict precision and recall on resolved cells, first set and held-out set.
| Engine | Precision, first set | Recall, first set | Precision, held out | Recall, held out |
| --- | --- | --- | --- | --- |
| Alfa | 0.737 (0.610 to 0.834) | 0.750 (0.623 to 0.845) | 0.862 (0.751 to 0.928) | 0.694 (0.580 to 0.789) |
| axe-core | 0.882 (0.657 to 0.967) | 0.231 (0.145 to 0.346) | 0.824 (0.665 to 0.917) | 0.333 (0.242 to 0.439) |
| HTML CodeSniffer | 0.571 (0.250 to 0.842) | 0.087 (0.034 to 0.203) | 0.381 (0.208 to 0.591) | 0.138 (0.072 to 0.249) |
| Marlo | 0.757 (0.599 to 0.866) | 0.301 (0.217 to 0.401) | 0.846 (0.725 to 0.920) | 0.393 (0.307 to 0.485) |

The finding that reshaped the project

## W3C's protocol will call an engine correct for answering "I don't know"

W3C defines how to grade an implementation against a rule's official test cases. Under that protocol `cantTell` is an allowed answer for _every_ example type.

So a tool that shrugs at all 1134 test cases is, officially, a correct implementation of all 91 rules that have them.

That is not a flaw in the protocol. It grades whether a tool _misleads_ you, and "I don't know" misleads nobody. It is just not the question a developer is asking, which is whether the violation actually gets found.

So the table publishes both views and computes the gap rather than leaving you to notice it. 8 entries currently grade as officially consistent while missing more than half the violations a real user would hit. **4 of them are ours.**

Officially a correct implementation. In practice, missing most of it.
| ACT rule | Engine | W3C verdict | Strict recall |
| --- | --- | --- | --- |
| [`36b590`](https://act-rules.github.io/rules/36b590) | Marlo | consistent | 0.000 |
| [`3ea0c8`](https://act-rules.github.io/rules/3ea0c8) | axe-core | consistent | 0.000 |
| [`4b1c6c`](https://act-rules.github.io/rules/4b1c6c) | Alfa | consistent | 0.000 |
| [`b20e66`](https://act-rules.github.io/rules/b20e66) | Alfa | consistent | 0.000 |
| [`cf77f2`](https://act-rules.github.io/rules/cf77f2) | Marlo | consistent | 0.000 |
| [`e88epe`](https://act-rules.github.io/rules/e88epe) | Marlo | consistent | 0.000 |
| [`fd3a94`](https://act-rules.github.io/rules/fd3a94) | Alfa | consistent | 0.000 |
| [`ye5d6e`](https://act-rules.github.io/rules/ye5d6e) | Marlo | consistent | 0.333 |

Auto-fix

## What Marlo may touch without asking

A rule qualifies only when the engine reporting it clears 0.950 strict precision over at least 6 official test cases.

The gate is precision, not recall, and the asymmetry is deliberate. A missed violation is a gap you already had. A wrong fix is a change to your code that you did not ask for.

Rules cleared for auto-fix 8 of 51 routed rules. Everything else gets flagged for a human.
