The numbers, including the ones that are bad for us
One harness, one corpus, one code path, four engines. Marlo's engine is one of the four and gets no exemption, because a table where the author's own engine happened to win would be worth nothing.
Marlo is third of four
Alfa and axe-core are both more precise and more sensitive. Which is why the router sends 18 rules to axe-core and only 20 to Marlo's own engine.
| Engine | Rules | Precision | Recall | False positives | cantTell, miss / cautious |
|---|---|---|---|---|---|
| Alfa | 40 | 0.964 | 0.835 | 1.5% | 21 / 30 |
| axe-core | 38 | 0.952 | 0.777 | 2.0% | 24 / 23 |
| Marlo | 41 | 0.853 | 0.640 | 5.6% | 37 / 50 |
| HTML CodeSniffer | 12 | 0.684 | 0.366 | 9.4% | 0 / 0 |
HTML CodeSniffer's recall is 0.366. It never returns a definite failure for any rule it claims. Its warnings and notices are advisory, which honestly means "cannot tell", and the adapter reads silence as a pass. That inference is the weakest step in the whole engines package. It stays in the table because it puts a number on exactly the gap the table exists to show.
W3C's protocol will call an engine correct for answering "I don't know"
W3C defines how to grade an implementation against a rule's official test cases.
Under that protocol cantTell is an allowed answer for
every example type.
So a tool that shrugs at all 1134 test cases is, officially, a correct implementation of all 91 rules that have them.
That is not a flaw in the protocol. It grades whether a tool misleads you, and "I don't know" misleads nobody. It is just not the question a developer is asking, which is whether the violation actually gets found.
So the table publishes both views and computes the gap rather than leaving you to notice it. 9 entries currently grade as officially consistent while missing more than half the violations a real user would hit. Two of the three are ours.
| ACT rule | Engine | W3C verdict | Strict recall |
|---|---|---|---|
36b590 |
Marlo | consistent | 0.000 |
3ea0c8 |
axe-core | consistent | 0.000 |
4b1c6c |
Alfa | consistent | 0.000 |
5c01ea |
Marlo | consistent | 0.000 |
b20e66 |
Alfa | consistent | 0.000 |
c487ae |
Marlo | consistent | 0.273 |
cf77f2 |
Marlo | consistent | 0.000 |
fd3a94 |
Alfa | consistent | 0.000 |
ye5d6e |
Marlo | consistent | 0.333 |
What Marlo may touch without asking
A rule qualifies only when the engine reporting it clears 0.950 strict precision over at least 6 official test cases.
The gate is precision, not recall, and the asymmetry is deliberate. A missed violation is a gap you already had. A wrong fix is a change to your code that you did not ask for.