Every number here comes from a benchmark you can re-run from the repository — including the ones that do not flatter us.
of the obvious mistakes surfaced — misspellings, doubled words, wrong words, wrong capitals, wrong dialect, missing punctuation. 421 planted errors, four languages. Betty suggests the right fix for 93% of those.
commas. That is the honest number, it is the largest hole in the product, and you should plan a human pass if commas matter to you.
Pooled across 617 planted errors in English, Danish, German and Spanish; a 95% interval is about four points. Counting commas alongside everything else, the same run surfaces 74% of all planted errors and fixes 69% — the two figures differ only in whether commas are in the denominator, and this page reports both rather than choosing the flattering one.
Three engines, measured the same afternoon on the same fixtures through the same pipeline. They land within 1.6 points of each other, on a measurement whose interval is four.
So choose on the things that actually differ: whether your machine can run a model at all, whether you want your manuscript to stay on it, and whether you would rather wait ten minutes or an hour. Not on quality. A model twice the size does not read your prose twice as well, and we would rather say so than sell you the bigger one.
Pooled across all four languages. The bars are honest about their own precision: a category with fifteen instances cannot tell you much, and says so.
Local Betty. The other two engines sit within a few points on every row — the differences between them are smaller than the intervals, which is the point of the section above.
A single accuracy figure hides the thing you actually care about, which is what a wrong suggestion costs you.
Betty never applies anything on its own, so the failure that matters is the one you cannot see. An error it finds but mis-fixes still arrives in front of you with a bad suggestion attached — you reject it and fix it yourself. An error it misses is invisible. That is why we report surfaced alongside fixed, and why the gap between them is not a hidden cost.
The same goes for the suggestions that turn out wrong. Our benchmark records one correct answer per error; real prose usually allows several. Of 235 wrong suggestions across all three engines, 25 introduced a word no dictionary knows. The other 210 left you with correct English that simply differed from the answer we wrote down.
A third. We are not going to dress that up.
Comma placement is the largest category in every fixture and the weakest result on this page, and it is weak in a specific way: it is not a size problem. A model twice as large finds no more of them, which is how we know the answer is not a bigger model.
It is also the rule that varies most between careful writers, and between houses. Betty is deliberately conservative here — it would rather leave your commas alone than reflow your sentences to a style you did not ask for. Plan a human pass on punctuation. If that is the part of copy editing you were hoping to hand over entirely, Betty is not there yet, and we would rather you knew before you bought.
Four short stories, written natively in each language rather than translated, so the mistakes are the ones that language actually makes. Errors are planted by a generator that refuses any edit it cannot prove creates exactly one error of exactly the intended kind — so the ground truth is constructed, not annotated afterwards, and cannot drift from the text.
Every run records the pipeline that produced it, and the benchmark refuses to start with its grammar checks disabled. Both of those exist because a run without them once produced numbers eight to twenty points low that looked entirely normal, and we published them.
Reproduce it with scripts/test-models.ts against the fixtures in sample_texts/. The report names every missed error and every false positive individually; that detail is where most of the fixes on this page came from.