What Betty catches

Pooled across 990 planted errors in English, Danish, German, Spanish and French. “Surfaced” counts only suggestions you are actually shown: Betty holds back the ones her own reviewer doubted, and those do not count here even when they happened to be right.

* On a strong PC or Mac — an NVIDIA graphics card with 16 GB or more, or an Apple Max or Ultra chip with 32 GB or more — Local Betty runs a larger model (Gemma 4 12B) instead. On the same benchmark, the same day, she surfaces 77% of mistakes commas aside, with the correct fix for 89%, and 71% of missing words.

Two Bettys, two different models

Local Betty and Betty in the Cloud run the same pipeline — the same checks, the same reviewer — on different models. So this page measures them separately, on the same fixtures, the same day.

Share of all 990 planted errors surfaced, commas included. On a strong machine, where Local Betty runs the larger model, 65%.

The cloud finds more — most of all the wrong-but-real words and the dropped words, which are the errors that need a model to understand the sentence — and raises fewer false alarms. Local Betty is a little more often right about the fix when she does find something, and she never sends a word of your manuscript anywhere. Choose on that: privacy and no cost per book, or the stronger read.

What it finds, by kind of error

Pooled across all five languages. Surfaced is out of the errors planted; correct fix is out of the ones Betty surfaced; false alarms are suggestions shown where nothing was wrong.

* Commas are scored as reconstructing the original text: each comma has exactly one right answer, the one in the text the error was planted into. Comma style legitimately varies between careful writers and between publishing houses, so this is not a true measure of comma quality — a pass that places commas differently but defensibly scores as missing some and raising false alarms on others.

A row with fifteen errors in it cannot tell you much, and the n is there so you can see which ones those are. False alarms are counted across 57,783 words of text, error-planted and clean alike. “Two errors side by side” are planted errors close enough together to be scored as one.

New: missing words

“They expect us to look each other in the same place.” A dropped word is invisible to a spell checker — every word that is there is spelled right — and it is the error an author reads straight past, because the mind supplies the word.

Betty now reads for them in a pass of its own: it looks only for sentences a small word has fallen out of, keeps only a suggestion that adds exactly one such word between two others, and asks whether the sentence parses with it and not without. Surfaced, by language:

This is where the two engines differ most — and where a strong machine closes most of the gap: with the larger model, Local Betty surfaces 71% of them. A suggested missing word is never applied on its own; it waits for you like any other suggestion. It is also a check that has to earn its false alarms, so we read it against 34,800 words of real novels outside the benchmark as well: on modern prose it suggested a missing word where none was missing twice on Local Betty and never in the cloud. Nineteenth-century prose, whose constructions can read as a dropped word, drew more.

What happens when it is wrong

A single accuracy figure hides the thing you actually care about, which is what a wrong suggestion costs you.

Betty never applies anything on its own, so the failure that matters is the one you cannot see. An error it finds but mis-fixes still arrives in front of you with a bad suggestion attached — you reject it and fix it yourself. An error it misses is invisible. That is why we report surfaced first, and the correct fix as a share of what was surfaced.

The same goes for the suggestions that turn out wrong. Our benchmark records one correct answer per error; real prose usually allows several. Of 60 wrong suggestions from Local Betty and 71 from the cloud, 4 and 8 introduced a word no dictionary knows. The rest left you with correct text that simply differed from the answer we wrote down.

The commas

Counted on their own, because a benchmark cannot tell a comma error from a comma choice. Split by what settles each comma, part of the picture is style — and part is a real weakness.

Commas are scored as reconstructing the original text: every planted comma error has exactly one right answer, the comma in the text it was planted into. For many commas that is not a true measure of quality. Comma placement is the rule that varies most between careful writers, and between houses, so a pass that punctuates differently but defensibly loses points here twice — once for the comma it did not restore, and once, as a false alarm, for the one it placed instead. So each of the 257 planted comma errors is sorted by what settles it:

The style row is not a list of errors. Style guides disagree with each other there, not with the writer, and Betty deliberately leaves those commas alone rather than reflow your sentences into a style you did not ask for. A low number in that row is closer to the intent than to a failure.

The grammar row is a real weakness, and it is not a size problem: the cloud's larger model finds no more of these than the model on your laptop. About one in three is found, most of them German subordinate clauses — and a comma the author put in by mistake is almost never flagged. Betty adds commas far more readily than she removes them. Plan a human pass on the commas grammar requires. If that is the part of copy editing you were hoping to hand over entirely, Betty is not there yet, and we would rather you knew before you bought.

Surfaced, out of the commas planted. Sorted by classifyComma in the app's backend/src/benchScoring.ts, on rules from the Chicago Manual of Style, Duden and the RAE that were fixed before any score was computed; the rows add up to the comma row in the tables above. Rows under 30 cannot tell you much, and the n is there so you can see which.

About one kettle.

Copy-editing a 60,000-word novel with Local Betty used a little less electricity than it takes to boil a litre of water — we measured it.*

In the cloud we estimate about the same, at most two. The bigger saving is the hours: the same pass by hand is twenty to forty at a desk. You still read every suggestion — but in about one.

* Measured on a desktop graphics card (RTX 5090): 60 Wh for 60,000 words, about 80–90 Wh for the whole computer; a litre boiled is about 110 Wh. The cloud's servers cannot be metered, so its figure is an estimate: 0.5 to 1 million tokens, at published energy figures for models of its size. Translation runs on a far larger model in a data centre, and is not one kettle.

How this was measured

Short stories written natively in each of the five languages rather than translated, so the mistakes are the ones that language actually makes, plus one text per language with only dropped words planted in it. Errors are planted by a generator that refuses any edit it cannot prove creates exactly one error of exactly the intended kind — so the ground truth is constructed, not annotated afterwards, and cannot drift from the text. Each story also runs clean, and everything Betty suggests there is a false alarm.

Both engines — and the larger local model for strong machines — ran the full pipeline on the same day: the Speed preset, grammar checks active, the reviewer on. The cloud run used a real credential, issued through the same checkout an author uses (with a 100% discount code). Every run records the pipeline that produced it, and the benchmark refuses to start with its grammar checks disabled — both of those exist because a run without them once produced numbers eight to twenty points low that looked entirely normal, and we published them.

Reproduce it with scripts/test-models.ts against the fixtures in sample_texts/, then scripts/performance-summary.ts, which computes every figure on this page and says exactly how. The report names every missed error and every false alarm individually; that detail is where most of the fixes on this page came from.