Last week Google announced two new Gemini models. We run four million pages of AI-transcribed historical text on the previous generation, so instead of reading the benchmarks in the press release, we ran our own: the same forty pages of blackletter, Fraktur, polytonic Greek, pointed Hebrew, classical Chinese, and 19th-century Armenian type, against every OCR-capable model we could reach. Two days and about thirteen dollars later, our reference dataset held 1,737 scored transcriptions across models from five labs, served three different ways. The most newsworthy result was the one we least expected: on historical documents, Google’s newest budget model is slightly — but systematically — worse than the model it replaces.
The test
The benchmark is the reference set we built for our memorization work: forty pinned pages from our own scans, each paired with an independent scholarly transcription of a passage printed on that page. The set has a property most OCR benchmarks lack — a memorization control. Twelve pages carry canonical text (the Iliad’s opening, Genesis 1, the Daodejing) that frontier models have plausibly memorized and can recite without reading; twenty-eight carry text almost certainly absent from training data (editors’ prefaces, biographical front matter, mid-text passages of works digitized nowhere else). Score the two classes separately and you can tell a model that reads from a model that remembers.
Scoring is deliberately conservative: the reference must appear as an in-order subsequence of the model’s output, a guard decides whether the passage is genuinely present (“aligned”), and character accuracy is measured only on the reference span, with orthographic conventions of early printing (u/v, i/j, long-s, ligatures) folded away. All comparisons below are paired per page — each model against the same page’s score for the reference model — with an exact sign test and a bootstrap confidence interval. The full dataset, including every raw transcription, is published as v0.3 on GitHub.
Finding 1: the newest model lost to its predecessor
The reference model is gemini-3.5-flash-lite, released the week of the test. Its predecessor, gemini-3.1-flash-lite — the model that runs our production pipeline — beat it on 19 of 38 shared pages and lost on 3 (mean +0.30 percentage points, exact sign test p = 0.0009). The effect is small, but it is not noise: it survives multiple-comparison correction across all eleven contrasts we ran. Whatever got better in the new generation — and the announcement emphasized coding, knowledge work, and chart parsing — reading 16th-century print was not part of it.
| vs gemini-3.5-flash-lite | mean Δ | 95% CI | W/T/L | sign p | |
|---|---|---|---|---|---|
| gemini-3.6-flash | +0.72pp | [+0.33, +1.20] | 26/9/3 | 1.5e-5 | 3× the price |
| gemini-3.1-flash-lite | +0.30pp | [−0.00, +0.57] | 19/16/3 | 8.6e-4 | the predecessor |
| claude-sonnet-5 | −0.17pp | [−1.37, +0.69] | 13/9/5 | 0.10 | |
| qwen3-vl-plus | −0.44pp | [−1.19, +0.13] | 5/12/6 | 1.0 | open weights; reads 24/41 pages |
| mistral-ocr | −0.71pp | [−1.56, −0.00] | 7/12/14 | 0.19 | $1 per 1,000 pages |
| qwen-vl-max | −0.83pp | [−1.67, −0.02] | 5/4/12 | 0.14 | |
| deepseek-ocr | −1.77pp | [−2.62, −1.08] | 0/0/16 | 3.1e-5 | reads 16/41 pages |
| gemma-4-31b (self-hosted) | −4.11pp | [−5.91, −2.40] | 2/4/19 | 2.2e-4 | L40S GPU, QAT quant |
| gemma-3-27b | −7.24pp | [−10.62, −4.14] | 1/2/21 | 1.1e-5 |
Pricing makes the regression sharper. The new lite model costs roughly four times its predecessor per output token. For a workload like ours — output-heavy transcription at library scale — the July generation is a price increase for slightly worse reading.
Finding 2: an open-weight model reached parity — with an asterisk the size of Armenia
Alibaba’s qwen3-vl-plus is statistically indistinguishable from Google’s newest flash-lite on the pages both can read: −0.44pp with a confidence interval straddling zero, five wins, six losses, twelve ties. On Greek, Latin, and classical Chinese it posts the best or near-best quality numbers in the entire table. A year ago no open-weight model was within five points of frontier OCR on this material.
The asterisk: “the pages both can read” is doing enormous work in that sentence. Qwen aligned on 24 of 41 pages. It read zero of the eight Armenian pages — and it did not fail honestly. It produced fluent, confidently formatted, entirely fabricated Armenian text, a failure mode we’ve written about before as the confident hallucinator. For a library, a model that invents plausible text on scripts it cannot read is worse than one that refuses.
Finding 3: coverage separates models; accuracy mostly doesn’t
Above the Gemma tier, every model’s accuracy on pages it can align clusters within about one percentage point. What actually separates the field is which scripts a model can read at all: the Gemini family aligns 36–40 of 41 pages, Mistral-OCR 34, Gemma-4 26, Qwen 24, DeepSeek-OCR 16. Armenian is the starkest column — our production model reads all eight pages at 97.2%; among everything else only Mistral-OCR (4/8) and self-hosted Gemma-4 (3/8) read any. And each model fails differently: Qwen fabricates; DeepSeek-OCR silently skips lines (its skip-penalizing score drops 4.5 points, the widest gap in the table); and the new Gemini generation introduced a failure the incumbents never showed — refusing pages of canonical text as suspected training-data recitation, killing 2 of 3 runs on one Hero of Alexandria page. Three failure modes, three different operational risks: fabrication corrupts silently, skipping loses text, refusal blocks exactly the famous passages readers most want.
Finding 4: the serving stack is part of the model
We ran Gemma three ways, and the same weights behaved as three different systems. Google’s free API served Gemma-4 beautifully — when it answered at all; most requests stalled for five minutes and died, a lottery that made systematic evaluation impossible. Scaleway’s paid serverless API ran Gemma-3 flawlessly. And self-hosting Gemma-4 on a rented GPU produced thousands of tokens of empty output — the runtime was routing the model’s entire answer into a hidden reasoning channel and discarding it (a known Ollama bug; one configuration flag fixed it, after which 120 of 120 calls succeeded). None of this is visible in any leaderboard. If you evaluate a model without recording who served it and how, you are not measuring what you think you are measuring — our dataset now tags serving provider on every row.
What it cost, and what we’re doing
The entire campaign — nine models, three serving providers, 1,678 scored runs, including renting and configuring a GPU server twice — cost about thirteen dollars and two days of wall-clock time, most of it waiting. That is the real finding for anyone running an AI-dependent pipeline: model churn is now cheap to answer empirically. Every “should we switch?” that used to be a meeting is an afternoon and pocket change, provided the reference set exists.
We are not switching anything. Our production pair — the year-old flash-lite for bulk work, flash for premium material — survived a nine-model challenge with its position strengthened: still the only stack that reads every script we hold, still the cheapest per page, still zero refusals. The eval’s job was to prove that with numbers rather than inertia, and to tell us exactly where the field is moving: open-weight quality is arriving script by script, and the next model we test will be judged on Armenian first.
Dataset v0.3 — forty-four pages (including four within-work canonicity pairs from a parallel workstream), licensed-or-hashed references, 1,737 raw transcriptions with re-derivable scores, and the statistics tooling — is on GitHub, alongside the harness that produced it. If you maintain a model we didn’t test, the bar is forty pages high and costs about a dollar to clear.