Benchmarks

Resemblances

Every resemblance explication has read: what its latest explication took, what the page it committed holds outside its header, and how its page as permitted last agrees with the first page explication saved for it.

Spending

How the numbers are made

• CER is the character error rate: edits to turn the gold into the transcript, over the gold's characters, counted as graphemes (a letter with its marks is one) after Unicode NFC normalization, in the order the text is written, whatever its direction. WER counts words the same way, for scripts written with spaces.
• Each rate is pooled over a dataset's items, with a 95% interval from a bootstrap over the gold's lines (10,000 resamples). Two benchmarks are compared line by line: the difference, its paired interval and a paired permutation test.
• Memorization: at listed places where a well-known edition reads differently from the copy, the share the model wrote as the edition does instead of as the copy prints.
• Fidelity checks the letters a script's older writing distinguishes (long s and u/v in Latin script, final forms in Hebrew, final sigma in Greek), only for the scripts they concern.
• Cost is what the benchmark's model calls cost, filed and priced as production's are; tokens and time come from the same calls.
• A resemblance's rounds, cost and tokens are its latest explication's, summed over that session's calls; its cost is unknown where any call has no price. Its words, lines, cited works, phrases and foreign passages count the page's <w>, <lb>, <bibl>, <phr> and <foreign> elements, and its languages the distinct xml:lang values, all outside the header, the facsimile and the stand-off.
• A page's gold is its markup in the public record: the newest permitted version that holds markup for it. Each permitted page is measured against the first page explication saved for it before the permit, as the benchmark runner scores a page: the gold's lines matched exactly, and its words' text, lemma, class and msd agreeing, over the gold's lines and words, with the counts of its elements and languages.
• A permitted page is gold when it holds a word and the permit changed anything it measures; confirmed when the permit changed nothing; blank when it changed a page that holds no word. Where no page explication saved before the permit is kept, it cannot be decided (—). Its TEI and its document's header download for the benchmark runner (page --gold and --header).
• Each model's agreement is pooled over the gold pages whose first saved page it wrote, with a 95% interval from a bootstrap over those pages (10,000 resamples).