Baselines

How well each process reads, baseline by baseline, and what it cost. Every dollar spent on models is counted below.

Gold

Gold: —.

A baseline is a run: one configuration reading every page of the gold, each reading scored against the page a person permitted. A page is gold when its markup is permitted in its record; nothing else adds one.

Baseline runs

Each run names the prompt vignette it read with, so a run after an amendment is a new row. A score is the share of the gold's units the run got wrong, by measure, with its change from the run before of the same model and effort.

Baseline runs, newest first
No baseline run yet.

Configurations

A configuration is a process vignette, a tool version (the pipeline's revision and the hash of the tool list its session sent), a model version (the provider's published version that served its runs) and a reasoning effort. Ability is on the model's scale, centered on the mean of the configurations: higher reads with fewer errors; beside it, the error rate it is expected to make on a typical page. Effort is a multiplier on a typical configuration, beside the amount it is expected to take on a typical page. Each with its 95% interval; — where the data cannot say. Two tool vignettes that offer the same tool list are one configuration here: the runs above tell them apart.

Configurations and their estimates
No run has been scored under a recorded configuration yet.

Gold pages

Difficulty is on the scale of the abilities, higher harder: the ability at which a configuration would get half the page's units wrong, so the further an ability is above it, the fewer errors. Discrimination says how sharply a page's errors separate abilities. The effort columns are what a typical configuration is expected to take on the page.

Gold pages and their parameters
No gold page yet.
Showing results 0–0 of 0
of 1

Revisers

Arbitration is a respondent kind of its own. Where an arbitration agent suggested markup for a page that is now permitted, the markup it suggested and the markup before it are each scored against the permitted version, whatever became of the suggestion; a configuration's improvement is the error that falls, on the pages it revised, and its ability gain over the markup before, on the scale of the abilities. Fitted apart from the explications above.

Arbitration's revisers and their improvement
No arbitration has revised a page that is permitted yet.

Datasets

A dataset is a set of transcriptions imported with its license and an account of how its gold was made, posted by an admin with the baseline harness. Its baselines read the whole set with one configuration and are scored by CER and WER, grouped by the language and script of the items.

No baselines yet.

Frames

Every frame explication has read: what its latest explication took, what the page it committed holds outside its header, and, for its page as permitted last, the writing it is in and whether it is gold.

Explicated frames
No frames explicated yet.
Showing results 0–0 of 0

Spending

$1.685 All model spending
$0.0000 Baselines (0.0% of all)
$1.685 Production
2 Unpriced calls

How the numbers are made

  • CER is the character error rate: edits to turn the gold into the transcript, over the gold's characters, counted as graphemes (a letter with its marks is one) after Unicode NFC normalization, in the order the text is written, whatever its direction. WER counts words the same way, for scripts written with spaces.
  • Each rate is pooled over a dataset's items, with a 95% interval from a bootstrap over the gold's lines (10,000 resamples). Two baselines are compared line by line: the difference, its paired interval and a paired permutation test.
  • Memorization: at listed places where a well-known edition reads differently from the copy, the share the model wrote as the edition does instead of as the copy prints.
  • Fidelity checks the letters a script's older writing distinguishes (long s and u/v in Latin script, final forms in Hebrew, final sigma in Greek), only for the scripts they concern.
  • Cost is what the baseline's model calls cost, filed and priced as production's are; tokens and time come from the same calls.
  • A frame's rounds, cost and tokens are its latest explication's, summed over that agent's calls; its cost is unknown where any call has no price. Its words, lines, cited works, phrases and foreign passages count the page's <w>, <lb>, <bibl>, <phr> and <foreign> elements, and its languages the distinct xml:lang values, all outside the header, the facsimile and the stand-off.
  • A page's correct answer is its markup in the public record: the newest permitted version that holds markup for it. Its TEI and its document's header download for the baseline runner (page --gold and --header).
  • Every run of a permitted page is scored against the page's newest permitted version, the answer key: production's explications and baseline runs alike, as the baseline runner scores a page—the gold's lines matched exactly, and the text, lemma, class and msd of its words agreeing. A page permitted again is scored afresh. Beside each, its effort: model calls, tool calls by tool, failed tool calls, input, cached and output tokens, cost and duration, from the ledger and the run's own record.
  • A configuration is a process vignette, a tool version (the pipeline's revision and a hash of the tool list the agent sent), a model and a reasoning effort, recorded on every run as it starts. A run resumed under another configuration belongs to none.
  • Errors are fitted by item response theory, one measure at a time: a run's errors are a share of the page's units, drawn from a beta-binomial around a two-parameter logistic curve, with an ability for each configuration and a difficulty and a discrimination for each page. A beta-binomial, not a binomial, because errors cluster within a page and a run. The fit is penalized by weakly informative priors, and each estimate's 95% interval comes from the curvature at the fit (a Laplace approximation): little data gives wide intervals, never extreme values. Only pages read by two configurations or more are fitted.
  • Effort is fitted apart, never blended with errors or with one another: counts by a negative binomial, tokens, cost and duration on a log scale, each with a demand for each page and a multiplier for each configuration.
  • Two configurations are compared on the pages both read: each one's error rate, and the difference with a 95% interval from a bootstrap over those pages (10,000 resamples); beside it, the model's ability difference, which uses every page and allows for its difficulty.
  • Gold is one rule applied to the current fit, never a number chosen. Configurations the data cannot tell apart (abilities within 1.96 standard errors of their difference) are one group, a twin that differs only in its code revision among them. The target is to tell apart the two groups whose abilities are closest, by a two-sided 5% test with 80% power. Pages are taken—each writing's most informative first, then the page that most closes the gap to that precision—until it is met, or every page where it never is. The page says how many of the permitted pages that makes. The fit is made again after each permit and each baseline run, and when the fitting code changes; each gold set is recorded, with the baseline runs that read it.
  • Arbitration is scored as a respondent kind of its own, a reviser: where an agent suggested markup for a page that is now permitted, the suggested markup and the markup before it are each scored against the permitted version, whatever became of the suggestion. A configuration's improvement is the error that falls on the pages it revised, with a 95% interval from a bootstrap over those pages, and its ability gain over the markup before, fitted apart from the explications; its effort is the session's.
  • A page's writing comes from what is declared only: its witness's carrier in its record, and for print the typeface class (roman, italic, blackletter, script, sans-serif, typewriter) its markup sets most of its text in. Print with no typeface class declared is its own group; nothing declared is —.
  • Run baseline (admins) reads the gold in force through the explication pipeline with the prompt vignette, model and effort chosen: the same agent, tools, trace and ledger as production, off-peak, as Computorium work, stopped by the drain like any other; each page's version is recorded as the run starts. Its pages are scored and the model fitted again when it has read them. A run's score on a measure is the units it got wrong over the units of the gold pages scored so far, pooled.