The generation benchmark

“Leading LLM programming language” is an empirical claim: a model receives a task, writes POLYTONE, and ptc test judges — deterministically, no human in the loop. Two numbers per task, both first-class: pass@1 (the first candidate passes) and pass@2e (after a failure the model sees the actual error and gets one repair attempt — POLYTONE’s thesis is that its errors teach).

Methodology

• The judge is ptc test: candidate + hidden tests → one file → pass/fail. The judge never trusts the model — tests are appended after generation.
• Capability-clean: effects appear only through mocks (mock_fs, fixed_rng, fixed_clock, mock_env, mock_http) — a run touches no network, disk, clock, or entropy.
• A CI gate keeps every reference solution green against its own hidden tests — the corpus cannot rot. CI never calls a model; runs are local and BYO-key.
• The report always shows every task × every attempt — no cherry-picking.
• The system prompt is the frozen IDE language card. Since Sprint 252 the harness sends the EVALUATED card — byte-identical with the IDE and the Pro CLI, pinned by a gate; runs 1–7 sent the raw source form (doubled backslashes). Tokens-to-green stays directly comparable only between runs of the same card form.
• Tier L (since Sprint 265) is multi-module: the task names every file, the model answers in the IDE loop's fence grammar (one block per file with path=), and the judge verifies EVERY file as its own entry — modules first, then the entry, each with its own hidden tests (the Sprint-244 rule). A path outside the project is dropped, never sanitized; a missing file teaches before ptc runs. One judge path (corpus.mjs) for run.mjs and the harness.
• Comparability across corpus sizes and generators: runs 1–6 measure corpus 33, runs 7–9 corpus 36, run 10 corpus 48 (36 + 12 × tier L), run 11 corpus 52 (48 + the four Sprint-306 tasks), and since Sprint 326 the corpus holds 58 (52 + six everyday tasks over the forms Sprints 319–320 opened — unmeasured until Baseline 12). A headline number compares directly only with runs over the same corpus; a 48-run's line on the 36 older tasks is the number that compares with runs 7–9 (run 10: 33/36 · 35/36 — the run-8 level). Tier L is reported on its own. Runs 1–10 were generated by claude-fable-5; run 11 (Sprint 314) ran on claude-opus-5 because Fable's credits were exhausted — the model is on every record, and numbers compare directly only within one generator (run 11 on the 48 run-10 tasks: 43/48 · 46/48 against Fable's 43/48 · 47/48; on the 36 older tasks 33/36 · 35/36, as run 10).
• Tasks added after the newest run are listed in the corpus table as not yet measured (since Sprint N) — the next run covers them; the coverage gate exempts them until then (the four Sprint-306 tasks trait_travel, culled_parts, model_page, label_card were first measured by run 11: 4/4 cold).

Results

The first delta proof (replays)

The phase's first failure dataset is the corpus-authoring session itself (Sprint 133) — an LLM writing POLYTONE cold, its first attempts recorded verbatim. Sprint 138 replays them against the hardened toolchain, CI-gated (gen_replays.rs):

• content_tag: failed on the missing Text.slice — now passes verbatim (the language grew to meet the model, Sprint 137).
• json_pluck: the error read like nonsense — now teaches the qualified form (the repair signal pass@2e depends on).
• log_scan: still fails (a semantics miss no compiler error can prevent) — but the assertion diff shows the actual values, and Card v6 teaches the rule.

The tier-L delta (Sprints 267–268)

Run 10 (corpus 48) had five first-attempt failures; two of them are the first multi-module replays (pkg_check_attempt1/, tune_mix_attempt1/ — directory form, every file its own entry) and web_toc_attempt2_run10.pt the residual. Sprint 268 rewrote two teaching errors from them (a partial variant pattern shows the exact named form; an element method on a list names the way to an element) and measured COLD over the same five tasks: 3/5 pass@1 (0/5 in run 10). The residual named the last gap — there was no way to write an arm that does nothing — so pass became the empty statement; the same candidates re-judged under it read 4/5, the repair round 5/5. All three replays are pinned in gen_replays.rs.

The run-11 dataset (Sprint 314)

Run 11 (corpus 52, claude-opus-5) had five first-attempt failures, three healed in one round (audio_probe sample_rate for the listed rate — run 10's class on a different model; json_pluck fields for entries — the run-9 residual, named by the error; model_parts shape lines without their indentation — the codec names the rule). The two residuals are ONE class, stacked twice: both TOML tasks first invented the shape of toml.parse (a document with .tables; a Value.Table variant), the repair fixed exactly that and hit the second error behind it — the guessed field entries on toml.Table (pairs). Two stacked errors cost the task under one repair round. Five replays (config_port_attempt{1,2}_run11.pt, pkg_check_attempt{1,2}_run11/, model_parts_attempt1_run11/) are pinned in gen_replays.rs.

The hardening from the run-11 dataset (Sprint 315)

The stacked class got two answers: a field read on a List[Record] now lists the ELEMENT's fields and the way to one (the Sprint-268 method error's twin) — the second error is visible in the first round; the task texts config_port/pkg_check name the shape of toml.parse the way the 266 corrections name every import, model_parts shows its shape line indented. Measured cold over the same five tasks (generator claude-opus-5 = run 11's): 3/5 pass@1 (0/5 in run 11), the repair round 5/5; the two untouched tasks failed on their exact one-round classes (stdlib field names the error names — no card candidate). The three cold candidates are pinned verbatim (*_cold_run11delta).

Fresh pass@1/pass@2e runs against frontier models stay local and BYO-key — this section shows only what has actually run.

Baseline 12 — the held-out protocol

The runs above measure the corpus POLYTONE grew up with. Baseline 12 measures against tasks the language never saw, and against Python and TypeScript on the same tasks. The protocol is stated here before any number exists; the runs wait for the owner's API keys.

• Sealed. 20 tasks (7 S · 9 M · 4 L, 283 vectors), written by an agent that never saw the language card, the spec or the corpus — issue-style tickets that name no module. SEAL.json hashes every task; preflight fails on any drift. It is never a tuning set: no card line, teaching error or stdlib change is ever made because of it.
• Three arms, one judge. The same language-neutral task (ticket + vectors as JSON + a Python reference) is derived into POLYTONE, Python and TypeScript; arms.mjs judges all three — ptc test, python3 -I, tsc --strict + node — over the same values. CI keeps all 60 references green.
• Intervals, not points. Every rate carries its Wilson 95 % interval; two arms differ only once their intervals separate.
• Ablation. POLYTONE runs three times: with the frozen card (--card full), with its first line and the eight syntax-core rules (minimal), and with no card (none) — exactly one line naming the language, as Python and TypeScript get. That separates what the language carries from what the card carries.
• Reproducible. A run directory is self-contained (model, commit, card and seal sha256, every reply verbatim). --rejudge re-judges every transcript without a key, --reproduce repeats the configuration on a second machine; published runs live under benchmarks/gen/published/.
• pass@1+fix, apart. A failed POLYTONE first attempt runs through ptc fix — the toolchain's own quick fixes for teaching errors, zero tokens, no second model call — and is judged again. That is its own column beside pass@1, never in its place; Python and TypeScript have no such fixes (—).
• Honest tokens. Only the provider's own usage report counts (reasoning tokens where billed). The planning ESTIMATE at n=3 over all three arms: ≈ 346 k input + 126 k output tokens per model (POLYTONE 224 k of the input — the card rides on every call). An estimate is never a measurement.

The corpus (58)

21 × tier S · 21 × tier M · 16 × tier L (multi-module: one file per module, each verified as its own entry) — each task: a prompt, a hidden judge, a reference solution.