The generation benchmark

“Leading LLM programming language” is an empirical claim: a model receives a task, writes POLYTONE, and ptc test judges — deterministically, no human in the loop. Two numbers per task, both first-class: pass@1 (the first candidate passes) and pass@2e (after a failure the model sees the actual error and gets one repair attempt — POLYTONE’s thesis is that its errors teach).

Methodology

• The judge is ptc test: candidate + hidden tests → one file → pass/fail. The judge never trusts the model — tests are appended after generation.
• Capability-clean: effects appear only through mocks (mock_fs, fixed_rng, fixed_clock, mock_env, mock_http) — a run touches no network, disk, clock, or entropy.
• A CI gate keeps every reference solution green against its own hidden tests — the corpus cannot rot. CI never calls a model; runs are local and BYO-key.
• The report always shows every task × every attempt — no cherry-picking.
• The system prompt is the frozen IDE language card. Since Sprint 252 the harness sends the EVALUATED card — byte-identical with the IDE and the Pro CLI, pinned by a gate; runs 1–7 sent the raw source form (doubled backslashes). Tokens-to-green stays directly comparable only between runs of the same card form.
• Tier L (since Sprint 265) is multi-module: the task names every file, the model answers in the IDE loop's fence grammar (one block per file with path=), and the judge verifies EVERY file as its own entry — modules first, then the entry, each with its own hidden tests (the Sprint-244 rule). A path outside the project is dropped, never sanitized; a missing file teaches before ptc runs. One judge path (corpus.mjs) for run.mjs and the harness.
• Comparability across corpus sizes: runs 1–6 measure corpus 33, runs 7–9 corpus 36, run 10 onward corpus 48 (36 + 12 × tier L). A headline number compares directly only with runs over the same corpus; a 48-run's line on the 36 older tasks is the number that compares with runs 7–9 (run 10: 33/36 · 35/36 — the run-8 level). Tier L is reported on its own.

Results

The first delta proof (replays)

The phase's first failure dataset is the corpus-authoring session itself (Sprint 133) — an LLM writing POLYTONE cold, its first attempts recorded verbatim. Sprint 138 replays them against the hardened toolchain, CI-gated (gen_replays.rs):

• content_tag: failed on the missing Text.slice — now passes verbatim (the language grew to meet the model, Sprint 137).
• json_pluck: the error read like nonsense — now teaches the qualified form (the repair signal pass@2e depends on).
• log_scan: still fails (a semantics miss no compiler error can prevent) — but the assertion diff shows the actual values, and Card v6 teaches the rule.

The tier-L delta (Sprints 267–268)

Run 10 (corpus 48) had five first-attempt failures; two of them are the first multi-module replays (pkg_check_attempt1/, tune_mix_attempt1/ — directory form, every file its own entry) and web_toc_attempt2_run10.pt the residual. Sprint 268 rewrote two teaching errors from them (a partial variant pattern shows the exact named form; an element method on a list names the way to an element) and measured COLD over the same five tasks: 3/5 pass@1 (0/5 in run 10). The residual named the last gap — there was no way to write an arm that does nothing — so pass became the empty statement; the same candidates re-judged under it read 4/5, the repair round 5/5. All three replays are pinned in gen_replays.rs.

Fresh pass@1/pass@2e runs against frontier models stay local and BYO-key — this section shows only what has actually run.

The corpus (48)

17 × tier S · 19 × tier M · 12 × tier L (multi-module: one file per module, each verified as its own entry) — each task: a prompt, a hidden judge, a reference solution.