Insights Goal D1: post-merge baseline, calendar-guard expression coverage, opus-5 as a selectable generator (2026-09-08)

Branch `insights/goal-d1` from `origin/main` 094cf05e (the merge of PR 98). Every measurement uses the goal-3 method: the frozen fixture, the sealed holdout v11-r2 or the Goal C real-usage sample, the goal-3 scorer, the clock anchor 2026-09-06 12:00 UTC in the Europe/London calendar, two workers, and an application snapshot frozen from Git blobs. Denominators: holdout 100 supported first turns, 250 attempts, 50 conversations; real usage 86 supported first turns, 101 attempts, 5 conversations. The noise floor is the Goal B control arm's spread over three repeats of one code snapshot; a difference smaller than it is not a finding.

Headline

• Step 0. The post-merge baseline on the holdout sits inside the control noise on every metric except correct completed (28/250 against the control's 33 to 39): first-turn errors 59/100 (original 60, control 54 to 57), correct first turns 14/100, precision 44.4%, silent wrong 35/250. Against the control repeats 0 of 17 stable-correct cases were lost; 22/36 original correct completions were retained (Goal A saw 26 and 24 of 36 on its two runs). On the real-usage sample the same code refused 23 of 86 first turns through the calendar guard, more than Goal C's 13 to 15, because PR 98's real physical-type loader now feeds the typed check that previously skipped.
• Step 1. The refused SQL had eight expression forms. The guard now proves aggregates (MIN/MAX with FILTER), branch expressions (COALESCE, GREATEST, LEAST, NULLIF, CASE) over same-typed timestamps, interval and integer date arithmetic, date spines cast to naive timestamps, ::date and DATE_TRUNC chains, CTE-, derived-table-, LATERAL- and subquery-derived timestamps, and generate_series spines, each with real-PostgreSQL tests on the London bucket, the spring skip, the autumn repeated hour and the month boundary. A date spine pushed through AT TIME ZONE stays refused: PostgreSQL resolves it as timestamptz in the session zone and the bound lands an hour off in BST on real rows, which the tests pin. Calendar refusals on the real-usage first turns went from 23/86 to 2/86 (target: at most 3). Every R08 counterexample still fails; on the holdout the calendar guard refuses the same multi-timezone questions as before (10 of 250 against 9), and the clean guard-head holdout run sits inside the control noise: first-turn errors 57/100 against the step-0 run's 59, correct first turns 17 against 14, precision 57.1% against 44.4%, silent wrong 30 against 35; against the control repeats 0 of 17 stable-correct cases were lost and 4 gained. No executed statement in any D1 run failed with an AT TIME ZONE chain in it.
• Step 2. With claude-opus-5 as the configured generator (the benchmark adapter now inside the application: pinned slot, tools proven disabled per call, empty working directory, file-backed transport, absolute deadline with kill and reap, no API key, fail-fast budget gate), the holdout over 3 repeats gives first-turn errors 39.7 [39.0 to 40.0] per 100 against the step-0 baseline's 59 (paired difference against the Goal B control -16.0 [-24.0 to -8.3]), correct first turns 25.3 [25.0 to 26.0] against 14 (10.3 [4.3 to 17.0]), precision 50.9% [50.0% to 51.8%] against 44.4% (0.2% [-12.9% to 13.3%]), silent wrong 55.7 [55.0 to 57.0] against 35 (21.3 [4.7 to 37.7]), full conversations 4.7 [4.0 to 5.0] against 2. On the real-usage sample opus-5 made 8/86 first-turn errors, 0/86 correct first turns, 35/101 silent wrong and 1/86 calendar refusals.
• Step 3. A fresh sealed slice (122 distinct questions, 31 Goal C families, 122 attempts) is sealed in the evaluator-only directory with seal SHA-256 d54a61d06882a38a9be0b2f4321688758cab15e585357329b75cee3bdce68067; 0 duplicate texts against 1496 prior texts. Not run.

1. Step 0: post-merge baseline (main 094cf05e, glm control, temperature 0, seed 42, two workers)

Holdout v11-r2

Stable-case view against the Goal B control repeats (a case is lost only if it was correct in all three control repeats and wrong or error in the candidate):

Calendar-guard refusals on the holdout, with their reason strings:

• D1 step 0: main 094cf05e, glm: 9 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 9, 'UNKNOWN-GOLD-COLUMN': 3}; reasons: 9 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'
• D1 step 1: guard head bb7e3938, glm (r1, zai 429s): 10 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 10, 'UNKNOWN-GOLD-COLUMN': 2}; reasons: 10 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'
• D1 step 1: guard head, glm (r2): 10 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 10, 'UNKNOWN-GOLD-COLUMN': 5}; reasons: 10 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'
• D1 step 2: opus-5 r1 (9fa89bcb): 10 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 10}; reasons: 10 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'
• D1 step 2: opus-5 r2: 10 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 10, 'UNKNOWN-GOLD-COLUMN': 1}; reasons: 10 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'
• D1 step 2: opus-5 r3: 10 calendar refusals of 250 (4 on supported first turns); refusal guards {'-': 1, 'CalendarSemanticsError': 10, 'UNKNOWN-GOLD-COLUMN': 1}; reasons: 10 × 'SQL still failed CalendarSemanticsError enforcement after regeneration: Unverified calendar semantics: more than one requested timezone'

Real-usage sample (Goal C corpus, 101 attempts)

2. Step 1: calendar-guard expression coverage

The forms found in the refused generated SQL (Goal C landed r1/r2 lost their SQL text: Goal C retained prompt hashes only; the opus r1 CLI transcripts and the D1 step-0 text trace retain it), what the extended guard now proves, and what stays refused:

Transitions of the calendar-refused cases of the step-0 run after the guard change:

• guard-glm-real-r3: {'silent_wrong': 11, 'comparison_unverified': 8, 'task_error': 3, 'refusal_unverified': 1}
• guard-glm-real-r2: {'silent_wrong': 8, 'comparison_unverified': 7, 'task_error': 7, 'refusal_unverified': 1}
• guard-glm-real-r1: {'silent_wrong': 10, 'correct_completed': 1, 'comparison_unverified': 8, 'task_error': 3, 'infrastructure_failure': 1}
• opus-real-r2: {'comparison_unverified': 8, 'silent_wrong': 11, 'task_error': 4}
• opus-real-r1: {'comparison_unverified': 9, 'silent_wrong': 10, 'task_error': 3, 'infrastructure_failure': 1}

Calendar refusals on supported first turns of the real-usage sample: D1 step 0: main 094cf05e, glm: 23/86; D1 step 1: guard head, glm (r1, 409 cascade): 0/86; D1 step 1: guard head, glm (r2, zai 429s): 2/86; D1 step 1: guard head, glm (r3): 2/86; D1 step 2: opus-5 (r1, 409 cascade): 0/86; D1 step 2: opus-5 (r2): 1/86.

3. Step 2: claude-opus-5 as the configured generator

Per-call generator statistics (CLI wall time includes the child's start-up and Claude Code's own request pipeline; runs overlapped on the host, so wall time is advisory):

4. Step 3: fresh sealed slice

Revision fresh-v1, split fresh_slice: 122 distinct question texts (113 standalone cases and 3 three-turn conversations, 122 attempts) across 31 Goal C families; 7 expected answers are empty on the fixture; 0 duplicate texts against 1496 prior texts. Seal manifest SHA-256 d54a61d06882a38a9be0b2f4321688758cab15e585357329b75cee3bdce68067. Sealed at 2026-09-08T14:30:56.271778+00:00 in the evaluator-only directory; not run.

Families: F01, F02, F03, F04, F10, F11, F12, F13, F14, F16, F17, F19, F20, F22, F23, F24, F25, F26, F27, F31, F33, F34, F38, P01, P02, P03, P04, P05, X01, X02, X03.

5. Method

• Code: branch `insights/goal-d1`; step 0 ran the exact Git blobs of main 094cf05e; step 1 the guard head bb7e3938 (main plus the calendar-guard change and `bench_metrics.py`); step 2 the adapter head 9fa89bcb and 9d64c468 (identical application bytes; the later revision only changes the real-usage runner). Snapshot manifests (exact blob hashes) are in every run's receipt.
• Runtime: `scripts/insights_d1_runtime.py`, the sealed goal-3 runtime plus generated-text capture (evaluator-only) and a configurable generator: `zai` reproduces the sealed LiveProvider call byte for byte (glm-5.2 requested through the zai coding endpoint, thinking disabled, temperature 0, seed 42, 25 s deadline); `claude-cli` leaves generation to the application's own `RoutedLLMClient` with `LORE_BI_SQLGEN_PROVIDER=claude-cli` (claude-opus-5, slot 3, thinking disabled, 75 s deadline). Auxiliary (non-SQL) generation stays zai glm-5.2 in every run. The 3-minute job deadline is unchanged.
• Runner: the sealed `scripts/insights_goal2_eval.py` for the holdout (with Goal B2's optional budget gate); `scripts/insights_real_usage_eval.py` (Goal C's copy) for the real-usage sample. During step 1 a real follow-up drew HTTP 409 'the parent answer has an incomplete cohort' before any job ran; the sealed runner treated it as unknown server work and stopped every later admission, leaving 13 and 11 attempts unscored in the first guard and opus real-usage runs. The real-usage runner now scores such a pre-dispatch 4xx terminal projection as the product outcome it is and continues; both runs were repeated with the fixed runner and the repeats are the figures used. The holdout runner is untouched; the baseline real-usage run never met a 409.
• Method binding: every run's `method-manifest.json` hashes the worktree's scorer, runner, runtime, clock library, corpus and reconciliation by absolute path; the runner re-hashes them before starting. Sealed E5 files that differ from the goal-3 seal are the fixture control-39 correction landed by PR 98 (`phase0_fixture_data.py`, the two fixture scripts) and, for the holdout runner, the B2 budget-gate option; the frozen fixture database is unchanged (control-39 correction not applied).
• Concurrency: up to six runs overlapped on the 16-CPU host (load average 1 to 6); quality metrics are compared on identical scoring inputs; per-call wall time is advisory. The zai coding endpoint rate-limited the generator (HTTP 429, code 1302) once about six evaluations shared it: the first guard-head holdout run recorded 34 rate-limited generator calls (16 attempts scored as provider errors) and the second guard-head real-usage run 38 (15 attempts); a second guard-head holdout repeat started under the same load was stopped at 49 attempts after 4 rate-limited calls and set aside. The clean repeats ran with at most two glm-generation runs on the endpoint; every run's provider trace lists its failed calls, and the tables name which runs were contaminated. The step-0 holdout run recorded 2 rate-limited calls (1 attempt).
• Opus 5 safeguards: two generator calls in the opus real-usage repeat returned Claude Code's safeguard refusal (stop reason refusal, exit 1) on the security-probe question; they are scored as provider errors, which is the expected outcome for that case.
• Statistics: noise floor and stable-case rules as in Goal B; the paired interval is the Goal B 10,000-draw intent-family cluster bootstrap (seed 2026090611) of the opus repeats against the three Goal B control repeats (different code snapshot; the D1 glm runs are single runs).

6. Unverified

• Single runs: the step-0 baseline and each glm after-run are one run each; the provider is not deterministic at temperature 0 and seed 42 (Goal A), so per-case transitions between single runs carry the control's noise. Only the opus arm has repeats.
• The opus paired interval pairs against the Goal B control repeats, which ran on the pre-merge Goal B snapshot, not on main; the D1 glm runs share main's code but are single runs.
• Per-call wall time of the CLI generator was measured while other runs shared the host and includes Claude Code's start-up and request pipeline; it is not comparable with the API arms.
• The calendar guard's new proofs are pinned by real-PostgreSQL tests on synthetic rows around the DST transitions and the month boundary; correctness on the fixture is not production usefulness, and the frozen scorer marks many real answers unverified because the product adds columns.
• The fresh slice's expected SQL follows the ratified definitions and house defaults as recorded in Goal C; 7 of its 122 expected answers are empty on the fixture; it has not been run and its questions never entered the repository.
• Isolation of the CLI generator is proven per call from Claude Code's own transcript (tool list, tool-use blocks, working directory); network isolation of the child is not established by the CLI banner.

7. Evidence

• `docs/evidence/insights-goal-d1-2026-09-08/results.json`: every number in this report.
• `docs/evidence/insights-goal-d1-2026-09-08/runs/<label>/`: compact attempt records (outcomes, classes, timings; no question text, SQL or values), summaries, run receipts with snapshot revision, load averages and slot usage, method manifests, runtime contracts, provider traces (hashes, tokens, served model, exit codes, wall time, tool list) and budget logs.
• `docs/evidence/insights-goal-d1-2026-09-08/generator/`: the tools-disabled proof and the live in-process proof of the configured generator.
• `docs/evidence/insights-goal-d1-2026-09-08/fresh-slice/seal-receipt.json`: counts, families and hashes of the sealed slice.
• Evaluator-only (never committed): `/tmp/lore-goal3-eval/goal-d1/runs/` (full attempt records, server state, generated-text traces) and `/tmp/lore-goal3-eval/sealed-d1/` (the fresh slice).