Session Triage: migrate Benchmark Failure

Durable triage report for $session-triage migrate benchmark failure. The fresh benchmark failure is verified, but the evidence does not justify editing the migrate skill contract directly. The benchmark did not reliably expose the installed skill to either runner, then failed on one route-preservation family and one literal phase-string evaluator family.

Table of Contents

Executive Verdict

Verdict

verified The user-identified benchmark failure is real: Claude and Codex each passed only 1 of 3 evaluated runs.

Primary diagnosis

The run cannot be treated as clean evidence against migrate itself because the raw sessions show the target skill was not visible to the benchmark agents.

Smallest durable next step

Route a narrow builder pass to the benchmark fixture/harness: skill visibility, semantic phase-structure recognition, and stricter final route prompting.

The current result is not a mass-rerun problem. Both agents created migration-plan.md and exited successfully, but four runs failed deterministic assertions. The strongest immediate evidence is setup-level: scripts/pack.sh which migrate reports that migrate is provided by the code-maintenance pack and is not installed, while both benchmark runners reported missing skill visibility during raw runs.

Scope and Sources

Target: the fresh migrate benchmark failure after the 2026-05-31 re-benchmark.

User-identified issue: $session-triage migrate benchmark failure.

Evidence sources inspected:

Verification

Question Verdict Evidence Interpretation
Did the benchmark fail? verified Curated report shows Claude 33.3% and Codex 33.3%, with no infrastructure-blocked runs. The observed failure is real and clean enough to triage; it is not a timeout-only artifact.
Did the agents create the required artifact? verified All six raw run JSON files list migration-plan.md in files, and all exited with code 0. The failure is not an artifact-creation failure.
Was the target skill visible? not reliably Claude failed runs state that no migrate skill is installed. Codex runs state that the available skill list did not include migrate. scripts/pack.sh which migrate says the code-maintenance pack is not installed. This makes the benchmark partly a prompt-compliance test rather than a true target-skill test.
Is there mirrored migrate contract drift? not found Claude and Codex migrate contracts are both v0.1 and share the same core workflow: audit, read migration guide/request, plan, approval, batched migration, verification. No direct skill mirror patch is justified by this evidence.
Are the assertion families meaningful? mixed Claude failed on missing $exec route. Codex failed on literal phases, even when artifacts had Phase 1, Phase 2, and later phase headings. Route failure is generated-output noncompliance under the fixture prompt; phase failure is an overly literal rubric for a semantically valid plan shape.

Timeline

  1. 2026-05-20/21: earlier Phase 41 records show migrate failing from route/artifact and infrastructure patterns, before the route-guidance fixture remediation.
  2. 2026-05-31: the re-benchmark ran pnpm --dir tests bench --skill migrate --agent both --runs 3 --chunk-size 3 --pause 0 after explicit spend approval.
  3. 2026-05-31: curated report recorded 1/3 hard-assertion pass for both Claude and Codex, no infrastructure blocks, and route/phase assertion misses.
  4. Session triage: raw JSON inspection found all six runs wrote migration-plan.md, while failing runs showed two distinct families: Claude substituted shell commands for $exec, and Codex used numbered phase headings without the exact plural word phases.
  5. Session triage: installed-skill checks found no local .codex/skills/migrate/SKILL.md or .claude/skills/migrate/SKILL.md, and scripts/pack.sh which migrate reported the provider pack as not installed.

Root Cause

Primary cause: benchmark skill visibility/setup gap

The fixture prompt says, "You have the migrate skill installed." The benchmark environment did not make that true for the observed runs. That gap matters more than the individual wording misses, because it means the benchmark did not reliably exercise the actual packs/code-maintenance skill contract.

Secondary cause: literal phase-string assertion

The fixture puts "phases" in expectedIncludes. The hard assertion and quality fact criterion then require that literal token in the artifact. Codex run #0 and #2 created structured multi-phase plans using headings such as ## Phase 1: Inventory Current Behavior and ## Phase 2: Introduce Grouped Setup Modules, but failed because the artifact did not contain the exact plural word. That is a benchmark false negative family.

Secondary cause: route-preservation variance

Claude run #0 and #2 wrote useful migration plans but ended with concrete shell commands instead of the fixture-required Recommended next command: $exec. The existing prompt says to end with that line, so this is generated-output noncompliance under an adequate but still improvable fixture prompt. Strengthening the prompt and requiring the exact final route would reduce this variance.

Evidence Matrix

Claim Evidence Inference Confidence Assumption Status Decision Impact
The benchmark failure is verified. benchmark/test-migrate-2026-05-31.md and both report.json files show 33.3% pass rates, 3 evaluated runs per agent, and no infrastructure-blocked runs. The triage should diagnose deterministic failure families rather than rerun immediately. High Evidence-backed Do not spend on a mass rerun before fixing the setup/evaluator.
The benchmark did not reliably expose the actual migrate skill. Raw Claude and Codex sessions report missing migrate skill visibility. scripts/pack.sh which migrate says migrate comes from code-maintenance and is not installed. Local installed-skill path search returns no migrate/SKILL.md. The clean-run benchmark is not a reliable test of the active pack-local skill contract. High Evidence-backed, with harness-internals not fully traced Fix benchmark skill exposure or fail fast on missing skill before changing the target skill.
Codex phase failures are false negatives. Failed Codex artifacts use ordered phase headings, including ## Phase 1 and ## Phase 2, while failing only Output includes phases. The evaluator requires a literal word instead of semantic phase structure. High Evidence-backed Replace the literal include with a phase-structure pattern or predicate and add retained positive/negative tests.
Claude route failures are output variance against the fixture route. Failed Claude artifacts end with shell commands; the fixture requests Recommended next command: $exec. Claude run #1 obeyed and passed. The skill contract is not the owner; the fixture needs stronger final-route wording and stricter final-route assertion. Medium-high Evidence-backed Harden the benchmark prompt/evaluator instead of editing migrate to emit benchmark-specific prose.
No mirrored migrate skill drift was found. Both active pack contracts are v0.1 and have matching workflow intent: audit, read migration guidance/request, plan, approval, batched execution, verification. There is no current reason to version-bump the skill itself for this benchmark failure. Medium-high Evidence-backed Route to fixture/harness owner first.

Responsible Contract Gap

Owner: benchmark setup and harness surfaces, not the active migrate skill contract.

The active packs/code-maintenance/{claude,codex}/migrate/SKILL.md contracts should remain unchanged unless a later rerun with actual skill visibility still proves a target-skill behavior gap.

Recommended Fix

Recommended next skill after approval: $targeted-skill-builder migrate benchmark fixture skill visibility and phase-route evaluator

Concrete behavior change to implement

  1. Make the benchmark either expose the actual code-maintenance migrate skill to both agents or fail fast/classify the run as setup invalid when the prompt claims the skill is installed but the runner cannot see it.
  2. Replace the literal "phases" expected include with a semantic phase-structure check. Accept ## Phases and multi-heading structures such as ## Phase 1 plus ## Phase 2. Reject generic unordered plans that do not show staged migration sequencing.
  3. Strengthen the route instruction to say the final line must be exactly Recommended next command: $exec and that shell commands must go in the plan body, not in the final handoff line.
  4. Use requireExactFinalRecommendedRoute: true or an equivalent final-route assertion if the prompt claims exact final-line behavior.
  5. Add layer1 retained fixtures for the positive Codex phase-heading shape, the negative generic-plan shape, the exact $exec route, and a shell-command route failure.
  6. Add a missing-skill visibility guard or retained test so output containing "could not find migrate skill" cannot be reported as a normal target-skill quality failure.

Candidate fixture wording

You have the migrate skill installed. Read migration-request.md and write migration-plan.md with a staged migration plan, file changes, compatibility risks, validation, rollback, and Next command.
Use either a "## Phases" section or numbered "Phase 1", "Phase 2", ... headings.
Put shell commands inside the plan body when useful.
The final line must be exactly: Recommended next command: $exec

Alternatives Considered

Alternative Decision Reason
Mass-rerun the paid benchmark now. Reject The raw sessions already show setup and evaluator issues. Rerunning would spend money without changing the failure mechanism.
Edit migrate/SKILL.md to require the literal word phases and exact $exec. Reject The target skill was not visible to the runners, and benchmark-specific wording would narrow a general migration skill.
Patch only the phases assertion. Insufficient It fixes a real false negative but leaves the missing-skill setup gap and Claude route variance intact.
Install the code-maintenance pack locally and rerun once. Diagnostic only Useful as confirmation after a harness fix, but not a durable repository-level benchmark contract.

Validation Plan

For the follow-up builder patch

Current triage verification already run

Confidence and Gaps

Conclusion Confidence Why What Could Change It
The failure is verified. High Curated report and persisted raw JSON align. A different persisted report source would need to contradict the current run IDs.
Skill visibility is a primary setup gap. High Raw runner output plus local pack resolution both show missing installed skill state. A harness trace proving the skill was injected despite those messages would narrow this to generated-output noise.
Codex phase failures are false negatives. High Artifacts show multiple phase headings and staged migration content. If the benchmark intentionally requires the exact heading ## Phases, the fixture should say that explicitly.
Claude route failures should be fixed in fixture/evaluator, not migrate. Medium-high The prompt already asks for $exec, and one Claude run passed. The skill was not visible. A rerun with visible skill that still substitutes shell commands could justify a target-skill output-contract tweak.

Broad $analyze-sessions is not needed for this decision. The relevant recurrence evidence is already local: recent tasks/todo.md entries record prior migrate route and phases failures, and tasks/lessons.md already captures route-guidance lessons.

Review Gates

Evidence Coverage Gate

Is the evidence sufficient to treat the failure as a benchmark setup/evaluator issue before editing the target skill?

Scope Gate

Should this triage explicitly avoid changing packs/code-maintenance/{claude,codex}/migrate/SKILL.md until a rerun proves a skill-contract gap?

Post-Approval Route Gate

Which next route should own the follow-up after this report is approved?

Validation Gate

After the builder patch, how should paid benchmark validation proceed?

Compile Feedback and Answers

Use feedback-only YAML to request revisions before answering every gate. Use final answers when the report is ready for agent review.

Required gates remaining: 4