Real-Repo Testbed — Phase 0 Decisions

Context Recap

SWE-bench's environment transfers to this project but its grading does not. SWE-bench grades a code patch by running hidden unit tests (FAIL_TO_PASS + PASS_TO_PASS); our skills emit markdown artifacts, run manifests, and routing decisions — there is no test suite to flip green. So the borrow is narrow: drop the agent into a pinned, realistic repo snapshot (a new repo-fixture setup tier) for real context/discovery complexity, then grade with layered artifact grading instead of pytest:

(a) deterministic state assertions on produced files   [backbone — gates the run]
(b) reference-set comparison (gold must-hit set, set-wise, paraphrase-tolerant)
(c) rubric-checklist + jury LLM judge  [open-ended residue only]
(d) pass^k reliability wrapper                           [wraps all of the above]

The through-line across all four decisions below: defer the expensive/risky half, prove the cheap deterministic half first. Each gate marks a recommended option; the four standing options (Other / None of the above and Need clarification) carry a notes field so you can override or push back.

Q1 — Snapshot Sourcing & Sanitization

The tension. Self-sourced snapshots (this repo, bismarck-v5) are the true distribution these skills run against — they actually exercise the planning surface (tasks/, research/, alignment/, the manifest conventions) that almost no public repo carries. But they need a sanitization pass for secrets and a provenance record (origin, commit SHA, capture date, license, canary string), and they would be vendored under tests/benchmarks/repos/<slug>@<shortsha>/. Synthesized/fake repos avoid the consent and secret-leak surface entirely but trade away ecological validity.

My lean (A). Self-source from this repo first — it is the highest-realism snapshot, already in front of us, and a sanitization pass on a repo you own is low-risk. Add bismarck-v5 only if Phase 0 surfaces a need for a second domain. This keeps the consent/sanitization surface to one repo for the pilot.

Q2 — LLM-Judge Budget

The tension. Layer (c) — the rubric-checklist + disjoint-family jury LLM judge — is the only grader that can score the open-ended residue (is the plan coherent? does the research cover the asked dimensions?). But it adds per-run model cost and, done honestly, requires a human-labeled calibration set with reported judge κ before its numbers can be trusted. Layers (a) state assertions + (b) reference-set + (d) pass^k are deterministic, cheap, and un-gameable by eloquence.

My lean (A). Defer it. Ship the deterministic backbone first; it carries the most weight and gates the run anyway. Add the rubric/jury judge in Phase 1 alongside the calibration harness that reports κ, so the judge arrives validated rather than trusted on faith. "Never" is too strong — the open-ended residue is real and deterministic graders cannot reach it forever.

Q3 — Scope Of v1

The tension. Designing the snapshot loader and grading layers for the whole product-design pack up front maximizes reuse but front-loads a lot of speculative generalization onto an unproven mode. Hardcoding everything for eval-ideas ships fastest but bakes in a rewrite the moment a second skill arrives.

My lean (A). Pilot eval-ideas only — that is where the originally requested run belongs — but build the snapshot loader and graders as reusable, generalization-aware modules from day one. You get a real first data point without committing to pack-wide authoring, and the second skill is a config, not a rewrite. The middle path beats both extremes.

Q4 — Task-Count Ambition

The tension. Statistical n is tasks × runs; ≥30–50 screened task instances per skill is the defensible target, below which Wilson CIs are ±30pt no matter how many reruns you do. But authoring 30–50 screened tasks is real, slow work — and gating Phase 0 on it risks never shipping the capability demo that proves the mode works at all.

My lean (A). Ship Phase 0 small-n, explicitly labeled "capability demo, not statistically definitive," so the mode earns its keep before you invest in task count. The honest label is what makes a small-n run acceptable; the ≥30–50 investment then becomes a deliberate Phase 2 decision rather than a blocker.