Skill Gaps & Manual Agent Asks — Session History Audit
alignment_status: review
Full-history audit of Claude Code and Codex usage (12,300 user messages, Dec 2025 – Jun 2026) answering: where are the gaps in our skills, and what is still asked of agents manually? Headline: the biggest historical manual-ask families (commit/push, phase execution) were already eliminated by /ship and the exec-loop pack; the strongest remaining gap is .env scaffolding/secrets handoff (23 recent asks, no skill exists), followed by three convention-level gaps (findings verification, fix-plan routing, research freshness).
- 1. Overview Stats
- 2. Token & Cost Check
- 3. Skill Command Usage
- 4. Closed Gaps (skills already working)
- 5. Open Gaps — Recent Manual Asks
- 6. Buckets Judged Not Skill Gaps
- 7. Ranked Recommendations
- 8. Highest-Impact View
- 9. Evidence Matrix
- 10. Confidence & Assumption Register
- 11. Glossary Additions Gate
- 12. Artifact Destination & Proposed File Changes
- 13. Post-Approval Route
- 14. Compile (bottom)
1. Overview Stats
| Metric | Claude | Codex |
|---|---|---|
| Date range | 2025-12-13 → 2026-06-10 | 2026-01-09 → 2026-06-10 |
| Messages | 10,208 | 2,092 |
| Sessions | 3,095 | 696 |
Top projects by message volume
| Project | Messages | Share |
|---|---|---|
| experiments/metternich-engine | 1,322 | 10.7% |
| apps/bismarck-ai/v0.3/AWS | 1,271 | 10.3% |
| poke/monorepo | 694 | 5.6% |
| apps/bismarck-ai/v0.3/monorepo | 694 | 5.6% |
| web/dev/bismarck-ai/bismarck-v0.4 | 670 | 5.4% |
| tools/dev/agentic-skills | 621 | 5.0% |
| web/dev/b4 | 423 | 3.4% |
| poke/dev/poke-productivity-suite | 370 | 3.0% |
| loadoutworks (apps + web) | 699 | 5.7% |
Monthly trend (messages)
| Month | Claude | Codex |
|---|---|---|
| 2025-12 | 663 | — |
| 2026-01 | 3,640 | 335 |
| 2026-02 | 1,577 | 14 |
| 2026-03 | 2,208 | — |
| 2026-04 | 1,136 | 1,359 |
| 2026-05 | 733 | 228 |
| 2026-06 (partial) | 251 | 156 |
2. Token & Cost Check
- Codex: 757 of 764 rollout files carried
token_countevents. Final cumulative session totals: total 2,020,107,036 · input 2,007,193,471 (cached 1,889,405,696) · output 12,913,565 (reasoning 4,319,262). Aggregated from finaltotal_token_usagesnapshots per session — no cumulative double-counting. - Claude:
history.jsonlcarries no usage metadata — token coverage gap for the Claude source. - Cost: unavailable. No cost fields in either log, no pricing table provided, no provider pricing verified this run. Per skill constraints, no estimate is made.
- Model attribution: not recoverable from
session_metain this parse — model-level breakdown not stated.
3. Skill Command Usage (all-time)
| Command | Uses | Command | Uses |
|---|---|---|---|
| /ship | 857 | /ship-then-plan | 31 |
| /clear | 552 | /workflow | 28 |
| /ship-end | 266 | /investigate | 21 |
| /resume | 166 | /exec | 20 |
| /usage | 162 | /expert-review | 16 |
| /sync | 124 | /session-triage | 14 |
| /run | 121 | /analyze-sessions | 10 |
| /commit-and-push-by-feature | 91 | /pack | 69 |
Long tail: /plan-interview 58, business/research skills (icp, gtm, journey-map, positioning, monetization) in single digits each. Codex $run invocations dominate Codex skill use (360+ across skill-path variants), with $ship/$sync next — Codex adoption is concentrated on run/ship/sync.
4. Closed Gaps — Manual Asks Skills Already Eliminated
The largest manual-ask families in history died out as skills were adopted — direct evidence the skill system works.
| Manual ask family | Volume | Timeline | Replaced by |
|---|---|---|---|
| "update docs, commit and push" / "commit this" / "push it" | ~552 | Jan 274 → Feb 206 → Mar 72 → Apr+ 0 | /ship (857), /ship-end (266), /commit-and-push-by-feature (91) |
| "implement phase N step N" / "phase N.N @docs/…plan.md" | ~279 | Jan 137 → Feb 111 → Mar 31 → Apr+ 0 | /ship-then-plan, /exec, exec-loop pack |
| "deploy to staging" | 59 | Jan 40 → Feb 19 → Mar+ 0 | release-ops/deploy skill (now with deployment-history tracking); bismarck project also wound down — confounder noted |
| "create a directory to house our new package" | 44 | Jan–Feb → Apr+ ~0 | monorepo pack (scaffold, mono-*) |
Verified follow-through: the user's 2026-04 ask — "update the deploy skill so that it maintains a historical record of what commits were part of what deployments" — was implemented: packs/release-ops/claude/deploy/SKILL.md now declares deployment history tracking in its description and body.
Q3 (required) — Do you confirm the closed-gap reading?
5. Open Gaps — Recent Manual Asks (Apr 1 – Jun 10, 2026)
Recent window: 3,863 messages, 1,892 freeform. Counts below are exact pattern matches with real examples from history.
5.1 .env scaffolding & secrets handoff — 23 recent · 69 all-time · no skill exists
Verified by inventory scan: no env-related skill in any pack. The pattern has a stable, automatable shape: derive required variables (from .env.example or code), scaffold .env.local with stubs, open it in VS Code (WSL-aware), never read secret values back. The repeated "don't look at it" warnings show the manual flow carries a recurring secret-exposure risk.
5.2 False-positive verification of findings — 10 recent · 15 all-time
expert-review already has a built-in false-positive filter step (verified in its SKILL.md), but these asks follow ad-hoc audits, not skill runs.
5.3 Findings → fix-plan handoff — 11 recent · 17 all-time
After any audit/review output, the user manually requests the fix plan.
5.4 Research data freshness — explicit recurring correction (May 2026)
Grep across packs: only pmf-engine and youtube-portfolio carry recency language today.
5.5 Document consistency validation — 5 recent · 12 all-time
docs-health pack (hygiene, reconcile-dev-docs) covers this but is not in this repo's enabled_packs (verified in .agents/project.json) and likely absent in the other projects where the asks occurred. Coverage exists; adoption is the gap.
5.6 Open-in-VS-Code ergonomics — 22 recent
Heavily overlaps with 5.1 — most are .env opens. Covered if the env skill includes the WSL-aware open step already documented in CLAUDE.md.
5.7 Codex-side observations
- 51 recent "I agree with your recommendations" messages are Codex-only — approval steering at recommendation gates. Inherent to gate design, not a gap; a Codex-side compiled-answer flow would compress it if desired.
- 38 recent manual git-ops asks on Codex vs 16 on Claude ("yes please and then commit it", "can you track and push the sync file") — Codex skill adoption lags outside run/ship/sync.
6. Buckets Reviewed and Judged NOT Skill Gaps
| Bucket | Recent freq | Why not a gap |
|---|---|---|
| Design/UI tweak feedback | 45 (all Claude) | Iterative taste feedback on animations/layout — inherently conversational; animation-design-planner + verify/run support it |
| Video/recording/content | 59 | guided-walkthrough + youtube-ops packs own these flows; asks were in-flow direction, not repeated workflow shapes |
| Research/investigate | 102 | Covered by /investigate, code-debug pack, deep-research, autoresearch |
| "What's next / status" | 22 | Covered by codebase-status, afps-status, roadmap skills |
| Run/restart dev server | 6 | Covered by /run; low frequency |
7. Ranked Recommendations
| # | Pattern | Recent freq | Type | Suggested action | Owner surface | Validation |
|---|---|---|---|---|---|---|
| 1 | .env scaffolding & secrets handoff | 23 (69 all-time) | Skill | env-setup: scaffold .env.local stubs from .env.example/code, WSL-aware VS Code open, never read secrets | gitops or repo-maintenance pack | layer1 contract test (no secret reads; stub file produced) |
| 2 | False-positive verify after ad-hoc audits | 10 | Standing instruction | Findings-verification convention: self-verify pass before presenting findings | CLAUDE.md / quality-gate contract doc | benchmark smoke on an audit prompt |
| 3 | Findings → fix-plan handoff | 11 | Standing instruction | Extend shipping-contract routing: audits end with a plan-to-fix offer | CLAUDE.md shared conventions | skill-next-step-contracts doc check |
| 4 | Research data freshness | recurring correction | Standing convention | Freshness contract bundled into research skills (current-year sources, date-stamped claims, stale-evidence flags) | docs/ + upgrade script | spot-check bundled skill text |
| 5 | Doc consistency validation | 5 | Pack adoption | /pack install docs-health in affected projects | docs-health pack | existing pack tests |
Q4 (required) — Recommendation 1: build the env-setup skill?
Q5 (required) — Recommendations 2–4 (findings-verification, fix-plan routing, research freshness conventions)?
Q6 (required) — Recommendation 5: adopt docs-health pack where doc-validation asks recur?
8. Highest-Impact View (avoided manual prompts)
- Historical win (already shipped): /ship + exec-loop eliminated ~830 manual commit/push/phase prompts — the single largest automation in the dataset.
- env-setup skill — ~23 prompts/quarter avoided, plus removes a recurring secret-exposure risk surface.
- Findings-verification convention — ~10/quarter avoided; improves audit quality everywhere at once.
- Fix-plan routing — ~11/quarter avoided; pure convention change, zero new code.
- Research freshness contract — prevents repeated stale-data corrections across all research skills.
9. Evidence Matrix
| Claim | Evidence (source · observation · date) | Inference | Confidence | Decision impact |
|---|---|---|---|---|
| Manual commit/push asks are eliminated | history.jsonl · 552 matches Jan–Mar, 0 from Apr; /ship 857 uses starting Mar · Dec 2025–Jun 2026 | Drop coincides exactly with /ship adoption month | High | No action needed on git-flow skills |
| Manual phase-exec asks are eliminated | history.jsonl · 279 matches Jan–Mar, 0 from Apr · same window | exec-loop/plan skills replaced typed phase invocations | High | No action needed |
| .env handoff is an uncovered recurring workflow | history.jsonl · 23 exact recent matches, 69 all-time; pack inventory scan shows no env skill · Apr–Jun 2026 | Stable repeated shape + explicit safety concern → skill-shaped | High | Approve/reject building env-setup (Q4) |
| Ad-hoc audits lack a verification pass | history.jsonl · 10 recent "false positives" follow-ups; expert-review SKILL.md already has the filter step · Apr 2026 | Gap is in ad-hoc behavior, not the skill — convention-level fix | Medium-High | Approve/reject convention (Q5) |
| Findings→fix-plan handoff is manual | history.jsonl · 11 recent "create a plan to fix" follow-ups · Apr–Jun 2026 | Routing convention would absorb these | Medium-High | Approve/reject convention (Q5) |
| Research skills lack a freshness rule | history.jsonl · explicit user correction May 2026 ("very stale data", "multiple instances"); grep shows recency language in only 2 skills | User explicitly says this recurs; coverage scan confirms absence | High (explicitly stated by user) | Approve/reject convention (Q5) |
| Doc-validation asks are an adoption gap, not a coverage gap | .agents/project.json · docs-health absent from enabled_packs; pack contains hygiene + reconcile-dev-docs · Jun 2026 | Skills exist; they were not installed where asked | Medium (other projects' pack state not directly inspected — inferred) | Approve/reject adoption rec (Q6) |
| Deploy-history ask was already implemented | packs/release-ops/claude/deploy/SKILL.md · description + body declare deployment history tracking · Jun 2026 | April user request landed | High | None — closes the loop |
Q1 (required) — Is the evidence base sufficient for these recommendations?
Q2 (required) — Could the missing context change the recommendations? (Claude token data, Codex model attribution, other projects' pack state were not parsed.)
10. Confidence & Assumption Register
| Conclusion | Status | What would change it |
|---|---|---|
| env-setup is the top remaining skill gap | Evidence-backed (exact counts, inventory scan) | An existing env skill in an uninspected location, or the asks stopping naturally |
| Closed-gap families stay closed | Evidence-backed (3 consecutive zero months) | Recurrence in future history |
| "Deploy to staging" closure credit | Provisional — confounded by bismarck wind-down | A new deploy-heavy project re-surfacing the asks |
| Doc-validation is an adoption gap | Provisional — other repos' pack state inferred, not inspected | Inspecting .agents/project.json in those repos |
| Codex skill adoption lags Claude | Evidence-backed (manual git-ops 38 vs 16; command mix) | Codex history showing broader $-skill use |
| Intent-bucket counts | Exact for shown patterns; buckets not mutually exhaustive (first-match keyword classification) | — |
Alternatives considered
- One mega "audit-followups" skill (verify + fix-plan in one): rejected — the two behaviors apply at different moments and conventions reach ad-hoc work that skills don't.
- Building a doc-consistency skill: rejected — docs-health already covers it; the gap is installation.
- A Codex approval-YAML flow for the 51 "I agree" messages: deferred — gate steering is by design; surfaced as an observation, not a recommendation.
Source coverage gaps
- Claude token usage and all dollar costs unavailable (no usage metadata in history.jsonl; no pricing basis).
- Codex model/provider attribution not recovered; token totals are aggregate only.
- Codex compact history lacks project paths; 764 sessions mapped via rollout
cwd, remainder "(unknown)". - Claude full transcripts (~/.claude/projects/*) were not parsed — history.jsonl user prompts only.
Downstream implications
- Approving Rec 1 routes to skill-dev pack (installed ✓ per
.agents/project.json) via targeted-skill-builder. - Approving Recs 2–4 means edits to shared convention docs (CLAUDE.md template source / quality-gate contract / a new freshness convention doc + bundler), which propagate to many skills.
- Approving Rec 5 is a per-project action, not a repo change here.
11. Glossary Additions Gate
Proposed for research/glossary.md (flat repo — single parent glossary). Only approved terms are appended during the confirmed-page write step.
| Term | Definition | Source | Category |
|---|---|---|---|
| manual ask | A recurring typed user instruction that a skill could perform end-to-end (e.g. "commit this and push") | analyze-sessions skill-gaps report | workflow |
| steering prompt | A short (≤30 char) confirmation or direction message ("y", "continue", "push it") that drives an agent between skill steps | analyze-sessions skill-gaps report | workflow |
Q7 (required) — Term: "manual ask"
Q8 (required) — Term: "steering prompt"
12. Artifact Destination & Proposed File Changes
On approval (Stage 3), the proposed canonical writes are:
- Write the approved report to
research/analyze-sessions-skill-gaps-manual-asks.md(flat research path, matching existing flat layout). - Append approved glossary terms to
research/glossary.md(create with standard header if missing). - Archive the working packet
research/_working/preliminary-analyze-sessions-research.mdtodocs/history/archive/YYYY-MM-DD/HHMMSS/…and remove the active copy. - Convert this page to
confirmedand updatealignment/index.html.
Already written this run (review-stage artifacts): this alignment page, the working packet, the prompt-history log prompts/analyze-sessions/skill-prompt-20260610-020338-skill-gaps-manual-asks.md, and the index entry.
Q9 (required) — Approve the canonical artifact destination and file-change scope above?
13. Post-Approval Route
If Recommendation 1 is approved, the natural follow-up is the skill-dev pack's builder, parameterized with the gap (owner surface: gitops or repo-maintenance pack; validation: layer1 contract test asserting no secret reads and stub-file output). skill-dev is installed per .agents/project.json.
Q10 (required) — Route after this page is confirmed?
14. Compile
Feedback-only path: select emphasize / 👎 / clarify on any section above, add notes, then compile feedback YAML here (or locally under that section) — available even with gate questions unanswered.
Final approval path: answer all 10 required questions, then compile answers.