Devtool Docs Audit — agentic-skills
review 2026-06-04 Third-pass documentation audit
Third-pass audit of developer-facing documentation for the agentic-skills shared skill library. Builds on the 2026-06-02 refresh. The prior audit's 5 backlog items remain unaddressed; this refresh deepens the analysis with new findings (prerequisites gap, skill template gap) and adds severity grading.
Table of Contents
Executive Verdict
P0: none. No critical documentation blockers. An experienced shell user can install and use the tool from existing docs.
P1 (3 findings):
- First-success route implicit — no named quickstart section guides a new user from clone to proof
- Troubleshooting fragmented — recovery information scattered across 5+ docs with no symptom-led page
- Prerequisites undocumented — jq is a hard dependency for pack operations, never listed upfront
P2 (4 findings): no skill authoring template, scattered script reference, missing team checklist, stale install.sh references in research
P3 (2 findings): no visual proof artifacts, static benchmark trends
Findings
First-Success Route Still Implicit
README explains the pieces (init.sh, pack install, CLI reload) but does not package a fastest proof path from clone to a visible working skill. A new user must read 3+ docs and mentally synthesize the sequence.
- README.md opens with
./init.shthen lists 15+ pack install commands with no guidance on which to try first rg "First successful|quickstart|getting started" README.md docs/*.mdreturns no results- The 2026-06-02 audit recommended a 6-step first-success section — not yet added
- docs/codex-workflow.md explains
$execbut is not linked from README onboarding
Recommendation: Add a README "First Successful Cycle" section before Repository Structure. Six steps: clone → init → pack install → CLI reload → run a skill → confirm artifact.
Gate: First-Success Route
Troubleshooting Fragmented
Recovery information exists across 5+ docs (README, packs.md, operating-modes.md, skills-reference.md, test-harness.md) but no symptom-led support path exists. A user hitting their first error must search multiple files.
- No
docs/troubleshooting.mdfile exists - Restart/refresh guidance: README.md lines 63-65
- Drift detection: docs/packs.md (doctor subcommand)
- Approval-packet failure: docs/operating-modes.md
- Validation commands: README.md lines 382-399
Recommendation: Create docs/troubleshooting.md with symptom-led table: symptom → likely cause → command → expected result → next fix. Minimum 7 rows covering skill visibility, drift, deps, versions, lock timeout, jq errors, and pinned archive issues.
Gate: Troubleshooting Format
Prerequisites Undocumented
Hard system dependencies are never listed before the user hits a failure. jq is required for all pack write operations. pnpm is required for tests. Claude Code or Codex must be installed. None are listed in a prerequisites section.
- pack.sh line 49:
die "jq required for write operations. Install with: brew install jq" - README.md line 385:
pnpm --dir tests test— pnpm assumed but not listed as prerequisite - No "Prerequisites" section in README.md
- Error messages are user-friendly when hit, but docs should prevent the failure path
Recommendation: Add a "Prerequisites" section to README.md before "Initialization" listing jq, Claude Code/Codex, and bash.
Gate: Prerequisites
No Skill Authoring Template
docs/skill-anatomy.md describes SKILL.md frontmatter fields and directory structure thoroughly, but provides no copyable skeleton. A first-time skill author must read 5+ docs and synthesize the format. The /skill-interview and /create-agentic-skill flows are the intended onboarding paths but are not surfaced in the docs.
- 1,349 SKILL.md files exist but all are full production skills, not minimal examples
- Required vs optional SKILL.md sections not explicitly listed
- No
docs/skill-template.mdexists /skill-interviewmentioned in skill-invocation-types.md but not in skill-anatomy.md
Recommendation: Add a minimal SKILL.md template with inline annotations to docs/skill-anatomy.md. Add a "First Time?" callout pointing to /skill-interview.
Script Reference Scattered
Common script commands are documented but spread across README.md, docs/packs.md, docs/skills-reference.md, docs/operating-modes.md, and docs/test-harness.md. No compact command index exists.
Recommendation: Create docs/scripts-reference.md or add a command index section to docs/skills-reference.md. Group by: Install/Global, Project Packs, Mode/Handoff, Skill Hygiene, Showcase/Benchmark.
Team Adoption Checklist Absent
Mechanics for team use are documented across docs/packs.md and docs/operating-modes.md, but no actionable checklist guides a team through initial setup. Checkout-path coupling, generated-root commit policy, and shipping workflow defaults require tacit knowledge.
Recommendation: Add a concise "Team Setup" checklist to docs/packs.md covering: checkout path convention, init per developer, commit policy (.agents/project.json committed, generated roots not committed), refresh after pull, CLI restart, and direct-to-primary workflow policy.
Stale install.sh References in Research
Six active devtool research artifacts still reference install.sh (renamed to init.sh). README and docs correctly use init.sh.
Recommendation: Run a reconciliation pass (sed -i '' 's/install\.sh/init.sh/g') with a dated note, or add historical-context caveat if research artifacts are treated as frozen snapshots.
No Visual Proof Artifacts
29 HTML alignment pages and 685KB of generated JS proof data exist, but no screenshots, diagrams, or recordings demonstrate the tool in action. README has no images. This is low priority — HTML pages serve power users, and the tool is primarily CLI-driven.
Benchmark Trends Static
docs/benchmark-results-matrix.md tracks 115+ skills with pass rates, latency, and cost, but is point-in-time only. No time-series or trend analysis across skill versions. Sufficient for current needs.
Progress Since Prior Audit (2026-06-02)
| Prior Backlog Item | Status | Notes |
|---|---|---|
| 1. Add README first-success and proof sections | Not started | Re-raised as P1-1 |
| 2. Add docs/troubleshooting.md | Not started | Re-raised as P1-2 |
| 3. Add script reference | Not started | Re-raised as P2-5 |
| 4. Add team adoption checklist | Not started | Re-raised as P2-6 |
| 5. Reconcile stale install.sh references | Not started | Re-raised as P2-7 |
All five prior items remain open. This audit adds 4 new findings: prerequisites (P1), skill template (P2), visual proof (P3), benchmark trends (P3).
Evidence Matrix
| Claim | Evidence | Inference | Confidence | Decision Impact |
|---|---|---|---|---|
| No P0 blocker | README covers init, pack install, restart, mode | Experienced users can install | High | No emergency rewrite |
| First-success implicit | No quickstart section; workflow split across 3+ docs | Activation requires synthesis | High | Add first-success route |
| Troubleshooting fragmented | Recovery info across 5 docs; no symptom table | Support is slower than needed | High | Add troubleshooting page |
| Prerequisites undocumented | jq hard-required; not in README | Users hit errors mid-workflow | High | Add prerequisites section |
| No skill template | Anatomy docs thorough; no skeleton | New authors must synthesize | Medium-high | Add minimal template |
| Script reference scattered | Commands in 5+ docs | Index would reduce friction | Medium-high | Add command index |
| Team checklist absent | Mechanics documented; checklist missing | Onboarding needs tacit knowledge | Medium-high | Add checklist |
| Stale installer refs | install.sh absent; research uses it | Could confuse readers | High | Reconcile or caveat |
Assumptions & Confidence
- High confidence: No first-success, troubleshooting, or prerequisites section exists in README or docs
- High confidence: jq is a hard dependency for pack operations and is not listed upfront
- High confidence: Prior audit's 5 backlog items remain unaddressed
- Medium-high confidence: A skill template would meaningfully help new contributors;
/skill-interviewpartially serves this role but is undiscoverable in docs - Medium confidence: Stale research references should be reconciled rather than caveated; caveating is an acceptable alternative if research artifacts are treated as historical snapshots
What Would Change These Conclusions
- If the target audience is exclusively the repo maintainer (single-user tool), P1 items drop to P2 — the maintainer already knows the workflow
- If research artifacts are treated as historical snapshots (never cited as current examples), the stale installer reference finding drops to P3
- If team adoption is not planned, P2-6 can be deferred indefinitely
Alternatives Considered
- Full documentation rewrite: Rejected. Docs are mostly accurate; highest value is navigation and activation packaging.
- Only fix stale references: Rejected as too narrow. Misses activation and support gaps.
- Separate website-first onboarding: Deferred. README and docs are the current adoption surface.
- Implement fixes in this audit: Rejected per staged workflow. Audit first, approve, then implement.
Recommended Backlog
| # | Priority | Item | Target File |
|---|---|---|---|
| 1 | P1 | Add Prerequisites section | README.md |
| 2 | P1 | Add "First Successful Cycle" section | README.md |
| 3 | P1 | Create symptom-led troubleshooting page | docs/troubleshooting.md |
| 4 | P2 | Add minimal SKILL.md template | docs/skill-anatomy.md |
| 5 | P2 | Add compact command index | docs/scripts-reference.md |
| 6 | P2 | Add team adoption checklist | docs/packs.md |
| 7 | P2 | Reconcile stale install.sh references | research/devtool-*.md (6 files) |
| 8 | P3 | Add README screenshot or architecture diagram | README.md |
Approval Gates
Evidence Coverage Gate
Is the evidence sufficient for these recommendations?
Scope & Non-Goals Gate
Is the audit scope correct? Should any findings be dropped or added?
Artifact Destination Gate
Where should the approved audit report be written?
Post-Approval Route Gate
After approval, what should happen next?
Compile Section
6 required questions unanswered