Idea Scope Brief — Task ROI
1. Scope Resolution & Branch Relationship
from repo The repo is currently in flat single-product mode: all of research/*.md describe one concept — the cost-intelligence platform aimed at the Platform Engineering Lead ICP (locked 2026-05-24, hidden-multiplier cost modeling).
from user This brief introduces a second, sibling product path aimed at a different beneficiary — project / product managers — with a different core job: deciding which tasks justify agentic token spend vs. hand-coding, plus auditing model performance.
| Question | Your answer |
|---|---|
| Relationship to cost-intelligence path | Sibling product path — both stay active; separate research tracks |
| Meaning of "auditing model performance" | Keep all three angles open (quality-vs-cost, model comparison, spend efficiency); research narrows to the most promising pain point |
| Source of "task" unit & quality signal | Not sure yet — flagged as a top unknown to test in research |
Because two concepts now coexist, this run will (on approval) create research/.progress.yaml recording both as product paths, rather than merging this into the existing brief.
cost-intelligence path.2. Concept Identity & Slug
inferred Proposed working slug: task-roi — it captures both pillars (return on a unit of agentic task spend, both forward-planning and backward-audit). Output paths derive from this slug, so it's worth locking now (it can be renamed later, but renaming moves files).
| Candidate slug | Reads as | Trade-off |
|---|---|---|
task-roi | Return on per-task agentic spend | Covers plan + audit; slightly abstract |
agentic-roi | ROI of agentic coding | Clear, but emphasizes coding over PM planning |
spend-planning | Planning token spend | Misses the model-audit pillar |
velocity-audit | Auditing AI-assisted velocity | Emphasizes audit over forward planning |
research/<slug>/ output paths)?3. Idea Assumptions Manifest
Each assumption is tagged by source. These are hypotheses, not findings — downstream research (starting with /icp) owns evidence.
| Dimension | Assumption | Source |
|---|---|---|
| Concept summary | A PM-facing tool answering two linked questions: which tasks are worth agentic token spend vs. hand-coding (forward planning) and did the model perform well on what we spent on (backward audit) | from prompt |
| Problem | PMs have no principled way to decide where agentic spend pays off; can't see which tasks burned tokens for low-quality output; planning is gut-feel, cost is a post-hoc surprise | inferred |
| Beneficiary | Project / product / delivery managers — the people planning work and defending budget — distinct from the Platform Eng Lead of the existing path | from prompt |
| Category | AI delivery/velocity planning + model-output QA; adjacent to FinOps-for-AI but framed around task ROI and output quality, not infra attribution | inferred |
| Value wedge | Per-task "agent it vs. hand-code it" decision support combined with model-performance auditing — spend × quality on one task ledger | inferred |
| Constraints | Solo dev, ship-fast; read-only provider data preferred — but quality auditing may need signals billing APIs don't expose; multi-provider | from repo |
| Non-goals | Not a prompt-engineering tool; not a request-path proxy; not a replacement for the cost-intelligence path | inferred |
| Riskiest unknown | Where the "task" unit and quality signal come from — likely needs task/outcome telemetry (PR merged/reverted, review churn) beyond read-only billing | inferred |
Sections 4–12 below are the proposed content of research/task-roi/idea-brief.md, rendered in full for review. Nothing is written until approval.
4. Summary
A planning-and-audit tool for project/product managers that turns LLM spend from an infrastructure line item into a per-task ROI decision. It answers two linked questions:
- Forward (plan): Which upcoming tasks are worth spending agentic tokens on, and which should be hand-coded?
- Backward (audit): Did the model actually perform well on the tasks we spent on — was the spend worth it?
This is a sibling product path to the existing CalcLLM cost-intelligence platform. Where cost-intelligence serves Platform Engineering Leads with hidden-multiplier cost attribution, this path serves delivery managers with task-level spend-vs-quality decision support.
5. Problem Hypothesis
Hypothesis Product/delivery managers are now accountable for AI-assisted velocity but have no principled way to decide where agentic coding spend pays off. Decisions about "should we let the agent do this or hand-code it?" are made on gut feel, and there's no feedback loop telling them whether past spend produced acceptable work.
- No build-decision rubric. No shared basis for "agent vs. hand-code" — it varies by engineer mood and deadline pressure.
- Spend ≠ outcome visibility. A task can burn large token spend and still be reverted; PMs don't see that linkage.
- Planning blind spot. Sprint/roadmap planning can't price agentic work because nobody knows the per-task cost or hit-rate.
- Quality is invisible until late. Whether the model "did a good job" surfaces only in code review or production, not at spend time.
6. Beneficiary Hypothesis
from prompt Primary beneficiary: project / product / delivery managers who plan work and defend budget. Distinct from the existing path's Platform Engineering Lead — these are people who think in tasks, tickets, sprints, and deliverables, not gateways and attribution.
| Candidate persona | Why they might care | Confidence |
|---|---|---|
| Product / project manager | Owns roadmap & estimates; needs to price agentic work and justify it | Medium |
| Eng manager / team lead | Owns delivery quality; wants to know if agentic output is trustworthy per task type | Medium |
| Tech lead / staff eng | Sets the "agent vs. hand-code" norms the team follows | Low |
Which of these is the true buyer/user is exactly what /icp resolves — this brief does not select it.
7. Product Category Guess
inferred AI delivery / velocity planning + model-output QA. Sits adjacent to FinOps-for-AI (the cost-intelligence path's category) but is framed around task ROI and output quality rather than infrastructure cost attribution. Think "the planning & scorecard layer for agentic work" rather than "the billing microscope."
8. Value Wedge
Spend × quality on one task ledger. Pure cost tools show what you spent; pure eval tools show output quality. Neither tells a PM "this kind of task is worth agentic spend and this kind isn't." The wedge is joining the two at the task unit for a planning audience.
The "auditing model performance" pillar spans three angles kept deliberately open — research will pick the most promising one:
| Angle | Question it answers | Data difficulty |
|---|---|---|
| Output quality vs. cost | Did the tokens we spent produce acceptable work (merged, low-rework)? | High — needs outcome telemetry |
| Model / provider comparison | Which model is best/cheapest for which task type? | High — needs labeled outcomes per model |
| Spend efficiency | Where is waste (retries, oversized context, redundant runs) at task granularity? | Medium — partly from usage data |
Note: the comparison angle crosses the existing path's stated non-goal ("Not a model evaluation or benchmarking platform"). That's intentional for this branch — see Non-Goals gate.
9. Constraints
- from repo Solo developer (George), ship-fast mindset.
- from repo Existing path prefers read-only provider data (no proxy). Tension: task-level quality auditing likely needs outcome telemetry that read-only billing APIs don't expose.
- from repo Multi-provider expectation (OpenAI, Anthropic, Google at minimum).
- inferred Must map spend to a "task" unit — requires integration with a work-tracking or agentic-tool boundary (ticket, PR, agent session).
- inferred No monetization model defined for this path yet.
10. Non-Goals
- Not a prompt-engineering tool — does not write or optimize prompts.
- Not a request-path proxy / gateway — does not sit inline in LLM traffic.
- Not a replacement for the cost-intelligence path — that path keeps serving Platform Eng Leads.
- Crosses one prior non-goal on purpose: the existing brief says "not a model evaluation/benchmarking platform." For this branch, lightweight model-performance auditing/comparison is in scope (pending the wedge decision above).
11. Assumptions & Unknowns
Assumptions
- PMs (not just engineers) want and will use a task-level spend/quality tool.
- A meaningful "task" boundary exists that spend can be attributed to.
- Output quality can be approximated from observable signals (merge/revert/review churn) without manual grading.
Riskiest unknowns
| # | Unknown | Impact |
|---|---|---|
| 1 | Task unit & quality-signal source. Where does the "task" boundary and the quality signal come from — VCS/PRs, agentic-tool session logs, or manual PM tagging? You marked this "not sure yet" | Blocks core value |
| 2 | Read-only data sufficiency. Can quality be inferred without instrumenting the request path or dev pipeline? | Shapes architecture |
| 3 | Buyer identity. Is the PM the buyer, or does eng leadership own this and the PM just consumes it? | Shapes GTM |
| 4 | Overlap with cost-intelligence. Where do the two paths share a spine vs. diverge — does this cannibalize or complement? | Shapes portfolio |
| 5 | Which audit angle wins. Quality-vs-cost, comparison, or efficiency — which is the sharpest pain? | Shapes wedge |
12. ICP Readiness
Status: Ready for /icp as a new product path, scoped to research/task-roi/.
Inputs /icp should use:
- Problem hypothesis (no build-decision rubric; spend ≠ outcome visibility)
- Beneficiary hypothesis (project/product/delivery managers vs. the Platform Eng Lead)
- Value wedge (spend × quality on one task ledger)
Assumptions to test first:
- The task-unit / quality-signal source (unknown #1) — the architecture hinges on it.
- Whether the PM is the buyer or eng leadership owns it.
- Which of the three audit angles is the sharpest pain.
/icp for the task-roi path?13. Artifact Destination & Proposed File Changes
On approval, these files would be created (the existing flat research/*.md cost-intelligence files are not touched):
| Path | Action | Contents |
|---|---|---|
research/task-roi/idea-brief.md | create | Sections 4–14 above |
research/task-roi/idea-brief-interview.md | create | This session's Q&A + decisions |
research/.progress.yaml | create | Two product paths: cost-intelligence (active) + task-roi (active, pipeline_stage: idea-scope-brief) |
alignment/idea-scope-brief-task-roi.html | convert to confirmed | This page, post-approval |
Slug shown as task-roi; if you pick a different slug in Gate 2, these paths update accordingly.
14. Coverage Checkpoint
Concept is shaped enough to enter research. The single biggest open item is the task-unit / quality-signal source (unknown #1), deliberately left for research. No competitive analysis, journey mapping, UX, or implementation planning was done here — that's downstream.
15. Proposed Next Steps
Proposed only — routing activates after approval, not now.
Primary (proposed): /icp research/task-roi — the business-discovery pack is already enabled, so ICP discovery is the next research step for this path.
Also useful: /product-line review once a third path appears (you'll have two after this).
Compile / Approve
Two paths:
- Send feedback first (no need to answer everything): set any 👍/👎/❓ above, then Compile Feedback YAML here. I'll revise the page.
- Approve: answer all required gate questions, then Compile Answers. Only this authorizes writing the canonical files.
8 required questions remaining.