Provider-Backed Planner Execution Spec

Status

Validated via spec interview on April 25, 2026. This spec covers the production planner adapter layer, superseding the contract-level planner sections of specs/agent-native-browser-qa-platform.md with implementation-level decisions.

Summary

Automium's planner layer connects model providers (Claude, GPT, Gemini) to the deterministic executor through a provider-agnostic adapter contract. Each adapter translates Automium's enriched semantic snapshots into provider-native tool-call prompts and parses structured tool-call responses back into the v1 intent vocabulary.

The contract evolves from v1 (string-based prompt/parse) to v2 (native tool calling with structured output). Claude/Anthropic is the first production adapter. GPT/OpenAI follows as a fast-follow in the same release cycle. Gemini is deferred to v1.1+.

Strategic Context

All three major providers (Anthropic, OpenAI, Google) now support native tool calling with structured output enforcement. This eliminates the need for fragile text-based JSON parsing and makes the planner adapter contract cleaner:

The adapter abstraction ensures Automium's executor and benchmark systems remain provider-agnostic while each adapter handles provider-specific API mechanics.

Contract Evolution: v1 → v2

v1 Contract (Deprecated)

interface PlannerAdapter {
  metadata(): PlannerAdapterMetadata;
  buildPrompt(input: PlannerPromptInput): string;
  parsePlannerOutput(output: string): readonly PlannerIntent[];
  compileIntent(intent: PlannerIntent): CompiledPlannerIntent;
  summarizeStep(intent: PlannerIntent, result: "pass" | "fail"): PlannerStepSummary;
}

String-based buildPrompt() and parsePlannerOutput() are fundamentally mismatched with native tool calling. Models return structured tool calls, not free-form text.

v2 Contract

interface PlannerAdapterV2 {
  metadata(): PlannerAdapterMetadata;
  toolDefinitions(): readonly ToolDefinition[];
  buildMessages(input: PlannerPromptInput): PlannerMessages;
  parseToolCalls(calls: readonly ToolCall[]): readonly PlannerIntent[];
  compileIntent(intent: PlannerIntent): CompiledPlannerIntent;
  summarizeStep(intent: PlannerIntent, result: "pass" | "fail"): PlannerStepSummary;
}

Changes from v1:

Metadata Extension

interface PlannerAdapterMetadata {
  readonly id: string;
  readonly vendor: "anthropic" | "openai" | "google" | "custom";
  readonly model: string;
  readonly intentSchemaVersion: "v1" | "v2";
  readonly supportsVision: boolean;
  readonly supportsToolCalls: boolean;
  readonly defaultModel: string;
}

Tool Definition Schema

Each planner intent becomes a tool definition:

interface ToolDefinition {
  readonly name: PlannerIntentName;
  readonly description: string;
  readonly parameters: JSONSchema;
}

Example for click:

{
  "name": "click",
  "description": "Click an interactive element identified by its stable element ID.",
  "parameters": {
    "type": "object",
    "properties": {
      "target": {
        "type": "string",
        "description": "Stable element ID from the semantic snapshot"
      },
      "reasoning": {
        "type": "string",
        "description": "Why this element should be clicked to advance the journey"
      }
    },
    "required": ["target", "reasoning"]
  }
}

Provider-Agnostic Message Types

interface PlannerMessages {
  readonly system: string;
  readonly messages: readonly PlannerMessage[];
}

type PlannerMessage =
  | { role: "user"; content: readonly ContentBlock[] }
  | { role: "assistant"; content: readonly ContentBlock[] }
  | { role: "tool_result"; toolCallId: string; content: readonly ContentBlock[] };

type ContentBlock =
  | { type: "text"; text: string }
  | { type: "image"; source: ImageSource; metadata?: VisionCropMetadata };

Each adapter translates these provider-agnostic types into provider-native formats (Anthropic messages, OpenAI Responses API input, Gemini Content objects).

Prompt Construction Strategy

System Prompt

The system prompt defines:

The system prompt is cacheable across steps within a journey:

User Message

Each step's user message contains:

  1. Semantic snapshot: enriched element list with stable IDs, roles, labels, values, actionability scores
  2. Mutations: changes since last step (elements added/removed/changed)
  3. Network events: relevant API responses, errors since last step
  4. Journey context: current checkpoint, remaining assertions, recovery state
  5. Pinned invariants: must-hold conditions
  6. Vision crops (when flagged): image content blocks with semantic annotations

Multi-Turn Context

The adapter maintains a sliding window of recent steps (previous assistant tool calls + tool results) within the provider's context window. Older steps are summarized by summarizeStep() and compressed into a context summary block in the system prompt. This implements the hierarchical context compaction strategy from the platform spec.

Vision Integration

Vision crops flow through the adapter as image content blocks in the user message:

  1. The semantic runtime flags vision_recommended on the snapshot with candidate element refs
  2. The executor captures targeted crops via BrowserRuntime.captureElementScreenshot()
  3. Crops are included in the PlannerPromptInput as visionCrops: VisionCrop[]
  4. The adapter's buildMessages() inserts image content blocks at the end of the user message, annotated with element context
  5. Vision-capable adapters (Claude, GPT, Gemini all support vision) include the blocks
  6. Non-vision adapters skip image blocks with a warning logged

Budget: max 2-3 crops per step, each under 100 KB. Enforced before the adapter receives input.

Provider Adapters

Package Structure

packages/
  planner-adapter/          # Registry/factory, v2 contract re-exports
  planner-adapter-anthropic/ # Claude adapter, depends on @anthropic-ai/sdk
  planner-adapter-openai/    # GPT adapter, depends on openai
  planner-adapter-google/    # Gemini adapter, depends on @google/generative-ai
  planner-adapter-fixture/   # Deterministic CI adapter, no provider SDK

Each provider package:

The base packages/planner-adapter/ becomes a registry:

function resolvePlannerAdapter(
  vendor: string,
  config: PlannerConfig
): PlannerAdapterV2;

Claude/Anthropic Adapter (First)

GPT/OpenAI Adapter (Fast-Follow)

Gemini/Google Adapter (v1.1+)

Fixture Adapter (CI/Benchmarks)

Cost Controls

Three-Layer Budget Enforcement

  1. Per-step token cap: Set max_tokens on each provider API call. Prevents any single step from consuming excessive tokens. Default: 4096 tokens.
  1. Per-run token budget: Track cumulative input_tokens + output_tokens across all steps in a journey run. If the budget is exceeded, the executor aborts the journey with a budget_exceeded verdict. Default: 100,000 tokens per run (configurable per journey or tenant).
  1. Per-tenant quota: Enforced at the orchestrator level. Checked before each planner call. Prevents a single tenant from consuming disproportionate resources. Quota period: daily or monthly, configurable.

Token Reporting

Each adapter extracts token usage from the provider response and returns it in PlannerStepTelemetry:

interface PlannerStepTelemetry {
  readonly promptTokens: number;
  readonly completionTokens: number;
  readonly totalTokens: number;
  readonly modelId: string;
  readonly modelVersion: string;
  readonly apiLatencyMs: number;
  readonly timeToFirstTokenMs: number;
  readonly toolCallCount: number;
  readonly visionIncluded: boolean;
  readonly retryCount: number;
}

The benchmark runner aggregates telemetry across runs for comparison reports.

Retry and Fallback Strategy

Provider-Level Retry

The adapter retries transient failures:

Journey-Level Fallback

If all provider retries fail, the failure surfaces to the journey executor's recovery policy:

  1. Retry the step with the same planner (executor-level retry, different from provider retry)
  2. Switch to a fallback planner if configured (e.g., Claude → GPT)
  3. Fail the journey with a planner_unavailable verdict

Content policy rejections (400) are not retried — they surface as step failures with the rejection reason in telemetry.

Credential Management

Provider API keys are stored in the same credential vault as application login credentials, under a separate scope:

At run time:

  1. Orchestrator resolves the planner credential for the tenant + vendor
  2. Credential is injected into the Firecracker microVM alongside app credentials
  3. The adapter reads the credential from the injected secret and initializes the provider SDK client
  4. Credential is never logged or included in artifacts

Model Selection

Each provider adapter has a default model. Tenants can override the model at two levels:

  1. Planner profile: tenant-level default (e.g., "always use Claude Opus for this tenant")
  2. Run submission: per-run override (e.g., "use GPT-4o for this benchmark run")

The adapter validates the requested model is supported by the provider and reports the actual model used in step telemetry. Benchmark reports show results at model granularity (Claude Sonnet vs GPT-4o vs Gemini Flash), not just provider level.

Data Privacy

By default, the full semantic snapshot is sent to the planner provider. The tenant consents to this data flow when they:

  1. Configure their own provider API key in the credential vault
  2. Add their application domain to the domain allowlist
  3. Submit runs targeting their own app

Optional Redaction

Tenants can define field-level redaction policies:

Redacted fields are replaced with [REDACTED] tokens in the snapshot before it reaches the adapter. Redaction metadata is preserved in the replay event stream so debugging is still possible (the debugger shows that a field was redacted, not its value).

Benchmark Integration

The benchmark runner's comparePlannerBackends() function evolves to use v2 adapters:

Comparison report metrics (unchanged from v1 contract):

New metrics from v2 telemetry:

v1 Scope Boundary

Ships with v1

  1. Claude/Anthropic adapter: full v2 contract, tool calling, vision, prompt caching, telemetry
  2. Fixture adapter: deterministic intent sequences, full v2 contract, CI/benchmark baseline
  3. v2 contract definition: in packages/contracts/
  4. Adapter registry/factory: in packages/planner-adapter/
  5. Multi-layer cost controls: per-step, per-run, per-tenant
  6. Rich per-step telemetry: normalized across providers

Fast-Follow (Same Release Cycle)

  1. GPT/OpenAI adapter: full v2 contract, enables first cross-provider benchmark comparison

Deferred (v1.1+)

  1. Gemini/Google adapter
  2. Advanced redaction policies (basic pattern redaction ships with v1)
  3. Model auto-selection (step complexity-based model routing)
  4. Streaming responses (current scope is request-response per step)

Open Questions