Production Persistence and Infrastructure Spec

Status

Validated via spec interview on April 25, 2026. This spec covers all infrastructure backing for Automium's production deployment, implementing the six adapter contracts defined in packages/adapters/.

Summary

Automium's production infrastructure is built on five external services: Postgres (Neon), Redis, Cloudflare R2, WorkOS, and Fly.io — plus bare-metal KVM hosts for Firecracker browser workers. The architecture preserves the existing adapter registry pattern: each infrastructure backing is a separate package implementing the contract from packages/adapters/.

The core pipeline for v1 enables: submit a journey → queue a run → dispatch to a Firecracker worker → execute via Playwright → store artifacts in R2 → persist results in Postgres → stream status via Redis pub/sub.

Infrastructure Stack

Layer Technology Purpose
Primary databasePostgres via NeonTenants, journeys, runs, steps, audit, RBAC, credential vault, search
Job queueBullMQ on RedisRun dispatch, worker leasing, async tasks
Object storageCloudflare R2Replay artifacts, screenshots, traces, downloads
Realtime transportRedis pub/sub + WebSocket gatewayLive run status, workspace events
Identity providerWorkOSMagic-link auth, SSO/SAML for enterprise
API frameworkHonoControl plane REST API
ORMDrizzle ORMType-safe schema, migrations
ObservabilityOpenTelemetry + Grafana CloudTraces, metrics, logs
Control plane hostingFly.ioAPI server, WebSocket gateway, BullMQ workers
Browser workersBare-metal KVM hostsFirecracker microVMs with Chromium/Playwright

Database: Postgres via Neon

Why Neon

Schema Design

Shared schema with row-level tenant scoping. Every table includes organization_id and workspace_id columns. Postgres RLS policies enforce that queries only return rows matching the authenticated tenant context.

Core Tables

organizations
  id, name, created_at

workspaces
  id, organization_id, name, created_at

memberships
  id, organization_id, workspace_id, principal_id, role, status, created_at

sessions
  id, identity_id, provider, state, expires_at, created_at

invites
  id, organization_id, workspace_id, email, status, invited_by, expires_at, created_at

journeys
  id, organization_id, workspace_id, name, natural_language_source, created_at, updated_at

journey_versions
  id, journey_id, version, compiled_graph, assertions, fixture_schema, policy_profile, created_at

runs
  id, journey_id, journey_version_id, organization_id, workspace_id,
  environment, planner_backend, status, final_verdict, metrics_json,
  artifact_manifest_path, created_at, started_at, completed_at

steps
  id, run_id, organization_id, workspace_id, sequence,
  planner_input_summary, planner_output_intent, executor_action,
  pre_state_snapshot_ref, post_state_snapshot_ref, visual_artifact_refs,
  timing_json, token_cost, result, created_at

assertions
  id, journey_id, type, condition, scope, severity

recovery_rules
  id, journey_id, trigger, strategy, retry_limit

artifact_manifests
  id, run_id, organization_id, workspace_id, root_path, entries_json,
  retention_expires_at, created_at

audit_events
  id, organization_id, workspace_id, actor_id, resource_type, resource_id,
  action, summary, metadata_json, occurred_at

credentials
  id, organization_id, workspace_id, scope, purpose, encrypted_value,
  created_at, rotated_at

files
  id, organization_id, workspace_id, owner_membership_id, name, storage_location,
  size_bytes, content_type, created_at

jobs
  id, organization_id, workspace_id, type, state, payload_json,
  priority, created_at, started_at, completed_at

Indexes

Row-Level Security

CREATE POLICY tenant_isolation ON runs
  USING (organization_id = current_setting('app.organization_id')::uuid);

Applied to all tenant-scoped tables. The application sets app.organization_id and app.workspace_id session variables after authenticating the request.

ORM: Drizzle

Schema Package

Schema definitions live in packages/persistence/src/schema/. Each domain area has its own schema file:

Migrations

Drizzle-kit generates migrations from schema diffs:

packages/persistence/
  src/
    schema/        # Drizzle schema definitions
    migrations/    # Generated SQL migrations
    connection.ts  # Neon connection pool
    index.ts       # Schema + migration exports

Migrations run at deployment time via a startup script before the API server accepts traffic.

Job Queue: BullMQ on Redis

Queue Architecture

┌──────────────────┐     ┌─────────┐     ┌────────────────┐
│  Control Plane   │────▶│  Redis   │◀────│  Worker Fleet  │
│  (enqueue runs)  │     │  BullMQ  │     │  (dequeue +    │
│                  │     │  queues  │     │   execute)     │
└──────────────────┘     └─────────┘     └────────────────┘

Queues

Job Data

interface RunJobData {
  runId: string;
  journeyId: string;
  journeyVersionId: string;
  organizationId: string;
  workspaceId: string;
  environment: string;
  plannerBackend: { vendor: string; model: string };
  credentialRefs: { scope: string; purpose: string }[];
  priority: "high" | "normal" | "low";
}

Worker Processing

  1. BullMQ worker dequeues a run job
  2. Resolves credentials from the vault
  3. Boots a Firecracker microVM with Chromium + Playwright
  4. Injects credentials into the VM
  5. Executes the journey via the BrowserRuntime adapter
  6. Collects artifacts and telemetry
  7. Uploads artifacts to R2 (via artifact-upload queue)
  8. Updates run status in Postgres
  9. Tears down the microVM

Concurrency and Rate Limiting

Object Storage: Cloudflare R2

Why R2

Bucket Structure

automium-artifacts/
  {organizationId}/
    {workspaceId}/
      runs/
        {runId}/
          manifest.json
          semantic-snapshots/
            step-{sequence}.json
          network-logs/
            step-{sequence}.json
          console-logs/
            step-{sequence}.json
          targeted-crops/
            step-{sequence}-{elementRef}.png
          assertion-traces/
            step-{sequence}.json
          planner-intents/
            step-{sequence}.json
          executor-actions/
            step-{sequence}.json
          playwright-trace.zip

Retention Enforcement

R2 lifecycle rules implement the artifact retention policy:

Objects are tagged with retention-class and expires-at at upload time. A lifecycle rule deletes objects past their expiration.

FileStorageAdapter Implementation

class R2FileStorageAdapter implements FileStorageAdapter {
  private client: S3Client; // @aws-sdk/client-s3 pointed at R2

  async store(fileId, data, metadata) {
    await this.client.send(new PutObjectCommand({
      Bucket: this.bucket,
      Key: this.keyFor(fileId),
      Body: data,
      Metadata: metadata,
      Tagging: `retention-class=${metadata.retentionClass}`
    }));
    return { stored: true, location: this.keyFor(fileId) };
  }

  async retrieve(fileId) { /* GetObjectCommand */ }
  async remove(fileId) { /* DeleteObjectCommand */ }
}

Realtime: Redis Pub/Sub + WebSocket Gateway

Architecture

┌──────────────┐    ┌─────────┐    ┌──────────────────┐    ┌─────────────┐
│ Control Plane │──▶│  Redis   │──▶│  WebSocket       │──▶│  Clients    │
│ (publish)     │   │  pub/sub │   │  Gateway         │   │  (browsers) │
└──────────────┘    └─────────┘    └──────────���───────┘    └─────────────┘

Topics

Mapped from the existing realtime contract:

WebSocket Gateway

Delivery Guarantees

The existing realtime contract specifies:

Authentication: WorkOS

Why WorkOS

IdentityProviderAdapter Implementation

class WorkOSIdentityProviderAdapter implements IdentityProviderAdapter {
  async authenticate(credentials) {
    // For magic-link: trigger WorkOS passwordless auth
    // For SSO: redirect to WorkOS SSO flow
    // Returns: { identityId, provider }
  }

  async validateToken(token) {
    // Validate WorkOS session token
    // Returns: { valid, identityId, expiresAt }
  }
}

Session Flow

  1. User requests magic-link → WorkOS sends email
  2. User clicks link → WorkOS validates, returns session
  3. Automium creates a Session record in Postgres (state: active)
  4. Session token is set as HTTP-only cookie
  5. API middleware validates session on every request

Credential Vault

Encrypted Postgres Table

Credentials are stored as AES-256-GCM encrypted values in the credentials table:

Access Control

RBAC Enforcement

Two-Layer Model

  1. Middleware layer: Hono middleware extracts the user's role from the session, calls checkPermission(role, resource, action), returns 403 with denial reason if not allowed.
  1. Database layer: Postgres RLS policies enforce tenant isolation at the query level. Even if middleware is bypassed, RLS prevents cross-tenant data access.

Middleware Application

const rbacMiddleware = (resource: string, action: string) => {
  return async (c: Context, next: Next) => {
    const { role } = c.get("session");
    const result = checkPermission(role, resource, action);
    if (!result.allowed) {
      return c.json({ error: result.reason }, 403);
    }
    await next();
  };
};

app.get("/journeys", rbacMiddleware("journey", "list"), listJourneys);

API Framework: Hono

Why Hono

Route Structure

The existing CONTROL_PLANE_ROUTES manifest maps to Hono routes:

/api/v1/
  journeys/        POST, GET, GET/:id, PUT/:id
  journeys/:id/compile  POST
  runs/            POST, GET, GET/:id
  runs/:id/status  GET
  runs/:id/artifacts  GET
  runs/:id/replay  GET
  benchmarks/      POST, GET/:id
  tenants/         GET, PUT (admin only)
  credentials/     POST, PUT/:id, DELETE/:id (no GET value)

Observability: OpenTelemetry + Grafana Cloud

Instrumentation

Logging

Key Dashboards

Deployment Topology

Control Plane (Fly.io)

┌──────────────────────────���──────────────────┐
│  Fly.io                                     │
│  ┌───────��─────┐  ┌─────────────────────┐  │
│  │  Hono API   │  │  WebSocket Gateway  │  │
│  │  (2+ nodes) │  │  (2+ nodes)         │  │
│  └─────────────┘  └─────────────────────┘  │
│  ┌─────────────┐  ┌─────────────────────┐  │
│  │  BullMQ     │  │  Data Lifecycle     │  │
│  │  Dispatcher │  │  Worker             │  │
│  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────┘
         │                    │
    ┌────▼────┐         ┌────▼────┐
    │  Neon   │         │  Redis  │
    │ Postgres│         │ (Upstash│
    │         │         │  or Fly)│
    └─────────┘         └─────────┘

Worker Fleet (Bare Metal)

┌──────────────────────────────────────┐
│  Bare Metal Host (KVM-capable)       │
│  ┌──────────────────────────────┐   │
│  │  Worker Process (Node.js)    │   │
│  │  - Dequeues from BullMQ      │   │
│  │  - Manages Firecracker VMs   │   │
│  │  - Uploads artifacts to R2   │   │
│  │  - Sends heartbeats via HTTP │   │
│  └──────────────────────────────┘   │
│  ┌────────┐ ┌────────┐ ┌────────┐  │
│  │ FC VM  │ │ FC VM  │ │ FC VM  │  │
│  │Chromium│ │Chromium│ │Chromium│  │
│  └────────┘ └────────┘ └────────┘  │
└──────────────────────────────────────┘

Workers connect to the control plane over the public internet:

Why Bare Metal for Workers

Data Lifecycle

Tiers

Tier Storage Duration Content
HotPostgres<30 daysFull run data: steps, telemetry, metadata
WarmR230 days to retention limitStep-level data migrated from Postgres, summary stays
Archive/DeleteR2 lifecyclePast retention limitArtifacts auto-deleted by R2 lifecycle rules

Migration Process

A scheduled BullMQ job (data-lifecycle queue) runs daily:

  1. Query Postgres for completed runs older than 30 days
  2. Export step-level data as JSONL to R2
  3. Delete step rows from Postgres (keep run summary row)
  4. Update run record with archived_at timestamp and R2 path
  5. Log migration in audit events

Deferred for v1

Data lifecycle tiering is deferred. In v1, all data stays in Postgres. The tiering infrastructure ships when run volume justifies it.

Search: Postgres Full-Text Search

v1 Implementation

Use Postgres tsvector columns with GIN indexes for full-text search on:

SearchBackendAdapter Implementation

class PostgresSearchAdapter implements SearchBackendAdapter {
  async index(entry) {
    // INSERT into search_entries with tsvector column
  }

  async search(query, filters) {
    // SELECT using tsquery with organization_id/workspace_id filters
  }
}

Upgrade Path

If search complexity outgrows Postgres FTS, migrate to Typesense or Meilisearch. The SearchBackendAdapter contract isolates the switch.

Package Structure

packages/
  adapters/                    # Contracts + registry (existing, unchanged)
  persistence/                 # NEW: Drizzle schema, migrations, connection
    src/
      schema/
        tenancy.ts
        auth.ts
        journeys.ts
        runs.ts
        artifacts.ts
        audit.ts
        credentials.ts
        files.ts
        jobs.ts
      migrations/
      connection.ts
      index.ts
  adapters-postgres/           # NEW: audit-sink, search-backend implementations
  adapters-r2/                 # NEW: file-storage implementation
  adapters-bullmq/             # NEW: job-queue implementation
  adapters-redis/              # NEW: realtime-transport implementation
  adapters-workos/             # NEW: identity-provider implementation

Each adapter package:

v1 Scope Boundary

Ships with v1

  1. Postgres persistence (Neon) — tenants, journeys, runs, steps, audit, credentials
  2. BullMQ job queue (Redis) — run dispatch, artifact upload, audit sink
  3. R2 file storage — artifact upload and retrieval
  4. Redis pub/sub + WebSocket gateway — live run status
  5. WorkOS auth — magic-link login, session management
  6. RBAC middleware — checkPermission() + RLS
  7. Hono API server — control plane routes
  8. Drizzle schema + migrations
  9. OpenTelemetry instrumentation
  10. Credential vault (encrypted Postgres)

Deferred

Open Questions