PRD — AutoResearch Primitive

Document Status

Draft V1

Purpose

This PRD defines AutoResearch as a first-class Cambium primitive: an autonomous experimentation loop that any workflow can invoke to iteratively optimize a target system against a measurable objective.

The core insight, borrowed from Karpathy’s autoresearch: program the researcher, not the research. Define the search space, the success metric, and the mutation strategy. Then let the system run experiments autonomously — propose, test, measure, keep or discard, repeat.

This is evolution. Mutation, selection, adaptation. It fits Cambium’s nature metaphor perfectly, and it maps cleanly onto Cambium’s existing infrastructure: Hypha for the experiment loop, Signals for metric feedback, Seed for rollback checkpoints, Immune for anomaly detection, Quorum for approval gates, and Composting for harvesting failed experiments into learning.

Problem

Optimization work today is manual and episodic. Someone has an idea, tries it, measures it (maybe), and moves on. The search is narrow because human attention is the bottleneck. Most of the parameter space goes unexplored. Failed experiments vanish without contributing to institutional knowledge.

This applies across domains: - Tuning prediction model parameters - Iterating on app store copy - Optimizing prompt quality - Adjusting infrastructure configuration

In every case, the pattern is identical: modify, measure, compare, decide. Only the target system and metric change. The loop itself is a reusable primitive.

Goals

Product goals

1. Provide a single primitive that any Cambium workflow can invoke to run autonomous experimentation.
2. Make experiments safe by default: sandboxed, cost-capped, and observable.
3. Capture every experiment outcome — successes and failures — as structured learning.
4. Support human-in-the-loop approval for high-risk mutations.
5. Surface results through Signals so dependent systems react to improvements automatically.

Engineering goals

1. Map the experiment loop to an existing Hypha cyclic workflow, not a new runtime.
2. Reuse Seed, Immune, Quorum, Signals, and Composting rather than building parallel systems.
3. Define a clean API surface that is target-agnostic (works for code, config, content, prompts, anything).
4. Keep the primitive composable: multiple AutoResearch sessions can run concurrently on different targets.

Non-Goals

This PRD does not require: - A specific experiment UI in Loom (that is a future enhancement) - Built-in modifier strategies for every domain (callers provide their own) - Multi-objective optimization (V1 optimizes a single metric per session) - Distributed experiment execution across multiple machines - Real-time A/B testing infrastructure (evaluators can wrap external A/B systems, but AutoResearch does not provide one)

Architecture

How AutoResearch maps to Cambium’s existing systems

AutoResearch is not a new subsystem. It is a workflow pattern composed from existing Cambium primitives.

┌─────────────────────────────────────────────────────────────────┐
│                    AutoResearch Session                          │
│                                                                 │
│   ┌──────────┐    ┌──────────┐    ┌──────────┐                 │
│   │ PROPOSE  │───>│  APPLY   │───>│ EVALUATE │                 │
│   │ modifier │    │ sandbox  │    │ evaluator│                 │
│   └──────────┘    └──────────┘    └──────────┘                 │
│        ^                               │                        │
│        │                               ▼                        │
│   ┌──────────┐                   ┌──────────┐                  │
│   │   LOG    │<──────────────────│ COMPARE  │                  │
│   │ compost  │                   │ baseline │                  │
│   └──────────┘                   └──────────┘                  │
│        │                               │                        │
│        │          ┌──────────┐         │                        │
│        └─────────>│  DECIDE  │<────────┘                        │
│                   │keep/disc.│                                   │
│                   └─────┬────┘                                  │
│                         │                                       │
│                    loops back to PROPOSE                         │
│                    (until budget/iterations exhausted)           │
└─────────────────────────────────────────────────────────────────┘

Cambium primitive mapping

Execution flow

1. Session initialization: Create a Hypha workflow from the AutoResearch template. Seed creates an initial checkpoint of the target system state. Baseline metric is recorded.

2. Propose: The modifier function generates a candidate change. If the mutation’s estimated risk exceeds approvalThreshold, a Quorum decision is created and the workflow pauses at an approval gate.

3. Apply: The candidate change is applied in a sandboxed environment. Seed creates a pre-experiment checkpoint.

4. Evaluate: The evaluator function runs against the modified system and returns a metric value.

5. Compare: The new metric is compared against the current baseline. Direction (higher-is-better or lower-is-better) is determined by the metric configuration.

6. Decide: If the new metric improves on the baseline, the change is kept and the baseline is updated. A autoresearch.improvement signal is emitted. If the metric regresses, the change is discarded and Seed restores from the pre-experiment checkpoint.

7. Log: The full experiment record (proposal, result, decision, duration, cost) is persisted. Failed experiments are composted. The iteration counter increments.

8. Loop or stop: If maxIterations is reached or the time budget is exhausted, the session completes. A final autoresearch.session_complete signal is emitted with the cumulative improvement summary.

Immune system integration

The Immune system monitors every AutoResearch session for:

API Surface

Core primitive

interface AutoResearchConfig {
  /** Human-readable name for this research session */
  name: string

  /** What system to optimize (identifier, file path, config key, etc.) */
  target: string

  /** How to measure success — returns a numeric score */
  metric: MetricFn

  /** Direction of optimization */
  direction: 'maximize' | 'minimize'

  /** How to propose changes — returns a candidate mutation */
  modifier: ModifierFn

  /** How to run an experiment — applies mutation and returns metric */
  evaluator: EvalFn

  /** Max wall-clock time for the entire session */
  budget: Duration

  /** Max time for a single experiment */
  experimentTimeout: Duration

  /** How many experiments before stopping */
  maxIterations: number

  /** Current best score (starting point) */
  baseline: number

  /** Max cost (tokens, API calls, dollars) per session */
  costCap: CostLimit

  /** Mutations above this risk score require Quorum approval (0-1, optional) */
  approvalThreshold?: number

  /** Workspace and run context */
  workspaceId: WorkspaceId
}

type MetricFn = (state: TargetState) => Promise<number>
type ModifierFn = (state: TargetState, history: ExperimentRecord[]) => Promise<Mutation>
type EvalFn = (state: TargetState, mutation: Mutation) => Promise<EvalResult>

interface Mutation {
  id: string
  description: string
  diff: unknown          // target-specific change representation
  estimatedRisk: number  // 0-1 scale
}

interface EvalResult {
  metric: number
  cost: CostRecord
  duration: number
  metadata?: Record<string, unknown>
}

interface ExperimentRecord {
  iteration: number
  mutation: Mutation
  result: EvalResult
  decision: 'kept' | 'discarded'
  baselineBefore: number
  baselineAfter: number
  timestamp: Date
}

interface AutoResearchResult {
  sessionId: string
  totalIterations: number
  improvements: number
  baselineStart: number
  baselineEnd: number
  totalCost: CostRecord
  totalDuration: number
  bestExperiment: ExperimentRecord
  allExperiments: ExperimentRecord[]
}

Launching a session

import { AutoResearch } from '@cambium/autoresearch'

const session = await AutoResearch.start({
  name: 'pam-strategy-hit-rate',
  target: 'strategies/momentum_v2.py',
  metric: async (state) => computeHitRate(state) * computeBps(state),
  direction: 'maximize',
  modifier: async (state, history) => proposeParamMutation(state, history),
  evaluator: async (state, mutation) => replayWithMutation(state, mutation),
  budget: Duration.hours(4),
  experimentTimeout: Duration.minutes(15),
  maxIterations: 50,
  baseline: currentHitRateBps,
  costCap: { maxTokens: 500_000, maxDollars: 10 },
  workspaceId,
})

// Session runs autonomously. Listen for results:
signals.on('autoresearch.session_complete', (signal) => {
  console.log(`Improved ${signal.data.baselineStart} → ${signal.data.baselineEnd}`)
})

Use Cases

1. Pam strategy optimization

2. Content optimization

3. ASO optimization

4. Prompt optimization

5. Infrastructure tuning

Governance and Safety

Sandboxing

Experiments never modify production directly. Every AutoResearch session operates against one of: - A copy of the target system (file, config, database snapshot) - A staging or preview environment - A simulation or replay harness

The evaluator function is responsible for environment isolation. The AutoResearch primitive enforces this by requiring the evaluator to declare its isolation mode (replay, staging, preview, simulation). Sessions with undeclared isolation mode are rejected.

Cost controls

Immune system monitoring

The Immune system treats each AutoResearch session as a monitored agent. Trust score mechanics apply: - Successful improvements: trust +0.02 - Neutral experiments (no improvement, no regression): trust unchanged - Regressions caught and rolled back: trust -0.03 - Cost anomaly: trust -0.10 - Metric collapse: trust -0.15, quarantine pending review

If trust drops below 0.2, the session is quarantined and a governance.escalation_required signal is emitted.

Human-in-the-loop

The approvalThreshold field controls when Quorum gates activate: - Mutations with estimatedRisk below the threshold proceed automatically - Mutations above the threshold create a Quorum decision and pause the workflow - If no approvalThreshold is set, all mutations proceed automatically (appropriate for low-risk, offline experiments)

Audit trail

Every experiment produces an immutable audit record containing: - Session ID and iteration number - Full mutation description and diff - Metric before and after - Decision (kept/discarded) with rationale - Cost and duration - Checkpoint IDs (pre-experiment and post-experiment if kept)

Integration with Obadiah

AutoResearch is a capability Obadiah can invoke across any project in his portfolio.

Morning brief

Overnight AutoResearch sessions report results in the morning brief:

AutoResearch: 3 sessions completed overnight
  - pam-momentum-v2: hit_rate*bps improved 12.4 → 14.1 (+13.7%) over 38 experiments
  - prompt-persona-arnold: quality score 6.2 → 7.8 (+25.8%) over 22 experiments
  - infra-cache-ttl: throughput/p95 improved 847 → 923 (+9.0%) over 18 experiments

Session management

Obadiah can: - Launch AutoResearch sessions on any project via natural language (“optimize Pam’s momentum strategy for hit rate”) - Monitor running sessions (“how’s the prompt optimization going?”) - Pause or stop sessions (“stop the ASO experiment, we’re doing a manual update”) - Review results and apply best experiments to production (with approval gates)

Strategy recommendations

Composted experiment data feeds Obadiah’s recommendations: - “The last 3 prompt optimization sessions all improved by adjusting the system instruction length. Consider standardizing on shorter system prompts.” - “Momentum strategy improvements have plateaued. The modifier may need a wider search space — consider adding new parameter dimensions.”

Signals cascade

AutoResearch results propagate through Cambium’s signal system: - autoresearch.improvement can trigger downstream workflows (e.g., auto-deploy to staging) - autoresearch.session_complete feeds into project status dashboards - autoresearch.cost_alert routes to relevant project channels

Implementation Plan

Phase 1: Core primitive

• Define AutoResearchConfig types in @cambium/core
• Build Hypha workflow template for the experiment loop
• Implement session lifecycle (start, pause, resume, stop)
• Seed integration for per-experiment checkpoints
• Basic experiment logging

Phase 2: Safety and governance

• Immune system anomaly detectors for AutoResearch patterns
• Quorum integration for approval-gated mutations
• Cost tracking and cap enforcement
• Composting integration for failed experiment harvesting

Phase 3: Observability

• Loom dashboard for active and completed sessions
• Experiment history visualization (metric over iterations)
• Signal-based notifications for improvements and alerts

Phase 4: Advanced features

• Multi-objective optimization (Pareto frontier)
• Modifier strategy library (common patterns: grid search, random, bayesian, LLM-guided)
• Cross-session learning (composted data informs future modifier strategies)
• Collaborative experiments (multiple sessions coordinating via Stigmergy)