Harness Agent SDK Migration: Empirical Analysis

1. Introduction

1.1 Heideggerian Framing

Per the CREATE Something philosophy, infrastructure should exhibit Zuhandenheit (ready-to-hand: when a tool disappears into transparent use, like a hammer during skilled carpentry)—receding into transparent use. The harness should be invisible when working correctly; failures should surface clearly with actionable context.

1.2 The Canon Redesign Project

The test project: removing --webflow-blue (#4353ff) from the Webflow Dashboard. This brand color polluted focus states, buttons, links, nav, and logos—43 violations across 19 files.

2. Architecture

2.1 Harness Flow

┌─────────────────────────────────────────────────────────┐
│                    HARNESS RUNNER                        │
│                                                          │
│  Spec Parser ──► Issue Creation ──► Session Loop         │
│                                                          │
│  ┌─────────────────────────────────────────────────┐    │
│  │  Session 1 ──► Session 2 ──► Session 3 ──► ...  │    │
│  │      │             │             │               │    │
│  │      ▼             ▼             ▼               │    │
│  │  Checkpoint    Checkpoint    Checkpoint          │    │
│  │      │             │             │               │    │
│  │      ▼             ▼             ▼               │    │
│  │  Peer Review   Peer Review   Peer Review         │    │
│  └─────────────────────────────────────────────────┘    │
│                                                          │
└──────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────┐
│              BEADS (Human Interface)                     │
│                                                          │
│  bd progress - Review checkpoints                        │
│  bd update   - Redirect priorities                       │
│  bd create   - Inject work                               │
└─────────────────────────────────────────────────────────┘

2.2 Session Spawning

Each session spawns Claude Code in headless mode with explicit configuration:

// packages/harness/src/session.ts
export async function runSession(
  issueId: string,
  prompt: string,
  options: SessionOptions = {}
): Promise<SessionResult> {
  const args = [
    '-p',
    '--allowedTools', HARNESS_ALLOWED_TOOLS,
    '--max-turns', options.maxTurns?.toString() ?? '100',
    '--output-format', 'json',
  ];

  if (options.model) {
    args.push('--model', options.model);
  }

  // Spawn claude process with captured stdout/stderr
  const result = await spawnClaude(args, prompt);

  // Parse structured JSON output
  const metrics = parseJsonOutput(result.stdout);

  return {
    issueId,
    outcome: determineOutcome(result),
    sessionId: metrics.sessionId,
    costUsd: metrics.costUsd,
    numTurns: metrics.numTurns,
  };
}

3. Migration Changes

3.1 Before: Legacy Pattern

const args = [
  '-p',
  '--dangerously-skip-permissions',
  '--output-format', 'json',
];

3.2 After: Agent SDK Pattern

const args = [
  '-p',
  '--allowedTools', HARNESS_ALLOWED_TOOLS,
  '--max-turns', '100',
  '--output-format', 'json',
  '--model', options.model,
];

3.3 Tool Categories

4. Peer Review Pipeline

The harness runs three peer reviewers at checkpoint boundaries:

const REVIEWERS: ReviewerConfig[] = [
  {
    name: 'security',
    prompt: 'Review the code changes for security vulnerabilities...',
    model: 'haiku',
    timeout: 30000,
  },
  {
    name: 'architecture',
    prompt: 'Review the code changes for architectural concerns...',
    model: 'haiku',
    timeout: 30000,
  },
  {
    name: 'quality',
    prompt: 'Review the code changes for quality issues...',
    model: 'haiku',
    timeout: 30000,
  },
];

4.1 Observed Review Outcomes

Finding: Architecture reviewer surfaces legitimate concerns (token consistency, pattern adherence) without blocking progress. This matches the intended "first-pass analysis" philosophy.

5. Empirical Observations

5.1 Security Improvements

Finding: No legitimate harness operations were blocked by the new restrictions. The allowlist is sufficient for all observed work patterns.

5.2 Runaway Prevention

--max-turns 100 prevents infinite loops. Observed session turn counts:

5.3 Cost Visibility

5.4 Model Selection Impact

6. Trade-offs Analysis

6.1 Pros

6.2 Cons

7. Recommendations

7.1 Immediate Adoption

1. Replace --dangerously-skip-permissions with --allowedTools: The security improvement has no operational cost.
2. Set --max-turns 100: Provides headroom without enabling runaways.
3. Parse JSON output for metrics: Even if not displayed, capture for future analysis.
4. Use Haiku for peer reviews: 95% cost reduction with equivalent quality.

7.2 Future Work

1. Implement --resume: Use captured session_id for task continuity within epics.
2. Model auto-selection: Use task complexity to choose Haiku/Sonnet/Opus.
3. Cost budgets: Set per-harness-run cost limits with automatic pause.
4. Streaming output: Use --output-format stream-json for real-time progress.

8. How to Apply This

Step 1: Audit Current Tool Access (Human)
List all tools your harness currently uses. Check headless session logs or code
that spawns Claude. If using --dangerously-skip-permissions, you have unlimited
access by default.

Step 2: Categorize Essential Tools (Human)
Group tools by purpose:
- Core file operations (Read, Write, Edit, Glob, Grep)
- Version control (Bash(git:*))
- Package management (Bash(pnpm:*), Bash(npm:*))
- Build/deploy (Bash(wrangler:*), domain-specific commands)
- MCP integrations (mcp__cloudflare__*, mcp__airtable__*, etc.)

Step 3: Create Explicit Allowlist (Human)
Build your ALLOWED_TOOLS string. Start conservative—add only what you've verified
is needed. You can expand later if sessions fail.

Step 4: Add Runaway Prevention (Human + Agent)
Set --max-turns based on observed session lengths. Use 2-3x your average as a
safety margin. If most sessions complete in 30-50 turns, set --max-turns 100.

Step 5: Enable Cost Tracking (Agent)
Add --output-format json to capture session metadata. Parse stdout to extract
costUsd, numTurns, sessionId. Store these for analysis.

Step 6: Test on Non-Critical Work First (Human)
Run the migrated harness on low-stakes tasks. Verify tools aren't blocked.
Check that turn limits don't prevent legitimate completion.

// Before: Unsafe pattern
const args = [
  '-p',
  '--dangerously-skip-permissions',
];

const result = await spawn('claude', args, { input: deployPrompt });
// No cost tracking, no runaway prevention, unrestricted tool access

// After: Agent SDK best practices
const DEPLOY_ALLOWED_TOOLS = [
  // Core file operations
  'Read', 'Write', 'Edit', 'Glob', 'Grep',

  // Version control (scoped)
  'Bash(git:status)', 'Bash(git:tag)', 'Bash(git:log)',

  // Deployment (scoped)
  'Bash(wrangler:deploy)', 'Bash(wrangler:tail)', 'Bash(wrangler:whoami)',

  // Cloudflare MCP (explicit)
  'mcp__cloudflare__worker_deploy',
  'mcp__cloudflare__kv_put',
  'mcp__cloudflare__d1_query',
].join(',');

const args = [
  '-p',
  '--allowedTools', DEPLOY_ALLOWED_TOOLS,
  '--max-turns', '50',  // Deployments are fast; low limit appropriate
  '--output-format', 'json',
  '--model', 'sonnet',  // Sonnet sufficient for deployments
];

const result = await spawn('claude', args, { input: deployPrompt });

// Parse metrics
const metrics = JSON.parse(result.stdout);
console.log(`Deployment cost: $${metrics.costUsd}`);
console.log(`Turns used: ${metrics.numTurns}/50`);

// Store for analysis
await db.deployments.create({
  sessionId: metrics.sessionId,
  costUsd: metrics.costUsd,
  numTurns: metrics.numTurns,
  timestamp: new Date(),
});

✓ Zero sessions blocked by missing tool permissions
✓ All sessions complete within turn limits (or fail for legitimate reasons)
✓ Cost tracking data populates correctly
✓ Model selection matches task complexity (Haiku for simple, Opus for complex)
✓ Peer reviews run and surface appropriate findings
✓ No degradation in harness capabilities compared to legacy approach

9. Conclusion

The Agent SDK migration improves the CREATE Something Harness without degrading operational capability. The explicit tool allowlist provides defense-in-depth security, while --max-turns prevents runaway sessions.

The key insight: restrictive defaults with explicit exceptions is more maintainable than permissive defaults with implicit risks.

Appendix A: Full Tool Allowlist

const HARNESS_ALLOWED_TOOLS = [
  // Core file operations
  'Read', 'Write', 'Edit', 'Glob', 'Grep', 'NotebookEdit',

  // Bash with granular patterns
  'Bash(git:*)', 'Bash(pnpm:*)', 'Bash(npm:*)', 'Bash(npx:*)',
  'Bash(node:*)', 'Bash(tsc:*)', 'Bash(wrangler:*)',
  'Bash(bd:*)', 'Bash(bv:*)',  // Beads CLI
  'Bash(grep:*)', 'Bash(find:*)', 'Bash(ls:*)', 'Bash(cat:*)',
  'Bash(mkdir:*)', 'Bash(rm:*)', 'Bash(cp:*)', 'Bash(mv:*)',
  'Bash(echo:*)', 'Bash(test:*)',

  // Orchestration
  'Task', 'TodoWrite', 'WebFetch', 'WebSearch',

  // CREATE Something
  'Skill',

  // MCP Cloudflare
  'mcp__cloudflare__kv_get', 'mcp__cloudflare__kv_put',
  'mcp__cloudflare__kv_list', 'mcp__cloudflare__d1_query',
  'mcp__cloudflare__d1_list_databases',
  'mcp__cloudflare__r2_list_objects', 'mcp__cloudflare__r2_get_object',
  'mcp__cloudflare__r2_put_object', 'mcp__cloudflare__worker_list',
  'mcp__cloudflare__worker_get', 'mcp__cloudflare__worker_deploy',
].join(',');

References

• Claude Code Agent SDK Documentation
• CREATE Something Harness Package
• Beads Patterns Documentation