AutoHarness: Why the Same AI Model Can Score 42% or 78% Depending on Its Harness

AutoHarness: Why the Same AI Model Can Score 42% or 78% Depending on Its Harness

There is a benchmark result that should make every AI power user stop and re-read it: the same underlying model, run against the same set of tasks, scores 42% under one configuration and 78% under another. Not a different model. Not a bigger one. The same weights, the same parameters — just a different harness.

This is the finding that sits at the heart of AutoHarness, a Claude Code plugin that is quietly reframing how developers think about AI productivity. And it has a lot of people rethinking where the real bottleneck is.

---

Big Model or Big Harness? The Question @swyx Is Asking

Developer and AI commentator @swyx has been circulating a framing that cuts through the noise: Big Model vs Big Harness. The conventional wisdom in 2025 was that model scale was everything — more parameters, better results, end of story. But the 42% → 78% delta tells a different story. The harness — the layer of scaffolding, memory, skill context, and guardrails that wraps the model — can matter more than the model itself.

The harness is not a new concept. Every serious AI agent deployment already has one. It is the CLAUDE.md file you maintain by hand, the system prompt you painstakingly tune, the SKILL.md files you write from scratch when you learn a new pattern. The harness is the accumulated tribal knowledge that sits between the raw model and a useful result.

The problem is that today, every harness is hand-maintained. It gets written once, drifts out of sync with how the model actually behaves, becomes a liability instead of an asset, and gets rewritten from scratch with every model generation. The ceiling on harness quality is the patience of the human maintaining it — and humans are not patient.

AutoHarness is built on a simple but radical premise: agents should be able to evolve their own harness.

---

What AutoHarness Actually Is

AutoHarness is a Claude Code plugin. You install it once. Then you work exactly as you always do.

In the background, AutoHarness observes each task episode — everything that happened, what tools were called, what worked, what got corrected. When a task ends, a lightweight Reflector subagent distills that episode into a proposed skill change: a new skill to add, an existing one to patch, or an obsolete one to retire.

That proposed change does not touch your disk immediately. It goes through a deterministic verification gate — a strict linter that checks structure, completeness, and provenance before anything is written. Only changes that pass the gate land in your .claude/skills directory, where Claude Code picks them up automatically in future sessions.

You never see the machinery. You just notice that Claude starts handling a certain class of tasks better than it did last week, without you doing anything.

---

The Three Things That Make This Different

1. Skills Earned in Real Work, Not Configured by Hand

This is the core differentiator, and it matters more than it sounds.

Current skill systems — SKILL.md files, CLAUDE.md sections, system prompt fragments — are written by humans based on intuition. They reflect what the human thinks the agent needs to know, not what the agent has actually learned from doing the work.

AutoHarness inverts this. Skills are never hand-authored and never evaluated against an offline benchmark dataset that was curated before the work started. Instead, when an episode ends, the Reflector reads the complete trajectory — user intention, tool calls, corrections, final outcome — and distills it into a concrete skill change. That change then has to survive: it gets recalled in later sessions and either proves its usefulness or gets retired.

The result is capability earned by working, not configured by hand. Skills that don't hold up in practice decay and disappear. Skills that repeatedly surface and help stay. The harness gets better the more you use the tool, without any manual curation step.

This is the mechanism behind that 42% → 78% gap. The higher-scoring configuration had a harness that had accumulated real operational intelligence — patterns that actually recur in that agent's actual workload, not patterns someone guessed might be useful.

2. Every Skill Is Fully Traceable — and the Agent Never Touches Yours

Here is the concern that kills most "AI self-improvement" proposals before they get started: what stops the AI from messing with my config?

AutoHarness addresses this directly with two mechanisms.

First, every skill it generates carries an LED — a provenance record containing the reason the skill was created or changed, and a precise slice of evidence pointing back to the exact conversation and episode it grew from. There are no black-box writes. Every change in the skill store is auditable: you can see exactly what triggered it, what the agent observed, and what conclusion it drew.

Second, AutoHarness operates under a self-authored-only gate. It tracks which skills it generated itself and it only touches those. Your hand-written CLAUDE.md, your manually crafted SKILL.md files, your system prompts — AutoHarness will not modify, override, or retire any of them. Ever. It maintains a strict separation between the skills it owns and the skills you own.

This is not a policy statement. It is enforced in the architecture. The Reflector subagent runs with minimum-privilege tooling — it can read and grep existing skills for deduplication checks, but it cannot write directly to any file. Writes happen only through a separate verified stage, and that stage's path guard physically prevents touches to any skill that lacks an AutoHarness provenance tag.

If you want to see exactly what AutoHarness has accumulated about your workflows, everything is in .claude/skills, labeled, dated, and linked back to the conversation it came from.

3. Install and Forget — No Workflow Changes Required

This is the part that sounds too good to be true, but it is how the plugin is actually designed.

AutoHarness hooks into Claude Code's existing event system. The Stop hook fires when a task episode ends; the SessionEnd hook flushes any pending work; the SessionStart hook runs lightweight maintenance (deduplication, retirement of stale skills). None of these hooks interrupt your session. They fire in the background as part of the normal session lifecycle.

The Reflector subagent — the component that does the actual reflection and skill proposal — runs detached. It does not block your next task. It does not consume your main session context. It uses a cheaper model (equivalent to Claude Haiku) because reflection does not require frontier reasoning capability. The cost is low; the latency is invisible.

From a user perspective: install the plugin, then work as usual. You will start noticing improvement in how Claude handles recurring patterns in your projects. That is it. There is no curation step, no "review these skill proposals" queue, no maintenance overhead pushed onto you.

---

How the Pipeline Works Under the Hood

For those who want to understand the architecture, here is how each piece connects:

CAP (Capture) — Hooks observe each session round: user intent, tool I/O, corrections. The raw trace is redacted for secrets and PII before any storage. The original trace is not stored; only a pointer into the host log is retained, keeping the footprint minimal.

REF (Reflection) — At the episode boundary, the Reflector subagent receives the episode window plus a description index of existing skills. It does a compare-first pass — checking for duplicates or existing skills that should be patched rather than replaced — before proposing any change. The proposal is sent through stage_skill, an MCP tool that queues it as an intent. Nothing touches the disk yet.

Verify + Store (Promoter) — The deterministic promoter reads the queued intent, assembles the full skill text in memory, runs six categories of linting checks (structure, provenance, completeness, path safety, self-authorship, format compliance), and only then writes to a temp file in the same directory before an atomic rename to the live path. If any check fails, nothing is written. The LED sidecar is appended to record the provenance chain.

MNG (Lifecycle Management) — Every skill carries usage counters. Skills that are not recalled, or that are recalled but not acted on, accumulate negative signal. The lifecycle manager periodically promotes or retires skills based on these counters, keeping the skill store from bloating into a graveyard of outdated patterns.

The result is a system where the model only proposes; it never directly writes. Every write is deterministic, auditable, and atomic.

---

The Symbolic Learning Bet Behind It

AutoHarness sits inside a broader intellectual movement that AI researcher Eddy (Zhihu) has called the symbolic learning renaissance. The argument: deep learning compresses experience into gradient updates to weights. Symbolic learning compresses experience into discrete, reusable, auditable artifacts — code, skills, rules.

The bet is that coding agents change the maintenance cost curve of symbolic systems. Expert systems failed in the 1980s not because symbols were wrong, but because maintaining a growing rule base by hand was impossible. When an agent can maintain its own rule base — pruning, deduplicating, retiring stale rules, adding new ones from direct experience — the maintenance cost curve flattens.

AutoHarness is that bet in working code. Every SKILL.md it generates is a symbol: discrete, inspectable, reusable, grounded in specific evidence from real work. The scarce resource is not generating symbols; any LLM can do that. The scarce resource is maintaining a symbol ecosystem that stays accurate and non-redundant as the environment evolves. That is what the LED + gate + lifecycle stack is designed to do.

---

Getting Started

AutoHarness installs through Claude Code's plugin system:

# Add from the self-hosted marketplace
/plugin marketplace add https://github.com/tigerless-labs/autoharness

Once installed, the plugin components drop into place automatically. There is no init script to run. The global and repo-level data directories are created lazily on first write. Your first task session is the first episode. The Reflector subagent begins accumulating on episode one.

The most convincing demo: do a real task that involves a pattern you repeat often, then check .claude/skills afterward. You will find a SKILL.md with a reason, an evidence slice pointing to the conversation, and a description written to surface in future session recalls. Do the same class of task again; watch the skill get recalled and applied without prompting.

That is the before/after that matters — not a benchmark number, but your own recurring workflow handled better the second time around because the harness remembers the first.

---

Frequently Asked Questions

Does AutoHarness work with any Claude Code project? Yes. The plugin is host-agnostic — it lives in ~/.claude/plugins/ and attaches to any Claude Code session via the hook system, regardless of the project or repository.

Will it ever modify my existing SKILL.md or CLAUDE.md files? No. The self-authored-only gate prevents AutoHarness from touching any skill it did not generate itself. Your hand-written files are completely safe.

How do I see what AutoHarness has learned? Check .claude/skills at the global or repo level. Every skill folder contains SKILL.md with provenance tags, plus a sidecar with usage counters and the LED chain.

What model does the Reflector use? A small, cheap model (comparable to Claude Haiku). Reflection does not require frontier capability. The main session model is unaffected.

Can I delete a skill AutoHarness created? Yes — delete the folder from .claude/skills and AutoHarness will not recreate it (unless the same pattern recurs strongly enough to trigger a new proposal). You can also pin skills you want to keep permanently.

Does the plugin send my code to external servers? Redaction rules strip secrets and PII from episode traces before any processing. The plugin runs entirely within your Claude Code session; no external telemetry is sent beyond what Claude Code itself processes.