---
name: autoresearch
description: "Autonomously optimize any skill by running it repeatedly, scoring outputs against binary evals, mutating the prompt, and keeping improvements. Based on Karpathy's autoresearch methodology. Use when: optimize this skill, improve this skill, run autoresearch on, make this skill better, self-improve skill, benchmark skill, eval my skill, run evals on. Outputs: an improved SKILL.md, a results log, and a changelog of every mutation tried."
---

# Autoresearch for Skills

Most skills work about 70% of the time. The other 30% you get garbage. The fix isn't to rewrite the skill from scratch. It's to let an agent run it dozens of times, score every output, and tighten the prompt until that 30% disappears.

This skill adapts Andrej Karpathy's autoresearch methodology (autonomous experimentation loops) to OpenClaw skills. Instead of optimizing ML training code, we optimize skill prompts.

---

## the core job

Take any existing skill, define what "good output" looks like as binary yes/no checks, then run an autonomous loop that:

1. Generates outputs from the skill using test inputs
2. Scores every output against the eval criteria
3. Mutates the skill prompt to fix failures
4. Keeps mutations that improve the score, discards the rest
5. Repeats until the score ceiling is hit or the user stops it

**Output:** An improved SKILL.md + `results.tsv` log + `changelog.md` of every mutation attempted + a live HTML dashboard presented via the `canvas` tool.

---

## before starting: gather context

**STOP. Do not run any experiments until all fields below are confirmed with the user. Ask for any missing fields before proceeding.**

1. **Target skill** — Which skill do you want to optimize? (exact path inside `~/.openclaw/workspace/skills/`)
2. **Test inputs** — What 3-5 different prompts/scenarios should we test the skill with? (variety matters — cover different use cases so we don't overfit to one scenario)
3. **Eval criteria** — What 3-6 binary yes/no checks define a good output? (see [references/eval-guide.md](references/eval-guide.md) for how to write good evals)
4. **Runs per experiment** — How many times to run the skill per mutation? Default: 5. (more = more reliable scores, but slower and more expensive)
5. **Run interval** — How often should experiments cycle? Default: every 2 minutes.
6. **Budget cap** — Optional. Max experiment cycles before stopping. Default: no cap (runs until you stop it).

---

## step 1: read the skill

Before changing anything, read and understand the target skill completely.

1. Read the full SKILL.md file
2. Read any files in `references/` that the skill links to
3. Identify the skill's core job, process steps, and output format
4. Note any existing quality checks or anti-patterns already in the skill

Do NOT skip this. You need to understand what the skill does before you can improve it.

---

## step 2: build the eval suite

Convert the user's eval criteria into a structured test. Every check must be binary — pass or fail, no scales.

**Format each eval as:**

```
EVAL [number]: [Short name]
Question: [Yes/no question about the output]
Pass condition: [What "yes" looks like — be specific]
Fail condition: [What triggers a "no"]
```

**Rules for good evals:**
- Binary only. Yes or no. No "rate 1-7" scales. Scales compound variability.
- Specific enough to be consistent. "Is the text readable?" is too vague.
- Not so narrow that the skill games the eval.
- 3-6 evals is the sweet spot. More than that and the skill starts parroting eval criteria back.

See [references/eval-guide.md](references/eval-guide.md) for detailed examples.

**Max score calculation:**
```
max_score = [number of evals] × [runs per experiment]
```

---

## step 3: generate the live dashboard

Before running any experiments, create a live HTML dashboard at `autoresearch-[skill-name]/dashboard.html` and present it via the `canvas` tool.

**OpenClaw dashboard delivery:**
- Create the dashboard as a single self-contained HTML file (inline CSS + JS)
- Use Chart.js from CDN for the line chart
- Present it with: `canvas(action=present, url="file:///root/.openclaw/workspace/skills/[skill-name]/autoresearch-[skill-name]/dashboard.html")`
- Update `results.json` after every experiment — canvas auto-refreshes

The dashboard must:
- Auto-refresh every 10 seconds (reads from results.json)
- Show a score progression line chart (experiment # on X, pass rate % on Y)
- Show colored bars per experiment: green = keep, red = discard, blue = baseline
- Show table of all experiments: #, score, pass rate, status, description
- Show per-eval breakdown: which evals pass/fail most across all runs
- Show current status: "Running experiment [N]..." or "Idle"
- Use clean styling: white background, pastel accents, clean sans-serif

**Update `results.json`** after every experiment. Format:

```json
{
  "skill_name": "[name]",
  "status": "running",
  "current_experiment": 3,
  "baseline_score": 70.0,
  "best_score": 90.0,
  "experiments": [
    {
      "id": 0,
      "score": 14,
      "max_score": 20,
      "pass_rate": 70.0,
      "status": "baseline",
      "description": "original skill — no changes"
    }
  ],
  "eval_breakdown": [
    {"name": "Text legibility", "pass_count": 8, "total": 10}
  ]
}
```

When run finishes, update `status` to `"complete"`.

---

## step 4: establish baseline

Run the skill AS-IS before changing anything. This is experiment #0.

1. **Ask the user what to name the new version.** Example: "What should I call the optimized version? (e.g., copywriting-v2)"
2. Create working directory: `autoresearch-[skill-name]/` inside the skill's folder
3. **Copy the original SKILL.md into the working directory as `[user-chosen-name].md`** — this is the copy you will mutate. NEVER edit the original SKILL.md.
4. Save `SKILL.md.baseline` in the working directory (identical to original — your revert target)
5. Create `results.tsv` with header row
6. Create `results.json` and `dashboard.html`, then present dashboard via canvas
7. Run the skill [N] times using the test inputs (use `[user-chosen-name].md` for all runs)
8. Score every output against every eval
9. Record baseline score, update results.tsv and results.json

**results.tsv format (tab-separated):**
```
experiment	score	max_score	pass_rate	status	description
0	14	20	70.0%	baseline	original skill — no changes
```

**IMPORTANT:** After establishing baseline, confirm the score with the user before proceeding. If baseline is already 90%+, ask if they want to continue.

---

## step 5: run the experiment loop

This is the core autoresearch loop. Once started, run autonomously until stopped.

**LOOP:**

1. **Analyze failures.** Which evals fail most? Read actual failing outputs. Identify the pattern — formatting issue? Missing instruction? Ambiguous directive?

2. **Form a hypothesis.** Pick ONE thing to change. Don't change 5 things at once.

   Good mutations:
   - Add a specific instruction addressing the most common failure
   - Reword an ambiguous instruction to be explicit
   - Add an anti-pattern ("Do NOT do X") for a recurring mistake
   - Move a buried instruction higher (priority = position)
   - Add or improve an example showing correct behavior
   - Remove an instruction causing over-optimization at the expense of something else

   Bad mutations:
   - Rewriting the entire skill from scratch
   - Adding 10 new rules at once
   - Making the skill longer without a specific reason
   - Adding vague instructions like "make it better"

3. **Make the change.** Edit `[user-chosen-name].md` (working dir only) with ONE targeted mutation. NEVER touch the original SKILL.md.

4. **Run the experiment.** Execute the skill [N] times with the same test inputs.

5. **Score it.** Run every output through every eval. Calculate total score.

6. **Decide: keep or discard.**
   - Score improved → **KEEP.** Log it. This is the new baseline for `[user-chosen-name].md`.
   - Score same → **DISCARD.** Revert to previous version. Complexity without improvement.
   - Score worse → **DISCARD.** Revert to previous version.

7. **Log the result** in results.tsv. Update results.json so dashboard refreshes.

8. **Repeat.**

**NEVER STOP.** Run autonomously until:
- The user manually stops you
- You hit the budget cap (if set)
- You hit 95%+ pass rate for 3 consecutive experiments

**If you run out of ideas:** Re-read failing outputs. Try combining two near-miss mutations. Try a completely different approach. Try removing things instead of adding. Simplification that maintains the score is a win.

---

## step 6: write the changelog

After each experiment (kept or discarded), append to `changelog.md`:

```markdown
## Experiment [N] — [keep/discard]

**Score:** [X]/[max] ([percent]%)
**Change:** [One sentence describing what was changed]
**Reasoning:** [Why this change was expected to help]
**Result:** [What actually happened — which evals improved/declined]
**Failing outputs:** [Brief description of what still fails, if anything]
```

This changelog is the most valuable artifact. Any future agent can pick it up and continue where the last one left off.

---

## step 7: deliver results

When the user returns or the loop stops, present:

1. **Score summary:** Baseline → Final (percent improvement)
2. **Total experiments run**
3. **Keep rate:** How many kept vs discarded
4. **Top 3 changes that helped most** (from changelog)
5. **Remaining failure patterns** (what the skill still gets wrong)
6. **Location of `[user-chosen-name].md`** — the improved version (original untouched)
7. **Location of results.tsv and changelog.md**

---

## output format

The skill produces these files in `autoresearch-[skill-name]/` inside the skill's folder:

```
autoresearch-[skill-name]/
├── dashboard.html       # live dashboard (presented via canvas tool)
├── results.json         # data powering the dashboard
├── results.tsv          # score log for every experiment
├── changelog.md         # detailed mutation log
├── SKILL.md.baseline    # original skill before optimization
└── [user-chosen-name].md  # the improved version
```

**The original SKILL.md is NEVER modified.** Do NOT offer to overwrite the original.

---

## example run

**Context gathered:**
- Target skill: `skills/landing-page-design-mengto/SKILL.md`
- Test inputs: "Landing page for AI productivity tool", "B2B SaaS landing page", "Creator tool landing page"
- Evals: (1) Headline includes specific number or result? (2) Free of buzzwords (revolutionary, cutting-edge, synergy)? (3) CTA uses specific verb phrase? (4) First line calls out specific pain point? (5) Total copy under 150 words?
- Runs per experiment: 5. Max score: 25.

**Baseline (experiment 0):** 14/25 (56%). Vague headlines, buzzword soup, weak CTAs.

**Experiment 1 — KEEP (18/25, 72%):** Added explicit rule: "Headline must include a specific number or result. Never use vague promises like 'Transform Your Business.'"

**Experiment 2 — KEEP (21/25, 84%):** Added banned buzzwords list: "NEVER use: revolutionary, cutting-edge, synergy, next-level, game-changing, leverage, unlock, transform."

**Experiment 3 — KEEP (23/25, 92%):** Added worked example of strong landing page section with pain point opener and CTA highlighted.

**Experiment 4 — DISCARD (21/25, 84%):** Tried tighter word count enforcement — copy got too thin, CTA suffered. Reverted.

**Final:** 14/25 → 23/25 (56% → 92%). 4 experiments, 3 kept, 1 discarded.

---

## the test

A good autoresearch run:

1. Started with a baseline — never changed anything before measuring
2. Used binary evals only — no scales, no vibes
3. Changed one thing at a time — so you know what helped
4. Kept a complete log — every experiment recorded
5. Improved the score — measurable improvement
6. Didn't overfit — skill got better at the actual job, not just passing tests
7. Ran autonomously — didn't stop to ask permission between experiments

If the skill "passes" all evals but actual output quality hasn't improved — the evals are bad, not the skill. Go back to step 2.

---

## our priority skills to run autoresearch on

High-value targets (skills with known execution gaps in REGISTRY.md):

| Skill | Why |
|-------|-----|
| `copywriting/` | 11 days of copy done without it — evals would sharpen the prompts |
| `landing-page-design-mengto/` | 6 days of LP work without it — most measurable output |
| `slide-layout/` | 22 days of deck work without it — high frequency |
| `social-content/` | X posting done ad-hoc — needs tightening |
| `cold-email/` | Measurable output, clear pass/fail criteria |