# Autoresearch Changelog — visual-research **Skill:** visual-research **Version:** visual-research-v2 **Started:** 2026-03-18 **Runs per experiment:** 5 **Max score per experiment:** 30 (6 evals × 5 runs) ## Eval Suite **EVAL 1: Positioning DNA First** Question: Does the output extract full positioning DNA (strategy, contrast, keywords, competitors) before any visual search? Pass: Positioning brief covering strategy, contrast, competitors, and affinity keywords appears before brand/reference capture Fail: Visual references appear before or without positioning brief **EVAL 2: Test Question** Question: Does the output define a single Test Question to filter all references? Pass: A specific, binary-style question is stated that will filter every reference choice Fail: No Test Question, or multiple questions, or only vague criteria **EVAL 3: Evocative Direction Names** Question: Does the output produce 2–4 directions with distinct, evocative names (not generic like "Option A" or "Direction 1")? Pass: 2–4 named directions with expressive, brand-relevant names (e.g. "Digital Rebel", "Quiet Earth") Fail: Fewer than 2 directions, more than 4, or generic names like "Option A / B" or "Modern Look" **EVAL 4: Named Brand References** Question: Does each direction include 3–5 real, named brand references? Pass: Every direction lists 3–5 actual brand names (not categories) as visual references Fail: Any direction has fewer than 3 named brand references, or lists category names instead of brands **EVAL 5: Positioning Connection** Question: Does each direction explicitly connect back to brand positioning? Pass: Each direction includes a "Why It Works" or equivalent sentence linking the visual direction to the positioning strategy Fail: Any direction lacks explicit positioning rationale **EVAL 6: HTML Output** Question: Is the output structured as visual boards/HTML output (not a plain text report)? Pass: Output includes an HTML file or explicit HTML structure for visual boards Fail: Output is plain text, markdown, or prose description without HTML --- ## Experiment 4 — keep (adversarial validation, no mutation) **Score:** 30/30 (100.0%) **Change:** None — validation run with adversarial underspecified inputs **Reasoning:** Confirming the score holds under worst-case conditions before declaring success **Result:** - All 5 runs pass all 6 evals including 2 adversarial inputs with minimal context - Anti-patterns provided guard-rails: agents self-corrected bad Test Questions and generic direction names - Confirmed: skill is robust under pressure **Failing outputs:** None. **Note:** This is the 3rd consecutive experiment at 95%+. **STOPPING CRITERION MET.** ## Experiment 3 — keep **Score:** 30/30 (100.0%) **Change:** Added direction name anti-patterns (❌ NEVER use single adjectives or Option A/B/C), naming formula ([Emotion/Stance] + [Object/Role]), and reinforced brand references must be "real named brands, not categories" **Reasoning:** E3 improved in Exp 2 via cascade but is still vulnerable if upstream Test Question isn't as strong. Adding explicit anti-patterns for direction names hardens E3 directly. Reinforcing "real named brands" targets the E4 failure mode (category names). **Result:** - E3 (Evocative names): maintained 5/5 — anti-patterns caught generic attempts; naming formula guided better names - E4 (Named brands): maintained 5/5 — "not categories" language prevented brand-category substitution - All other evals maintained 5/5 - Stress tests (minimal input) passed all evals **Failing outputs:** None. **Note:** This is the 2nd consecutive experiment at 95%+. One more needed to stop. ## Experiment 2 — keep **Score:** 30/30 (100.0%) **Change:** Added "Test Question — Rules" block with explicit anti-patterns (❌ bad Test Questions) and examples (✅ good Test Questions) to Phase 1 **Reasoning:** E2 failed when agents wrote vague filter criteria. Anti-patterns show what NOT to write, preventing the lazy shortcut. Expected +1 on E2; hoped for cascade to E3. **Result:** - E2 (Test Question): 4/5 → 5/5 (+1) — anti-patterns caught vague questions; agents revised to specific tension-based questions - E3 (Evocative names): 4/5 → 5/5 (+1) — unexpected cascade: better Test Questions upstream led to more precise direction naming downstream - All other evals maintained 5/5 **Failing outputs:** None in this experiment. **Note:** This is the 1st consecutive experiment at 95%+ (100.0%). ## Experiment 1 — keep **Score:** 28/30 (93.3%) **Change:** Added mandatory "MANDATORY OUTPUT FORMAT: HTML Direction Boards" section after Phase 4 **Reasoning:** E6 was 0/5 in baseline — the biggest gap. Skill referenced an HTML template but never instructed agents to create HTML. Added explicit section with HTML structure, creation rules, and "plain-text is NOT acceptable" language. **Result:** - E6 (HTML output): 0/5 → 5/5 (+5) — all runs now produce HTML - E4 (named brands): 3/5 → 5/5 (+2) — clearer template forced explicit brand naming - E2 (Test Question): still 4/5 — Run 4 still produces vague filtering criteria - E3 (Evocative names): still 4/5 — Run 4 still produces functional/generic names **Remaining failures:** Run 4 pattern: when agent goes shallow on positioning, both Test Question and direction names become generic. Root cause is upstream in Phase 1. --- ## Final Summary **Baseline → Final:** 21/30 (70.0%) → 30/30 (100.0%) — **+30 percentage points improvement** **Experiments run:** 5 (0 baseline + 4 loop) **Keep rate:** 4/4 (100%) **Consecutive 95%+ experiments:** 3 (Experiments 2, 3, 4) **Stop reason:** 3 consecutive experiments at 95%+ **Top 3 changes:** 1. **Mandatory HTML output section** (+23.3pp) — Single biggest gain; skill never instructed HTML creation 2. **Test Question anti-patterns** (+6.7pp) — Prevented vague filter criteria; cascaded to better direction names 3. **Direction name anti-patterns + formula** (robustness) — Hardened E3 against underspecified inputs **Optimized skill location:** `visual-research-v2.md` (this directory) **Original untouched:** `SKILL.md` in `/root/.openclaw/workspace/skills/visual-research/` --- ## Experiment 0 — baseline **Score:** 21/30 (70.0%) **Change:** None — original skill AS-IS **Reasoning:** Establishing baseline before any mutations **Result:** - E1 (DNA first): 5/5 PASS ✓ - E2 (Test Question): 4/5 — fails when agent skips to vague filtering criteria - E3 (Evocative names): 4/5 — fails when agent defaults to generic names ("Modern Banking", "Digital Services") - E4 (3-5 named brands): 3/5 — fails when agent names categories or produces thin lists - E5 (Connects to positioning): 5/5 PASS ✓ - E6 (HTML output): 0/5 FAIL — skill never instructs HTML creation; only references existing template path **Primary failure:** E6 (HTML) at 0% is the biggest gap. E4 (named brands) also weak at 60%. **Failing outputs:** All 5 runs fail E6. Runs 2, 5 fail E4 (category names instead of brand names). Run 4 fails E2 + E3 (vague Test Question, generic direction names).