Merge pull request #11 from binghe2727/main

Add a-evolve tb2 harness for clawcode

Author: binghe2727

artifacts/tb2_clawcode_opus46/TB2_README.md

@@ -0,0 +1,118 @@
+# Terminal-Bench 2 (TB2) — Experiment Report
+
+Terminal-Bench 2 is an 89-task benchmark where an LLM agent solves command-line challenges inside Docker containers. Each task provides a prompt, a container image, and a `test.sh` evaluation script. The agent interacts via `bash()`, `python()`, `read_skill()`, and `submit()` tools in a ReAct loop.
+
+This document describes how we run TB2, the task-guidance prompt and evolved skills used, and the results we obtained.
+
+---
+
+## Task-Guidance Prompt
+
+Before each task, the agent receives the following system prompt (see [`prompt.md`](prompt.md)):
+
+```text
+You are an AI assistant tasked with solving command-line tasks in a Linux environment.
+You will be given a task description and the output from previously executed commands.
+You have several tools available to help with finding the solution. You are running as
+root inside a Docker container. Do not use sudo — it is not installed and not needed
+since you already have root privileges.
+
+Your goal is to solve the task by providing batches of shell commands. If you need to
+perform multiple actions, you can always send more messages with additional tool calls.
+
+Before taking action, you should:
+  1. ANALYZE the current state based on previous tool outputs
+  2. PLAN your next steps — what commands will you run and why
+
+Format your reasoning as follows before calling tools:
+
+  **Analysis:** [Analyze the current state.]
+  **Plan:** [Describe your plan for the next steps.]
+
+Then call the appropriate tools to execute your plan.
+
+Important: Each bash() call is independent — no state is preserved between calls.
+If you need to run commands in sequence, chain them with &&
+
+After you think you have completed the task, read the self-verification skill to
+verify your solution.
+
+When you have completed and verified the task, call the submit() tool with "DONE"
+as argument to report that the task is complete.
+```
+
+Key design choices in this prompt:
+
+- Explicit Analysis → Plan reasoning structure before every tool call.
+- Reminder that `bash()` calls are stateless — commands must be chained with `&&`.
+- A self-verification nudge: the agent is told to `read_skill("self-verification")` before submitting, which loads a checklist that catches common mistakes.
+
+---
+
+## Evolved Skills (7 Total)
+
+Skills are Markdown documents loaded on-demand via the `read_skill(name)` tool. Seven skills were produced by the A-EVOLVE adaptive_skill pipeline.
+
+### Skill Overview
+
+| # | Skill | Note | Purpose |
+|---|-------|--------|---------|
+| 1 | `self-verification` | Seed | Checklist to verify every requirement before calling `submit()` |
+| 2 | `environment-discovery` | Seed | Quickly discover tools, languages, and files in the container |
+| 3 | `debug-and-fix` | Seed | Fix build failures, write structured output, solve constraint tasks |
+| 4 | `scientific-computing` | Seed | Numerical methods, bioinformatics, ML training, logic circuits |
+| 5 | `build-compiled-extensions` | Seed | Build C/C++/Cython/Fortran extensions and frameworks from source |
+| 6 | `python-data-analysis` | Seed | Multi-step Python, HuggingFace datasets, stateless `python()` patterns |
+| 7 | `systematic-exploration` | Seed | Avoid dead ends, backtrack when stuck, verify interpretations |
+
+Skills are stored in `skills/<name>/SKILL.md` and injected into the conversation when the agent calls `read_skill("<name>")`.
+
+### Skills 
+
+- `self-verification` — A four-step checklist (re-read requirements → verify each one → check assumptions → review common pitfalls). Invoked automatically before submission thanks to the prompt nudge.
+- `environment-discovery` — One-liner commands to probe the container (`which`, `ls /app/`, `pip list`, etc.) so the agent doesn't waste turns guessing what's installed.
+- `debug-and-fix` — Recipes for C/C++ build fixes, binary inspection without `xxd`, writing complete structured files (ICS/JSON/XML) via Python to avoid heredoc truncation, and systematic constraint-satisfaction solvers.
+- `scientific-computing` — Patterns for bioinformatics (primer design, FASTA parsing), logic-circuit simulation, numerical optimization with SciPy, and ML training on CPU-only containers.
+- `build-compiled-extensions` — Step-by-step workflows for Cython, GCC manual builds, large framework compilation (Caffe, OpenCV), and protobuf version mismatch fixes.
+- `python-data-analysis` — Encodes the critical rule that each `python()` call starts with a blank slate (no state carries over). Default strategy: write a complete `.py` script to a file, then execute it via `bash()`. Includes patterns for HuggingFace dataset loading, token counting, and saving intermediate state to disk between calls.
+- `systematic-exploration` — Meta-strategies for when the agent gets stuck: don't reject an approach after a single failure (vary parameters widely first), backtrack after 5+ unproductive turns, verify task interpretation before committing, and treat independent requirements separately.
+
+---
+
+## Results on TB2
+
+We compare two configurations:
+
+- **Baseline** — Vanilla harness, no skills, default system prompt.
+- **Opus 4.6 + Evolved Skills** — The full setup described above: updated task-guidance prompt with self-verification nudge and all 7 skills.
+
+### Per-Run Results
+
+| Run | Configuration | Pass Rate |
+|-----|---------------|-----------|
+| 1 | Baseline | 67.1% |
+| 2 | Baseline | 67.0% |
+| 3 | Baseline | 69.4% |
+| 4 | Opus 4.6 + Evolved Skills | 74.1% |
+| 5 | Opus 4.6 + Evolved Skills | 69.4% |
+| 6 | Opus 4.6 + Evolved Skills | 75.3% |
+
+### Summary
+
+| Configuration | Avg Pass Rate | Range |
+|---------------|---------------|-------|
+| Baseline | 67.8% | 67.0–69.4% |
+| Opus 4.6 + Evolved Skills | 72.9% | 69.4–75.3% |
+| **Improvement** | **+5.1 pp** | |
+
+---
+
+## What Drove the Improvement
+
+The gain comes from four changes layered on top of the baseline:
+
+1. **The evolved skills** — Skills address the common failure modes (e.g., stateless `python()` confusion and premature approach abandonment.) 
+2. **Self-verification nudge in prompt**  — The explicit instruction to read the `self-verification` skill before submitting catches last-mile mistakes.
+
+---
+

artifacts/tb2_clawcode_opus46/prompt.md

@@ -0,0 +1,23 @@
+You are an AI assistant tasked with solving command-line tasks in a Linux environment.
+You will be given a task description and the output from previously executed commands.
+You have several tools available to help with finding the solution. You are running as root inside a Docker container. Do not use sudo - it is not installed and not needed since you already have root privileges.
+
+Your goal is to solve the task by providing batches of shell commands. If you need to perform multiple actions, you can always send more messages with additional tool calls.
+
+Before taking action, you should:
+  1. ANALYZE the current state based on previous tool outputs
+  2. PLAN your next steps - what commands will you run and why
+
+Format your reasoning as follows before calling tools:
+
+  **Analysis:** [Analyze the current state. What do you see? What has been accomplished? What still needs to be done?]
+
+  **Plan:** [Describe your plan for the next steps. What commands will you run and why? Be specific about what you expect each command to accomplish.]
+
+Then call the appropriate tools to execute your plan.
+
+Important: Each bash() call is independent - no state is preserved between calls. If you need to run commands in sequence, chain them with &&
+
+After you think you have completed the task, read the self-verification skill to verify your solution.
+
+When you have completed and verified the task, call the `submit()` tool with "DONE" as argument to report that the task is complete.

artifacts/tb2_clawcode_opus46/skills/build-compiled-extensions/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: build-compiled-extensions
+description: How to build C/C++/Cython/Fortran extensions and frameworks like Caffe, OpenCV, protobuf-dependent projects from source.
+---
+
+# Building Compiled Extensions & Frameworks
+
+## Pre-flight
+```bash
+pip install setuptools wheel cython 2>/dev/null
+apt-get install -y build-essential python3-dev 2>/dev/null
+```
+
+## Cython extension workflow
+```bash
+cython file.pyx   # Cythonize to .c first
+python setup.py build_ext --inplace 2>&1
+# Manual fallback:
+gcc -shared -fPIC -O2 $(python3-config --includes) -o mod$(python3-config --extension-suffix) mod.c
+```
+
+## Building large C++ frameworks (Caffe, etc.)
+1. Check protobuf version: `protoc --version` and `dpkg -l | grep protobuf`
+2. Fix API incompatibilities before building (see debug-and-fix skill)
+3. Use `make -j$(nproc)` but capture errors: `make 2>&1 | tee build.log`
+4. For Caffe specifically:
+   - Set `CPU_ONLY := 1` in Makefile.config for non-GPU environments
+   - Fix protobuf SetTotalBytesLimit calls for newer protobuf
+   - Use `make all -j$(nproc) && make test && make runtest`
+
+## When pip install times out
+1. Install dependencies individually first: `pip install numpy scipy`
+2. Build extensions only: `python setup.py build_ext --inplace`
+3. Or: `pip install --no-deps -e .`
+
+## Common pitfalls
+- **Missing numpy headers**: `pip install numpy` before building numpy-dependent C extensions
+- **Import from wrong dir**: Test from `/tmp`, not the source directory
+- **Protobuf mismatch**: Check version and fix API calls before compiling
+
+## Verification
+```bash
+find . -name "*.so" -newer setup.py
+cd /tmp && python -c "import mypackage; print('OK')"
+```

artifacts/tb2_clawcode_opus46/skills/debug-and-fix/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: debug-and-fix
+description: Strategies for debugging build failures, runtime errors, writing complete output files (ICS/JSON/config), and constraint satisfaction tasks.
+---
+
+# Debug and Fix Skill
+
+## 1. C/C++ build fixes
+```bash
+apt-get update && apt-get install -y build-essential pkg-config
+```
+### Protobuf API fix
+```bash
+find . -name "*.cc" -o -name "*.cpp" | xargs sed -i \
+  's/SetTotalBytesLimit(\([^,]*\),[^)]*)/SetTotalBytesLimit(\1)/g'
+```
+
+## 2. Binary inspection without xxd
+```bash
+od -A x -t x1z -v file.dat | head -100
+```
+
+## 3. Writing complete structured output files
+Use Python to write ICS/JSON/XML — avoids heredoc truncation:
+```python
+from datetime import datetime
+lines = ['BEGIN:VCALENDAR','VERSION:2.0','BEGIN:VEVENT',
+    f'DTSTART:{start:%Y%m%dT%H%M%S}',
+    f'DTEND:{end:%Y%m%dT%H%M%S}',
+    f'SUMMARY:{title}',
+    'END:VEVENT','END:VCALENDAR']
+with open('/app/out.ics','w') as f:
+    f.write('\r\n'.join(lines)+'\r\n')
+```
+
+## 4. Scheduling / constraint satisfaction
+- Parse ALL constraints before attempting solutions
+- Write a systematic solver to a .py file (enumerate candidates, check all constraints)
+- Verify solution satisfies EVERY constraint before writing output
+```python
+# Write complete solver to file to avoid truncation:
+with open('/app/solver.py','w') as f:
+    f.write('''...all imports, logic, output...''')
+# Then run: python3 /app/solver.py
+```
+
+## 5. Implementation code
+- Write complete files — don't build up fragments
+- For complex logic, use Python to write files programmatically
+- Test immediately after writing
+- If tests fail, rewrite the COMPLETE file

artifacts/tb2_clawcode_opus46/skills/environment-discovery/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: environment-discovery
+description: Strategies for quickly discovering what tools, languages, and files are available in a Terminal-Bench container.
+---
+
+# Environment Discovery Skill
+
+When starting a new Terminal-Bench challenge, quickly assess the environment.
+
+## 1. Check available tools and languages
+```bash
+which python python3 pip node npm gcc g++ make cmake perl ruby 2>/dev/null
+```
+
+## 2. Check the filesystem
+```bash
+ls /app/ 2>/dev/null
+find /app -type f 2>/dev/null | head -30
+```
+
+## 3. Check pre-installed task-specific tools
+Many containers have specialized tools already installed:
+```bash
+# Security/crypto tools
+which john hashcat 7z 7za openssl 2>/dev/null
+ls /app/john/run/ 2>/dev/null
+# Bio tools
+which oligotm primer3_core samtools 2>/dev/null
+# Data tools
+pip list 2>/dev/null | head -30
+```
+
+## 4. If python3 is missing, try alternatives
+```bash
+which python perl 2>/dev/null  # python2 may exist as 'python'
+# Use perl for text processing if python unavailable
+# Use /app/john/run/john for password cracking (has its own perl scripts)
+```
+
+## 5. Pre-installed tool locations
+- John the Ripper scripts: `/app/john/run/*.pl` (perl-based, use `perl` not `python3`)
+- Task binaries often in `/app/` or `/usr/local/bin/`
+
+## Tips
+- Check `/app/` first — it often contains task-specific files and tools
+- Don't assume python3 exists — verify first, use bash/perl alternatives
+- Use `dpkg -l` or `apt list --installed` to see system packages

artifacts/tb2_clawcode_opus46/skills/python-data-analysis/SKILL.md

@@ -0,0 +1,57 @@
+---
+name: python-data-analysis
+description: Best practices for multi-step Python tasks including data analysis, HuggingFace datasets, token counting, and any task requiring state across multiple python() calls.
+---
+
+# Python Multi-Step Tasks
+
+## RULE #1: Each python() call starts fresh — NO state carries over
+Variables, imports, data from previous calls DO NOT EXIST. This is the #1 source of NameError.
+
+**DEFAULT STRATEGY: Write a complete .py script to a file, then run it.**
+
+```python
+with open('/app/solve.py', 'w') as f:
+    f.write('''#!/usr/bin/env python3
+import numpy as np
+# ALL logic in one file
+data = np.load("/app/data.npy")
+result = process(data)
+with open("/app/answer.txt","w") as out:
+    out.write(str(result))
+''')
+```
+Then: `bash("python3 /app/solve.py && cat /app/answer.txt")`
+
+## If you must use multiple python() calls:
+1. **Save state to files between calls:**
+```python
+import json, numpy as np
+# Save: json.dump(obj, open('/tmp/state.json','w'))
+# Save: np.save('/tmp/arr.npy', array)
+# Load: obj = json.load(open('/tmp/state.json'))
+```
+2. **Every call MUST re-import and re-load** — never reference prior variables
+3. **NameError = forgot to re-define** — add missing definitions, don't just re-run
+
+## HuggingFace Datasets + Token Counting (ONE call)
+```python
+from datasets import load_dataset
+from transformers import AutoTokenizer
+ds = load_dataset("org/name", split="train")
+tok = AutoTokenizer.from_pretrained("model-name")
+total = sum(len(tok.encode(r["text"])) for r in ds if r["domain"] == "science")
+with open("/app/answer.txt","w") as f: f.write(str(total))
+```
+
+## For iterative exploration, keep blocks self-contained
+```python
+# Each block: imports + load + process + print
+import pandas as pd
+df = pd.read_csv('f.csv')
+print(df.describe())
+```
+
+## Verification
+- Print/cat output files BEFORE submitting
+- Confirm formats match expected schema exactly

artifacts/tb2_clawcode_opus46/skills/scientific-computing/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: scientific-computing
+description: Strategies for scientific computing, numerical methods, bioinformatics/DNA tasks, logic circuit design, algorithmic challenges, and ML training tasks.
+---
+
+# Scientific Computing & Algorithmic Tasks
+
+## CRITICAL: Write complete scripts to files for complex tasks
+Multi-step scientific analysis MUST be written as a single .py file to avoid state loss:
+```python
+with open('/app/solve.py', 'w') as f:
+    f.write("""#!/usr/bin/env python3
+import numpy as np
+# ... complete self-contained solution ...
+""")
+# Then run: bash("python3 /app/solve.py")
+```
+
+## Bioinformatics / DNA primer design
+- Use `oligotm` CLI if available for Tm calculation (check with `which oligotm`)
+- Use `primer3_core` if available for automated primer design
+- Key Tm parameters: `-tp 1 -sc 1 -mv 50 -dv 2 -n 0.8 -d 500` (Santa Lucia, salt-corrected)
+- Primer length: 15-30 bp, GC content 40-60%, Tm 55-65°C
+- For Gibson/Golden Gate assembly: add overlaps/BsaI sites to 5' end of primers
+- Parse FASTA files: handle multi-line sequences, strip whitespace
+- Write results in FASTA format to the expected output path
+
+## Logic circuit / hardware design
+- Read simulator code FIRST to understand gate semantics and timing
+- Key patterns: ripple-carry adder, mux, shift register
+- For Fibonacci mod 2^N: doubling method or iterative with registers
+
+## Numerical / distribution tasks
+- For optimization: use scipy.optimize (fsolve, minimize, root)
+- When searching distributions: verify KL divergence, entropy constraints
+- Always validate: check sums to 1, non-negative, constraint satisfaction
+
+## ML training tasks
+- Check GPU: `python3 -c "import torch; print(torch.cuda.is_available())"`
+- CPU-only: smaller batch size, simpler models, fewer epochs
+- Save checkpoints frequently
+
+## Verification
+- Verify output format matches task specification exactly
+- Save results to exact paths mentioned in task prompt
+- Compare against reference values for correctness