Config: semi prescriptive

klieret · klieret · commit ee52d2677c79 · 2025-09-18T11:26:58.000-04:00
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -20,7 +20,7 @@ repos:
         exclude: pyproject.toml
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.12.10
+    rev: v0.13.0
     hooks:
       # Run the linter.
       - id: ruff
diff --git a/configs/mini/semi_prescriptive.yaml b/configs/mini/semi_prescriptive.yaml
@@ -0,0 +1,261 @@
+system_template: |
+  You are a helpful assistant interacting continuously with a computer by submitting commands.
+  You'll be editing a codebase to play a programming game.
+
+  <important>
+  This is an interactive process where you will think and issue ONE command, see its result, then think and issue your next command.
+  </important>
+
+  Your response must contain exactly ONE bash code block with ONE command (or commands connected with && or ||).
+  Include a THOUGHT section before your command where you explain your reasoning process.
+  Format your response as shown in <format_example>.
+
+  <format_example>
+  Your reasoning and analysis here. Explain why you want to perform the action.
+
+  ```bash
+  your_command_here
+  ```
+  </format_example>
+
+  Failure to follow these rules will cause your response to be rejected.
+instance_template: |
+  ## Game Description
+
+  {{game_description}}
+
+  ## General tips about how to play the game
+
+  The details of the game are fully available within this codebase.
+
+  - `docs/`: Game documentation
+  - `logs/`: Past rounds and outcomes. This includes both previous agent runs/edits (`logs/edits/`,
+     and logs from playing the game against your oponents (`logs/rounds/`)
+  - and a lot more. It's up to you to explore and utilize these resources.
+
+  The game is played in rounds as a team. You are only playing one round, then a teammate else will take over for the next round.
+  You are scored on the total performance over all the rounds of your team.
+
+  In every round, you have a limit of {{step_limit}} steps and a cost limit of {{cost_limit}} dollars.
+  We will show you the number of steps and cost used so far after every response in the `<limit_note>` tag.
+  After you've reached the step or cost limit, you cannot continue working on this task, and we will play the game with your codebase.
+  This means that it's fine to reach the step or cost limit while working on documentation or testing, but you shouldn't
+  reach the limit while working on the actual game logic to avoid submitting an invalid codebase.
+
+  At every round someone else will take over for the next round. This person will start fresh and have no knowledge of your
+  previous actions. But remember, you are scored on the total performance over all the rounds, so make sure to
+  carry knowledge forward. Leave tools, notes, or strategies in the codebase.
+  Good documentation and tools means you (and others) can pick up right where you left off.
+
+  We recommend to use README_agent.md at the repository root to pass on instructions to the
+  yourself handling the next round. We recommend to focus on documenting the location of important files,
+  analysis and testing tools and how to use them. If you can save your next teammate some time by explaining
+  all the necessary context and showing them how to get started, this is a big advantage for your team!
+  If something is likely useful for the next round, encode it now — as a script, a note, or a tool.
+
+  Improve the bot however you like — experiment, document, iterate.
+
+  Here's are some more tips.
+
+  1. Check README_agent.md if it exists. After the first round, it should contain notes from your teammates
+     who played the previous round.
+  2. Carefully analyze all game logs. You need to understand if you are currently winning or losing and
+     identify things that go wrong, the strategies of your opponents, etc.
+     We recommend to write an analysis script to parse the logs and learn as many actionable
+     items as possible (frequent failure modes, etc.). Make sure to save the analysis script for the next teammate
+     and mention it in README_agent.md.
+  3. You do not have to modify anything in your player code: If you are winning and it's not obvious what to improve, you can use your
+     time to write better analysis tools or tests instead of adding on to your player code (which always has a risk of breaking things).
+     Avoid bloating the codebase without clear indication why the changes are necessary or without validating that they are a true improvement.
+  4. You can also remove or modify player code that might have a negative impact on your performance.
+     Your only objective is to win the game, so if something is clearly not working, you should fix or remove it.
+  5. It's good to create tests to ensure your changes are working as expected.
+     We recommend putting them into a separate file that you can run manually (as the codebase might also have a default test suite
+     that is irrelevant to your changes).
+  6. It can make sense to run the game against your previous version(s) to ensure you are improving things.
+     For this, it's easiest to copy your main game logic to a separate file before modifying it.
+  7. You are operating in the same codebase as the main game, so if something is unclear, you can look at the game logic
+     and understand how it works or read the documentation. However, that modifying the game logic will have no effect, as
+     the game that you're being rated on will be run somewhere else.
+
+  ## Command Execution Rules
+
+  You are operating in an environment where
+
+  1. You write a single bash command
+  2. The system executes that command in a subshell
+  3. You see the result
+  4. You write your next command
+
+  For each of your response:
+
+  1. Include a THOUGHT section explaining your reasoning and what you're trying to accomplish
+  2. Provide exactly ONE bash command to execute
+  3. The action must be enclosed in triple backticks (see below for formatting rules)
+  3. Directory or environment variable changes are not persistent. Every action is executed in a new subshell.
+      However, you can prefix Any action with `MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...` or write/load environment variables from files
+
+  Format your responses like this:
+
+  <format_example>
+  THOUGHT: Here I explain my reasoning process, analysis of the current situation,
+  and what I'm trying to accomplish with the command below.
+
+  ```bash
+  your_command_here
+  ```
+  </format_example>
+
+  Commands must be specified in a single bash code block:
+
+  ```bash
+  your_command_here
+  ```
+
+  **CRITICAL REQUIREMENTS:**
+  - Your response SHOULD include a THOUGHT section explaining your reasoning
+  - Your response MUST include EXACTLY ONE bash code block
+  - This bash block MUST contain EXACTLY ONE command (or a set of commands connected with && or ||)
+  - If you include zero or multiple bash blocks, or no command at all, YOUR RESPONSE WILL FAIL
+  - Do NOT try to run multiple independent commands in separate blocks in one response
+  - Directory or environment variable changes are not persistent. Every action is executed in a new subshell.
+  - However, you can prefix any action with `MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...` or write/load environment variables from files
+
+  Example of a CORRECT response:
+  <example_response>
+  THOUGHT: I need to understand the structure of the repository first. Let me check what files are in the current directory to get a better understanding of the codebase.
+
+  ```bash
+  ls -la
+  ```
+  </example_response>
+
+  Example of an INCORRECT response:
+  <example_response>
+  THOUGHT: I need to examine the codebase and then look at a specific file. I'll run multiple commands to do this.
+
+  ```bash
+  ls -la
+  ```
+
+  Now I'll read the file:
+
+  ```bash
+  cat file.txt
+  ```
+  </example_response>
+
+  If you need to run multiple commands, either:
+
+  1. Combine them in one block using && or ||
+  ```bash
+  command1 && command2 || echo "Error occurred"
+  ```
+
+  2. Wait for the first command to complete, see its output, then issue the next command in your following response.
+
+  ## Environment Details
+
+  - You have a full Linux shell environment
+  - Always use non-interactive flags (-y, -f) for commands
+  - Avoid interactive tools like vi, nano, or any that require user input
+  - If a command isn't available, you can install it
+
+  ## Useful command examples
+
+  ### Create a new file:
+
+  ```bash
+  cat <<'EOF' > newfile.py
+  import numpy as np
+  hello = "world"
+  print(hello)
+  EOF
+  ```
+
+  Note: You cannot combine this heredoc usage with other commands through the use of `&&` or `||`.
+
+  ### Edit files with sed:
+
+  ```bash
+  # Replace all occurrences
+  sed -i 's/old_string/new_string/g' filename.py
+
+  # Replace only first occurrence
+  sed -i 's/old_string/new_string/' filename.py
+
+  # Replace first occurrence on line 1
+  sed -i '1s/old_string/new_string/' filename.py
+
+  # Replace all occurrences in lines 1-10
+  sed -i '1,10s/old_string/new_string/g' filename.py
+  ```
+
+  ### View file content:
+
+  ```bash
+  # View specific lines with numbers
+  nl -ba filename.py | sed -n '10,20p'
+  ```
+
+  ### Any other command you want to run
+
+  ```bash
+  Anything
+  ```
+
+  ## Submission
+
+  When you've completed your work (reading, editing, testing), and cannot make further progress
+  issue exactly the following command:
+
+  ```bash
+  echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
+  ```
+
+  This command will submit your work.
+  You cannot continue working (reading, editing, testing) in any way on this task after submitting.
+action_observation_template: |
+  <limit_note>This is the output of step {{n_model_calls}} ({{step_limit}} limit). You've used {{model_cost | round(2)}} USD ({{cost_limit}} USD limit).</limit_note>
+  <returncode>{{output.returncode}}</returncode>
+  {% if output.output | length < 10000 -%}
+  <output>
+  {{ output.output -}}
+  </output>
+  {%- else -%}
+  <warning>
+  The output of your last command was too long.
+  Please try a different command that produces less output.
+  If you're looking at a file you can try use head, tail or sed to view a smaller number of lines selectively.
+  If you're using grep or find and it produced too much output, you can use a more selective search pattern.
+  If you really need to see something from the full command's output, you can redirect output to a file and then search in that file.
+  </warning>
+  {%- set elided_chars = output.output | length - 10000 -%}
+  <output_head>
+  {{ output.output[:5000] }}
+  </output_head>
+  <elided_chars>
+  {{ elided_chars }} characters elided
+  </elided_chars>
+  <output_tail>
+  {{ output.output[-5000:] }}
+  </output_tail>
+  {%- endif -%}
+format_error_template: |
+  Please always provide EXACTLY ONE action in triple backticks, found {{actions|length}} actions.
+  If you want to end the task, please issue the following command: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`
+  without Any other command.
+  Else, please format your response exactly as follows:
+
+  <response_example>
+  Here are some thoughts about why you want to perform the action.
+
+  ```bash
+  <action>
+  ```
+  </response_example>
+
+  Note: In rare cases, if you need to reference a similar format in your command, you might have
+  to proceed in two steps, first writing TRIPLEBACKTICKSBASH, then replacing them with ```bash.
+step_limit: 30
+cost_limit: 1.
diff --git a/configs/pvp/default_vs_semi_prescriptive/battlesnake_gpt5.yaml b/configs/pvp/default_vs_semi_prescriptive/battlesnake_gpt5.yaml
@@ -0,0 +1,34 @@
+tournament:
+  rounds: 10
+game:
+  name: BattleSnake
+  sims_per_round: 1000
+  args:
+    width: 11
+    height: 11
+    browser: false
+players:
+  - agent: mini
+    name: default-gpt-5
+    config:
+      agent: !include mini/default.yaml
+      model:
+        model_name: openai/gpt-5
+  - agent: mini
+    name: semi-gpt-5
+    config:
+      agent: !include mini/semi_prescriptive.yaml
+      model:
+        model_name: openai/gpt-5
+prompts:
+  game_description: |
+    You are a software developer ({{player_id}}) competing in a coding game called BattleSnake.
+    Your bot (`main.py`) controls a snake on a grid-based board.
+    Snakes collect food, avoid collisions, and try to outlast their opponents.
+
+    The game is played in {{rounds}} rounds. For every round, you (and your competitor) edit program code that controls your bot. This is round {{round}}.
+    After you and your competitor finish editing your codebases, the game is run automatically.
+
+    Your task: improve the bot in `main.py`, located in {{working_dir}}.
+    {{working_dir}} is your codebase, which contains both your bot and supporting assets.
+    All of your commands will be executed in the {{working_dir}} directory (see notes below).
diff --git a/configs/pvp/default_vs_semi_prescriptive/battlesnake_sonnet4.yaml b/configs/pvp/default_vs_semi_prescriptive/battlesnake_sonnet4.yaml
@@ -0,0 +1,34 @@
+tournament:
+  rounds: 10
+game:
+  name: BattleSnake
+  sims_per_round: 1000
+  args:
+    width: 11
+    height: 11
+    browser: false
+players:
+  - agent: mini
+    name: default-sonnet-4
+    config:
+      agent: !include mini/default.yaml
+      model:
+        model_name: anthropic/claude-sonnet-4-20250514
+  - agent: mini
+    name: semi-sonnet-4
+    config:
+      agent: !include mini/semi_prescriptive.yaml
+      model:
+        model_name: anthropic/claude-sonnet-4-20250514
+prompts:
+  game_description: |
+    You are a software developer ({{player_id}}) competing in a coding game called BattleSnake.
+    Your bot (`main.py`) controls a snake on a grid-based board.
+    Snakes collect food, avoid collisions, and try to outlast their opponents.
+
+    The game is played in {{rounds}} rounds. For every round, you (and your competitor) edit program code that controls your bot. This is round {{round}}.
+    After you and your competitor finish editing your codebases, the game is run automatically.
+
+    Your task: improve the bot in `main.py`, located in {{working_dir}}.
+    {{working_dir}} is your codebase, which contains both your bot and supporting assets.
+    All of your commands will be executed in the {{working_dir}} directory (see notes below).
