|
| 1 | +--- |
| 2 | +description: Writing the viewer/inspector/trajectory viewer |
| 3 | +globs: |
| 4 | +alwaysApply: false |
| 5 | +--- |
| 6 | + |
| 7 | +You are building a trajectory viewer for an AI agent benchmark that pits |
| 8 | +two AI agent in a round-based game against each other. |
| 9 | + |
| 10 | +## Framework notes |
| 11 | + |
| 12 | +* Write a python server using flask |
| 13 | +* As much output as possible should be formatted uses jinja templates |
| 14 | +* JS should be kept to a minimum |
| 15 | +* Your input is a game folder (described below) |
| 16 | +* Use clean modern CSS, but don't go too crazy, simplicity is key |
| 17 | +* This is a single page application |
| 18 | + |
| 19 | +## Input folder |
| 20 | + |
| 21 | +The input folders are all subfolders of `logs/` |
| 22 | + |
| 23 | +The input folder has the following files |
| 24 | + |
| 25 | +* results.json |
| 26 | +* main.log |
| 27 | +* round_$i.log, where $i is the round. Infer the number of rounds from how many of these files we have |
| 28 | +* .*_p$i_r$j.traj.log, where $i$ is the player (there's always 1 or 2 players and $j is the round number |
| 29 | +* Trajectory files do not exist for the last round |
| 30 | + |
| 31 | +Here's how a `.traj.log` file looks: |
| 32 | + |
| 33 | + |
| 34 | +``` |
| 35 | +{ |
| 36 | + "info": { |
| 37 | + "exit_status": null, |
| 38 | + "submission": null, |
| 39 | + "memory": null, |
| 40 | + "model_stats": { |
| 41 | + "instance_cost": 0.0661002, |
| 42 | + "api_calls": 11 |
| 43 | + } |
| 44 | + }, |
| 45 | + "messages": [ |
| 46 | + { |
| 47 | + "role": "system", |
| 48 | + "content": "You are a helpful assistant that can interact with a computer.\n\nYour response must contain exactly ONE bash code block with ONE command (or commands connected with && or ||).\nInclude a THOUGHT section before your command where you explain your reasoning process.\nFormat your response as shown in <format_example>.\n\n<format_example>\nYour reasoning and analysis here. Explain why you want to perform the action.\n\n```bash\nyour_command_here\n```\n</format_example>\n\nFailure to follow these rules will cause your response to be rejected.\nTo finish, issue the following command: `echo MINI_SWE_AGENT_FINAL_OUTPUT`\nwithout any other command." |
| 49 | + }, |
| 50 | + { |
| 51 | + "role": "user", |
| 52 | + "content": [ |
| 53 | + { |
| 54 | + "type": "text", |
| 55 | + "text": "Please solve this issue: in gitconfig, add a new alias ldc which copies the last diff ..." |
| 56 | + } |
| 57 | + ] |
| 58 | + }, |
| 59 | + { |
| 60 | + "role": "assistant", |
| 61 | + "content": "THOUGHT: I need to first look at the current directory to see the gitconfig file and understand its structure. Then I'll examine the existing aliases to understand the pattern and add the new 'ldc' alias that combines 'ld' (last diff) with clipboard functionality.\n\n```bash\nls -la\n```" |
| 62 | + }, |
| 63 | + { |
| 64 | + "role": "user", |
| 65 | + "content": [ |
| 66 | + { |
| 67 | + "type": "text", |
| 68 | + "text": "<returncode>0</returncode>\n<output>\ntotal 1232\ndrwxr-xr-x ..." |
| 69 | + } |
| 70 | + ] |
| 71 | + }, |
| 72 | + { |
| 73 | + "role": "assistant", |
| 74 | + "content": "THOUGHT: I can see there's a `gitconfig.sh` file in the current directory. Let me examine its contents to understand the current structure and find the existing aliases, particularly looking for the 'ld' alias and any clipboard-related aliases that I need to combine.\n\n```bash\ncat gitconfig.sh\n```" |
| 75 | + }, |
| 76 | + "trajectory_format": "mini-swe-agent-1" |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +## Page design |
| 81 | + |
| 82 | +### Header |
| 83 | + |
| 84 | +* Here's some control elements to choose the input folder or visualization settings |
| 85 | +* There's a dark/light mode switch |
| 86 | + |
| 87 | +### Main body |
| 88 | + |
| 89 | +#### Overall results |
| 90 | + |
| 91 | +* At the top, we show the content of `metadata.json` |
| 92 | +* There's a foldout that displays main.log (hidden by default) |
| 93 | + |
| 94 | +#### Rounds |
| 95 | + |
| 96 | +First, show a foldout of `round_1.log`. |
| 97 | + |
| 98 | +Then, show a foldout with the trajectory view of player 1 round 1 (see below for trajectory view) |
| 99 | +Then, show a foldout with the content of trajectory player 1 round 1 ["submission"] |
| 100 | +Then, show a foldout with the content of trajectory player 1 round 1 ["memory"] |
| 101 | +Then, show a foldout with the trajectory view of player 2 round 1 (see below for trajectory view). |
| 102 | +Then, show a foldout with the content of trajectory player 2 round 1 ["submission"] |
| 103 | +Then, show a foldout with the content of trajectory player 2 round 1 ["memory"] |
| 104 | + |
| 105 | +Then, show a foldout of `round_2.log`. |
| 106 | + |
| 107 | +Then, show a foldout with the trajectory view of player 1 round 2 (see below for trajectory view) |
| 108 | +Then, show a foldout with the trajectory view of player 2 round 2 (see below for trajectory view). |
| 109 | +Then, show a foldout with the content of trajectory player 2 round 2 ["submission"] |
| 110 | +Then, show a foldout with the content of trajectory player 2 round 2 ["memory"] |
| 111 | + |
| 112 | +etc. |
| 113 | + |
| 114 | +Note that he last round only has the `round_n.log`, but there's no more trajectories for the last one. |
| 115 | + |
| 116 | +#### Trajectory view |
| 117 | + |
| 118 | +This should be its own jinja template. |
| 119 | + |
| 120 | +First show the following properties from the `info` section |
| 121 | + |
| 122 | +* Number of LM calls (api calls) |
| 123 | +* Cost (instance cost) |
| 124 | + |
| 125 | +Then, show all the messages. Each message should be its own block together with the role. |
| 126 | +Because the content can be long, make sure to have a foldout for the content. |
0 commit comments