Skip to content

Commit 087220b

Browse files
committed
Improve modern Codex profiler breakdowns
1 parent 2d265e8 commit 087220b

3 files changed

Lines changed: 662 additions & 63 deletions

File tree

docs/agent-conversation-profile.md

Lines changed: 34 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ The output directory contains:
4848
SnakeViz.
4949
- `<left-label>.snakeviz.prof`: SnakeViz profile for the left conversation.
5050
- `<right-label>.snakeviz.prof`: SnakeViz profile for the right conversation.
51+
- `combined.snakeviz.stats.md`: SnakeViz-style stats table for the combined
52+
synthetic profile.
53+
- `<left-label>.snakeviz.stats.md`: SnakeViz-style stats table for the left
54+
conversation.
55+
- `<right-label>.snakeviz.stats.md`: SnakeViz-style stats table for the right
56+
conversation.
5157
- `combined.speedscope.json`: combined duration-weighted Speedscope profile.
5258
- `<left-label>.speedscope.json`: Speedscope profile for the left conversation.
5359
- `<right-label>.speedscope.json`: Speedscope profile for the right
@@ -57,7 +63,7 @@ The output directory contains:
5763
- `<right-label>.folded`: folded-stack file for the right conversation.
5864
- `summary.json`: machine-readable totals by kind, tool, timing quality, and
5965
longest events. It also includes high-level `totals_by_activity`,
60-
`tool_groups`, and excluded `metadata`.
66+
`tool_groups`, `tool_commands`, and excluded `metadata`.
6167
- `summary.md`: human-readable comparison tables.
6268

6369
Open any `*.perfetto.json` file in Perfetto or another Chrome trace viewer.
@@ -73,6 +79,12 @@ Open SnakeViz profiles with:
7379
snakeviz <file>.snakeviz.prof
7480
```
7581

82+
Open the adjacent `*.snakeviz.stats.md` file for a sortable-source text view of
83+
the same synthetic pstats frames using cProfile-style columns: `ncalls`,
84+
`tottime`, `percall`, `cumtime`, `percall`, frame, and callers. The table also
85+
includes a `Tool Command Breakdown` section so loc commands appear as
86+
`tool_group=loc` with subcommands such as `push`, `status`, and `create-page`.
87+
7688
Render folded stacks with Brendan Gregg's FlameGraph tooling:
7789

7890
```bash
@@ -82,26 +94,34 @@ flamegraph.pl --countname=us <file>.folded > <file>.svg
8294
If two labels sanitize to the same filename, the split trace, SnakeViz,
8395
Speedscope, and folded-stack filenames are deconflicted with their input side.
8496
The exact paths are listed in `summary.json` under `outputs.split`,
85-
`outputs.snakeviz`, `outputs.speedscope`, and `outputs.flamegraph`.
97+
`outputs.snakeviz`, `outputs.snakeviz_stats`, `outputs.speedscope`, and
98+
`outputs.flamegraph`.
8699

87100
## Activity Model
88101

89102
Viewer profiles and the `Time By Activity` summary use derived activity buckets
90103
that are meant for high-level agent timing:
91104

92-
- `tool`: time spent calling and waiting for a tool call to finish. This is
93-
derived from tool-call records and, when a matching result exists, spans from
94-
the call timestamp to the result timestamp. Tool-result records remain in the
95-
raw kind summary but are not counted as separate tool activity.
105+
- `tool`: time spent running an agent action until it completes. This is
106+
derived from tool-call and file-change records and, when a matching result
107+
exists, spans from the start timestamp to the result timestamp. Tool-result
108+
and file-change-result records remain in the raw kind summary but are not
109+
counted as separate activity because their time is represented by the parent
110+
tool span.
96111
- `reasoning`: reasoning or thinking records.
97112
- `user_query`: user-message records.
98113
- `agent_response`: assistant text response records.
99114
- `system`: system-message records.
100115
- `other`: non-metadata records that do not fit the activity model.
101116

102-
Tool activity is additionally grouped in `tool_groups`. Bash calls whose shell
103-
segment invokes a `loc` executable are reported as `bash_loc`; other Bash calls
104-
are reported as `bash_other`; non-Bash tools keep their tool name.
117+
Tool activity uses a two-level stack hierarchy. The first frame is the broad
118+
tool group: `loc` for shell calls that invoke the Locality CLI, or `non_loc` for
119+
everything else. The second frame is the command detail. For `loc`, that detail
120+
is the loc subcommand, for example `status`, `diff`, `push`, or `create-page`.
121+
Shell commands that run more than one loc subcommand join the subcommands with
122+
`+`, for example `diff+status`. For non-loc shell calls, the detail is the
123+
executable name such as `git`, `gh`, `sed`, or `find`; for non-shell tools, it is
124+
the tool name such as `list_issues` or `API-post-search`.
105125

106126
Harness metadata records, such as Claude terminal `result:success` records,
107127
`system:turn_duration`, `system:local_command`, attachments, file-history
@@ -127,7 +147,8 @@ present. Metadata timestamps can still act as run-boundary markers, but metadata
127147
durations do not extend wall time; this keeps terminal aggregate records from
128148
doubling the apparent runtime.
129149

130-
Every trace slice and summary bucket carries `timing_quality` as `measured` or
131-
`inferred`. Treat inferred reasoning time as a useful approximation, not ground
132-
truth. Raw `Time By Kind` totals still include metadata durations for auditability
133-
and therefore may not sum to wall time.
150+
Every trace slice and summary entry carries `timing_quality` as `measured` or
151+
`inferred`, but timing quality is not included as a Speedscope, SnakeViz, or
152+
folded-stack frame. Treat inferred reasoning time as a useful approximation, not
153+
ground truth. Raw `Time By Kind` totals still include metadata durations for
154+
auditability and therefore may not sum to wall time.

0 commit comments

Comments
 (0)