Skip to content

chore(agent): add shell efficiency section to appended instructions#3207

Merged
charlesvien merged 2 commits into
mainfrom
posthog-code/shell-efficiency-instructions
Jul 7, 2026
Merged

chore(agent): add shell efficiency section to appended instructions#3207
charlesvien merged 2 commits into
mainfrom
posthog-code/shell-efficiency-instructions

Conversation

@MattPua

@MattPua MattPua commented Jul 6, 2026

Copy link
Copy Markdown
Member

Problem

Bash is the single biggest LLM cost line in PostHog Code sessions (in a sampled 30-day window: 16k Bash generations at ~111k avg input tokens each, ~half of all spend). Each tool call re-sends the entire conversation, so sequential one-command-per-call shell usage multiplies full-context round trips.

Changes

Adds a SHELL_EFFICIENCY section to APPENDED_INSTRUCTIONS telling the model to batch related commands with &&, emit independent tool calls in one response, read multiple files at once, and not rerun commands for output it already has.

Follow-up commits add the same section to the workspace-server local session prompt and the cloud agent-server prompt, so Codex sessions (which don't receive APPENDED_INSTRUCTIONS) and cloud runs carry it too.

How did you test this?

  • pnpm --filter @posthog/agent typecheck passes.
  • Benchmarked with a local eval harness, details below.
Benchmarks: -19% Bash calls, -12% turns, -15% input tokens, no quality loss

Each run scaffolds a fresh git fixture repo in a temp dir and executes a fixed task through the Claude Agent SDK (claude-sonnet-4-6, settingSources: [] so local config can't contaminate results). Control = no Shell Efficiency section, treatment = this PR's wording. Metrics captured per run: Bash calls, turns, input tokens, cost, task success (with post-hoc verification where applicable).

Sweep 1: wording comparison (5 synthetic tasks, 4 variants, n=6 each, 120 runs)

Variant Bash calls Turns Success
control (no section) 3.0 4.8 29/30
bullet list, no example 2.6 4.3 29/30
+ concrete && example (shipped) 2.4 4.1 30/30
"ALWAYS chain" directive 2.5 4.1 30/30

The concrete example outperformed a more forceful directive; shipped wording is the example variant.

Sweep 2: real-workload tasks (n=6 each, 36 runs)

Tasks modeled on the top Bash patterns mined from 5k real PostHog Code session commands (grep/search ~17%, git operations ~12%, per-package script runs ~4%).

Task Control bash/turns Treatment bash/turns
code-search (grep + context) 1.0 / 2.0 1.0 / 2.0
git-workflow (review, stage, commit, verify) 3.7 / 5.0 2.5 / 4.0
multi-package-checks (4 npm scripts) 4.7 / 5.7 4.0 / 5.0
Overall 3.1 / 4.2 2.5 / 3.7

-19% Bash calls, -12% turns, -15% input tokens, 36/36 success in both variants.

Example collapse (git-workflow, typical run):

control (4 calls)                      treatment (2 calls)
$ git diff && git status               $ git diff README.md && cat src/discount.js
$ cat src/discount.js                  $ git add ... && git commit -m "..." \
$ git add ... && git commit -m "..."       && git status && git log --oneline -3
$ git status && git log --oneline -3

Since each tool call re-sends the full conversation (~111k avg input tokens in production sessions), fewer round trips compounds: the turns reduction is the number that translates to spend.

Known limit: models still run check/build/test scripts individually to attribute failures per step (unchanged across all wordings, including "ALWAYS chain"). Arguably correct behavior; not worth forcing via prompt.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Created with PostHog Code

@trunk-io

trunk-io Bot commented Jul 6, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

React Doctor found no issues in the changed files. 🎉

Reviewed by React Doctor for commit f9a4d02.

@MattPua MattPua force-pushed the posthog-code/shell-efficiency-instructions branch 5 times, most recently from 9efcb25 to 34b8e29 Compare July 6, 2026 21:41
@MattPua MattPua marked this pull request as ready for review July 6, 2026 22:39
Nudges the model to batch sequential shell commands with && and emit
independent tool calls in one response, cutting full-context round trips.

Generated-By: PostHog Code
Task-Id: 2e63ca05-ef07-4251-9348-687a07ecfa93
@MattPua MattPua changed the title feat(agent): add shell efficiency section to appended instructions chore(agent): add shell efficiency section to appended instructions Jul 6, 2026
@MattPua MattPua force-pushed the posthog-code/shell-efficiency-instructions branch from 34b8e29 to 6394e8e Compare July 6, 2026 22:39
@greptile-apps

greptile-apps Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(agent): add shell efficiency sectio..." | Re-trigger Greptile

@charlesvien charlesvien merged commit 606215a into main Jul 7, 2026
24 checks passed
@charlesvien charlesvien deleted the posthog-code/shell-efficiency-instructions branch July 7, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants