-
Notifications
You must be signed in to change notification settings - Fork 50
feat(agent): bundle rtk with the agent package #3145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
pauldambra
wants to merge
19
commits into
main
Choose a base branch
from
posthog-code/bundle-rtk-with-agent
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,223
−134
Open
Changes from 2 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
a9b04fc
feat(agent): bundle rtk with the agent package
pauldambra 0146ce3
fix(agent): verify rtk archive checksum and harden test teardown
pauldambra 28c5868
refactor(agent): tidy rtk bundling from review feedback
pauldambra c016f35
fix(agent): preserve explicit POSTHOG_RTK over the desktop bundle def…
pauldambra 2168f7e
feat(agent): emit rtk token-savings telemetry at end of cloud run (#3…
pauldambra 4e5352d
fix(agent): stop rtk savings tests probing the machine's real rtk
pauldambra 1fced7a
fix(agent): clarify at-most-once emit semantics and harden rtk binary…
pauldambra 20cb39c
refactor: apply simplify pass
pauldambra 1bcf3c5
fix(agent): keep rtk cache rename on one filesystem and pin error-pat…
pauldambra 258e019
refactor: reuse signClaudeBinary for the bundled rtk binary
pauldambra 99e4db6
fix(agent): download rtk into the per-process temp dir and polish rev…
pauldambra 4763eb5
docs(agent): correct two review-pass comments
pauldambra 5fe748e
feat(agent): emit rtk savings as a gauge snapshot, not a per-run delta
pauldambra 2516812
fix(agent): key the rtk savings counter to the task and drop the pct …
pauldambra e6bd533
feat(agent): report the rtk savings gauge from desktop daily
pauldambra 7b77dfd
refactor(agent): register the rtk gauge in the typed analytics event map
pauldambra 36ed5d2
fix(agent): polish the desktop rtk gauge wiring from review
pauldambra b9c14cc
fix(agent): scrub the env passed to rtk gain
pauldambra bc53fab
fix(agent): probe rtk before rewriting commands through it
pauldambra File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| import { createHash } from "node:crypto"; | ||
| import { | ||
| chmodSync, | ||
| copyFileSync, | ||
| createReadStream, | ||
| createWriteStream, | ||
| existsSync, | ||
| mkdirSync, | ||
| readFileSync, | ||
| rmSync, | ||
| writeFileSync, | ||
| } from "node:fs"; | ||
| import { dirname, join, resolve } from "node:path"; | ||
| import { pipeline } from "node:stream/promises"; | ||
| import { setTimeout as sleep } from "node:timers/promises"; | ||
| import { fileURLToPath } from "node:url"; | ||
| import { unzipSync } from "fflate"; | ||
| import { extract } from "tar"; | ||
| import { targetArch, targetPlatform } from "./native-binary.mjs"; | ||
|
|
||
| /** | ||
| * Shared build-time helper for vendoring the RTK binary | ||
| * (https://github.com/rtk-ai/rtk) into the agent package's `dist/rtk/`. | ||
| * | ||
| * RTK compresses the output of common dev commands before it reaches the model | ||
| * (see `src/adapters/claude/session/rtk.ts`). We pin a single version here so | ||
| * every host — desktop and cloud runs — uses the same RTK, rather than | ||
| * depending on whatever happens to be on the machine's PATH. | ||
| * | ||
| * There is no npm package for RTK, so we download the pinned release from | ||
| * GitHub at build time and cache it under `node_modules/.cache`. Bundling is | ||
| * best-effort: if the download fails (e.g. offline build) the caller warns and | ||
| * continues, and the runtime resolver falls back to PATH. | ||
| */ | ||
|
|
||
| export const RTK_VERSION = "0.43.0"; | ||
|
|
||
| const __dirname = dirname(fileURLToPath(import.meta.url)); | ||
|
|
||
| // Rust target triples for the RTK GitHub release assets. Linux x64 ships musl | ||
| // (static, portable); linux arm64 ships gnu — RTK publishes no arm64 musl or | ||
| // Windows arm64 asset, so those combinations resolve to undefined and skip. | ||
| const RTK_TARGETS = { | ||
| darwin: { arm64: "aarch64-apple-darwin", x64: "x86_64-apple-darwin" }, | ||
| linux: { | ||
| arm64: "aarch64-unknown-linux-gnu", | ||
| x64: "x86_64-unknown-linux-musl", | ||
| }, | ||
| win32: { x64: "x86_64-pc-windows-msvc" }, | ||
| }; | ||
|
|
||
| // SHA-256 of each release archive (from the release's checksums.txt), pinned | ||
| // alongside RTK_VERSION so a swapped release asset or a corrupt/truncated | ||
| // download fails the build instead of being cached and bundled. Update both | ||
| // RTK_VERSION and these hashes together when bumping rtk. | ||
| const RTK_SHA256 = { | ||
| "aarch64-apple-darwin": | ||
| "8a17e49acbd378997eb21d0eb6f7f861111f35b4fc9b1c74edf4c7448e576c65", | ||
| "aarch64-unknown-linux-gnu": | ||
| "5519f7ca12e5c143a609f0d28a0a77b97413a8dce31c2681f1a41c24519a8731", | ||
| "x86_64-apple-darwin": | ||
| "a85f60e2637811be68366208b8d8b9c5ba1b748cb5df4477ab20cd73d3c5d9f8", | ||
| "x86_64-pc-windows-msvc": | ||
| "7c5e4a2ef816a4d4ed947ddd74ca3df851fc39ea87d49a3ca2bf3abc515a016b", | ||
| "x86_64-unknown-linux-musl": | ||
| "ff8a1e7766496e175291a85aeca1dc97c9ff6df33e51e5893d1fbc78fea2a609", | ||
| }; | ||
|
|
||
| export function rtkBinName(platform = targetPlatform()) { | ||
| return platform === "win32" ? "rtk.exe" : "rtk"; | ||
| } | ||
|
|
||
| export function rtkReleaseTarget( | ||
| platform = targetPlatform(), | ||
| arch = targetArch(), | ||
| ) { | ||
| return RTK_TARGETS[platform]?.[arch]; | ||
| } | ||
|
|
||
| function rtkAssetUrl(target) { | ||
| const ext = target.includes("windows") ? "zip" : "tar.gz"; | ||
| return `https://github.com/rtk-ai/rtk/releases/download/v${RTK_VERSION}/rtk-${target}.${ext}`; | ||
|
pauldambra marked this conversation as resolved.
|
||
| } | ||
|
|
||
| const MAX_DOWNLOAD_ATTEMPTS = 5; | ||
| const RETRIABLE_HTTP_STATUSES = new Set([408, 425, 429, 500, 502, 503, 504]); | ||
|
|
||
| class NonRetriableError extends Error {} | ||
|
|
||
| function backoffDelayMs(attempt) { | ||
| // Deterministic backoff — build scripts avoid Math.random for reproducibility. | ||
| return Math.min(1000 * 2 ** (attempt - 1), 15000); | ||
| } | ||
|
|
||
| async function downloadFile(url, destPath) { | ||
| for (let attempt = 1; attempt <= MAX_DOWNLOAD_ATTEMPTS; attempt++) { | ||
| try { | ||
| const response = await fetch(url, { redirect: "follow" }); | ||
| if (!response.ok) { | ||
| const message = `HTTP ${response.status}: ${response.statusText}`; | ||
| if (RETRIABLE_HTTP_STATUSES.has(response.status)) | ||
| throw new Error(message); | ||
| throw new NonRetriableError(message); | ||
| } | ||
| await pipeline(response.body, createWriteStream(destPath)); | ||
| return; | ||
| } catch (error) { | ||
| if ( | ||
| error instanceof NonRetriableError || | ||
| attempt === MAX_DOWNLOAD_ATTEMPTS | ||
| ) { | ||
| throw error; | ||
| } | ||
| await sleep(backoffDelayMs(attempt)); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| async function verifyChecksum(archivePath, target) { | ||
| const expected = RTK_SHA256[target]; | ||
| if (!expected) { | ||
| throw new Error(`No pinned checksum for rtk target ${target}`); | ||
| } | ||
| const hash = createHash("sha256"); | ||
| await pipeline(createReadStream(archivePath), hash); | ||
| const actual = hash.digest("hex"); | ||
| if (actual !== expected) { | ||
| throw new Error( | ||
| `rtk checksum mismatch for ${target}: expected ${expected}, got ${actual}`, | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| async function extractArchive(archivePath, destDir, target) { | ||
| if (target.includes("windows")) { | ||
| const entries = unzipSync(readFileSync(archivePath)); | ||
| const bin = rtkBinName("win32"); | ||
| const data = entries[bin] ?? entries[`rtk-${target}/${bin}`]; | ||
| if (!data) throw new Error(`rtk binary not found in ${archivePath}`); | ||
| writeFileSync(join(destDir, bin), data); | ||
|
greptile-apps[bot] marked this conversation as resolved.
|
||
| } else { | ||
| await extract({ file: archivePath, cwd: destDir }); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Downloads and caches the pinned RTK binary for the build target, then copies | ||
| * it into `destDir`. Returns the path to the bundled binary, or null when RTK | ||
| * publishes no asset for the current platform/arch. Throws on download failure. | ||
| */ | ||
| export async function ensureRtkBinary(destDir) { | ||
| const target = rtkReleaseTarget(); | ||
| if (!target) return null; | ||
|
|
||
| const binName = rtkBinName(); | ||
| const cacheDir = resolve( | ||
| __dirname, | ||
| "../../../node_modules/.cache/posthog-rtk", | ||
| RTK_VERSION, | ||
| target, | ||
| ); | ||
| const cachedBinary = join(cacheDir, binName); | ||
|
|
||
| if (!existsSync(cachedBinary)) { | ||
| mkdirSync(cacheDir, { recursive: true }); | ||
| const url = rtkAssetUrl(target); | ||
| const archivePath = join( | ||
| cacheDir, | ||
| url.endsWith(".zip") ? "rtk.zip" : "rtk.tar.gz", | ||
| ); | ||
| await downloadFile(url, archivePath); | ||
| try { | ||
| await verifyChecksum(archivePath, target); | ||
| } catch (error) { | ||
| // Don't leave an unverified archive on disk to be reused. | ||
| rmSync(archivePath, { force: true }); | ||
| throw error; | ||
| } | ||
| await extractArchive(archivePath, cacheDir, target); | ||
|
pauldambra marked this conversation as resolved.
Outdated
|
||
| rmSync(archivePath, { force: true }); | ||
| if (!existsSync(cachedBinary)) { | ||
| throw new Error(`rtk binary missing after extraction: ${cachedBinary}`); | ||
| } | ||
| } | ||
|
|
||
| mkdirSync(destDir, { recursive: true }); | ||
| const dest = join(destDir, binName); | ||
| copyFileSync(cachedBinary, dest); | ||
| if (targetPlatform() !== "win32") chmodSync(dest, 0o755); | ||
| return dest; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.