You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add extract-to-cache build mode for the Copilot CLI (#1450)
* Add extract-to-cache build mode to copilot-sdk
Splits CLI provisioning into two modes selected by the existing
`bundled-cli` cargo feature:
- `bundled-cli` on (default, release): unchanged. build.rs downloads,
extracts on first use via embeddedcli::install_at.
- `bundled-cli` off (dev): build.rs downloads, SHA-verifies, and
extracts the binary directly into <cache>/github-copilot-sdk/cli/
<version>/ (staging dir + atomic rename). Emits a build-time
COPILOT_CLI_DEV_PATH env var that resolve.rs returns directly.
Replaces the three-source version resolution chain in build.rs
(COPILOT_CLI_VERSION env, bundled_cli_version.txt snapshot, lockfile
fallback) with a two-source chain:
- cli-version.txt at the crate root (published crates, vendored slots).
- ../nodejs/package-lock.json (mono-repo contributor build) — matches
the .NET _GetCopilotCliVersion MSBuild target and Go cmd/bundler
conventions.
Cache layout flips from <cache>/github-copilot-sdk-{version}/ to the
shared <cache>/github-copilot-sdk/cli/<version>/ so both build modes
populate the same directory. Old per-user caches become orphaned
(harmless; the next launch re-populates the new path).
COPILOT_CLI_PATH env override still wins in both modes. Stale env
override falls through to the next source with a warn! log.
Drops:
- rust/scripts/snapshot-bundled-cli-version.sh (replaced by an inline
node one-liner in the publish workflow that writes cli-version.txt
from nodejs/package-lock.json before cargo publish).
- COPILOT_CLI_VERSION env override path.
- bundled_cli_version.txt snapshot path + parse_snapshot.
- cargo:rerun-if-env-changed=COPILOT_CLI_VERSION.
Adds tests/cli_resolution_test.rs covering pin-file format, env
override (both modes), stale env override fallthrough, and dev-mode
extracted binary path. The existing rust-sdk-tests CI job runs with
--no-default-features so dev-mode resolution is exercised on every PR;
the bundle job exercises embed mode. Both jobs now share the
BUNDLED_CLI_CACHE_DIR archive cache to avoid double-downloading.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop 'mono-repo' jargon in favor of 'github/copilot-sdk repo'
Term isn't established in this repo's prose. The fallback path that
reads ../nodejs/package-lock.json fires for contributor builds inside
github/copilot-sdk itself; spelling it out is clearer than introducing
a new term.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore per-platform SHA snapshot to keep publish-time trust boundary
Without a committed SHA snapshot, build.rs would fetch both
`SHA256SUMS.txt` and the archive from the same release URL on every
consumer build. An attacker who later compromises a release tag could
swap both files together and the hash check would pass against their
own sum — the verification would only protect against transit
corruption, not against upstream tampering.
Restore the snapshot model:
- New `rust/scripts/write-cli-shas.sh` writes `cli-shas.txt` with one
`<asset>=<hex sha>` line per supported platform, fetched from the
release matching `cli-version.txt`. Gitignored locally.
- `build.rs::read_expected_sha` prefers the snapshot when present;
falls back to live `SHA256SUMS.txt` only when absent (contributor
builds inside github/copilot-sdk).
- Publish workflow runs `write-cli-shas.sh` after writing
cli-version.txt and before `cargo publish`, so the published crate
carries hashes captured at publish time. Same flow as the old
snapshot-bundled-cli-version.sh, just split across one-line pin file
+ sibling SHAs file.
- `Cargo.toml` includes `cli-shas.txt` in the crate package.
Verified locally:
- Snapshot path: build with both `cli-version.txt` + `cli-shas.txt`
present — skips the live SHA fetch, downloads + verifies against the
committed hash.
- Tamper detection: corrupting a hash in `cli-shas.txt` causes the
build to panic with the existing "could indicate a supply-chain
attack" message.
- Fallback path: with neither pin file present, build.rs reads version
from `../nodejs/package-lock.json` and fetches SHA256SUMS.txt live
— unchanged from the pre-snapshot contributor flow.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Collapse pin to single combined cli-version.txt; restore proven script
Reduces churn by going back to one combined snapshot file:
- File: cli-version.txt (matches the design-doc name) holds both the
version line and per-platform SHA-256 hashes. Same combined format
the proven bundled_cli_version.txt used.
- Script: snapshot-bundled-cli-version.sh restored byte-for-byte from
history except for the OUTPUT filename. Same logic, same name, same
output structure.
- build.rs: restored the proven resolve_version_and_hash +
parse_snapshot pair (minus the dropped COPILOT_CLI_VERSION env path).
Drops the cli-shas.txt / read_expected_sha / parse_sha_snapshot
helpers I introduced in the previous commit — those split the snapshot
into two files for no real benefit.
- Publish workflow: one step calling the script, like before.
Cargo.toml include[] and .gitignore drop cli-shas.txt. Test for pin
format updated to validate the combined lines.
Verified locally: cargo check passes in both feature modes with the
snapshot present, cargo test --no-default-features passes (4/4).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update lib.rs docs for new resolver order and dev-cache layout
Self-review (GPT 5.5) catch:
- CliProgram::Resolve docstring still said 'PATH + common locations' for
the third source; correct it to 'dev cache'.
- ClientOptions docstring still implied bundled-cli was the only
auto-resolve path; spell out the env var + dev-cache fallback.
- bundled_cli_extract_dir field doc had the old hyphenated cache path
(github-copilot-sdk-{version}); update to the shared
github-copilot-sdk/cli/<version>/ layout.
- bundled_cli_extract_dir + with_bundled_cli_extract_dir now explicitly
document that the option is ignored in default-features=false dev
builds (build.rs has already extracted; use CliProgram::Path or
COPILOT_CLI_PATH for an override).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use file-level atomic rename in extract_to_cache (Windows correctness)
Previous version staged a whole per-version directory and renamed it
into place. fs::rename for files is atomic on Windows
(MoveFileExW + MOVEFILE_REPLACE_EXISTING), but fs::rename for
directories is NOT — if install_dir already exists (partial-extract
recovery, stale install from a crashed build, etc.) Windows refuses the
rename and we'd panic.
Switch to file-level staging:
- create_dir_all(install_dir) up front (idempotent, fine on Windows
even if it exists with content).
- Write extracted binary to a staging *file* alongside its final
location (.copilot[.exe].staging-<pid>-<nanos>).
- chmod 755 on Unix only.
- fs::rename(staging_file, final_path) — atomic file-replace on both
Unix and Windows. If a concurrent build won the race their bytes are
the same (SHA-verified inputs), so replacement is safe.
- On error, remove the staging file (a file, not remove_dir_all of a
directory).
Verified extraction still works end-to-end on macOS (dev-mode test
passes, extracted copilot --version prints expected output). Windows
release zip confirmed to be the same single-file layout
(copilot.exe at root, no subdir) so the extraction code's name match
applies cleanly there too. Windows CI already exercises both
default-features and --no-default-features jobs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop staging+rename from extract_to_cache; match proven embed install()
User caught this on review: embed mode's embeddedcli::install (which
already ships to Windows) doesn't stage. It does:
create_dir_all(install_dir)?
if final_path.is_file() { return Ok(final_path); }
write_binary(final_path, bytes)
That's it. My extract_to_cache invented a staging-file + atomic-rename
dance that doesn't exist in the proven path, then I 'discovered' my own
over-engineering didn't work on Windows (directory rename), 'fixed' it
with a file-level rename, and was still strictly more complex than what
already works.
Match the embed pattern exactly. Same idempotency check, same direct
write, same #[cfg(unix)] chmod gate. Race semantics across concurrent
build.rs invocations are identical to the embed install at runtime, and
both write SHA-verified-identical bytes. If two builds race, last write
wins with identical contents.
Verified end-to-end on macOS: dev-mode test passes, extracted
copilot --version runs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sanitize version in extract_to_cache to match embed-mode cache path
embeddedcli::default_install_dir runs sanitize_version on the version
string before using it as a path component, replacing anything outside
[a-zA-Z0-9._-] with _. extract_to_cache did not.
For today's CLI versions (1.0.55-1) both resolve to identical paths
because the version chars are already safe. But the invariant we
documented — embed mode and dev mode share the same per-version cache
directory — would silently break for any future version containing an
unusual character: embed would write to <cache>/.../1.0.55_rc1/copilot
while dev wrote to <cache>/.../1.0.55-rc1/copilot (etc).
Duplicate sanitize_version into build.rs (small, can't share with the
crate it's building) and note it's kept in sync with the embed-mode
copy. No other behavior change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on tests + docs
rust/tests/cli_resolution_test.rs (resolves 5 review threads):
- Replace `display.contains("/cli/")` with a Path::components() walk so
the assertion is portable across Windows backslashes. This was the
actual Windows CI failure (PR #1450 run 26493115251).
- Re-gate dev_mode_extracted_binary_exists on cfg(has_dev_cli) so it
compiles cleanly on unsupported target platforms where build.rs
doesn't emit COPILOT_CLI_DEV_PATH.
- Replace substring "BinaryNotFound" / "not bundled" check in
stale_env_override_falls_through with a variant match on
Error::BinaryNotFound (the Display output is lowercase, so the old
check passed even when fallthrough was broken).
- Rewrite the comment in env_override_resolves_to_pointed_file: the
test never chmods the tempfile and doesn't need to (resolver only
does is_file()).
- Switch env-touching tokio tests to flavor = "current_thread" and
rewrite the safety comment on set_env/unset_env to honestly describe
what serial + current_thread give us (and what they don't).
rust/src/resolve.rs:
- Drop the "matches the .NET and TypeScript SDKs" claim from the
module doc; this resolver now has an extra dev-cache step.
rust/README.md:
- Add a callout warning that default-features = false produces a build
that is *not* self-contained; only intended for local development.
Production / distributed builds should keep bundled-cli on or supply
COPILOT_CLI_PATH at runtime.
Validated: cargo test (both feature configs), clippy, nightly fmt,
RUSTDOCFLAGS="-D warnings" cargo doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refine extract-to-cache: skip envvar, extract-dir envvar, single cache, staging+rename
Drop 'extract-to-cache mode' phrasing; clarify caveats when bundled-cli is off
Phrase the disabled-bundled-cli case in terms of the feature flag itself
rather than coining a 'mode' name. Strengthen the README caveat: the
consumer must supply a compatible CLI via CliProgram::Path /
COPILOT_CLI_PATH and is responsible for version compatibility; the
build-machine auto-resolution is a convenience that does not carry over
to other machines.
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
* Log extracted CLI path when extract_to_cache actually writes
The download path emits a 'Downloading <url>' cargo warning on cache
miss, but the extract step that follows is silent. With the separate
download cache gone (extracted binary is the cache), a first build
gave no surface telling the contributor where the ~160 MB binary
landed. Mirror the existing warning style with a one-line
'Extracted Copilot CLI to <path>' after the atomic rename succeeds.
Quiet on the hot path: caller short-circuits on is_file() so this
only fires on a true cache miss.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Gate package-lock.json rerun-if-changed on lockfile presence
Cargo treats a missing rerun-if-changed path as 'always rerun', so the
unconditional 'cargo:rerun-if-changed=../nodejs/package-lock.json'
declaration was forcing build.rs to re-run on every cargo build for
consumers without a sibling nodejs/ directory (vendored slots,
published crates). cli-version.txt is the source of truth in those
deployments; the lockfile is only a fallback inside this repo.
Now: declare the lockfile rerun only when the file actually exists.
In-tree contributor builds keep re-running when @github/copilot is
bumped via the lockfile; vendored consumers get true zero-cost
rebuilds when nothing has changed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Steve Sanderson <SteveSandersonMS@users.noreply.github.com>
Copy file name to clipboardExpand all lines: rust/README.md
+64-16Lines changed: 64 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -756,30 +756,59 @@ none of them are scheduled for removal.
756
756
757
757
## Embedded CLI
758
758
759
-
The SDK bundles the Copilot CLI binary inside the consumer's compiled crate by default. No env var setup, no separate install — just `cargo build` and you get a self-contained binary.
759
+
The SDK provisions the Copilot CLI binary at build time. By default the `bundled-cli` feature embeds the verified binary directly in your compiled crate, so end-user binaries are self-contained — no env var setup, no separate install, just `cargo build`.
760
760
761
-
To opt out (e.g. for binary-size-sensitive consumers, or environments that provide the CLI via PATH), set `default-features = false`:
761
+
For builds that prefer a smaller artifact, disable the `bundled-cli` feature:
762
762
763
763
```toml
764
764
github-copilot-sdk = { version = "0.1", default-features = false }
765
765
```
766
766
767
+
> **You become responsible for supplying the CLI at runtime.** With
768
+
> `bundled-cli` disabled, the produced binary does not contain the CLI
769
+
> and will not search the system for one. You must point it at a
770
+
> compatible CLI via [`CliProgram::Path`] (on `ClientOptions`) or the
771
+
> `COPILOT_CLI_PATH` environment variable, and you are responsible for
772
+
> guaranteeing the supplied CLI version is compatible with this SDK
773
+
> release. Do **not** assume that whatever CLI happens to be installed
774
+
> on the target system will work — the SDK and CLI are versioned
775
+
> together.
776
+
>
777
+
> **Convenience on the build machine only.** As a special case,
778
+
> `build.rs` downloads and SHA-verifies the compatible CLI version and
779
+
> drops it into the build machine's per-user cache; the runtime
780
+
> resolver on that same machine will pick it up automatically. This
781
+
> makes local development and CI ergonomic, but it does **not** carry
782
+
> over when you copy the built binary to another machine — distributed
783
+
> builds (release artifacts, signed installers, container images, etc.)
784
+
> must either keep `bundled-cli` enabled or ship the CLI alongside and
785
+
> set `CliProgram::Path` / `COPILOT_CLI_PATH`.
786
+
767
787
### How it works
768
788
769
-
1.**Pinned at publish time.** When the rust crate is published, a workflow step writes `bundled_cli_version.txt` (CLI version + per-platform SHA-256 hashes) into the crate from the in-effect `nodejs/package-lock.json` and the matching GitHub Release's `SHA256SUMS.txt`. This file is gitignored locally; it only exists in the published crate tarball.
789
+
1.**Version pin.**`build.rs` reads the CLI version from one of two sources:
790
+
-`cli-version.txt` at the crate root (present in published crate tarballs and vendored slots).
791
+
- Otherwise, `../nodejs/package-lock.json` (contributor build inside the github/copilot-sdk repo — matches the .NET and Go SDK conventions here).
792
+
793
+
The resolved version is baked into the crate via `cargo:rustc-env=COPILOT_SDK_CLI_VERSION` regardless of mode. The runtime resolver consumes it to recompute the on-disk path by convention, so no absolute paths leak into the rlib.
794
+
795
+
2.**Build time:**`build.rs` downloads the platform-appropriate archive from the [`github/copilot-cli` GitHub Releases](https://github.com/github/copilot-cli/releases) (`copilot-{platform}.tar.gz` on macOS/Linux, `.zip` on Windows), live-fetches the matching `SHA256SUMS.txt`, and verifies the archive hash. Then:
796
+
-**`bundled-cli` on (default, release):** embeds the raw archive bytes via `include_bytes!()`. Runtime extracts on first `Client::start()`.
797
+
-**`bundled-cli` off:** extracts the binary directly into the platform cache (staging file + atomic rename), idempotent across rebuilds. If the extracted binary is already present at the expected path, the download is skipped entirely — the extracted binary *is* the cache.
770
798
771
-
2.**Build time:** The SDK's `build.rs` resolves the version + per-platform SHA-256:
772
-
-`COPILOT_CLI_VERSION` env var (advanced override; fetches live `SHA256SUMS.txt`).
773
-
- Otherwise, `bundled_cli_version.txt` from the published crate.
774
-
- Otherwise (mono-repo contributor build), live read from `../nodejs/package-lock.json` + live fetch of `SHA256SUMS.txt`.
799
+
3.**Runtime:** in both modes the binary lives at:
775
800
776
-
It then downloads the platform-appropriate archive from the [`github/copilot-cli` GitHub Releases](https://github.com/github/copilot-cli/releases) (`copilot-{platform}.tar.gz` on macOS/Linux, `.zip` on Windows), verifies the SHA-256, extracts the `copilot` binary, compresses it with zstd, and embeds via `include_bytes!()`.
| Linux |`${XDG_CACHE_HOME:-~/.cache}/github-copilot-sdk/cli/<version>/copilot`|
805
+
| Windows |`%LOCALAPPDATA%\github-copilot-sdk\cli\<version>\copilot.exe`|
777
806
778
-
3.**Runtime:** On the first call to `github_copilot_sdk::Client::start()`, the embedded archive is lazily extracted to the platform cache dir (`%LOCALAPPDATA%\github-copilot-sdk-{version}\` on Windows, `~/Library/Caches/github-copilot-sdk-{version}/` on macOS, `$XDG_CACHE_HOME/github-copilot-sdk-{version}/` (or `~/.cache/...`) on Linux). Subsequent runs reuse the extracted binary.
807
+
Old version directories accumulate in siblings; clean them up at your leisure.
779
808
780
809
### Overriding the extraction location
781
810
782
-
Use [`ClientOptions::with_bundled_cli_extract_dir`]when you need to place the extracted binary somewhere other than the platform cache dir (CI runners with ephemeral homes, sandboxes that disallow cache paths, etc.):
811
+
[`ClientOptions::with_bundled_cli_extract_dir`]redirects embed-mode extraction to a custom directory (CI runners with ephemeral homes, sandboxes that disallow cache paths, etc.):
783
812
784
813
```rust,ignore
785
814
use std::path::PathBuf;
@@ -790,15 +819,34 @@ let options = ClientOptions::new()
790
819
let client = Client::start(options).await?;
791
820
```
792
821
822
+
With `bundled-cli` disabled the equivalent knob is the **`COPILOT_CLI_EXTRACT_DIR`** environment variable, which is honored symmetrically at build time (where `build.rs` writes the binary) and at runtime (where the resolver reads it). When set, the binary lives directly under the named directory (no per-version subdir). The most ergonomic way to pin it from a consumer crate is `.cargo/config.toml`:
823
+
824
+
```toml
825
+
# .cargo/config.toml at the consumer's repo root
826
+
[env]
827
+
COPILOT_CLI_EXTRACT_DIR = { value = "vendor/copilot", relative = true, force = true }
828
+
```
829
+
830
+
`relative = true` resolves the path against the config file's directory, so the value is stable regardless of where `cargo build` is invoked from. `force = true` makes the value visible to invocations of the produced binary under `cargo run` / `cargo test`, keeping build and runtime in sync. For runtime invocations outside cargo (e.g. a deploy script running the binary directly), either export the same env var or use [`CliProgram::Path`] / `COPILOT_CLI_PATH` at runtime.
831
+
832
+
### Skipping the bundle entirely
833
+
834
+
Set `COPILOT_SKIP_CLI_DOWNLOAD=1` at build time to disable the entire download / bundle / cache mechanism — `build.rs` returns immediately without touching the network. Use this when you always supply the CLI at runtime via `ClientOptions::program = CliProgram::Path(...)` or `COPILOT_CLI_PATH`. Works regardless of the `bundled-cli` feature state; runtime resolution falls through to `Error::BinaryNotFound` unless one of those explicit sources resolves.
835
+
793
836
### Resolution priority
794
837
795
-
`copilot_binary()` checks these sources in order:
838
+
`Client::start` resolves the CLI in this order:
839
+
840
+
1. Explicit `CliProgram::Path(path)` on `ClientOptions::program`.
841
+
2.`COPILOT_CLI_PATH` environment variable, if it points at a real file.
842
+
3.**`bundled-cli` on:** the embedded archive, lazily extracted on first call.
843
+
4.**`bundled-cli` off:** the build-time-extracted binary in the per-user cache, located by recomputing the convention from `COPILOT_SDK_CLI_VERSION` + OS + optional `COPILOT_CLI_EXTRACT_DIR`.
844
+
845
+
There is no PATH scanning. If none of the above resolves, `Client::start` returns `Error::BinaryNotFound`.
796
846
797
-
1. Explicit `CliProgram::Path(path)` on `ClientOptions::program`
798
-
2.`COPILOT_CLI_PATH` environment variable
799
-
3. Embedded CLI (when the `bundled-cli` feature is enabled, which it is by default)
847
+
### Download cache (build-time, embed mode)
800
848
801
-
There is no PATH scanning. If both 1+2 are unset and the SDK was built with `default-features = false`, `Client::start` returns `Error::BinaryNotFound`.
849
+
In embed mode `build.rs` re-downloads on every clean build by default. Set `BUNDLED_CLI_CACHE_DIR=<path>` to cache the verified archive between builds (CI keys this on `<os>-<version>` for ~zero-cost rebuilds on cache hits). With `bundled-cli` disabled there is no separate archive cache — the extracted binary itself is the cache.
|`bundled-cli`| ✓ | Build-time CLI embedding. Pulls in `dirs`, `tar`+`flate2` (Linux/macOS), or `zip` (Windows). Disable via `default-features = false` to opt out (e.g. when shipping a smaller binary or when always supplying the CLI via `CliProgram::Path` / `COPILOT_CLI_PATH`). |
859
+
|`bundled-cli`| ✓ | Build-time CLI embedding. Pulls in `tar`+`flate2` (Linux/macOS) or `zip` (Windows). Disable via `default-features = false` to opt out (e.g. when shipping a smaller binary or when always supplying the CLI via `CliProgram::Path` / `COPILOT_CLI_PATH`). |
812
860
|`derive`| — |`schema_for::<T>()` for generating JSON Schema from Rust types (adds `schemars`). Enable when defining [tool parameters](#tool-registration). |
0 commit comments