feat: sync API specs from infra and belt with Copybara (#1564)

The specs in `spec/` were copied from their source repos by hand and had
drifted ~2,400 lines behind infra, so they are now imported with
Copybara (`copy.bara.sky`, run in a pinned Docker image by
`scripts/fetch-spec.sh`): `make codegen` re-fetches them at the commits
pinned in `spec/infra-ref` and `spec/belt-ref` before generating, and
the generated-files CI check fails if the tracked copies don't match the
pins. Regenerating from the current pins picks up the accumulated spec
changes in the generated JS/Python clients (renamed request schemas,
`SandboxNetworkConfig`, `SandboxIam` workload identity,
`FILE_TYPE_SYMLINK`, access-token auth deprecation, volume path-metadata
tweaks). The one handwritten SDK change follows from that: the public
`FileType` enums gain a `SYMLINK` member (JS and both Python surfaces)
so entries envd reports as symlinks show up in `files.list()` and
`getInfo()`/`get_info()` instead of being silently skipped as unknown
types. The custom `spec/remove_extra_tags.py` tag-filtering script is
replaced by Redocly CLI's `filter-in` decorator (`redocly.yaml`), which
produces identical generated JS output; a `filter-out` decorator
additionally drops any operation or component schema the upstream specs
mark `x-not-implemented: true` (currently the SOCKS5
`SandboxEgressProxyConfig`/`egressProxy` surface, which infra flagged as
spec-only); each SDK's bundle now goes to its own gitignored
`spec/openapi_generated.<api>.yml` instead of both pipelines overwriting
one shared file; Python client models now list fields in spec order
instead of alphabetical (mechanical reordering only — construct models
with keyword args). Spec fetches try whatever GitHub token is available
and fall back to the tracked copies with a warning (the public infra
specs also fetch anonymously); in CI a short-lived belt-scoped token is
minted from the org-wide Autofixer GitHub App (no new secrets), so fork
PRs simply fall back for the belt spec; the CI workflows also cache the
Copybara image alongside the codegen image, and the previously ignored
`CODEGEN_IMAGE` env is honored by the Makefile.

## Usage

```sh
# update the specs: bump a pin, then regenerate
echo <infra-commit-sha> > spec/infra-ref
make codegen

# fetch a single spec without regenerating
pnpm fetch:api-spec     # spec/openapi.yml from infra
pnpm fetch:envd-spec    # spec/envd/ from infra
pnpm fetch:volume-spec  # spec/openapi-volumecontent.yml from belt

# try the latest spec without touching the pin
E2B_INFRA_REF=main pnpm fetch:api-spec

# change which endpoint tags an SDK exposes
$EDITOR redocly.yaml && make codegen
```

```ts
// symlinks are now visible in the filesystem API (JS; same shape in Python)
const entries = await sandbox.files.list('/home/user')
const link = entries.find((e) => e.type === FileType.SYMLINK)
console.log(link?.symlinkTarget)
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: mishushakov

.changeset/lucky-donuts-fetch.md

@@ -0,0 +1,6 @@
+---
+'e2b': patch
+'@e2b/python-sdk': patch
+---
+
+Regenerate API clients from the latest specs, which are now synced with Copybara from their source-of-truth repositories (e2b-dev/infra@e2255f0 for the REST and envd specs, belt for the volume-content spec) instead of being copied by hand. Picks up the latest spec changes: named `SandboxTimeoutRequest`/`SandboxSnapshotRequest`/`SandboxRefreshRequest` request schemas, `SandboxNetworkConfig` and `SandboxIam` workload-identity models, the `FILE_TYPE_SYMLINK` filesystem entry type, and deprecation of access-token auth in favor of API keys. Anything the upstream specs mark `x-not-implemented: true` (currently the SOCKS5 egress-proxy config) is excluded from the generated clients. Generated Python client models now list fields in spec order instead of alphabetical order (the tag filtering moved from a custom script to Redocly CLI); construct them with keyword arguments if you don't already

.changeset/symlink-file-type.md

@@ -0,0 +1,6 @@
+---
+'e2b': minor
+'@e2b/python-sdk': minor
+---
+
+Add `FileType.SYMLINK` to the sandbox filesystem types. Newer envd versions report symlinks with a dedicated `FILE_TYPE_SYMLINK` entry type; previously the SDKs treated it as unknown, so `files.list()` silently omitted symlink entries and `getInfo()`/`get_info()` returned an `undefined`/`None` type for them. Symlinks now surface as `FileType.SYMLINK` (`'symlink'`) with `symlinkTarget`/`symlink_target` populated, in JS and both sync and async Python.

.github/workflows/codegen_image_cache.yml

@@ -1,13 +1,15 @@
 name: Codegen image cache
 
-# Seeds the buildkit cache for codegen.Dockerfile from main so pull request
-# runs of generated_files.yml warm-start. PR-scoped cache writes are not
-# readable by other PRs, so main is the only scope worth writing to.
+# Seeds the buildkit caches for codegen.Dockerfile and copybara.Dockerfile
+# from main so pull request runs of generated_files.yml warm-start. PR-scoped
+# cache writes are not readable by other PRs, so main is the only scope worth
+# writing to.
 on:
   push:
     branches: [main]
     paths:
       - 'codegen.Dockerfile'
+      - 'copybara.Dockerfile'
       - '.github/workflows/codegen_image_cache.yml'
   workflow_dispatch:
 
@@ -32,3 +34,11 @@ jobs:
           file: codegen.Dockerfile
           cache-from: type=gha
           cache-to: type=gha,mode=max
+
+      - name: Build copybara image and export cache
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: copybara.Dockerfile
+          cache-from: type=gha,scope=copybara
+          cache-to: type=gha,mode=max,scope=copybara

.github/workflows/generated_files.yml

@@ -25,6 +25,10 @@ jobs:
             generated:
               - 'spec/**/!(*.md)'
               - 'codegen.Dockerfile'
+              - 'copybara.Dockerfile'
+              - 'copy.bara.sky'
+              - 'redocly.yaml'
+              - 'scripts/fetch-spec.sh'
               - 'Makefile'
               - 'packages/**/!(*.md)'
               - 'pnpm-lock.yaml'
@@ -86,8 +90,43 @@ jobs:
           # the repo's 10 GB Actions cache, evicting other caches.
           cache-from: type=gha
 
+      - name: Build copybara image with caching
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: copybara.Dockerfile
+          tags: e2b-copybara:latest
+          load: true # makes the image available for `docker run`
+          # Cache is seeded from main by codegen_image_cache.yml (see the
+          # cache-to note above for why there is no cache-to here).
+          cache-from: type=gha,scope=copybara
+
+      # scripts/fetch-spec.sh needs read access to the private e2b-dev/belt
+      # repo to fetch the volume-content spec. The default `secrets.GITHUB_TOKEN`
+      # is scoped to this repository only, so mint a short-lived token from
+      # the Autofixer GitHub App instead, scoped to every repo the fetches
+      # touch. Fork PRs have no access to the App secret, so they skip the
+      # mint, run the public infra fetches with the default token, and fall
+      # back to the tracked volume-content spec (make fetch-specs warns
+      # instead of failing).
+      - name: Generate token for spec repos
+        id: spec-repos-token
+        if: github.event.pull_request.head.repo.full_name == github.repository
+        uses: actions/create-github-app-token@v2
+        with:
+          app-id: ${{ vars.AUTOFIXER_APP_ID }}
+          private-key: ${{ secrets.AUTOFIXER_APP_SECRET }}
+          owner: e2b-dev
+          repositories: belt,infra
+          permission-contents: read
+
       - name: Run codegen
-        run: CODEGEN_IMAGE=codegen-env:latest make codegen
+        env:
+          GITHUB_TOKEN: ${{ steps.spec-repos-token.outputs.token || github.token }}
+        run: |
+          CODEGEN_IMAGE=codegen-env:latest \
+          COPYBARA_IMAGE=e2b-copybara:latest \
+          make codegen
 
       - name: Check for uncommitted changes
         run: |

.gitignore

@@ -293,5 +293,8 @@ cython_debug/
 sdk_ref/
 sdkRefRoutes.json
 
-# SDK Client generated spec
+# SDK Client generated specs (per-SDK redocly bundle outputs; the bare
+# openapi_generated.yml pattern covers stale copies from older checkouts)
+spec/openapi_generated.*.yml
 spec/openapi_generated.yml
+

CLAUDE.md

@@ -1,7 +1,7 @@
 Use pnpm for node and uv for python to install and update dependencies.
 Run `pnpm run format`, `pnpm run lint` and `pnpm run typecheck` before committing changes.
 When modifying the SDK packages, ensure equivalent changes are applied to both JS as well as sync and async Python implementations.
-To re-generate the API client run `make codegen` in the repository root when modifying spec/.
+Never edit the API, envd, or volume-content specs in spec/ by hand; they are synced with Copybara from e2b-dev/infra and e2b-dev/belt at the commits pinned in spec/infra-ref and spec/belt-ref (see copy.bara.sky and spec/README.md). To update the specs, bump the pins and re-run `make codegen`, which re-fetches them before generating.
 Create or update tests covering affected codepaths and run them using `pnpm run test`.
 Generate a changeset when updating packages/cli, packages/js-sdk, packages/python-sdk with `pnpm changeset` in the repository root.
 When creating a pull request, add usage examples for user-facing changes to the PR description.

Makefile

@@ -1,9 +1,25 @@
+# Specs are fetched from their source-of-truth repositories (e2b-dev/infra
+# and e2b-dev/belt) at the commits pinned in spec/infra-ref and spec/belt-ref.
+# To update the specs, bump the pins and re-run `make codegen`.
+# When a fetch fails (no GitHub token, no access to the private belt repo),
+# fall back to the tracked copy in spec/ with a warning.
+.PHONY: fetch-specs
+fetch-specs:
+	@for spec in api-spec envd-spec volume-api-spec; do \
+		./scripts/fetch-spec.sh $$spec || \
+			echo "WARNING: could not fetch $$spec; using the tracked copy in spec/"; \
+	done
+
+# Set CODEGEN_IMAGE to skip the image build and use a prebuilt image instead
+# (CI builds it separately with a warm buildkit cache).
 .PHONY: codegen
-codegen:
+codegen: fetch-specs
+ifeq ($(CODEGEN_IMAGE),)
 	@echo "Building codegen image"
 	docker build -q -t codegen-env . -f codegen.Dockerfile
+endif
 	@echo "Generating code"
-	docker run -v $$PWD:/workspace codegen-env make generate
+	docker run -v $$PWD:/workspace $(or $(CODEGEN_IMAGE),codegen-env) make generate
 
 generate: generate-js generate-python
 

codegen.Dockerfile

@@ -21,7 +21,7 @@ ENV PATH="/go/bin:${PATH}"
 # https://github.com/openapi-generators/openapi-python-client/pull/1296
 # protoc-gen-py generates the Python envd protobuf-py messages and
 # protoc-gen-connectrpc the Connect stubs.
-RUN pip install black==26.3.1 pyyaml==6.0.2 e2b-openapi-python-client==0.26.2 datamodel-code-generator==0.34.0 protoc-gen-connectrpc==0.11.1 protoc-gen-py==0.1.1
+RUN pip install black==26.3.1 e2b-openapi-python-client==0.26.2 datamodel-code-generator==0.34.0 protoc-gen-connectrpc==0.11.1 protoc-gen-py==0.1.1
 
 # Install Node.js (pinned to match .tool-versions)
 ENV NODE_VERSION=22.18.0
@@ -38,6 +38,7 @@ ENV PNPM_VERSION=9.15.5
 RUN npm install -g \
     pnpm@${PNPM_VERSION} \
     @connectrpc/protoc-gen-connect-es@1.6.1 \
-    @bufbuild/protoc-gen-es@2.6.2
+    @bufbuild/protoc-gen-es@2.6.2 \
+    @redocly/cli@2.39.0
 
 CMD ["make", "generate"]

copy.bara.sky

@@ -0,0 +1,79 @@
+# Copybara config that imports API specs from their source-of-truth
+# repositories into spec/. The imported files are tracked here but must not
+# be edited by hand — `make codegen` re-fetches the infra ones at the commit
+# pinned in spec/infra-ref before generating the clients, and the
+# generated-files CI check fails if the tracked copies don't match the pin.
+# See spec/README.md.
+#
+# scripts/fetch-spec.sh runs these workflows with `--folder-dir` pointing at
+# spec/ itself. Each workflow's destination_files glob declares the spec/
+# paths its upstream owns: after a successful fetch, Copybara replaces
+# exactly those paths (so files deleted upstream are deleted here too) and
+# leaves the repo-owned files in spec/ untouched.
+
+INFRA_REPO = "https://github.com/e2b-dev/infra.git"
+BELT_REPO = "https://github.com/e2b-dev/belt.git"
+
+AUTHORING = authoring.pass_thru("E2B <hello@e2b.dev>")
+
+# REST API spec -> spec/openapi.yml
+core.workflow(
+    name = "api-spec",
+    origin = git.github_origin(
+        url = INFRA_REPO,
+    ),
+    destination = folder.destination(),
+    origin_files = glob(["spec/openapi.yml"]),
+    destination_files = glob(["openapi.yml"]),
+    authoring = AUTHORING,
+    mode = "SQUASH",
+    transformations = [
+        core.move("spec/openapi.yml", "openapi.yml"),
+    ],
+)
+
+# envd spec (HTTP API + protobufs) -> spec/envd/
+core.workflow(
+    name = "envd-spec",
+    origin = git.github_origin(
+        url = INFRA_REPO,
+    ),
+    destination = folder.destination(),
+    origin_files = glob(
+        [
+            "packages/envd/spec/envd.yaml",
+            "packages/envd/spec/filesystem/**",
+            "packages/envd/spec/process/**",
+        ],
+        # This repo has its own buf generation templates in spec/envd.
+        exclude = ["packages/envd/spec/buf*.yaml"],
+    ),
+    destination_files = glob([
+        "envd/envd.yaml",
+        "envd/filesystem/**",
+        "envd/process/**",
+    ]),
+    authoring = AUTHORING,
+    mode = "SQUASH",
+    transformations = [
+        core.move("packages/envd/spec", "envd"),
+    ],
+)
+
+# Volume-content API spec -> spec/openapi-volumecontent.yml
+# belt is private, so the fetch authenticates with a GitHub token
+# (see scripts/fetch-spec.sh).
+core.workflow(
+    name = "volume-api-spec",
+    origin = git.github_origin(
+        url = BELT_REPO,
+    ),
+    destination = folder.destination(),
+    origin_files = glob(["packages/volume-content/openapi.yml"]),
+    destination_files = glob(["openapi-volumecontent.yml"]),
+    authoring = AUTHORING,
+    mode = "SQUASH",
+    transformations = [
+        core.move("packages/volume-content/openapi.yml", "openapi-volumecontent.yml"),
+    ],
+)

copybara.Dockerfile

@@ -0,0 +1,27 @@
+FROM eclipse-temurin:21-jre
+
+RUN apt-get update && apt-get install -y --no-install-recommends git && \
+    rm -rf /var/lib/apt/lists/*
+
+# gh serves as the git credential helper for private origins; it reads the
+# token from GH_TOKEN in the environment.
+ENV GH_VERSION=2.96.0
+RUN ARCH=$(uname -m) && \
+    case "$ARCH" in \
+        x86_64) GH_ARCH="amd64" ;; \
+        arm64|aarch64) GH_ARCH="arm64" ;; \
+        *) echo "Unsupported architecture: $ARCH" && exit 1 ;; \
+    esac && \
+    curl -fsSL https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${GH_ARCH}.tar.gz | \
+    tar -xz --strip-components=2 -C /usr/local/bin gh_${GH_VERSION}_linux_${GH_ARCH}/bin/gh && \
+    git config --system credential.helper '!gh auth git-credential'
+
+ENV COPYBARA_VERSION=v20260720
+ENV COPYBARA_SHA256=e94448c702addc17cfc45d4bbfc8509d458b9a25f4715e3e77207ad570e1075d
+RUN curl -fsSL -o /opt/copybara.jar \
+    https://github.com/google/copybara/releases/download/${COPYBARA_VERSION}/copybara_deploy.jar && \
+    echo "${COPYBARA_SHA256}  /opt/copybara.jar" | sha256sum -c -
+
+WORKDIR /workspace
+
+ENTRYPOINT ["java", "-jar", "/opt/copybara.jar"]