Skip to content

Commit 07c3ce9

Browse files
isaacrowntreeclaude
andcommitted
refactor: Simplify R2 cache population to use local wrangler dev with remote binding
Reworks the R2 cache upload approach per reviewer feedback. Instead of deploying the worker with a token, populating, then redeploying, we now use a local wrangler dev worker with a remote R2 binding. Key changes: - Replace deploy/populate/redeploy cycle with local wrangler dev approach - Remove --cacheMethod flag and dual code paths - Remove token auth system (local worker needs no auth) - Remove esbuild compilation step (wrangler accepts TS natively) - Revert deploy.ts, worker.ts, cloudflare-context.ts, build.ts, run-wrangler.ts to main - Delete compile-cache-populate-handler.ts The new flow derives a temp wrangler config with the R2 binding set to remote: true, starts wrangler dev locally, sends batched cache entries to the worker, then stops it. Single code path for all cache sizes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 11473ee commit 07c3ce9

11 files changed

Lines changed: 284 additions & 754 deletions

File tree

.changeset/fix-r2-cache-upload.md

Lines changed: 12 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,17 @@
22
"@opennextjs/cloudflare": minor
33
---
44

5-
fix: Use worker binding for R2 cache uploads to avoid API rate limits
6-
7-
For large deployments with 500+ prerendered pages, the R2 incremental cache upload now uses the worker's R2 binding directly instead of wrangler's `r2 bulk put` command. This bypasses the Cloudflare API rate limit of 1,200 requests per 5 minutes.
8-
9-
**New `--cacheMethod` flag:**
10-
```bash
11-
# Auto-select based on cache size (default) - uses binding for 500+ entries
12-
opennextjs-cloudflare deploy --cacheMethod=auto
13-
14-
# Force wrangler CLI (original behavior)
15-
opennextjs-cloudflare deploy --cacheMethod=wrangler
16-
17-
# Force worker binding (bypasses API rate limits)
18-
opennextjs-cloudflare deploy --cacheMethod=binding
19-
```
20-
21-
The new deploy flow for large R2 caches (when using `binding` method):
22-
1. Deploy worker with a temporary cache populate token
23-
2. Send cache entries directly to the worker's `/_open-next/cache/populate` endpoint
24-
3. Worker writes to R2 using its binding (no API rate limits)
25-
4. Redeploy without the token to secure the endpoint
26-
27-
Features:
28-
- Automatic threshold: Uses binding approach for caches with 500+ entries (in `auto` mode)
29-
- Batched uploads with configurable batch size (default: 100)
30-
- Retry logic with exponential backoff
31-
- Secure: Temporary token is removed after cache population
32-
- Full backward compatibility: Use `--cacheMethod=wrangler` to force original behavior
33-
34-
This fixes the "500 Internal Server Error" during R2 cache upload for applications with thousands of prerendered pages.
5+
fix: Use remote R2 binding for cache population to avoid API rate limits
6+
7+
For deployments with prerendered pages, the R2 incremental cache is now populated
8+
using a local wrangler dev worker with a remote R2 binding instead of `wrangler r2 bulk put`.
9+
This bypasses the Cloudflare API rate limit of 1,200 requests per 5 minutes that caused
10+
failures for large applications with thousands of prerendered pages.
11+
12+
The new approach:
13+
1. Derives a temporary wrangler config from the project's config with the R2 binding set to `remote: true`
14+
2. Starts a local worker via `wrangler dev` with a POST endpoint
15+
3. Sends batched cache entries to the local worker, which writes to R2 via the binding
16+
4. No deployment or authentication tokens required
3517

3618
Closes #1088

packages/cloudflare/src/api/cloudflare-context.ts

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,6 @@ declare global {
3434
// Prefix used for the R2 incremental cache bucket
3535
[R2_CACHE_PREFIX_ENV_NAME]?: string;
3636

37-
// Token for cache population during deployment (auto-generated, temporary)
38-
// This token is used by the CLI to authenticate cache population requests
39-
OPEN_NEXT_CACHE_POPULATE_TOKEN?: string;
40-
4137
// D1 db used for the tag cache
4238
NEXT_TAG_CACHE_D1?: D1Database;
4339

packages/cloudflare/src/cli/build/build.ts

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ import { OpenNextConfig } from "../../api/config.js";
1212
import type { ProjectOptions } from "../project-options.js";
1313
import { bundleServer } from "./bundle-server.js";
1414
import { compileCacheAssetsManifestSqlFile } from "./open-next/compile-cache-assets-manifest.js";
15-
import { compileCachePopulateHandler } from "./open-next/compile-cache-populate-handler.js";
1615
import { compileEnvFiles } from "./open-next/compile-env-files.js";
1716
import { compileImages } from "./open-next/compile-images.js";
1817
import { compileInit } from "./open-next/compile-init.js";
@@ -83,7 +82,6 @@ export async function build(
8382
await compileInit(options, wranglerConfig);
8483
await compileImages(options);
8584
await compileSkewProtection(options, config);
86-
await compileCachePopulateHandler(options);
8785

8886
// Compile middleware
8987
await createMiddleware(options, { forceOnlyBuildOnce: true });

packages/cloudflare/src/cli/build/open-next/compile-cache-populate-handler.ts

Lines changed: 0 additions & 26 deletions
This file was deleted.

0 commit comments

Comments
 (0)