feat: use remote dev for r2 cache population

[edmundhung]

Fixes #1110, #1088
Close #1121, #1099

This is based on the work @vicb and @isaacrowntree did in the PRs above, with the following changes:

• Use Wrangler remote dev for a more reliable remote R2 cache population path as remote bindings fail under large payload volume (Shared WebSocket RPC sessions in workerd seem to fail under large payload volume cloudflare/capnweb#158)
• Auto-provision the R2 bucket when remote cache population runs and the bucket does not exist
• Set config: "" when starting the helper worker to avoid auto-discovering the project wrangler config and leaking unrelated bindings into the cache-populate worker
• gradually ramp up concurrency to avoid overwhelming the worker
• simplify the upload request to reduce overhead during cache population
• add retry/backoff for upstream 5xx responses

[changeset-bot]

🦋 Changeset detected

Latest commit: fbdae21

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@opennextjs/cloudflare	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@opennextjs/cloudflare@1159

commit: fbdae21

[grabmateusz]

Hi @edmundhung, I've decided to give it a go and unfortunately current state of PR does not resolve #1110
It fails on our end with:

Populating remote R2 incremental cache...
Monorepo detected at <path>
  0%|                                             | 1/9599 [00:00:<00:00:, 0.00it/s]
  0%|                                             | 2/9599 [00:00:<00:23:, 400.00it/s]

Some additional progress and:

  1%|                                             | 129/9599 [00:13:<16:11:, 9.75it/s]
ERROR Failed to populate remote R2 bucket "<bucket>" for binding "NEXT_INC_CACHE_R2_BUCKET": fetch failed

Or

Populating remote R2 incremental cache...
Monorepo detected at <path>
  0%|                                             | 1/123 [00:00:<00:00:, 0.00it/s]
  1%|                                             | 2/123 [00:00:<00:00:, 400.00it/s]

Some additional progress and:

 21%|█████████                                    | 26/123 [00:00:<00:00:, 1733.33it/s]
ERROR Failed to populate remote R2 bucket "<bucket>" for binding "NEXT_INC_CACHE_R2_BUCKET": Unexpected token '<', "<!DOCTYPE "... is not valid JSON

Tested on Next.js 15.5.14 and 16.1.6.

[edmundhung]

@grabmateusz Thanks for giving this a try 🙏🏼

I have made a few additional changes to make the remote cache populate path more resilient. Would you mind giving the pre-release another try?

npm i https://pkg.pr.new/@opennextjs/cloudflare@92b3cf2

If it still fails, could you also try lowering --cache-chunk-size from the default 25 to something smaller like 10 or 5 and let me know whether that fixes the issue?

[grabmateusz]

Hi @edmundhung, this looks much better, I've run 6 parallel deployments (without adjusting --cache-chunk-size) and results are positive now, among them rather big ISR caches were populated successfully:

100%|█████████████████████████████████████████████| 9599/9599 [01:19:<00:00:, 121.05it/s]
Successfully populated cache with 9599 entries

or

100%|█████████████████████████████████████████████| 12859/12859 [03:21:<00:00:, 63.91it/s]
Successfully populated cache with 12859 entries

Test performed so far only on Next.js 15.5.14 but looks promising, will update this comment tomorrow with Next.js 16.1.6 test results.

[james-elicx]

Been through some of the logic and it looks good / makes sense. Just have a couple of minor questions/notes.

[petebacondarwin]

A bunch of comments. Hope they are helpful.

[petebacondarwin]

Thanks for addressing all my feedback

[edmundhung]

Thanks for the review, @petebacondarwin and @james-elicx!

[edmundhung]

@grabmateusz We are ready to ship this. If you get a chance, feel free to give the latest build another try.

[grabmateusz]

Hi @edmundhung, newest package tested o both Next.js 15.5.14 and 16.1.6.
Looking forward for stable package with those changes!

[edmundhung]

Thanks for all the feedbacks, @grabmateusz. This is released in v1.18.0.