feat(api): discover orchestrators via nomad service by wj-e2b · Pull Request #3176 · e2b-dev/infra

wj-e2b · 2026-07-02T00:09:39Z

Switch discovery to use nomad's service discovery instead of blindly checking every node in the default nodepool. Lets us use multiple nodepools for cutting over orchestrator from system jobs to service jobs.

Existing orchestrator-ee system job already registers to the service.

cursor · 2026-07-02T00:09:45Z

PR Summary

Medium Risk
Wrong or partial discovery would stop the API from reaching orchestrators for sandbox work; the legacy merge reduces rollout ordering risk but both backends must succeed on each sync cycle.

Overview
Nomad orchestrator discovery now uses configured Nomad-native service registrations (address and bound port) instead of treating every ready node in the default pool as an orchestrator on a fixed port, so orchestrators can live in other pools and job layouts. Startup wires service-based discovery with an optional union of the old default-pool node listing for jobs whose registrations still have an empty address, controlled by new environment settings, and tests follow the service API.

^{Reviewed by Cursor Bugbot for commit 4c073c2. Bugbot is set up for automated code reviews on this repo. Configure here.}

gemini-code-assist

Code Review

The error check !errors.As(err, &sde) in the deferred wait block will always evaluate to false when a service fails because startService always returns a serviceDoneError regardless of whether the service function f() succeeded or failed. This causes all service failures to be silently ignored, and the orchestrator will incorrectly exit with a success status instead of failing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov · 2026-07-02T00:11:40Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3183	2	3181	8

View the top 2 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestMaskRequestHostAPIParameter

Stack Traces | 4.68s run time

=== RUN   TestMaskRequestHostAPIParameter
=== PAUSE TestMaskRequestHostAPIParameter
=== CONT  TestMaskRequestHostAPIParameter
    mask_request_host_test.go:44: Command [python3] output: event:{start:{pid:1108}}
    mask_request_host_test.go:68: Command [cat] output: event:{start:{pid:1109}}
    mask_request_host_test.go:68: Command [cat] output: event:{data:{stderr:"cat: /tmp/nc_output.txt: No such file or directory\n"}}
    mask_request_host_test.go:68: Command [cat] output: event:{end:{exit_code:1 exited:true status:"exit status 1" error:"exit status 1"}}
    mask_request_host_test.go:69: 
        	Error Trace:	.../tests/proxies/mask_request_host_test.go:69
        	Error:      	Received unexpected error:
        	            	command cat in sandbox iqpkzpcwe8gah1bb0kmkp failed with exit code 1
        	Test:       	TestMaskRequestHostAPIParameter
--- FAIL: TestMaskRequestHostAPIParameter (4.68s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestSandboxCreateWithTag

Stack Traces | 40.5s run time

=== RUN   TestSandboxCreateWithTag
=== PAUSE TestSandboxCreateWithTag
=== CONT  TestSandboxCreateWithTag
    template_tags_test.go:115: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestSandboxCreateWithTag (40.48s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

cursor

Stale comment

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c82c8df518

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-06T18:17:06Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

chatgpt-codex-connector · 2026-07-06T18:17:33Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

chatgpt-codex-connector · 2026-07-06T18:23:48Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

cursor

Stale comment

…llback Discover orchestrators by listing registrations of Nomad-native services (GET /v1/service/<name> per configured name, unioned and deduped by node; names configurable via NOMAD_ORCHESTRATOR_SERVICE_NAMES, default "orchestrator") instead of listing ready Nomad nodes in the hardcoded "default" pool. Service discovery is node-pool- and job-agnostic, which is required while orchestrators migrate to the health-gated service job on the dedicated "orchestrator" pool. ShortID stays the truncated Nomad node ID, so node identity is stable across the backend switch. Already-running orchestrators were deployed from a jobspec that registers the service with an empty Address, so service discovery alone would miss them until redeployed. To avoid that rollout ordering constraint, the old node-pool listing is kept as a fallback and unioned with service discovery: dedup is by ShortID with service entries winning (they carry the real bound port), and a failure of either backend fails the whole listing rather than silently degrading to a subset of nodes. The fallback also keeps draining nodes (still Status=="ready") discoverable after their services deregister. The fallback is on by default; set NOMAD_ORCHESTRATOR_LEGACY_DISCOVERY_ENABLED=false once no legacy jobs remain.

chatgpt-codex-connector · 2026-07-06T23:16:33Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

wj-e2b requested review from ValentaTomas, dobrac and jakubno as code owners July 2, 2026 00:09

cla-bot Bot added the cla-signed label Jul 2, 2026

gemini-code-assist Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/factories/run.go Outdated

cursor Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread packages/api/internal/orchestrator/discovery/nomad.go Outdated

chatgpt-codex-connector Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/factories/run.go Outdated

djeebus approved these changes Jul 2, 2026

View reviewed changes

dobrac approved these changes Jul 2, 2026

View reviewed changes

wj-e2b force-pushed the wj-orchestrator-rollout branch 2 times, most recently from 46e04d9 to 750ab19 Compare July 6, 2026 18:17

wj-e2b force-pushed the wj-orchestrator-rollout branch from 750ab19 to 37a7881 Compare July 6, 2026 18:17

wj-e2b force-pushed the wj-orchestrator-rollout branch from 37a7881 to fddf68c Compare July 6, 2026 18:23

cursor Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread packages/api/internal/orchestrator/discovery/nomad.go

wj-e2b changed the title ~~fix(api): discover orchestrators via nomad service~~ feat(api): discover orchestrators via nomad service Jul 6, 2026

wj-e2b force-pushed the wj-orchestrator-rollout branch from fddf68c to 4c073c2 Compare July 6, 2026 23:16

dobrac reviewed Jul 7, 2026

View reviewed changes

Comment thread packages/api/internal/cfg/model.go

dobrac approved these changes Jul 7, 2026

View reviewed changes

Uh oh!

Conversation

wj-e2b commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

codecov Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 6, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wj-e2b commented Jul 2, 2026 •

edited

Loading

cursor Bot commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading