feat(self-hosted): background tasks, scheduler & real-time on the server flavour#5414
Open
adrians5j wants to merge 33 commits into
Open
feat(self-hosted): background tasks, scheduler & real-time on the server flavour#5414adrians5j wants to merge 33 commits into
adrians5j wants to merge 33 commits into
Conversation
… to the server flavour Wire the AWS-only background-tasks, scheduler, and file-manager capabilities into the self-hosted Node server flavour, plus the transport fixes needed to run them in a single long-running process. Background tasks: - Register BackgroundTasksServerFeature in the server handler root (in-process worker_thread runner + /background-task route). Root, not per-request, so the shared InternalToken gate is a stable singleton. - WorkerService reads the runtime PORT env instead of the (unset) SERVER_PORT build param. Scheduler: - Wire registerSchedulerServerExtension via the registerSchedulerTransport hook. - Register its RequestContextInitializer via register() (not registerInstance) so it runs after SchedulerModelContextualSchema registers ScheduledActionModel. File manager: - Add a registerFileManagerTransport hook to the shared request stack (replaces the hardcoded FileManagerS3Feature); AWS passes S3, server passes FileManagerServerFeature. - FileManagerServerFeature: register local asset delivery, the upload GraphQL schema (mirrors S3 query names so the SDK is unchanged), and the domain Read/WriteFileMetadata features; resolve config lazily (build params are not registered at register() time). - Add Infra.FileStorage (bakes WEBINY_LOCAL_STORAGE_PATH + WEBINY_UPLOAD_SECRET build params). - resolveServerUrl derives the URL from WEBINY_API_URL/PORT (the DI-native gql context exposes no request headers to resolvers). - SettingsInstaller falls back to WEBINY_API_URL for srcPrefix when there is no CloudFront manifest (AWS-safe). Transport (event-handler-server): - Read binary request bodies as raw Buffers (multipart uploads were utf8-corrupted). - Write Buffer response bodies raw (image delivery was JSON-serialized -> ORB-blocked). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nsports`
Collapse the three structurally-identical hooks (registerRealtimeTransport /
registerSchedulerTransport / registerFileManagerTransport) into a single
`transports: { realtime?, scheduler?, fileManager? }` object on
RegisterApiRequestStackConfig, plus a shared `TransportRegistrar` type. Behaviour
unchanged — the adapters still run at the same interleave points. Makes "these are all
transport adapters" one named concept and adding a future request-seam transport a single
key rather than a new top-level param. Updates the AWS and server handlers to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `assetDelivery` config knob was never set (both S3 and server callers pass no config), so it was speculative parity. Remove the interface + param; register with `createLocalAssetDeliveryFeature()` defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rver-bg-scheduler-fm # Conflicts: # packages/project-server/src/infra.ts
…ess.env API runtime code must not read build-time config from process.env. Add Infra.ApiUrl (bakes the WEBINY_API_URL build param) and consume it via BuildParams: - FileManagerServerConfig gains `apiUrl` (from BuildParams); the upload use cases use it, and the process.env-reading `resolveServerUrl` helper is deleted. - SettingsInstaller reads the build param (falling back after the CloudFront manifest) for the file srcPrefix instead of process.env. process.env stays only in webiny.config (build-time, where it belongs) and for genuinely runtime values (PORT). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three `transports` entries read inconsistently because each package exposed a different
surface (raw impl / function / Feature). Give each a `createFeature` Feature so every entry
in the handlers' `transports: {}` reads identically as `XFeature.register(c)`:
- api-websockets-server: add WebsocketsServerFeature (registers ServerWebsocketsTransport)
- api-scheduler-server: add SchedulerServerFeature (wraps registerSchedulerServerExtension)
- api-scheduler-aws: add SchedulerAwsFeature (wraps registerSchedulerAwsExtension with the
default scheduler client)
Both the AWS and server handlers updated. The underlying functions/impls stay exported for
custom wiring. Behaviour unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…api-scheduler-server The Bree scheduler was rebuilt per request (new instance + re-arm each request) and api-scheduler-server imported event-handler-core (RequestContextInitializer) — a transport coupling in what should be a pure adapter. Rework it into one long-lived root singleton, and move the transport glue to the composition layer. api-scheduler-server (now PURE — no event-handler-core import): - BreeSchedulerService tracks tenant per job and fires onTrigger(id, namespace, tenant), so a single instance serves all tenants (timers fire outside any request, so tenant can't be read from context). `recover()` is public for boot re-arming. - Remove context.ts (the per-request RequestContextInitializer) and SchedulerServerFeature; the package now exports only BreeSchedulerService. Drop the event-handler-core + feature deps. api-event-handler-server (composition layer owns the transport wiring): - registerSchedulerServer (root): one BreeSchedulerService, registered as SchedulerService (so per-request create/update/delete hit the one live timer set) + as SchedulerSingleton; plus the internal token and the run/recover routes. Counterpart of the WebSockets connection manager. - ScheduledActionRunRoute: the singleton POSTs here when a timer fires; it rebuilds the tenant's full request context (initializers + contextual schemas) and runs ExecuteScheduledAction — mirrors the background-task run route. - ScheduledActionRecoverRoute: re-arms a tenant's persisted pending actions at boot (root tenant; multi-tenant boot recovery is a noted follow-up). - onServer starts the timers and defers the recover POST until the server is listening. - Scheduler leaves the per-request `transports` block (it's now root + onServer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ildparams-transport-surface # Conflicts: # packages/api-event-handler-aws/src/createWebinyApiHandler.ts # packages/api-event-handler-server/src/createWebinyApiHandler.ts # packages/api-file-manager-server/src/features/CreateMultiPartUpload/CreateMultiPartUploadUseCase.ts # packages/api-file-manager-server/src/features/FileManagerServerConfig/FileManagerServerConfig.ts # packages/api-file-manager-server/src/features/GetUploadPayload/GetUploadPayloadUseCase.ts # packages/api-file-manager-server/src/utils/resolveServerUrl.ts # packages/api-file-manager/src/features/settings/SettingsInstaller/SettingsInstaller.ts # packages/project-server/src/infra.ts # webiny.config.server.tsx
…adrian/self-hosted-scheduler-boot-singleton
…adrian/self-hosted-scheduler-boot-singleton
#5409's optimization skips the cache→dist restore when a dist marker (`.webiny-build-hash`) matches the package's source hash. The marker can go stale when something writes dist out of band (e.g. `webiny watch`, or a dependency changes while the dependent's own source hash is unchanged), which surfaces as a stale dist / dangling imports. Until that class of staleness is fully addressed, make the optimization opt-in. Default behavior reverts to always restoring from cache (the original, always-correct behavior). Enable with `WEBINY_EXPERIMENTAL_BUILD_CACHE=true`. When disabled, the orchestrator neither reads nor writes the marker, so any markers left on disk are inert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…skip' into adrian/self-hosted-scheduler-boot-singleton
…e publicPath asset
The task worker was spawned via `new Worker(new URL("../worker/workerEntry.js", import.meta.url))`.
The app bundler rewrites that `new URL(...)` to a publicPath-based asset URL
("/static/assets/workerEntry.<hash>.js"), which `new Worker()` then tries to load as a filesystem
path from root and fails ("Cannot find module '/static/assets/...'") — so every background task on
the server flavour stayed stuck "pending" (the worker never started).
`import.meta.url` resolves to this module inside its own dist (the bundler preserves it and does not
inline the package), and the compiled worker + its deps live beside it under dist/worker with only
relative/node-builtin imports. So resolve the worker with `path.join` off `import.meta.url` — an
expression the bundler leaves untouched — pointing straight at the real, node-resolvable dist worker.
Verified end-to-end on the server flavour: image upload → the worker spawns, POSTs the background-task
route, and the AI-image-enrichment + metadata-extract tasks run to completion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… shared runner validates + surface worker errors
…heduler-boot-singleton # Conflicts: # packages/api-event-handler-server/src/createWebinyApiHandler.ts # packages/api-scheduler-server/package.json # packages/api-scheduler-server/src/index.ts # packages/api-scheduler-server/tsconfig.build.json # packages/api-scheduler-server/tsconfig.json # yarn.lock
adrians5j
changed the title
adrians5j
changed the base branch from
adrian/self-hosted-buildparams-transport-surface
to
next
July 16, 2026 11:30
… so targeted sends route Server-flavour WebSocket connections were registered with an empty identity and were never persisted to the ConnectionRegistry, so SendToIdentity (which looks connections up by identity id) matched nothing — targeted server→client notifications (e.g. AI image enrichment) never reached the admin. - ServerConnectionManager.add now registers the connection in the ConnectionRegistry (connectionId → identity/tenant): the row SendToIdentity/ListConnections query by identity. - WebsocketsServer decodes the `?token` JWT + `?tenant` from the upgrade URL and registers the connection under the real identity via a new `authenticate` callback (guarded; a failure falls back to an anonymous, unaddressable connection rather than crashing the upgrade). - The server handler supplies `authenticate` via a per-connection request-scoped child container — AuthenticationContext lives in the per-request stack, not at root; the token→identity step is tenant-independent, so it's safe to run at connection time. Also folds into this PR: - fix(ai-powerups): send image bytes as base64 to the AI provider instead of a URL; a URL forces the provider to fetch private/local files, which fails off-cloud. - fix(app-websockets): on the server flavour, derive the ws:// URL from the admin API URL when REACT_APP_WEBSOCKET_URL is unset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…heduler-boot-singleton # Conflicts: # packages/cli-core/files/references.json
…enant-aware API
Align the unit tests with this branch's redesign:
- BreeSchedulerService: onTrigger now receives (id, namespace, tenant); recovery
moved from start(actions) to start() + recover(actions); internal `namespaces`
map is now `jobs` keyed with { namespace, tenant }.
- ServerConnectionManager: add() now registers the connection in the
ConnectionRegistry, so the test asserts register() is called (not skipped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adrians5j
force-pushed
the
adrian/self-hosted-scheduler-boot-singleton
branch
from
July 16, 2026 13:15
22ed432 to
e593b28
Compare
adrians5j
changed the title
…tAuthorization at boot Boot recovery runs outside any request, so it has no identity. ListScheduledActions performs a permission check, which rejected the anonymous boot identity with "Not authorized!" — so persisted schedules were never re-armed after a restart. Wrap the list in identityContext.withoutAuthorization(), the same escape hatch the run route (ExecuteScheduledActionUseCase) already uses. Also: - Add boot/trigger observability to the scheduler: log the recovered count and surface non-ok run/recover responses (fetch only rejects on network errors, so a 403/500 was previously invisible). - Inline the per-request stack back into the handler's `request` hook and give the WebSocket auth child a transport-less registerApiRequestStack (it only needs the core auth stack) — removes the shared helper without duplicating the transport config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-box Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ct currentDir Extract currentDir in WorkerTaskService for readability, and add pointers to issue #5429 (standalone server build must externalize the worker-file packages) next to both runtime sidecar-file loaders (background-tasks worker, scheduler bree pollWorker). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adrians5j
force-pushed
the
adrian/self-hosted-scheduler-boot-singleton
branch
from
July 17, 2026 06:37
5e2df0a to
207820e
Compare
…in.WebsocketsUrl Add an `Admin.WebsocketsUrl` extension (mirrors `Admin.ApiUrl`) that bakes a dedicated WebSocket URL into the admin bundle as `WEBINY_ADMIN_WS_API_URL`. getUrl now resolves, in order: the AWS API-Gateway URL (REACT_APP_WEBSOCKET_URL), this dedicated server URL (if configured), else derives ws(s):// from the API URL as before. The dedicated URL is optional — only needed when WebSockets are served from a different origin than the API — and, like the API URL, is a build param populated from any env the config chooses, so users aren't tied to a fixed env var name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ss.env in app code Move all WebSocket-URL env resolution to resolveWebsocketUrl() at the admin composition root (the sanctioned process.env spot, alongside resolveApiUrl), feed it into EnvConfig.websocketUrl, and have WebsocketsContextProvider read it via useFeature(EnvConfigFeature). Removes the process.env reads from getUrl.ts (deleted) so app code depends on the config abstraction instead of raw env vars. Resolution order is unchanged: REACT_APP_WEBSOCKET_URL (AWS) -> WEBINY_ADMIN_WS_API_URL (dedicated server) -> derived from the API URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…visibility only) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single PR for the remaining self-hosted (server) flavour work — wiring the server-specific packages into the Node HTTP handler and fixing the gaps found end-to-end. (
#5412build-param + transport-surface work and#5420are folded in here.)What changed
The self-hosted server flavour now runs the capabilities that previously worked only on AWS:
Changelog
Self-hosted server flavour: background tasks, scheduler, and real-time notifications
The self-hosted (non-AWS) server now runs background jobs, scheduled actions, and live notifications end-to-end. Background tasks such as AI image enrichment run to completion, scheduled actions fire on time and survive restarts, and real-time updates are delivered to the correct signed-in user in the admin app.
Squash Merge Commit