feat(query-runner): storage config + websocket notifications#27542
feat(query-runner): storage config + websocket notifications#27542Khairajani wants to merge 14 commits intomainfrom
Conversation
Adds schema + infrastructure for the Collate Query Runner to stream results
to object storage and notify the UI via websocket when a query completes.
- queryRunnerRequest: runtime-injected storageConfig {bucketName, prefix}
and resultPath. Credentials never enter the payload — the worker fetches
them via a server callback so secret: refs resolve in-worker via
CustomSecretStr.
- queryRunnerResponse: resultPath for S3-backed results (mutually exclusive
with inline results).
- WebSocketManager: queryRunnerChannel constant.
- WebsocketNotificationHandler + QueryRunnerMessage: COMPLETED/FAILED
notifications for the UI hook.
- SOCKET_EVENTS.QUERY_RUNNER_CHANNEL so the UI hook can subscribe.
Paired with open-metadata/openmetadata-collate query_runner_socket_flow.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
🟡 Playwright Results — all passed (17 flaky)✅ 3980 passed · ❌ 0 failed · 🟡 17 flaky · ⏭️ 86 skipped
🟡 17 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
Code Review ✅ ApprovedIntegrates storage configuration and websocket notifications into the query runner. No issues were found. OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
|



Summary
Adds the OSS-side pieces that let the Collate Query Runner stream results to object storage and notify the UI via websocket when a query completes. All Collate-specific logic lives in
open-metadata/openmetadata-collate#query_runner_socket_flow; this PR only lands the schema/infrastructure pieces that must live in OSS.queryRunnerRequest.json: runtime-injectedstorageConfig(bucket + prefix, no credentials) andresultPath. Worker fetches credentials via a separate callback sosecret:refs resolve in-worker viaCustomSecretStr.queryRunnerResponse.json:resultPathfor S3-backed results (mutually exclusive with inlineresults).WebSocketManager: newqueryRunnerChannelconstant.WebsocketNotificationHandler+ newQueryRunnerMessage: COMPLETED / FAILED notifications for the UI hook.SOCKET_EVENTS.QUERY_RUNNER_CHANNEL: TS constant so the UI hook can subscribe.Test plan
query_runner_socket_flowbuilds against this branch end-to-end (test-connection + real BigQuery query verified locally — S3 streaming, WebSocket → UI fetch).🤖 Generated with Claude Code
Summary by Gitar
handleTaskNotification(Task task)inWebsocketNotificationHandlerto support real-time updates for the new task system.TEAMentity relationships.This will update automatically on new commits.