Skip to content

feat(query-runner): storage config + websocket notifications#27542

Open
Khairajani wants to merge 14 commits intomainfrom
query_runner_socket_flow
Open

feat(query-runner): storage config + websocket notifications#27542
Khairajani wants to merge 14 commits intomainfrom
query_runner_socket_flow

Conversation

@Khairajani
Copy link
Copy Markdown
Contributor

@Khairajani Khairajani commented Apr 20, 2026

Summary

Adds the OSS-side pieces that let the Collate Query Runner stream results to object storage and notify the UI via websocket when a query completes. All Collate-specific logic lives in open-metadata/openmetadata-collate#query_runner_socket_flow; this PR only lands the schema/infrastructure pieces that must live in OSS.

  • queryRunnerRequest.json: runtime-injected storageConfig (bucket + prefix, no credentials) and resultPath. Worker fetches credentials via a separate callback so secret: refs resolve in-worker via CustomSecretStr.
  • queryRunnerResponse.json: resultPath for S3-backed results (mutually exclusive with inline results).
  • WebSocketManager: new queryRunnerChannel constant.
  • WebsocketNotificationHandler + new QueryRunnerMessage: COMPLETED / FAILED notifications for the UI hook.
  • SOCKET_EVENTS.QUERY_RUNNER_CHANNEL: TS constant so the UI hook can subscribe.
  • Regenerated TS from the two schema changes (picked up by the type-generation CI).

Test plan

  • Type-generation workflow stays green (no drift).
  • Collate PR query_runner_socket_flow builds against this branch end-to-end (test-connection + real BigQuery query verified locally — S3 streaming, WebSocket → UI fetch).

🤖 Generated with Claude Code


Summary by Gitar

  • Task Notifications:
    • Added handleTaskNotification(Task task) in WebsocketNotificationHandler to support real-time updates for the new task system.
    • Implemented logic to broadcast notifications to all task assignees, including team members via TEAM entity relationships.

This will update automatically on new commits.

Adds schema + infrastructure for the Collate Query Runner to stream results
to object storage and notify the UI via websocket when a query completes.

- queryRunnerRequest: runtime-injected storageConfig {bucketName, prefix}
  and resultPath. Credentials never enter the payload — the worker fetches
  them via a server callback so secret: refs resolve in-worker via
  CustomSecretStr.
- queryRunnerResponse: resultPath for S3-backed results (mutually exclusive
  with inline results).
- WebSocketManager: queryRunnerChannel constant.
- WebsocketNotificationHandler + QueryRunnerMessage: COMPLETED/FAILED
  notifications for the UI hook.
- SOCKET_EVENTS.QUERY_RUNNER_CHANNEL so the UI hook can subscribe.

Paired with open-metadata/openmetadata-collate query_runner_socket_flow.
@Khairajani Khairajani requested a review from a team as a code owner April 20, 2026 09:39
@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@Khairajani Khairajani added Ready for testing Tickets that are fixed and ready for testing before closing them safe to test Add this label to run secure Github workflows on PRs and removed Ready for testing Tickets that are fixed and ready for testing before closing them labels Apr 20, 2026
@github-actions
Copy link
Copy Markdown
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Copy Markdown
Contributor

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 20, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.42% (62824/100643) 42.71% (33817/79165) 45.72% (10013/21897)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 20, 2026

🟡 Playwright Results — all passed (17 flaky)

✅ 3980 passed · ❌ 0 failed · 🟡 17 flaky · ⏭️ 86 skipped

Shard Passed Failed Flaky Skipped
🟡 Shard 1 295 0 4 4
🟡 Shard 2 750 0 4 8
🟡 Shard 3 745 0 2 7
🟡 Shard 4 773 0 2 18
🟡 Shard 5 685 0 2 41
🟡 Shard 6 732 0 3 8
🟡 17 flaky test(s) (passed on retry)
  • Features/DataAssetRulesDisabled.spec.ts › Verify the Database Service entity item action after rules disabled (shard 1, 1 retry)
  • Flow/Tour.spec.ts › Tour should work from help section (shard 1, 1 retry)
  • Flow/Tour.spec.ts › Tour should work from welcome screen (shard 1, 1 retry)
  • Flow/Tour.spec.ts › Tour should work from URL directly (shard 1, 1 retry)
  • Features/ActivityAPI.spec.ts › Activity event is created when description is updated (shard 2, 1 retry)
  • Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
  • Features/BulkImport.spec.ts › Database Schema (shard 2, 1 retry)
  • Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard 2, 1 retry)
  • Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
  • Features/Workflows/WorkflowOssRestrictions.spec.ts › clicking a node in view mode opens read-only config sidebar (no save or delete buttons) (shard 3, 1 retry)
  • Pages/CustomProperties.spec.ts › Should clear search and show all properties for apiCollection in right panel (shard 4, 1 retry)
  • Pages/Domains.spec.ts › Domain Rbac (shard 4, 1 retry)
  • Pages/Entity.spec.ts › Inactive Announcement create & delete (shard 5, 1 retry)
  • Pages/EntityDataConsumer.spec.ts › Tier Add, Update and Remove (shard 5, 1 retry)
  • Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
  • Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
  • Pages/ServiceEntity.spec.ts › Tier Add, Update and Remove (shard 6, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@github-actions
Copy link
Copy Markdown
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@Khairajani Khairajani self-assigned this Apr 30, 2026
@gitar-bot
Copy link
Copy Markdown

gitar-bot Bot commented Apr 30, 2026

Code Review ✅ Approved

Integrates storage configuration and websocket notifications into the query runner. No issues were found.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@sonarqubecloud
Copy link
Copy Markdown

@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant