feat(connector): delegate add_resource imports to external Connector#3084
Open
zihengli-bytedance wants to merge 1 commit into
Open
feat(connector): delegate add_resource imports to external Connector#3084zihengli-bytedance wants to merge 1 commit into
zihengli-bytedance wants to merge 1 commit into
Conversation
Opt-in integration that routes add_resource data fetching and parsing to external Connector service; the Connector stages source data and calls back into OV through the standard add_resource pipeline. - add ConnectorClient wrapping the control plane's inner doc/add and task/info endpoints - add [connector] config section: enable, connector/tracker endpoint URLs, main_account_id, timeout_seconds, poll_interval_ms, allowed_add_types - route add_resource via Connector when enabled and args.add_type is in allowed_add_types; otherwise fall back to the standard pipeline with an info log - track imports as connector_import TaskRecords and poll Connector task status in the background until terminal state or timeout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Opt-in integration that routes
add_resourcedata fetching and parsing to an external Connector service. When Connector integration is enabled and the requestedadd_typeis allowed, OpenViking delegates the import task to Connector. Connector stages source data and calls back into OV through the standardadd_resourcepipeline, so the existing ingestion flow remains the source of truth.The integration is disabled by default and falls back to the existing import path when not enabled or when the requested
add_typeis not configured for Connector.Changes Made
ConnectorClientfor Connectordoc/addandtask/infoAPIs.[connector]config for enablement, endpoints, account ID, timeout, polling interval, and allowedadd_typevalues.ResourceService.add_resourcethrough Connector when enabled and allowed.connector_importtask tracking and background Connector task polling.RequestContextfor Connector callbacks.Testing
Checklist