feat(open-metadata): enable Airbyte connector for QA#4654
Open
quazi-h wants to merge 3 commits into
Open
Conversation
- Add airbyte block to QA SOPS secrets file with internal K8s service URL (airbyte-airbyte-server-svc.airbyte.svc.cluster.local:8001) - Enable airbyte CronOMJob in substructure QA stack config Airbyte built-in auth is disabled; username/password are empty strings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Enables the Airbyte metadata ingestion connector for the QA OpenMetadata deployment by turning on the existing Airbyte ingestion CronOMJob and adding corresponding connector configuration to QA secrets.
Changes:
- Enable the Airbyte ingestion CronOMJob in the OpenMetadata QA substructure stack config.
- Add an
airbyteconnector block (host/port + credentials fields) to the QA OpenMetadata SOPS secrets file.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/ol_infrastructure/substructure/open_metadata/Pulumi.QA.yaml |
Turns on the pre-existing Airbyte connector CronOMJob via enable_airbyte_connector. |
src/bridge/secrets/open_metadata/secrets.qa.yaml |
Adds the Airbyte connector configuration (host/port + username/password fields) to QA bridge secrets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+17
to
+18
| username: "" | ||
| password: "" |
SOPS was storing empty strings as plaintext. Replaced with placeholder 'airbyte' values that are properly AES256-GCM encrypted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
Closes mitodl/ol-data-platform#1355
Description (What does it do?)
Enables the Airbyte metadata ingestion connector for the QA OpenMetadata deployment.
airbyteblock to the QA SOPS secrets file (src/bridge/secrets/open_metadata/secrets.qa.yaml) with the internal Kubernetes service URL (airbyte-airbyte-server-svc.airbyte.svc.cluster.local:8001). Username and password are empty strings because Airbyte's built-in auth is disabled (auth.enabled: false) and auth is handled externally by APISIX/OIDC.open_metadata:enable_airbyte_connector: "true"in the substructure QA stack config (Pulumi.QA.yaml), activating the pre-existingairbyteCronOMJob (daily at 03:00) that runs theairbyte_metadata.pyingestion script.Both the
_all_connector_configsentry in the application stack and theairbyte_metadata.pyingestion script were already in place from prior merged work.How can this be tested?
pulumi upon the application stack (QA) to provision theom-connector-airbyteK8s secret via Vault/VSO.pulumi upon the substructure stack (QA) to create the CronOMJob.Additional Context
Production follow-up (separate PR): add
airbyteblock tosecrets.production.yamland setenable_airbyte_connector: "true"inPulumi.Production.yaml.