feat(sinks): add new databricks_zerobus for Databricks ingestion by flaviofcruz · Pull Request #24840 · vectordotdev/vector

flaviofcruz · 2026-03-03T23:42:49Z

Summary

Databricks provides a Zerobus ingest connector [1], a push based API that writes data directly into Unity Catalog Delta tables. This PR introduces a new vector sink that integrates with Databricks, allowing Vector to push data into Databricks. We use the Databricks provided SDK to implement the sink [2].

Zerobus supports row level ingestion and that's we do here. Zerobus also has arrow batch in experimental mode but we didn't add support for it. We will swap the row level ingestion once it becomes stable and that will be the future default.

With row based ingestion, we extended the BatchSerializerConfig to support a batch serializer that creates vector's of protocol buffer bytes. This makes it the second option for doing batch serialization, along arrow batch.

Users do not have to specify the schema at all, we will fetch the schema for them from Unity Catalog and then use on the API. If users want to do schema changes, they should update their table as needed. We don't have a lot of support for dynamic schema changes at the moment.

Vector configuration

[sinks.databricks_zerobus]
type = "databricks_zerobus"
inputs = ["logs"]
ingestion_endpoint = "https://91041497925470.zerobus.us-west-2.cloud.databricks.com"
table_name = "main.default.zerobus_table"
unity_catalog_endpoint = "https://logfood-us-west-2-mt.cloud.databricks.com/"
[sinks.databricks_zerobus.auth]
strategy = "oauth"
client_id = "<client id>"
client_secret = "<secret>"

How did you test this PR?

Unit tests, running small toy examples and using it in production for actual traffic.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

[1] https://docs.databricks.com/aws/en/ingestion/zerobus-overview
[2] https://github.com/databricks/zerobus-sdk

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

pront · 2026-04-03T15:46:15Z

Thanks @flaviofcruz for this new integration! Apologies for the slow review on this one.

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 436d0da4bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-04-10T17:11:01Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9dcb5d1e71

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pront · 2026-04-10T18:03:25Z

FYI I am waiting for @hsuanyi to sign the CLA (see comment) before reviewing this further. Also, there is an resolved review comment.

pront · 2026-04-13T19:01:34Z

FYI I am waiting for @hsuanyi to sign the CLA (see comment) before reviewing this further. Also, there is an resolved review comment.

@flaviofcruz in case you missed the above, we will require all profiles who contributed to this PR to sign the CLA. Happy to review once that is done.

hsuanyi · 2026-04-13T19:35:20Z

I have read the CLA Document and I hereby sign the CLA

pront · 2026-04-13T19:38:07Z

recheck

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 050409defd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

flaviofcruz · 2026-04-13T22:06:34Z

@pront really appreciate your work for the review. However, I was looking at the zerobus SDK license and the license could be problematic: https://github.com/databricks/zerobus-sdk/blob/main/LICENSE Do you know if this could be a blocker? Let me know if that is.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 36a74ef530

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d7b620bfb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…ck up the environment variables

…f no URL is set up

flaviofcruz · 2026-05-04T14:01:45Z

LGTM, thank you!

There are a few minor issues remaining (please followup and resolve any open threads), but approving to unblock this effort.

Reminder to rebase on #25340 once that is merged.

Thanks, I have rebased and force pushed. I will follow up with the VRL fixes which should address codex comment above. Will also follow up with arrow batch support once I get the greenlight from the zerobus team.

pront · 2026-05-04T16:24:17Z

Hi @flaviofcruz, there are a few open threads and it's unclear if they are resolved or not. Ping me when ready for a final review.

flaviofcruz · 2026-05-04T17:55:18Z

Hi @flaviofcruz, there are a few open threads and it's unclear if they are resolved or not. Ping me when ready for a final review.

Threads should be closed now I think.

pront

Left a few review comments from the local pass.

pront · 2026-05-04T19:36:09Z

+    }
+
+    /// Encode a batch of events into a `BatchOutput`.
+    pub fn encode_batch(&self, events: &[Event]) -> Result<BatchOutput, Error> {


Just noting down here that the 10MB limit enforced in config.rs might not hold in all cases. For example, a numeric-heavy schema right at the 10MB boundary could realistically encode over 10MB and fail at the SDK call.

Should we just a comment on the sink documentation? Or you have some other proposal?

A short mention in the docs is a good idea. But

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d564d98f70

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 141f680972

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-05-04T23:30:09Z

+				When disabled (the default), events are marked as delivered as soon as the
+				ingestion call completes without error, without waiting for an explicit
+				server acknowledgement.


Correct ack-disabled behavior description

This paragraph says that with acknowledgements disabled the sink marks events delivered without waiting for server acknowledgement, but the implementation always waits on wait_for_offset after every ingest and even errors on missing offsets (MissingAckOffset) regardless of sink ack config (src/sinks/databricks_zerobus/service.rs:390-393). Users who disable acknowledgements to avoid server-ack latency/failures will get different runtime behavior than documented, so this guidance should be updated to match the actual sink semantics.

Useful? React with 👍 / 👎.

github-actions Bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Mar 3, 2026

flaviofcruz changed the title ~~feat(databricks zerobus): add new databricks_zerobus for ingesting da…~~ feat(databricks zerobus): add new databricks_zerobus for Databricks ingestion Mar 3, 2026

flaviofcruz force-pushed the upstream-databricks-zerobus branch 3 times, most recently from 2368e4a to 42bf043 Compare March 12, 2026 17:04

flaviofcruz marked this pull request as ready for review March 12, 2026 17:05

flaviofcruz requested review from a team as code owners March 12, 2026 17:05

flaviofcruz changed the title ~~feat(databricks zerobus): add new databricks_zerobus for Databricks ingestion~~ feat(sinks): add new databricks_zerobus for Databricks ingestion Mar 12, 2026

github-actions Bot added the domain: ci Anything related to Vector's CI environment label Mar 12, 2026

drichards-87 self-assigned this Mar 12, 2026

drichards-87 approved these changes Mar 12, 2026

View reviewed changes

drichards-87 removed their assignment Mar 12, 2026

chatgpt-codex-connector Bot reviewed Apr 3, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

pront added the meta: awaiting author Pull requests that are awaiting their author. label Apr 3, 2026

flaviofcruz force-pushed the upstream-databricks-zerobus branch from 436d0da to 9dcb5d1 Compare April 10, 2026 17:10

github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Apr 10, 2026

chatgpt-codex-connector Bot reviewed Apr 10, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/config.rs Outdated

Comment thread lib/codecs/src/encoding/format/proto_batch.rs

flaviofcruz force-pushed the upstream-databricks-zerobus branch from 050409d to 36a74ef Compare April 13, 2026 22:47

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/service.rs Outdated

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

flaviofcruz added 11 commits May 4, 2026 14:00

Some cosmetic changes

7f7842e

Add generated file

8cc5073

Force direct connection when proxy is disabled to avoid the sdk to pi…

d1c09e0

…ck up the environment variables

Throw a config error is proxy url is invalid. Use direct connection i…

2bd587f

…f no URL is set up

Add test to confirm we always cap at 10MB for the batch size

c3dedf3

Bump the zerobus SDK

bd0d791

Bump the zerobus SDK again

431d218

Use the new zerobus sdk builder

d30a5c6

Address pront's comments

e4d0754

Try to fix Cargo.lock

91c957b

Ensure we always close the stream even when we get retryable errors

31bcdcb

flaviofcruz force-pushed the upstream-databricks-zerobus branch from f643aaf to 31bcdcb Compare May 4, 2026 14:00

flaviofcruz force-pushed the upstream-databricks-zerobus branch 2 times, most recently from c1e2cc3 to 31bcdcb Compare May 4, 2026 16:13

pront enabled auto-merge May 4, 2026 18:15

pront added this pull request to the merge queue May 4, 2026

pront removed this pull request from the merge queue due to a manual request May 4, 2026

pront reviewed May 4, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/unity_catalog_schema.rs Outdated

Comment thread src/sinks/databricks_zerobus/service.rs Outdated

Comment thread Cargo.lock Outdated

Always wait for acknowledgments from the SDK

ef462b7

pront reviewed May 4, 2026

View reviewed changes

Use 2.1 UC API

d564d98

flaviofcruz force-pushed the upstream-databricks-zerobus branch from dd346b6 to d564d98 Compare May 4, 2026 19:50

github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label May 4, 2026

Bump VRL which was downgraded due to the cargo proxy

ced054b

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread src/sinks/databricks_zerobus/service.rs

Do not downgrade rustls-webpki

141f680

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Conversation

flaviofcruz commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

pront commented Apr 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pront commented Apr 10, 2026

Uh oh!

pront commented Apr 13, 2026

Uh oh!

hsuanyi commented Apr 13, 2026

Uh oh!

pront commented Apr 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

flaviofcruz commented Apr 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

flaviofcruz commented May 4, 2026

Uh oh!

pront commented May 4, 2026

Uh oh!

flaviofcruz commented May 4, 2026

Uh oh!

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pront May 4, 2026

Choose a reason for hiding this comment

Uh oh!

flaviofcruz May 4, 2026

Choose a reason for hiding this comment

Uh oh!

pront May 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

flaviofcruz commented Mar 3, 2026 •

edited

Loading

github-actions Bot commented Apr 10, 2026 •

edited

Loading