I/O: Migrate Kafka->CrateDB recipe from ingestr to omniload

[amotl]

About

ingestr v1 migrated to Go, but omniload inherited the v0 codebase based on dlt and Python. It also integrated the long-awaited pending patch to improve the Kafka decoder, so everything we need to transfer data from Kafka to CrateDB is now nicely wrapped into a package going forward.

Review

This patch mostly includes a refactoring and renaming operation s/ingestr/omniload/ for one particular data i/o nozzle exercise that transfers data from a Kafka topic into a CrateDB table, which needs some more advanced transformations than the standard data path would provide.

Other than the layout refactoring by extracting the kafka-to-cratedb exercise into the application/omniload folder, no other changes were made to the database workload or validation code: kafka-compose.yml and kafka-demo.xsh were moved to the new folder almost unmodified.

References

• Improve Kafka adapter to provide features closer to Kafka Connect bruin-data/ingestr#306
• Kafka: Expand default_msg_processor into a miniature decoding unit panodata/omniload#51

/cc @florinutz, @matriv

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 35730e62-a398-4d7d-9ea5-52b55b8d7925

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[amotl]

This custom repository reference has been dissolved now. 👍
All what is needed here is packaged into omniload v0.2 today. 🌻

[florinutz]

so basically we're moving away from ingestr, towards omniload, right?

[amotl]

Hi Florin,

in CrateDB Toolkit, this other patch moves away from ingestr, mainly due to licensing concerns.

• I/O: Migrate from ingestr to omniload cratedb-toolkit#842

Here, we are just migrating one exercise from ingestr to omniload, but both ETL applications will continue to have their representation within cratedb-examples. The reason to migrate only one example is because the Kafka exercise depended on a feature patch in crate-workbench/ingestr.git@kafka-decoder that never saw the light of the day within ingestr, but is now shipping with omniload.

All what's needed is packaged into omniload today. 🌻

With kind regards,
Andreas.