Volga and OpenMLDB

[anovv]

Hi OpenMLDB community!

I’ve been following the development of OpenMLDB for some time and have done extensive research on your papers and codebase and would like to thank the team for the great work you've done in this field.

I'd like to share the project in the same space I've been working on:

Volga — a data processing engine for real-time AI/ML systems written in Rust.

In short, it is a Flink/Spark/Arroyo alternative tailored for AI/ML pipelines and similar to systems like OpenMLDB and Chronon.

Key features:

• SQL-based Pipelines: Powered by Apache DataFusion.
• Unified Streaming + Batch Processing: Consistent watermark-based execution for real-time and backfills with Apache Arrow.
• Remote State Storage : LSM-Tree-on-S3 via SlateDB for compute-storage separation and cheap checkpoints.
• Request Mode (inspired by OpenMLDB): Point-in-time correct queryable state to serve data directly within the dataflow (no external KV/serving workers). Write/Read compute separation.
• ML-Specific Aggs (inspired by OpenMLDB): Built-in topk, _cate, and _where functions
• Long-Window Optimizations via Tiling
• Standalone: No complex stitching (Flink+Spark+Redis+custom services) required.

I have recently finished writing a blog post about Volga's re-write in Rust, diving deep into architecture, including a detailed comparison to OpenMLDB and some design decisions influenced by your approach to SQL syntax and feature-serving:

Technical Deep Dive Post

I would love to hear general feedback on the project (especially regarding the trade-offs between Volga’s streaming execution and OpenMLDB db-style approach, in-memory storage vs S3-backed remote state, replication vs cloud durability, latency, ease of use, etc.) and general ideas on possible future direction for the project and potential improvements.

Thank you!