Feature Request: Are recently ingested records query-visible before ArrowBuffer flush?...

[ccwxl]

Hi Arc team,

I’m trying to understand the query visibility semantics around the in-memory ArrowBuffer.

From reading the code, my current understanding is:

1. Write requests are first buffered in ArrowBuffer.
2. Data is flushed to Parquet when max_buffer_size or max_buffer_age_ms is reached.
3. Query paths rewrite logical tables to read_parquet(...) and execute them through DuckDB.
4. QueryHandler does not appear to hold a reference to ArrowBuffer, and normal query paths do not seem to call FlushAll() before executing the query.

So my question is:

Are records that have been accepted by the write API but have not yet been flushed to Parquet expected to be invisible to normal queries?

In other words, is Arc’s intended consistency model “eventual query visibility after flush”, rather than “read-your-write” visibility?

I also noticed that DuckDB can query Apache Arrow objects/datasets directly in some bindings, for example:

https://duckdb.org/docs/current/guides/python/sql_on_arrow#apache-arrow-datasets

Has Arc considered querying both persisted Parquet files and the current in-memory Arrow buffer, e.g. by combining:

SELECT * FROM read_parquet(...)
UNION ALL
SELECT * FROM <hot_arrow_buffer>
Or is the current design intentionally optimized around batching writes to Parquet and accepting a short freshness delay?

Thanks!

[xe-nvdk]

Good morning! Thanks for the thoughtful write-up — your reading of the code is correct.

Yes, Arc's consistency model is "eventual visibility after flush", not read-your-write. Records sit in ArrowBuffer until either max_buffer_size or max_buffer_age_ms is hit, then land in Parquet and become queryable through DuckDB's read_parquet(...). The query path deliberately does not hold a reference to the buffer and does not call FlushAll() before executing.

We actually prototyped exactly the UNION ALL design you're describing — read_parquet(...) UNION ALL arc_buffer_scan(database, measurement) via a DuckDB table UDF — on branch feat/query-buffer-scan, and decided not to ship it. Full write-up here: #249

The short version:

• The shipped default for max_buffer_age_ms is already 100ms (see arc.toml [ingest]), so out of the box maximum staleness is ≤ 100ms — well under any dashboard refresh interval. You don't need to tune anything to get this.
• Adding the buffer scan puts a non-zero cost on every query: a nil-check, path parsing, a regex validation, an RLock/map lookup on the buffer, and — when the buffer has data — a UNION ALL that perturbs DuckDB's query plan even when the buffered rows don't contribute to the result.
• Compaction already handles the small-file churn from a 100ms flush window cleanly, so the "lots of small files" concern doesn't really materialize.
• And there are real correctness traps in the implementation (SQL injection at the UDF call site, RBAC isolation across database/measurement keys) that we'd have to keep guarding forever.

So the design is intentionally optimized around batched Parquet writes plus a short, fixed freshness delay — rather than paying query-path overhead to close a sub-100ms gap.

That said — what's your use case? Specifically, what kind of query are you running where ≤100ms freshness isn't enough? Read-your-write for a UI flow, an alerting/streaming pipeline, agent memory recall, something else? If there's a real workload that needs sub-flush visibility, we'd love to hear it — it might justify revisiting #249 with an opt-in design (e.g. ?include_buffer=true) so the cost only lands on queries that actually need it.

Thanks again for digging into the code!