Skip to content

memory_model.md

Latest commit

 

History

History
36 lines (24 loc) · 3.43 KB

memory_model.md

File metadata and controls

36 lines (24 loc) · 3.43 KB
Raw

Memory Model

Arrow data is represented as arrays backed by buffers, usually shared through Arc. A RecordBatch owns a schema and column arrays; cloning a batch is generally shallow because arrays share underlying buffers.

The server is zero-copy aware, not zero-copy absolute. Arrow Flight decode may allocate. Validation reads Arrow schema and array metadata without rewriting columns. Backends may need copies to create tensor layouts, align memory, convert dtypes, transfer CPU data to GPU, or materialize output arrays.

Implemented low-copy zones:

  • Passing RecordBatch through validation and scheduler.
  • Model manifests and compiled execution plans. These avoid repeated schema/field construction, output-schema construction, and normal-case input column name lookups on every request.
  • ONNX input fast path for non-null float32 / int64 primitive arrays and fixed_size_list columns. These are passed to ort as borrowed TensorRef views over Arrow value buffers when dtype, shape, and contiguous CPU layout match the model input.
  • ONNX output binding for known float32 output shapes. The backend preallocates ORT output tensors and runs via IoBinding, avoiding ONNX Runtime-managed output allocation for these outputs.
  • ONNX inference is executed in spawn_blocking, which prevents synchronous ORT work from blocking Tokio worker threads. This is a scheduling boundary, not a memory-copy optimization.
  • Stateful Flight stream decode/encode. A DoExchange stream can carry one schema followed by multiple RecordBatch values, so schema IPC framing is not repeated for every batch in that stream.

Potential low-copy zones:

  • Returning backend outputs already stored in Arrow-compatible buffers.

Likely copy zones:

  • FlightData to RecordBatch decoding.
  • Arrow arrays to backend tensors when the array has nulls, needs null substitution, has an unsupported layout, needs dtype conversion, or is not a supported primitive/fixed-size-list tensor representation.
  • CPU to GPU transfer.
  • Backend-specific temporary allocations.
  • Output tensor to Arrow array construction. Current ONNX output binding reduces ORT-side allocation, but Arrow output arrays are still materialized from ORT output data.
  • ONNX timeout cancellation bookkeeping. Each ORT call with scheduler timeout metadata creates RunOptions and a small timer thread that can set ORT's terminate flag.
  • Splitting or coalescing batches once dynamic batching is implemented. Coalescing multiple Arrow batches into one backend tensor will improve backend throughput, but may introduce batch concatenation copies unless the runtime can consume chunked Arrow buffers directly.

Full zero-copy is not guaranteed because ML runtimes have layout, dtype, device, and ownership constraints that differ from Arrow. Copy-count instrumentation is planned so operators can see where allocations happen per backend.

Current Runtime Implication

For larger dense batches, Arrow Flight can be substantially cheaper than protobuf tensor marshalling in the benchmark harness because the client and server already exchange Arrow IPC payloads. The current server benefits most when the client sends a single larger RecordBatch, for example batch_size = 128.

Sending batches_per_stream = 8 with batch_size = 16 reduces Flight stream setup overhead, but it still performs eight ONNX Runtime calls. Planned stream coalescing will merge compatible batches before backend execution so the runtime sees one larger tensor batch.