Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **Projected historical frames keep the trading `date`.** A response whose wire sends one `Timestamp` header split into a time-of-day field and `date` (every EOD and trade/quote/greeks history endpoint) no longer drops `date` from the Arrow / Polars frame, so rows spanning multiple days are distinguishable instead of collapsing to a near-constant time-of-day.
- **Projected snapshot frames keep the per-row `symbol`.** A multi-symbol snapshot response no longer labels every row with the first row's symbol; the broadcast symbol column is emitted only when the response's `symbol` is provably constant across all rows.
- **Flat-files block decode fails loud on two drifted-header shapes.** A header declaring zero columns over a non-empty DATA block, and a row carrying more fields than the header's column count, now return a typed decode error instead of silently emitting zero rows or clipping the surplus field. This matches the FPSS delta path's width guard and the existing mid-row truncation guard.

## [13.0.0-rc.13] - 2026-07-02

Expand Down
1 change: 1 addition & 0 deletions docs-site/docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **Projected historical frames keep the trading `date`.** A response whose wire sends one `Timestamp` header split into a time-of-day field and `date` (every EOD and trade/quote/greeks history endpoint) no longer drops `date` from the Arrow / Polars frame, so rows spanning multiple days are distinguishable instead of collapsing to a near-constant time-of-day.
- **Projected snapshot frames keep the per-row `symbol`.** A multi-symbol snapshot response no longer labels every row with the first row's symbol; the broadcast symbol column is emitted only when the response's `symbol` is provably constant across all rows.
- **Flat-files block decode fails loud on two drifted-header shapes.** A header declaring zero columns over a non-empty DATA block, and a row carrying more fields than the header's column count, now return a typed decode error instead of silently emitting zero rows or clipping the surplus field. This matches the FPSS delta path's width guard and the existing mid-row truncation guard.

## [13.0.0-rc.13] - 2026-07-02

Expand Down
46 changes: 45 additions & 1 deletion thetadatadx-rs/src/flatfiles/decode.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,17 @@ pub(crate) fn decode_block(
out: &mut Vec<Vec<i32>>,
) -> Result<(), Error> {
out.clear();
if block.is_empty() || n_columns == 0 {
if block.is_empty() {
return Ok(());
}
if n_columns == 0 {
// A zero-column schema over non-empty DATA is a drifted header: every
// row would decode to nothing while the block still carries bytes.
// Fail loud rather than emit zero rows, matching the truncation guard.
return Err(Error::decode_codec(
"flatfiles: zero-column header with non-empty data",
));
}

let mut reader = FitReader::new(block);
let mut prev: Vec<i32> = vec![0; n_columns];
Expand All @@ -55,6 +63,15 @@ pub(crate) fn decode_block(
"flatfiles: FIT block truncated mid-row",
));
}
if n > n_columns {
// Row carries more fields than the blob's column schema, so the
// FIT reader already dropped the surplus into a silent clip. A
// wider row is a drifted header; reject it rather than emit a
// truncated row, matching the FPSS delta path's width guard.
return Err(Error::decode_codec(
"flatfiles: FIT row wider than header column count",
));
}
if reader.is_date {
// DATE marker row — no user-visible data. Vendor's writer
// skips DATE rows before they reach `toCSV2`, so we do too.
Expand Down Expand Up @@ -165,6 +182,33 @@ mod tests {
);
}

#[test]
fn zero_column_header_with_data_is_rejected() {
// A header claiming zero columns over a non-empty DATA block is a
// drifted schema: silently decoding to zero rows would hide the drift.
let buf = vec![pack(1, END)];
let mut out = Vec::new();
let err = decode_block(&buf, 0, &mut out).unwrap_err();
assert!(
err.to_string().contains("zero-column"),
"expected a zero-column decode error, got: {err}"
);
}

#[test]
fn over_width_row_is_rejected() {
// "12,34<END>" carries 2 fields against a 1-column schema: the extra
// field would be silently clipped. The FPSS delta path rejects the
// same width drift; decode_block must too.
let buf = vec![pack(1, 2), pack(FIELD_SEP, 3), pack(4, END)];
let mut out = Vec::new();
let err = decode_block(&buf, 1, &mut out).unwrap_err();
assert!(
err.to_string().contains("wider"),
"expected an over-width decode error, got: {err}"
);
}

#[test]
fn date_marker_is_skipped() {
// DATE marker row, then absolute "7"
Expand Down