Skip to content

fix(flatfiles): reject zero-column and over-width header drift in block decode#1101

Merged
userFRM merged 1 commit into
mainfrom
fix/flatfiles-header-drift
Jul 3, 2026
Merged

fix(flatfiles): reject zero-column and over-width header drift in block decode#1101
userFRM merged 1 commit into
mainfrom
fix/flatfiles-header-drift

Conversation

@userFRM

@userFRM userFRM commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Problem

decode_block in thetadatadx-rs/src/flatfiles/decode.rs passed two drifted/hostile-header shapes silently, unlike the mid-row truncation guard that already fails loud on the same surface:

  1. A header declaring fmt_count == 0 over a non-empty DATA block decoded to zero rows with no error. A drifted zero-column header vanished silently.
  2. A row carrying more fields than the header's column count was silently clipped (the FIT reader drops surplus fields past the buffer length). The FPSS sibling decode path (fpss/delta.rs) rejects the same width drift.

Solution

Both now return Error::decode_codec, matching the FPSS delta path's width guard and the existing truncation guard:

  • n_columns == 0 over a non-empty block returns a zero-column decode error (an empty block still returns Ok with no rows).
  • A row with n > n_columns returns an over-width decode error. Delta rows legitimately carry n < n_columns (trailing carry-forward), so only wider rows are rejected.

Regression tests cover both shapes.

Impact

A drifted flat-files header fails loud instead of producing silently wrong output. No behavior change on well-formed blocks; no version bump.

🤖 Generated with Claude Code

…ck decode

decode_block silently emitted zero rows for a header declaring zero columns over a non-empty DATA block, and silently clipped rows carrying more fields than the header column count. Both now return a typed decode error, matching the FPSS delta path width guard and the existing mid-row truncation guard so a drifted header fails loud.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens flat-file FIT block decoding so drifted/hostile headers fail fast instead of silently producing empty output or clipped rows, aligning behavior with the FPSS delta decoding path and the existing mid-row truncation guard.

Changes:

  • Rejects non-empty DATA blocks when n_columns == 0 with a typed codec decode error.
  • Rejects rows whose decoded field count exceeds the header column count (n > n_columns) to prevent silent truncation.
  • Adds regression tests for both drifted-header shapes and documents the behavior change in both changelogs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
thetadatadx-rs/src/flatfiles/decode.rs Adds explicit guards for zero-column headers over non-empty blocks and over-width rows, plus regression tests.
docs-site/docs/changelog.md Documents the new “fail loud” behavior for drifted headers in the docs changelog.
CHANGELOG.md Documents the new “fail loud” behavior for drifted headers in the root changelog.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@userFRM userFRM merged commit 2a7a7a9 into main Jul 3, 2026
53 checks passed
@userFRM userFRM deleted the fix/flatfiles-header-drift branch July 3, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants