DRC-3025: stamp catalog float/decimal types so decimals compare exactly#1466
Draft
danyelf wants to merge 1 commit into
Draft
DRC-3025: stamp catalog float/decimal types so decimals compare exactly#1466danyelf wants to merge 1 commit into
danyelf wants to merge 1 commit into
Conversation
…tly (DRC-3025)
Numeric values reach the frontend as untyped JSON strings (dbt's agate flattens
DECIMAL and DOUBLE alike to Decimal -> string), so the grid compared them with
bare string-inequality: genuine DOUBLE/aggregate float noise ("0.30000000000000004"
vs "0.3") read as a phantom "modified".
Backend: add an approximate-float column type (DataFrameColumnType.FLOAT, wire
label "float", distinct from exact NUMBER) and stamp each result column's true
comparison type from the catalog -- the offline artifact recce already loads for
lineage. DbtAdapter.catalog_column_types() resolves a model's column DB types;
DataFrameColumnType.from_db_type() maps DECIMAL/NUMERIC -> NUMBER (exact),
DOUBLE/FLOAT/REAL -> FLOAT (approximate), INT* -> INTEGER. value_diff (detail
rows) and query_diff (current_model) stamp model columns; profile stamps its
computed avg/median/proportions FLOAT by construction. Unmatched ad-hoc/expression
columns and catalog-absent cases keep the from_agate type -> exact (safe fallback).
A DECIMAL data column maps to NUMBER, never FLOAT, so a 1-cent change is never
float-collapsed (the cycle-4 guard, by construction). Values stay exact strings;
only the column-type label changes.
Frontend: isCellChanged(base, current, colType) dispatches on the stamped type --
FLOAT parses the numeric strings and applies a magnitude-relative epsilon (with
deep traversal for floats nested in JSON/ARRAY/STRUCT); everything else is exact
deep-equal. All five bare _.isEqual cell-change sites (determineRowStatus,
getCellClass, createCellClassBase/Current, detectModifications) route through it.
FLOAT renders like NUMBER.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Danyel Fisher <danyel@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Fixes DRC-3025 — numeric/decimal values arriving as untyped strings, so decimals were compared as text and flagged as changed on meaningless float noise.
** Still TODO ** could use careful review, and making sure this is ACTUALLY the underlying problem. (We spent a long time on numerical comparison before realizing that this was a string issue.)
Approach: the backend stamps each column's comparison type from
catalog.jsononto the diff payload; the frontend then routes all cell-compare sites through a singleisCellChanged(…, colType)so thatDOUBLE/float columns tolerate floating-point noise whileDECIMALstays exact.Changes
recce/adapter/dbt_adapter/__init__.py,recce/tasks/dataframe.py,profile.py,query.py,valuediff.py; newbase.tstype field.js/packages/ui/src/utils/dataGrid/(gridUtils.ts,rowBuilders.ts,toDiffColumn.tsx) and the dataGrid header/render-cell components.tests/tasks/test_drc3025_serialization.py(backend), plus frontendgridUtils/rowBuildersunit tests.Status / notes
recce-fixesworkflow (validation passed; aggressive-self-review gate not yet run).explore/drc-3025-float-precisionbranch. Those are dead ends; this branch is the live fix.pnpm run buildbefore backendtest_server.pySPA-route tests pass in a fresh checkout.🤖 Generated with Claude Code