Skip to content

[BUG]: Reconcile fails with DELTA_EXCEED_CHAR_VARCHAR_LIMIT when source has CHAR/VARCHAR padded values #2389

@moomindani

Description

@moomindani

Is there an existing issue for this?

  • I have searched the existing issues

Category of Bug / Issue

Reconcile bug

Current Behavior

When reconciling data from external sources via Lakehouse Federation (e.g., Teradata), the reconcile process fails with DELTA_EXCEED_CHAR_VARCHAR_LIMIT when writing intermediate DataFrames to Delta volumes.

Teradata CHAR(n) columns contain space-padded values. The intermediate Delta table/volume preserves the VARCHAR(n) type constraint from the source schema, causing the write to fail when padded values exceed the declared length.

Version: v0.12.2

Expected Behavior

Reconcile intermediate data should be written successfully regardless of source column padding. The intermediate tables/volumes are temporary and should use STRING (unbounded) instead of VARCHAR(n)/CHAR(n) to avoid length constraint violations.

Steps To Reproduce

  1. Set up Lakehouse Federation connection to Teradata
  2. Create a Foreign Catalog pointing to a Teradata database with CHAR(n) columns
  3. Run Lakebridge Reconcile with data_source="databricks" using the Foreign Catalog as source
  4. Source table has CHAR(16) columns with space-padded values
  5. Reconcile fails when writing intermediate DataFrame to Volume

Relevant log output or Exception details

```
[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Value "4297690382031830 " exceeds char/varchar type length limitation.
Failed check: ((src_kyf IS NULL) OR (length(src_kyf) <= 16)).

File: recon_capture.py, line 92, in _write_df_to_volumes
df.write.format(self._format).save(path)
```

Operating System

macOS

Version

latest via Databricks CLI

Metadata

Metadata

Labels

feat/reconmaking sure that remorphed query produces the same results as original

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions