Is there an existing issue for this?
Category of Bug / Issue
Reconcile bug
Current Behavior
When reconciling data from external sources via Lakehouse Federation (e.g., Teradata), the reconcile process fails with DELTA_EXCEED_CHAR_VARCHAR_LIMIT when writing intermediate DataFrames to Delta volumes.
Teradata CHAR(n) columns contain space-padded values. The intermediate Delta table/volume preserves the VARCHAR(n) type constraint from the source schema, causing the write to fail when padded values exceed the declared length.
Version: v0.12.2
Expected Behavior
Reconcile intermediate data should be written successfully regardless of source column padding. The intermediate tables/volumes are temporary and should use STRING (unbounded) instead of VARCHAR(n)/CHAR(n) to avoid length constraint violations.
Steps To Reproduce
- Set up Lakehouse Federation connection to Teradata
- Create a Foreign Catalog pointing to a Teradata database with
CHAR(n) columns
- Run Lakebridge Reconcile with
data_source="databricks" using the Foreign Catalog as source
- Source table has
CHAR(16) columns with space-padded values
- Reconcile fails when writing intermediate DataFrame to Volume
Relevant log output or Exception details
```
[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Value "4297690382031830 " exceeds char/varchar type length limitation.
Failed check: ((src_kyf IS NULL) OR (length(src_kyf) <= 16)).
File: recon_capture.py, line 92, in _write_df_to_volumes
df.write.format(self._format).save(path)
```
Operating System
macOS
Version
latest via Databricks CLI
Is there an existing issue for this?
Category of Bug / Issue
Reconcile bug
Current Behavior
When reconciling data from external sources via Lakehouse Federation (e.g., Teradata), the reconcile process fails with
DELTA_EXCEED_CHAR_VARCHAR_LIMITwhen writing intermediate DataFrames to Delta volumes.Teradata
CHAR(n)columns contain space-padded values. The intermediate Delta table/volume preserves theVARCHAR(n)type constraint from the source schema, causing the write to fail when padded values exceed the declared length.Version: v0.12.2
Expected Behavior
Reconcile intermediate data should be written successfully regardless of source column padding. The intermediate tables/volumes are temporary and should use
STRING(unbounded) instead ofVARCHAR(n)/CHAR(n)to avoid length constraint violations.Steps To Reproduce
CHAR(n)columnsdata_source="databricks"using the Foreign Catalog as sourceCHAR(16)columns with space-padded valuesRelevant log output or Exception details
```
[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Value "4297690382031830 " exceeds char/varchar type length limitation.
Failed check: ((src_kyf IS NULL) OR (length(src_kyf) <= 16)).
File: recon_capture.py, line 92, in _write_df_to_volumes
df.write.format(self._format).save(path)
```
Operating System
macOS
Version
latest via Databricks CLI