Skip to content

feat(query): support Latin-1 binary input format - #20238

Draft
dbsid wants to merge 1 commit into
databendlabs:mainfrom
dbsid:codex/ticdc-canal-json-latin1
Draft

feat(query): support Latin-1 binary input format#20238
dbsid wants to merge 1 commit into
databendlabs:mainfrom
dbsid:codex/ticdc-canal-json-latin1

Conversation

@dbsid

@dbsid dbsid commented Jul 31, 2026

Copy link
Copy Markdown

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

TiCDC Canal-JSON intentionally represents binary column bytes as ISO-8859-1 code points in a JSON string. Converting the decoded string as UTF-8 changes bytes above 0x7f; for example, 0x80 becomes c280 and 0xff becomes c3bf.

This PR adds latin1, latin-1, and iso-8859-1 formats to TO_BINARY and TRY_TO_BINARY so consumers can restore the original bytes without changing the Canal-JSON pipeline.

SELECT TO_HEX(
  TO_BINARY(
    AS_STRING(GET(PARSE_JSON('{"v":"\u0000A\u007f\u0080\u00ff"}'), 'v')),
    'latin1'
  )
);
-- 00417f80ff

TiCDC documents this encoding for binary columns and implements it with an ISO-8859-1 decoder:

Implementation

  • Encode each Unicode code point in U+0000..U+00FF as its corresponding single byte.
  • Make TO_BINARY reject code points above U+00FF instead of truncating them.
  • Make TRY_TO_BINARY return NULL for unrepresentable input.
  • Preserve the existing nullable and vectorized function behavior.

The change is backward compatible: existing hex, base64, and utf-8 formats are unchanged.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Validation performed on origin/main at b6a88c6930:

  • cargo fmt --check -- src/query/functions/src/scalars/binary.rs src/query/functions/tests/it/scalars/binary.rs
  • cargo test -p databend-common-functions --test it test_binary -- --nocapture
  • cargo clippy -p databend-common-functions --lib --tests -- -D warnings
  • cargo build --bin databend-query --bin databend-sqllogictests
  • databend-sqllogictests for 03_0041_insert_into_binary.test with both MySQL and HTTP handlers (20 assertions per handler)

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding agent researched the TiCDC/Canal encoding contract, implemented the scalar function and regression tests, and drafted this PR description.
  • Responsible human: @dbsid
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

Decode Unicode code points U+0000 through U+00FF into their original single-byte values when TO_BINARY uses the latin1 format.

Reject out-of-range code points in TO_BINARY and return NULL from TRY_TO_BINARY, with unit and SQL regression coverage for Canal-JSON input.
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jul 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The ## AI assistance section is incomplete. Review is blocked until it is filled in.
@dbsid please update it 🙏.

  • the checkbox "The responsible human has read every line of this diff and can explain each change" is not checked exactly as written

Required format (see AI_POLICY.md):

## AI assistance

- AI usage: An AI coding agent drafted the patch; I reviewed and added logic tests (or "None")
- Responsible human: @actual-github-id
- [x] The responsible human has read every line of this diff and can explain each change

The responsible human is the author-side owner — the person who has read the diff,
can explain each change, and will answer questions during review.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis (Retry 1)

Workflow: 30631694889

📊 Summary

  • Total Jobs: 90
  • Failed Jobs: 1
  • Retryable: 0
  • Code Issues: 1

NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • linux / test_private_tasks: Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant