feat(query): support Latin-1 binary input format - #20238
Draft
dbsid wants to merge 1 commit into
Draft
Conversation
Decode Unicode code points U+0000 through U+00FF into their original single-byte values when TO_BINARY uses the latin1 format. Reject out-of-range code points in TO_BINARY and return NULL from TRY_TO_BINARY, with unit and SQL regression coverage for Canal-JSON input.
Contributor
|
The
Required format (see AI_POLICY.md): The responsible human is the author-side owner — the person who has read the diff, |
Contributor
🤖 CI Job Analysis (Retry 1)
📊 Summary
❌ NO RETRY NEEDEDAll failures appear to be code/test issues requiring manual fixes. 🔍 Job Details
🤖 AboutAutomated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
TiCDC Canal-JSON intentionally represents binary column bytes as ISO-8859-1 code points in a JSON string. Converting the decoded string as UTF-8 changes bytes above
0x7f; for example,0x80becomesc280and0xffbecomesc3bf.This PR adds
latin1,latin-1, andiso-8859-1formats toTO_BINARYandTRY_TO_BINARYso consumers can restore the original bytes without changing the Canal-JSON pipeline.TiCDC documents this encoding for binary columns and implements it with an ISO-8859-1 decoder:
Implementation
U+0000..U+00FFas its corresponding single byte.TO_BINARYreject code points aboveU+00FFinstead of truncating them.TRY_TO_BINARYreturnNULLfor unrepresentable input.The change is backward compatible: existing
hex,base64, andutf-8formats are unchanged.Tests
Validation performed on
origin/mainatb6a88c6930:cargo fmt --check -- src/query/functions/src/scalars/binary.rs src/query/functions/tests/it/scalars/binary.rscargo test -p databend-common-functions --test it test_binary -- --nocapturecargo clippy -p databend-common-functions --lib --tests -- -D warningscargo build --bin databend-query --bin databend-sqllogictestsdatabend-sqllogictestsfor03_0041_insert_into_binary.testwith both MySQL and HTTP handlers (20 assertions per handler)Type of change
AI assistance
This change is