Commit 76c6270
committed
Handle unicode whitespace characters in lexer
Add isClickHouseWhitespace() to recognize characters that ClickHouse
treats as whitespace but Go's unicode.IsSpace() does not: U+FEFF BOM,
U+180E Mongolian Vowel Separator, U+200B Zero Width Space, U+200C Zero
Width Non-Joiner, U+200D Zero Width Joiner, and U+2060 Word Joiner.
This fixes the 01280_unicode_whitespaces_lexer test case stmt3.1 parent d875f47 commit 76c6270
2 files changed
Lines changed: 24 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
77 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
78 | 79 | | |
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
82 | 103 | | |
83 | 104 | | |
84 | 105 | | |
| |||
Lines changed: 1 addition & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
5 | | - | |
| 1 | + | |
0 commit comments