Skip to content

Commit 7df9c70

Browse files
derek73claude
andcommitted
Document the interpunct transcription reading (#298)
docs/release_log.rst gains the 2.1.0 Behavior Changes bullet for U+00B7: the 间隔号 is a token separator between characters of a classified script, the name it divides keeps its source order and is never segmented, the Catalan punt volat is untouched, the nakaguro is deliberately not a transcription marker, and rendering stays space-joined with a custom string_format reinstating the dot. Placed directly after the katakana-dot bullet, whose closing sentence hands off to #298. docs/usage.rst gains a paragraph in the East Asian section's Chinese flow — the 间隔号 convention as background, then the separator behavior and the never-segmented reading, with the nakaguro distinguished — and the Boundaries limits paragraph states the two limitations: the spaced form (威廉 莎士比亚) and the nakaguro-typed transcription (威廉・莎士比亚) both keep family-first, because only the Chinese dot marks a transcription. No new doctest blocks; the case rows carry the examples. docs/migrate.rst completes the "one exception" sentence: two dots are now decided in tokenization rather than by policy — the katakana dot unconditionally, the interpunct only between classified-script characters. The nakaguro doctest and its explanation are unchanged. docs/concepts.rst extends the first/last decider list: the interpunct walks the script_orders entry back, marking a transcription that keeps its source order. docs/customize.rst needs no edit: its tokenization-not-policy sentence is scoped to the nakaguro and stays true, and under script_orders={} everything is positional already, so the dot adds nothing switchable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 96832ce commit 7df9c70

4 files changed

Lines changed: 35 additions & 7 deletions

File tree

docs/concepts.rst

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,9 @@ layer, which assigns purely by where a word sits: the first unclaimed
5858
word is the given name, the last is the family name, and anything
5959
between them is the middle name. ``name_order``, an explicit comma,
6060
and — for a name written wholly in one East Asian script —
61-
``script_orders`` change what "first" and "last" mean here; nothing
61+
``script_orders`` change what "first" and "last" mean here; the
62+
Chinese interpunct ``·`` dividing such a name walks the last of those
63+
back, marking a transcription that keeps its source order; nothing
6264
else does.
6365

6466
This is the whole parser in two sentences, and it explains its

docs/migrate.rst

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -429,7 +429,10 @@ now.
429429
surface is frozen for 2.x — so the way out is the 2.0 API:
430430
``Parser(policy=Policy(script_orders={}, segment_scripts=()))``
431431
restores 1.4's reading of every shape above that turns on order or
432-
splitting. The middle dot is the one exception: it is decided in
433-
tokenization rather than by policy, so a name written with one still
434-
divides at the dot, and still renders with a space, whatever those two
435-
fields are set to.
432+
splitting. The middle dots are the exception: both the katakana dot
433+
and the Chinese interpunct ``·`` (U+00B7, dividing a transcription
434+
like ``威廉·莎士比亚``, and a separator only between characters of a
435+
classified script where the katakana dot is unconditional) are decided
436+
in tokenization rather than by policy, so a name written with either
437+
still divides at the dot, and still renders with a space, whatever
438+
those two fields are set to.

docs/release_log.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Release Log
2424
- Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272)
2525
- Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
2626
- Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
27+
- Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
2728
- Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
2829

2930
* 2.0.0 - July 27, 2026

docs/usage.rst

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,21 @@ language is up to you. When you know the data is Chinese, apply the
188188
>>> parser_for(locales.ZH).parse("毛泽东").family
189189
'毛'
190190

191+
Chinese also has a transcription convention, and it is written in the
192+
punctuation: a foreign name transcribed into Han characters keeps its
193+
source order and divides its parts with the 间隔号, the interpunct
194+
``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name
195+
first. The dot itself is the marker, so nameparser reads U+00B7 as a
196+
token separator when it sits between characters of a classified
197+
script, and a name it divides keeps the order it was written in and is
198+
never segmented — the role pure katakana plays for Japanese
199+
transcriptions, played here by the divider instead of the script.
200+
Because only a classified script on both sides makes it a divider, the
201+
same codepoint interior to a Latin-script name — the Catalan punt
202+
volat in ``Gal·la`` — is untouched. The Japanese middle dot ・
203+
(covered next) is a different mark carrying a different convention,
204+
and keeps its own reading.
205+
191206
Japanese
192207
~~~~~~~~~
193208

@@ -266,8 +281,15 @@ positional rules — order genuinely varies in romanized data, so
266281
nothing script-based applies. A name written wholly in katakana stays
267282
positional for the reason given above, pack or no pack: it is
268283
predominantly a transcription, and a transcription is already in the
269-
order it should be read in. And a comma disables the script behaviors
270-
entirely, on the reasoning ``name_order`` already follows: whoever
284+
order it should be read in. A Han transcription written with a space
285+
instead of the 间隔号 (威廉 莎士比亚) carries nothing to distinguish it
286+
from a native two-token name and keeps the family-first reading — the
287+
dot is the marker, and without it there is no signal. The same holds
288+
for a transcription typed with the Japanese middle dot (威廉・莎士比亚):
289+
each dot carries its own script's convention, the nakaguro's is
290+
Japanese roster formatting rather than transcription, so only the
291+
Chinese dot rescues the source order. And a comma disables the script
292+
behaviors entirely, on the reasoning ``name_order`` already follows: whoever
271293
wrote the comma has already said where the family name ends.
272294

273295
A division the parser had to choose is reported rather than hidden.

0 commit comments

Comments
 (0)