support DefineTokens with AloneTokenOption and IgnoreCaseTokenOption#38

einsitang wants to merge 4 commits into

bzick:masterbzick/tokenizer:masterfrom

einsitang:codespace-improved-waffle-r74qv7vjx7c5v54einsitang/tokenizer:codespace-improved-waffle-r74qv7vjx7c5v54Copy head branch name to clipboard