aboutsummaryrefslogtreecommitdiff
path: root/tools/gen_wordpiece_unicode.py (follow)
Commit message (Expand)AuthorAgeFilesLines
* tools: point wordpiece unicode generator at src/tokenizer_unicode.hOnur Gündüz2026-06-161-2/+2
* tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz2026-06-161-6/+52
* tokenizer: full-Unicode punctuation and control classificationOnur Gündüz2026-06-161-0/+93