aboutsummaryrefslogtreecommitdiff
path: root/tools/gen_wordpiece_unicode.py (follow)
Commit message (Expand)AuthorAgeFilesLines
* devtools: add scripts for unicode generation, model inspection, embedding r...Onur Gündüz2026-06-211-139/+0
* tools: point wordpiece unicode generator at src/tokenizer_unicode.hOnur Gündüz2026-06-161-2/+2
* tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz2026-06-161-6/+52
* tokenizer: full-Unicode punctuation and control classificationOnur Gündüz2026-06-161-0/+93