aboutsummaryrefslogtreecommitdiff
path: root/tools/gen_wordpiece_unicode.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* devtools: add scripts for unicode generation, model inspection, embedding ↵Onur Gündüz2026-06-211-139/+0
| | | | ranking, and more
* tools: point wordpiece unicode generator at src/tokenizer_unicode.hOnur Gündüz2026-06-161-2/+2
|
* tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz2026-06-161-6/+52
|
* tokenizer: full-Unicode punctuation and control classificationOnur Gündüz2026-06-161-0/+93