aboutsummaryrefslogtreecommitdiff
path: root/devtools/gen_wordpiece_unicode.py (unfollow)
Commit message (Collapse)AuthorFilesLines
2026-06-21devtools: add scripts for unicode generation, model inspection, embedding ↵Onur Gündüz1-3/+3
ranking, and more
2026-06-16tools: point wordpiece unicode generator at src/tokenizer_unicode.hOnur Gündüz1-2/+2
2026-06-16tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz1-6/+52
2026-06-16tokenizer: full-Unicode punctuation and control classificationOnur Gündüz1-0/+93