| Commit message (Collapse) | Author | Files | Lines | ||
|---|---|---|---|---|---|
| 2026-06-21 | devtools: add scripts for unicode generation, model inspection, embedding ↵ | Onur Gündüz | 1 | -3/+3 | |
| ranking, and more | |||||
| 2026-06-16 | tools: point wordpiece unicode generator at src/tokenizer_unicode.h | Onur Gündüz | 1 | -2/+2 | |
| 2026-06-16 | tokenizer: Unicode case-fold and accent strip for multilingual WordPiece | Onur Gündüz | 1 | -6/+52 | |
| 2026-06-16 | tokenizer: full-Unicode punctuation and control classification | Onur Gündüz | 1 | -0/+93 | |
