aboutsummaryrefslogtreecommitdiff
path: root/wordpiece_tokenizer.c (follow)
Commit message (Collapse)AuthorAgeFilesLines
* chore: move sources to src/ with proper names, drop qwen_ prefixOnur Gündüz2026-06-161-678/+0
|
* tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz2026-06-161-96/+67
|
* tokenizer: full-Unicode punctuation and control classificationOnur Gündüz2026-06-161-46/+31
|
* tokenizer: add WordPiece backend for BERT-family modelsOnur Gündüz2026-06-151-0/+722