aboutsummaryrefslogtreecommitdiff
path: root/libffwd/wordpiece.c (unfollow)
Commit message (Expand)AuthorFilesLines
2026-06-27chore: clean up comment formatting and wordingOnur Gündüz1-38/+43
2026-06-21all: split Makefile and reorg library/toolsOnur Gündüz1-42/+38
2026-06-16chore: move sources to src/ with proper names, drop qwen_ prefixOnur Gündüz1-2/+2
2026-06-16tokenizer: Unicode case-fold and accent strip for multilingual WordPieceOnur Gündüz1-96/+67
2026-06-16tokenizer: full-Unicode punctuation and control classificationOnur Gündüz1-46/+31
2026-06-15tokenizer: add WordPiece backend for BERT-family modelsOnur Gündüz1-0/+722