diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-06-27 00:58:05 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-06-27 01:03:41 +0200 |
| commit | af55803d0dff9eb692c760c8d486e736c945061d (patch) | |
| tree | 0875a1112bf98304e445e0544c4309670d6f4769 /tests/tokenizer_dump.c | |
| parent | 0f861b7f3098750ddee4f6570f3be228d1da5320 (diff) | |
| download | ffwd-af55803d0dff9eb692c760c8d486e736c945061d.tar.gz ffwd-af55803d0dff9eb692c760c8d486e736c945061d.zip | |
chore: clean up comment formatting and wording
Diffstat (limited to 'tests/tokenizer_dump.c')
| -rw-r--r-- | tests/tokenizer_dump.c | 17 |
1 files changed, 11 insertions, 6 deletions
diff --git a/tests/tokenizer_dump.c b/tests/tokenizer_dump.c index f6e26aa..4e8e8ec 100644 --- a/tests/tokenizer_dump.c +++ b/tests/tokenizer_dump.c @@ -1,9 +1,14 @@ -/* Parity driver for the BPE tokenizer. Loads vocab.json (merges.txt is read - * from the same directory), encodes each argument through the three encode - * entry points (and cross-checks they agree), then prints "<count> <id> <id> - * ..." per line. check_tokenizer_parity.py builds this with - * `make parity-tokenizer-driver` and compares the ids against the stored - * reference vectors in tests/test-vectors/bpe/. Links libtok. */ +/* BPE tokenizer parity driver. + * Loads vocab.json; merges.txt is read from the same directory. + * Encodes each argument through the three encode entry points and checks that + * they agree. + * + * Prints one line per argument: "<count> <id> <id> ...". + * + * check_tokenizer_parity.py builds this with `make parity-tokenizer-driver` and + * compares ids against reference vectors in tests/test-vectors/bpe/. + */ + #include "bpe.h" #include <stdio.h> |
