diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 13:11:00 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 13:11:00 +0200 |
| commit | 37cbaef9ceb828c667da8f398027ba5415ee7475 (patch) | |
| tree | 6103e92a9030766e78a35df00f7a1155d0d0ecf5 /README.md | |
| parent | cf7d8169ec12cb3adf90d4ff78b3257e74985302 (diff) | |
| download | langdet-37cbaef9ceb828c667da8f398027ba5415ee7475.tar.gz langdet-37cbaef9ceb828c667da8f398027ba5415ee7475.zip | |
Rename CPU library API and sources to langdet
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 25 |
1 files changed, 21 insertions, 4 deletions
@@ -1,7 +1,7 @@ # XLM-R language detection on macOS CPU -A native C CLI for `papluca/xlm-roberta-base-language-detection`, using ffwd's -CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language +A native C CLI for `papluca/xlm-roberta-base-language-detection`, using the +liblangdet CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python runtime is required for inference. @@ -31,7 +31,7 @@ when there are no text arguments, produces one JSON line in input order: - `ms` measures tokenization and inference, excluding model loading and output. `--all` includes a `scores` object with probabilities for every language. -`-t N` selects 1 to 16 ffwd kernel threads; the default is the detected CPU count, +`-t N` selects 1 to 16 CPU kernel threads; the default is the detected CPU count, capped at 16. Accelerate manages its own BLAS threads. `-h` prints help. Use `--` before text that begins with a hyphen. Diagnostics go to stderr. @@ -44,7 +44,7 @@ The encoder uses first-token pooling without L2 normalization. The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer, and stable softmax. Dropout is disabled during inference. -The CPU encoder and tokenizer are in `libffwd/`; the JSON parser is in +The CPU encoder and tokenizer are in `liblangdet/`; the JSON parser is in `deps/yyjson/`. Their license files are included. The build uses only local source files and the macOS SDK, with no dependency on another repository. @@ -54,6 +54,23 @@ accepted, but their predictions are not useful. The model always chooses among its 20 labels; it has no unknown-language result, and scores are not a guarantee of accuracy on ambiguous or unsupported text. +## CPU library + +`make` builds both `langdetect` and the static archive `liblangdet.a`. The public +encoder and tokenizer API is declared in `liblangdet/langdet.h`, with the +`langdet_` prefix. Each encoder call returns the first token's final hidden state; +the language-classification head and JSON output remain in `langdetect.c`. + +Link a C application with: + +```sh +cc -Iliblangdet application.c liblangdet.a -framework Accelerate -lm -pthread -o application +``` + +The archive includes its JSON parser. No separate parser library is needed. +Models own their memory-mapped weights; free their workspaces before freeing the +model. A tokenizer or workspace must not be used concurrently by multiple calls. + ## Download Python 3 and internet access are the only requirements. No Hugging Face token or |
