aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-09-29 13:11:00 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-09-29 13:11:00 +0200
commit37cbaef9ceb828c667da8f398027ba5415ee7475 (patch)
tree6103e92a9030766e78a35df00f7a1155d0d0ecf5 /README.md
parentcf7d8169ec12cb3adf90d4ff78b3257e74985302 (diff)
downloadlangdet-37cbaef9ceb828c667da8f398027ba5415ee7475.tar.gz
langdet-37cbaef9ceb828c667da8f398027ba5415ee7475.zip
Rename CPU library API and sources to langdet
Diffstat (limited to 'README.md')
-rw-r--r--README.md25
1 files changed, 21 insertions, 4 deletions
diff --git a/README.md b/README.md
index a7f5aae..e74eab9 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
# XLM-R language detection on macOS CPU
-A native C CLI for `papluca/xlm-roberta-base-language-detection`, using ffwd's
-CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
+A native C CLI for `papluca/xlm-roberta-base-language-detection`, using the
+liblangdet CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python
runtime is required for inference.
@@ -31,7 +31,7 @@ when there are no text arguments, produces one JSON line in input order:
- `ms` measures tokenization and inference, excluding model loading and output.
`--all` includes a `scores` object with probabilities for every language.
-`-t N` selects 1 to 16 ffwd kernel threads; the default is the detected CPU count,
+`-t N` selects 1 to 16 CPU kernel threads; the default is the detected CPU count,
capped at 16. Accelerate manages its own BLAS threads. `-h` prints help.
Use `--` before text that begins with a hyphen. Diagnostics go to stderr.
@@ -44,7 +44,7 @@ The encoder uses first-token pooling without L2 normalization.
The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer,
and stable softmax. Dropout is disabled during inference.
-The CPU encoder and tokenizer are in `libffwd/`; the JSON parser is in
+The CPU encoder and tokenizer are in `liblangdet/`; the JSON parser is in
`deps/yyjson/`. Their license files are included. The build uses only local source
files and the macOS SDK, with no dependency on another repository.
@@ -54,6 +54,23 @@ accepted, but their predictions are not useful. The model always chooses among
its 20 labels; it has no unknown-language result, and scores are not a guarantee
of accuracy on ambiguous or unsupported text.
+## CPU library
+
+`make` builds both `langdetect` and the static archive `liblangdet.a`. The public
+encoder and tokenizer API is declared in `liblangdet/langdet.h`, with the
+`langdet_` prefix. Each encoder call returns the first token's final hidden state;
+the language-classification head and JSON output remain in `langdetect.c`.
+
+Link a C application with:
+
+```sh
+cc -Iliblangdet application.c liblangdet.a -framework Accelerate -lm -pthread -o application
+```
+
+The archive includes its JSON parser. No separate parser library is needed.
+Models own their memory-mapped weights; free their workspaces before freeing the
+model. A tokenizer or workspace must not be used concurrently by multiple calls.
+
## Download
Python 3 and internet access are the only requirements. No Hugging Face token or