aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'README.md')
-rw-r--r--README.md41
1 files changed, 29 insertions, 12 deletions
diff --git a/README.md b/README.md
index e74eab9..bdab9bc 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
# XLM-R language detection on macOS CPU
A native C CLI for `papluca/xlm-roberta-base-language-detection`, using the
-liblangdet CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
+liblangdet prediction API, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python
runtime is required for inference.
@@ -37,8 +37,8 @@ Use `--` before text that begins with a hyphen. Diagnostics go to stderr.
## Inference
-`langdetect.c` loads and validates the four F32 classifier tensors and the
-`id2label` mapping. The encoder accepts F32 XLM-R weights with exact GeLU activation;
+`liblangdet/langdet.c` loads and validates the four F32 classifier tensors and the
+`id2label` mapping. `langdetect.c` handles only command-line input and JSON output. The encoder accepts F32 XLM-R weights with exact GeLU activation;
other model families and reduced-precision weights are not supported.
The encoder uses first-token pooling without L2 normalization.
The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer,
@@ -56,20 +56,35 @@ of accuracy on ambiguous or unsupported text.
## CPU library
-`make` builds both `langdetect` and the static archive `liblangdet.a`. The public
-encoder and tokenizer API is declared in `liblangdet/langdet.h`, with the
-`langdet_` prefix. Each encoder call returns the first token's final hidden state;
-the language-classification head and JSON output remain in `langdetect.c`.
+`make` builds both `langdetect` and the static archive `liblangdet.a`. Include
+`liblangdet/langdet.h` to use the complete prediction API from C or C++:
-Link a C application with:
+- `langdet_open(model_dir, n_threads, err, errlen)` loads all inference resources.
+ Use `0` threads for automatic selection, or an explicit count from 1 to 16.
+- `langdet_predict(detector, text, &result, err, errlen)` returns the best language,
+ its score, all 20 language scores, token count, and truncation flag.
+- `langdet_free(detector)` releases the resources. Passing NULL is safe.
+
+Opening returns NULL on failure. Prediction returns 0 on success or -1 on failure
+and clears the result on failure. The optional error buffer receives a message;
+model-file readers may also print diagnostics to stderr. Result label strings
+remain valid until the detector is freed. Numeric scores are stored in the result
+and do not change when another prediction runs.
+
+Use one API call at a time per process, including when multiple detectors are
+loaded. Internal CPU workers may cooperate on a prediction, but simultaneous calls
+from application threads are not supported.
+
+`examples/detect.c` is a complete application using only the public header:
```sh
-cc -Iliblangdet application.c liblangdet.a -framework Accelerate -lm -pthread -o application
+cc -Iliblangdet examples/detect.c liblangdet.a -framework Accelerate -lm -pthread -o build/detect-example
+./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.'
```
-The archive includes its JSON parser. No separate parser library is needed.
-Models own their memory-mapped weights; free their workspaces before freeing the
-model. A tokenizer or workspace must not be used concurrently by multiple calls.
+The archive includes its JSON parser. No separate parser library or private
+inference headers are needed. The encoder, tokenizer, classifier, and thread
+configuration are managed by the detector.
## Download
@@ -102,6 +117,8 @@ byte offset.
single-thread inference, and 512-token truncation.
- Invalid thread counts, a missing model directory, and NUL bytes in stdin
produce errors.
+- Standalone C and C++ applications link using only `langdet.h` and the static
+ archive. Prediction results, error returns, and detector lifecycle checks pass.
- The downloaded checkpoint files match Hugging Face's sizes and hashes.
- Numerical equivalence with Hugging Face Transformers has not been established.
These smoke checks are not an accuracy benchmark or exhaustive tokenizer test.