aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-09-29 15:41:41 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-09-29 16:06:03 +0200
commit939b9a8deae07c2f06aa16c08f3ddad70f97b261 (patch)
tree00245f8e747349ddcca171be878e5ac19e8110a4 /README.md
parent91c6aa07597462705a7f30ee5b97e2c7943b5616 (diff)
downloadlangdet-master.tar.gz
langdet-master.zip
Add Linux OpenBLAS builds and Docker validationHEADmaster
Diffstat (limited to '')
-rw-r--r--README.md168
1 files changed, 84 insertions, 84 deletions
diff --git a/README.md b/README.md
index 0735571..cf952c4 100644
--- a/README.md
+++ b/README.md
@@ -1,15 +1,31 @@
-# XLM-R language detection on macOS CPU
+# XLM-R language detection
-A native C CLI for `papluca/xlm-roberta-base-language-detection`, using the
-liblangdet prediction API, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
-predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python
-runtime is required for inference.
+A native C library and CLI for `papluca/xlm-roberta-base-language-detection`.
+It uses F32 CPU inference with Apple Accelerate on macOS and OpenBLAS on Linux,
+and returns predictions for the checkpoint's 20 languages.
## Build and run
-Use an Apple Silicon Mac with Xcode Command Line Tools installed
-(`xcode-select --install` if needed). Python 3 is used for downloading the model
-and running the optional reference checks, not for native inference.
+### macOS (Apple Silicon)
+
+Install Xcode Command Line Tools (`xcode-select --install` if needed).
+Apple Accelerate is included with macOS.
+
+### Linux (ARM64 or x86-64)
+
+Install a C compiler, Make, `pkg-config`, and pthread OpenBLAS development files.
+On Ubuntu or Debian:
+
+```sh
+sudo apt-get update
+sudo apt-get install build-essential pkg-config libopenblas-pthread-dev python3 ca-certificates
+```
+
+The Makefile selects OpenBLAS on Linux using `pkg-config`. ARM64 uses NEON.
+The default Linux build does not require AVX2; OpenBLAS selects its CPU kernels
+at runtime. Optional compiler flags can enable AVX2/FMA in the attention code.
+
+### Download and run
```sh
python3 download_model.py
@@ -21,9 +37,6 @@ printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdet
Run these commands from this directory. The default model path is `./model`;
use `-d DIRECTORY` for a different path.
-The Makefile currently targets macOS ARM64. The source includes Linux/OpenBLAS
-and x86 paths, but Linux still needs build configuration and testing.
-
The model is loaded once per process. Each text argument, or each stdin line
when there are no text arguments, produces one JSON line in input order:
@@ -34,32 +47,19 @@ when there are no text arguments, produces one JSON line in input order:
- `ms` measures tokenization and inference, excluding model loading and output.
`--all` includes a `scores` object with probabilities for every language.
-`-t N` selects 1 to 16 CPU kernel threads; the default is the detected CPU count,
-capped at 16. Accelerate manages its own BLAS threads. `-h` prints help.
-Use `--` before text that begins with a hyphen. Diagnostics go to stderr.
-
-## Inference
-
-`liblangdet/langdet.c` loads and validates the four F32 classifier tensors and the
-`id2label` mapping. `langdetect.c` handles only command-line input and JSON output. The encoder accepts F32 XLM-R weights with exact GeLU activation;
-other model families and reduced-precision weights are not supported.
-The encoder uses first-token pooling without L2 normalization.
-The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer,
-and stable softmax. Dropout is disabled during inference.
-
-The native loader requires `config.json`, one `model.safetensors` file, and
-`sentencepiece.bpe.model`. Sharded weights and alternate tokenizer files are not
-supported. The safetensors and configuration readers both use `yyjson`.
+`-t N` selects 1 to 16 CPU threads; the default is the detected CPU count,
+capped at 16. On macOS, it sets the internal kernel worker count while Accelerate
+manages its own BLAS threads. On Linux, it sets the OpenBLAS thread count and
+keeps the auxiliary kernel pool serial to avoid nested parallelism.
+`-h` prints help. Use `--` before text that begins with a hyphen. Diagnostics go
+to stderr.
-The CPU encoder and tokenizer are in `liblangdet/`; the JSON parser is in
-`deps/yyjson/`. Their license files are included. The build uses only local source
-files and the macOS SDK, with no dependency on another repository.
+## Input limits
-Inputs are processed individually, not as a parallel batch. Long inputs are
-truncated to 512 tokens while preserving the final `</s>`. Empty inputs are
-accepted, but their predictions are not useful. The model always chooses among
-its 20 labels; it has no unknown-language result, and scores are not a guarantee
-of accuracy on ambiguous or unsupported text.
+Inputs longer than 512 tokens are truncated while preserving the final `</s>`.
+Empty inputs are accepted, but their predictions are not useful. The model
+always chooses among its 20 labels; it has no unknown-language result, and
+scores are not a guarantee of accuracy on ambiguous or unsupported text.
## CPU library
@@ -82,16 +82,24 @@ Use one API call at a time per process, including when multiple detectors are
loaded. Internal CPU workers may cooperate on a prediction, but simultaneous calls
from application threads are not supported.
-`examples/detect.c` is a complete application using only the public header:
+`examples/detect.c` is a complete application using only the public header.
+Link it on macOS with:
```sh
cc -Iliblangdet examples/detect.c liblangdet.a -framework Accelerate -lm -pthread -o build/detect-example
-./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.'
```
-The archive includes its JSON parser. No separate parser library or private
-inference headers are needed. The encoder, tokenizer, classifier, and thread
-configuration are managed by the detector.
+On Linux, link OpenBLAS instead:
+
+```sh
+cc -Iliblangdet examples/detect.c liblangdet.a $(pkg-config --libs openblas) -lm -pthread -o build/detect-example
+```
+
+Then run it on either platform:
+
+```sh
+./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.'
+```
## Download
@@ -104,16 +112,13 @@ python3 download_model.py
python3 download_model.py ./models/language-detection
```
-The default destination is `model/` beside the script. The download uses revision
-`9865598389ca9d95637462f743f683b51d75b87b` and includes safetensors weights,
-configuration, tokenizer files, and the model card. It excludes duplicate PyTorch
-and TensorFlow weights. Allow about 1.13 GB of disk space for a fresh download.
+The default destination is `model/` beside the script. The checkpoint revision is
+`9865598389ca9d95637462f743f683b51d75b87b`. Allow about 1.13 GB of disk space.
+The library needs `config.json`, `model.safetensors`, and `sentencepiece.bpe.model`
+in the model directory.
-The script checks each file's size and hash against Hugging Face metadata. Large
-LFS files use SHA-256; ordinary files use Git blob SHA-1. Verified files are reused.
-Downloads go to `.part` files and become final files only after verification.
-Interrupted files restart on the next run; they do not resume from their previous
-byte offset.
+The downloader verifies file hashes and reuses verified files. Interrupted
+downloads restart on the next run.
## Tests
@@ -125,43 +130,38 @@ make test
```
Use `make test MODEL=/path/to/model PYTHON=python3.12` to select a different
-checkpoint directory or Python executable. The first run needs internet access
-for the test-only Python packages. `uv` manages the pinned dependencies declared
-in `tools/verify.py`; its cache stays in `.uv-cache/` inside this project.
-The Transformers reference uses only the local model files and runs offline on
-CPU in F32, with dropout disabled and eager attention.
+checkpoint directory or Python executable. Tests check the public API and
+compare token IDs and language scores with Transformers on CPU.
+The first run needs internet access for Python dependencies, which `uv` caches
+in `.uv-cache/` inside this project.
-The target builds and runs:
+### Linux tests through Docker
-- `tools/test_api.c`: repeated predictions, score normalization, result ownership,
- empty input, truncation, invalid arguments, error buffers, and detector lifecycle.
-- `tools/test_safetensors.c`: single-file tensor loading, metadata, lookup, and
- rejection of malformed shapes, offsets, and JSON headers.
-- `tools/token_ids.c`: a test-only adapter that returns the native token IDs.
-- `tools/verify.py`: exact token-ID comparison and all 20 language scores against
- Transformers on 31 inputs. These cover the 20 languages, empty and whitespace
- input, Unicode normalization, multiple lines, and lengths below, at, and above
- the 128-token BLAS threshold and the 512-token limit. Scores must differ by no more than `1e-4` in absolute value,
- and the predicted labels must match, with both one and four CPU kernel threads.
+With Docker running and `model/` downloaded, run:
+
+```sh
+./tools/test-linux.sh arm64
+./tools/test-linux.sh amd64
+```
+
+Without an architecture argument, the script uses the Docker engine's native
+architecture. On an ARM64 Mac, `amd64` uses Docker Desktop's x86-64 emulation.
+Docker Desktop must have emulation enabled for that command.
+
+The script uses Ubuntu 24.04 and mounts the source and `model/` read-only.
+Build products stay inside the container. The first image build needs internet
+access; tests run with networking disabled.
+
+Each run performs `make test`, then builds and runs the public API example as
+both C and C++. Additional arguments are passed to Make. To exercise AVX2/FMA
+on an x86-64 CPU or emulator that supports those instructions:
+
+```sh
+./tools/test-linux.sh amd64 CFLAGS='-O3 -march=x86-64-v3'
+```
-The Python checks also verify stdin behavior, invalid CLI arguments, NUL rejection,
-and failures for missing or invalid configuration, weights, and tokenizer files.
-They also check rejection of sharded weights, alternate tokenizer filenames, and
-BERT model configuration.
-No Python package is linked into or required by the native library.
+## License
-## Verification status
+MIT License.
-- The executable builds and runs as native macOS ARM64, linked to Accelerate and
- the system C library, with no GPU runtime.
-- CLI smoke checks pass for one sample in each of the 20 supported languages,
- normalized scores, stdin and argument input, empty and whitespace-only input,
- single-thread inference, and 512-token truncation.
-- Invalid thread counts, a missing model directory, and NUL bytes in stdin
- produce errors.
-- Standalone C and C++ applications link using only `langdet.h` and the static
- archive. Prediction results, error returns, and detector lifecycle checks pass.
-- The downloaded checkpoint files match Hugging Face's sizes and hashes.
-- The reference comparisons pass on the 31 test inputs: token IDs match exactly,
- labels agree, and probability differences stay below the `1e-4` tolerance.
- These regression checks are not an accuracy benchmark or exhaustive tokenizer test.
+The model weights are `papluca/xlm-roberta-base-language-detection` which is based on `FacebookAI/xlm-roberta-base`; see the [model card](https://huggingface.co/papluca/xlm-roberta-base-language-detection).