diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 15:41:41 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 16:06:03 +0200 |
| commit | 939b9a8deae07c2f06aa16c08f3ddad70f97b261 (patch) | |
| tree | 00245f8e747349ddcca171be878e5ac19e8110a4 /README.md | |
| parent | 91c6aa07597462705a7f30ee5b97e2c7943b5616 (diff) | |
| download | langdet-master.tar.gz langdet-master.zip | |
Diffstat (limited to '')
| -rw-r--r-- | README.md | 168 |
1 files changed, 84 insertions, 84 deletions
@@ -1,15 +1,31 @@ -# XLM-R language detection on macOS CPU +# XLM-R language detection -A native C CLI for `papluca/xlm-roberta-base-language-detection`, using the -liblangdet prediction API, ARM NEON kernels, and Apple Accelerate BLAS. It returns language -predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python -runtime is required for inference. +A native C library and CLI for `papluca/xlm-roberta-base-language-detection`. +It uses F32 CPU inference with Apple Accelerate on macOS and OpenBLAS on Linux, +and returns predictions for the checkpoint's 20 languages. ## Build and run -Use an Apple Silicon Mac with Xcode Command Line Tools installed -(`xcode-select --install` if needed). Python 3 is used for downloading the model -and running the optional reference checks, not for native inference. +### macOS (Apple Silicon) + +Install Xcode Command Line Tools (`xcode-select --install` if needed). +Apple Accelerate is included with macOS. + +### Linux (ARM64 or x86-64) + +Install a C compiler, Make, `pkg-config`, and pthread OpenBLAS development files. +On Ubuntu or Debian: + +```sh +sudo apt-get update +sudo apt-get install build-essential pkg-config libopenblas-pthread-dev python3 ca-certificates +``` + +The Makefile selects OpenBLAS on Linux using `pkg-config`. ARM64 uses NEON. +The default Linux build does not require AVX2; OpenBLAS selects its CPU kernels +at runtime. Optional compiler flags can enable AVX2/FMA in the attention code. + +### Download and run ```sh python3 download_model.py @@ -21,9 +37,6 @@ printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdet Run these commands from this directory. The default model path is `./model`; use `-d DIRECTORY` for a different path. -The Makefile currently targets macOS ARM64. The source includes Linux/OpenBLAS -and x86 paths, but Linux still needs build configuration and testing. - The model is loaded once per process. Each text argument, or each stdin line when there are no text arguments, produces one JSON line in input order: @@ -34,32 +47,19 @@ when there are no text arguments, produces one JSON line in input order: - `ms` measures tokenization and inference, excluding model loading and output. `--all` includes a `scores` object with probabilities for every language. -`-t N` selects 1 to 16 CPU kernel threads; the default is the detected CPU count, -capped at 16. Accelerate manages its own BLAS threads. `-h` prints help. -Use `--` before text that begins with a hyphen. Diagnostics go to stderr. - -## Inference - -`liblangdet/langdet.c` loads and validates the four F32 classifier tensors and the -`id2label` mapping. `langdetect.c` handles only command-line input and JSON output. The encoder accepts F32 XLM-R weights with exact GeLU activation; -other model families and reduced-precision weights are not supported. -The encoder uses first-token pooling without L2 normalization. -The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer, -and stable softmax. Dropout is disabled during inference. - -The native loader requires `config.json`, one `model.safetensors` file, and -`sentencepiece.bpe.model`. Sharded weights and alternate tokenizer files are not -supported. The safetensors and configuration readers both use `yyjson`. +`-t N` selects 1 to 16 CPU threads; the default is the detected CPU count, +capped at 16. On macOS, it sets the internal kernel worker count while Accelerate +manages its own BLAS threads. On Linux, it sets the OpenBLAS thread count and +keeps the auxiliary kernel pool serial to avoid nested parallelism. +`-h` prints help. Use `--` before text that begins with a hyphen. Diagnostics go +to stderr. -The CPU encoder and tokenizer are in `liblangdet/`; the JSON parser is in -`deps/yyjson/`. Their license files are included. The build uses only local source -files and the macOS SDK, with no dependency on another repository. +## Input limits -Inputs are processed individually, not as a parallel batch. Long inputs are -truncated to 512 tokens while preserving the final `</s>`. Empty inputs are -accepted, but their predictions are not useful. The model always chooses among -its 20 labels; it has no unknown-language result, and scores are not a guarantee -of accuracy on ambiguous or unsupported text. +Inputs longer than 512 tokens are truncated while preserving the final `</s>`. +Empty inputs are accepted, but their predictions are not useful. The model +always chooses among its 20 labels; it has no unknown-language result, and +scores are not a guarantee of accuracy on ambiguous or unsupported text. ## CPU library @@ -82,16 +82,24 @@ Use one API call at a time per process, including when multiple detectors are loaded. Internal CPU workers may cooperate on a prediction, but simultaneous calls from application threads are not supported. -`examples/detect.c` is a complete application using only the public header: +`examples/detect.c` is a complete application using only the public header. +Link it on macOS with: ```sh cc -Iliblangdet examples/detect.c liblangdet.a -framework Accelerate -lm -pthread -o build/detect-example -./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.' ``` -The archive includes its JSON parser. No separate parser library or private -inference headers are needed. The encoder, tokenizer, classifier, and thread -configuration are managed by the detector. +On Linux, link OpenBLAS instead: + +```sh +cc -Iliblangdet examples/detect.c liblangdet.a $(pkg-config --libs openblas) -lm -pthread -o build/detect-example +``` + +Then run it on either platform: + +```sh +./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.' +``` ## Download @@ -104,16 +112,13 @@ python3 download_model.py python3 download_model.py ./models/language-detection ``` -The default destination is `model/` beside the script. The download uses revision -`9865598389ca9d95637462f743f683b51d75b87b` and includes safetensors weights, -configuration, tokenizer files, and the model card. It excludes duplicate PyTorch -and TensorFlow weights. Allow about 1.13 GB of disk space for a fresh download. +The default destination is `model/` beside the script. The checkpoint revision is +`9865598389ca9d95637462f743f683b51d75b87b`. Allow about 1.13 GB of disk space. +The library needs `config.json`, `model.safetensors`, and `sentencepiece.bpe.model` +in the model directory. -The script checks each file's size and hash against Hugging Face metadata. Large -LFS files use SHA-256; ordinary files use Git blob SHA-1. Verified files are reused. -Downloads go to `.part` files and become final files only after verification. -Interrupted files restart on the next run; they do not resume from their previous -byte offset. +The downloader verifies file hashes and reuses verified files. Interrupted +downloads restart on the next run. ## Tests @@ -125,43 +130,38 @@ make test ``` Use `make test MODEL=/path/to/model PYTHON=python3.12` to select a different -checkpoint directory or Python executable. The first run needs internet access -for the test-only Python packages. `uv` manages the pinned dependencies declared -in `tools/verify.py`; its cache stays in `.uv-cache/` inside this project. -The Transformers reference uses only the local model files and runs offline on -CPU in F32, with dropout disabled and eager attention. +checkpoint directory or Python executable. Tests check the public API and +compare token IDs and language scores with Transformers on CPU. +The first run needs internet access for Python dependencies, which `uv` caches +in `.uv-cache/` inside this project. -The target builds and runs: +### Linux tests through Docker -- `tools/test_api.c`: repeated predictions, score normalization, result ownership, - empty input, truncation, invalid arguments, error buffers, and detector lifecycle. -- `tools/test_safetensors.c`: single-file tensor loading, metadata, lookup, and - rejection of malformed shapes, offsets, and JSON headers. -- `tools/token_ids.c`: a test-only adapter that returns the native token IDs. -- `tools/verify.py`: exact token-ID comparison and all 20 language scores against - Transformers on 31 inputs. These cover the 20 languages, empty and whitespace - input, Unicode normalization, multiple lines, and lengths below, at, and above - the 128-token BLAS threshold and the 512-token limit. Scores must differ by no more than `1e-4` in absolute value, - and the predicted labels must match, with both one and four CPU kernel threads. +With Docker running and `model/` downloaded, run: + +```sh +./tools/test-linux.sh arm64 +./tools/test-linux.sh amd64 +``` + +Without an architecture argument, the script uses the Docker engine's native +architecture. On an ARM64 Mac, `amd64` uses Docker Desktop's x86-64 emulation. +Docker Desktop must have emulation enabled for that command. + +The script uses Ubuntu 24.04 and mounts the source and `model/` read-only. +Build products stay inside the container. The first image build needs internet +access; tests run with networking disabled. + +Each run performs `make test`, then builds and runs the public API example as +both C and C++. Additional arguments are passed to Make. To exercise AVX2/FMA +on an x86-64 CPU or emulator that supports those instructions: + +```sh +./tools/test-linux.sh amd64 CFLAGS='-O3 -march=x86-64-v3' +``` -The Python checks also verify stdin behavior, invalid CLI arguments, NUL rejection, -and failures for missing or invalid configuration, weights, and tokenizer files. -They also check rejection of sharded weights, alternate tokenizer filenames, and -BERT model configuration. -No Python package is linked into or required by the native library. +## License -## Verification status +MIT License. -- The executable builds and runs as native macOS ARM64, linked to Accelerate and - the system C library, with no GPU runtime. -- CLI smoke checks pass for one sample in each of the 20 supported languages, - normalized scores, stdin and argument input, empty and whitespace-only input, - single-thread inference, and 512-token truncation. -- Invalid thread counts, a missing model directory, and NUL bytes in stdin - produce errors. -- Standalone C and C++ applications link using only `langdet.h` and the static - archive. Prediction results, error returns, and detector lifecycle checks pass. -- The downloaded checkpoint files match Hugging Face's sizes and hashes. -- The reference comparisons pass on the 31 test inputs: token IDs match exactly, - labels agree, and probability differences stay below the `1e-4` tolerance. - These regression checks are not an accuracy benchmark or exhaustive tokenizer test. +The model weights are `papluca/xlm-roberta-base-language-detection` which is based on `FacebookAI/xlm-roberta-base`; see the [model card](https://huggingface.co/papluca/xlm-roberta-base-language-detection). |
