XLM-R language detection
A native C library and CLI for papluca/xlm-roberta-base-language-detection.
It uses F32 CPU inference with Apple Accelerate on macOS and OpenBLAS on Linux,
and returns predictions for the checkpoint's 20 languages.
Build and run
macOS (Apple Silicon)
Install Xcode Command Line Tools (xcode-select --install if needed).
Apple Accelerate is included with macOS.
Linux (ARM64 or x86-64)
Install a C compiler, Make, pkg-config, and pthread OpenBLAS development files.
On Ubuntu or Debian:
sudo apt-get update
sudo apt-get install build-essential pkg-config libopenblas-pthread-dev python3 ca-certificates
The Makefile selects OpenBLAS on Linux using pkg-config. ARM64 uses NEON.
The default Linux build does not require AVX2; OpenBLAS selects its CPU kernels
at runtime. Optional compiler flags can enable AVX2/FMA in the attention code.
Download and run
python3 download_model.py
make -j8
./langdetect 'This is an English sentence.' 'Bugün hava çok güzel.'
printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdetect
Run these commands from this directory. The default model path is ./model;
use -d DIRECTORY for a different path.
The model is loaded once per process. Each text argument, or each stdin line when there are no text arguments, produces one JSON line in input order:
labelis the predicted language code.scoreis its softmax probability.tokensincludes the leading<s>and trailing</s>.truncatedindicates whether the input exceeded 512 tokens.msmeasures tokenization and inference, excluding model loading and output.
--all includes a scores object with probabilities for every language.
-t N selects 1 to 16 CPU threads; the default is the detected CPU count,
capped at 16. On macOS, it sets the internal kernel worker count while Accelerate
manages its own BLAS threads. On Linux, it sets the OpenBLAS thread count and
keeps the auxiliary kernel pool serial to avoid nested parallelism.
-h prints help. Use -- before text that begins with a hyphen. Diagnostics go
to stderr.
Input limits
Inputs longer than 512 tokens are truncated while preserving the final </s>.
Empty inputs are accepted, but their predictions are not useful. The model
always chooses among its 20 labels; it has no unknown-language result, and
scores are not a guarantee of accuracy on ambiguous or unsupported text.
CPU library
make builds both langdetect and the static archive liblangdet.a. Include
liblangdet/langdet.h to use the complete prediction API from C or C++:
langdet_open(model_dir, n_threads, err, errlen)loads all inference resources. Use0threads for automatic selection, or an explicit count from 1 to 16.langdet_predict(detector, text, &result, err, errlen)returns the best language, its score, all 20 language scores, token count, and truncation flag.langdet_free(detector)releases the resources. Passing NULL is safe.
Opening returns NULL on failure. Prediction returns 0 on success or -1 on failure and clears the result on failure. The optional error buffer receives a message; model-file readers may also print diagnostics to stderr. Result label strings remain valid until the detector is freed. Numeric scores are stored in the result and do not change when another prediction runs.
Use one API call at a time per process, including when multiple detectors are loaded. Internal CPU workers may cooperate on a prediction, but simultaneous calls from application threads are not supported.
examples/detect.c is a complete application using only the public header.
Link it on macOS with:
cc -Iliblangdet examples/detect.c liblangdet.a -framework Accelerate -lm -pthread -o build/detect-example
On Linux, link OpenBLAS instead:
cc -Iliblangdet examples/detect.c liblangdet.a $(pkg-config --libs openblas) -lm -pthread -o build/detect-example
Then run it on either platform:
./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.'
Download
Python 3 and internet access are the only requirements. No Hugging Face token or third-party Python packages are needed for this public checkpoint.
python3 download_model.py
# Alternatively, choose a destination:
python3 download_model.py ./models/language-detection
The default destination is model/ beside the script. The checkpoint revision is
9865598389ca9d95637462f743f683b51d75b87b. Allow about 1.13 GB of disk space.
The library needs config.json, model.safetensors, and sentencepiece.bpe.model
in the model directory.
The downloader verifies file hashes and reuses verified files. Interrupted downloads restart on the next run.
Tests
With the model downloaded, install uv and have Python 3.10 through 3.13
available, then run:
make test
Use make test MODEL=/path/to/model PYTHON=python3.12 to select a different
checkpoint directory or Python executable. Tests check the public API and
compare token IDs and language scores with Transformers on CPU.
The first run needs internet access for Python dependencies, which uv caches
in .uv-cache/ inside this project.
Linux tests through Docker
With Docker running and model/ downloaded, run:
./tools/test-linux.sh arm64
./tools/test-linux.sh amd64
Without an architecture argument, the script uses the Docker engine's native
architecture. On an ARM64 Mac, amd64 uses Docker Desktop's x86-64 emulation.
Docker Desktop must have emulation enabled for that command.
The script uses Ubuntu 24.04 and mounts the source and model/ read-only.
Build products stay inside the container. The first image build needs internet
access; tests run with networking disabled.
Each run performs make test, then builds and runs the public API example as
both C and C++. Additional arguments are passed to Make. To exercise AVX2/FMA
on an x86-64 CPU or emulator that supports those instructions:
./tools/test-linux.sh amd64 CFLAGS='-O3 -march=x86-64-v3'
License
MIT License.
The model weights are papluca/xlm-roberta-base-language-detection which is based on FacebookAI/xlm-roberta-base; see the model card.
