aboutsummaryrefslogtreecommitdiff
path: root/README.md
blob: cf952c450d573100521eb5a626d199609f95a49d (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
# XLM-R language detection

A native C library and CLI for `papluca/xlm-roberta-base-language-detection`.
It uses F32 CPU inference with Apple Accelerate on macOS and OpenBLAS on Linux,
and returns predictions for the checkpoint's 20 languages.

## Build and run

### macOS (Apple Silicon)

Install Xcode Command Line Tools (`xcode-select --install` if needed).
Apple Accelerate is included with macOS.

### Linux (ARM64 or x86-64)

Install a C compiler, Make, `pkg-config`, and pthread OpenBLAS development files.
On Ubuntu or Debian:

```sh
sudo apt-get update
sudo apt-get install build-essential pkg-config libopenblas-pthread-dev python3 ca-certificates
```

The Makefile selects OpenBLAS on Linux using `pkg-config`. ARM64 uses NEON.
The default Linux build does not require AVX2; OpenBLAS selects its CPU kernels
at runtime. Optional compiler flags can enable AVX2/FMA in the attention code.

### Download and run

```sh
python3 download_model.py
make -j8
./langdetect 'This is an English sentence.' 'Bugün hava çok güzel.'
printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdetect
```

Run these commands from this directory. The default model path is `./model`;
use `-d DIRECTORY` for a different path.

The model is loaded once per process. Each text argument, or each stdin line
when there are no text arguments, produces one JSON line in input order:

- `label` is the predicted language code.
- `score` is its softmax probability.
- `tokens` includes the leading `<s>` and trailing `</s>`.
- `truncated` indicates whether the input exceeded 512 tokens.
- `ms` measures tokenization and inference, excluding model loading and output.

`--all` includes a `scores` object with probabilities for every language.
`-t N` selects 1 to 16 CPU threads; the default is the detected CPU count,
capped at 16. On macOS, it sets the internal kernel worker count while Accelerate
manages its own BLAS threads. On Linux, it sets the OpenBLAS thread count and
keeps the auxiliary kernel pool serial to avoid nested parallelism.
`-h` prints help. Use `--` before text that begins with a hyphen. Diagnostics go
to stderr.

## Input limits

Inputs longer than 512 tokens are truncated while preserving the final `</s>`.
Empty inputs are accepted, but their predictions are not useful. The model
always chooses among its 20 labels; it has no unknown-language result, and
scores are not a guarantee of accuracy on ambiguous or unsupported text.

## CPU library

`make` builds both `langdetect` and the static archive `liblangdet.a`. Include
`liblangdet/langdet.h` to use the complete prediction API from C or C++:

- `langdet_open(model_dir, n_threads, err, errlen)` loads all inference resources.
  Use `0` threads for automatic selection, or an explicit count from 1 to 16.
- `langdet_predict(detector, text, &result, err, errlen)` returns the best language,
  its score, all 20 language scores, token count, and truncation flag.
- `langdet_free(detector)` releases the resources. Passing NULL is safe.

Opening returns NULL on failure. Prediction returns 0 on success or -1 on failure
and clears the result on failure. The optional error buffer receives a message;
model-file readers may also print diagnostics to stderr. Result label strings
remain valid until the detector is freed. Numeric scores are stored in the result
and do not change when another prediction runs.

Use one API call at a time per process, including when multiple detectors are
loaded. Internal CPU workers may cooperate on a prediction, but simultaneous calls
from application threads are not supported.

`examples/detect.c` is a complete application using only the public header.
Link it on macOS with:

```sh
cc -Iliblangdet examples/detect.c liblangdet.a -framework Accelerate -lm -pthread -o build/detect-example
```

On Linux, link OpenBLAS instead:

```sh
cc -Iliblangdet examples/detect.c liblangdet.a $(pkg-config --libs openblas) -lm -pthread -o build/detect-example
```

Then run it on either platform:

```sh
./build/detect-example ./model 'This is an English sentence.' 'Bugün hava çok güzel.'
```

## Download

Python 3 and internet access are the only requirements. No Hugging Face token or
third-party Python packages are needed for this public checkpoint.

```sh
python3 download_model.py
# Alternatively, choose a destination:
python3 download_model.py ./models/language-detection
```

The default destination is `model/` beside the script. The checkpoint revision is
`9865598389ca9d95637462f743f683b51d75b87b`. Allow about 1.13 GB of disk space.
The library needs `config.json`, `model.safetensors`, and `sentencepiece.bpe.model`
in the model directory.

The downloader verifies file hashes and reuses verified files. Interrupted
downloads restart on the next run.

## Tests

With the model downloaded, install `uv` and have Python 3.10 through 3.13
available, then run:

```sh
make test
```

Use `make test MODEL=/path/to/model PYTHON=python3.12` to select a different
checkpoint directory or Python executable. Tests check the public API and
compare token IDs and language scores with Transformers on CPU.
The first run needs internet access for Python dependencies, which `uv` caches
in `.uv-cache/` inside this project.

### Linux tests through Docker

With Docker running and `model/` downloaded, run:

```sh
./tools/test-linux.sh arm64
./tools/test-linux.sh amd64
```

Without an architecture argument, the script uses the Docker engine's native
architecture. On an ARM64 Mac, `amd64` uses Docker Desktop's x86-64 emulation.
Docker Desktop must have emulation enabled for that command.

The script uses Ubuntu 24.04 and mounts the source and `model/` read-only.
Build products stay inside the container. The first image build needs internet
access; tests run with networking disabled.

Each run performs `make test`, then builds and runs the public API example as
both C and C++. Additional arguments are passed to Make. To exercise AVX2/FMA
on an x86-64 CPU or emulator that supports those instructions:

```sh
./tools/test-linux.sh amd64 CFLAGS='-O3 -march=x86-64-v3'
```

## License

MIT License.

The model weights are `papluca/xlm-roberta-base-language-detection` which is based on `FacebookAI/xlm-roberta-base`; see the [model card](https://huggingface.co/papluca/xlm-roberta-base-language-detection).