diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 12:54:13 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 12:54:13 +0200 |
| commit | 91871be6ece9317c5d49353f6474583e7e28168d (patch) | |
| tree | 8cabf2546322a9e1b724bf8413f32de54378824f /README.md | |
| parent | 35ab9c12ba4d396e6cba99d6ea947e26becedad4 (diff) | |
| download | langdet-91871be6ece9317c5d49353f6474583e7e28168d.tar.gz langdet-91871be6ece9317c5d49353f6474583e7e28168d.zip | |
Add native CPU language detection CLI
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 116 |
1 files changed, 52 insertions, 64 deletions
@@ -1,65 +1,56 @@ -# XLM-R language detection: ffwd compatibility +# XLM-R language detection on macOS CPU -`papluca/xlm-roberta-base-language-detection` is **not supported as a language -classifier by the inspected ffwd implementation**. Its XLM-R encoder is supported, -but ffwd does not load or execute the classification head. Loading the checkpoint -as an embedding model would produce 768 numbers, not 20 language scores. +A native C CLI for `papluca/xlm-roberta-base-language-detection`, using ffwd's +CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language +predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python +runtime is required for inference. -## Compatibility findings +## Build and run -The reference source is `/Users/ogz/x/ffwd`. +Use an Apple Silicon Mac with Xcode Command Line Tools installed +(`xcode-select --install` if needed). Python 3 is required only for downloading +the model. -- `README.md` lists XLM-R support for text embeddings. -- `libffwd/config.c` recognizes `xlm-roberta`, sets the correct position offset, - and defaults to mean pooling. It does not interpret the classification - architecture or `id2label`. -- `libffwd/model.c`, in `load_bert_weights`, accepts the checkpoint's `roberta.` - tensor prefix and loads the encoder, but not `classifier.*`. -- `libffwd/forward.c` implements the encoder and embedding pooling, including - first-token pooling (`FFWD_POOL_CLS`), but no classification head. -- `libffwd/ffwd.h` exposes embeddings and reranking, not sequence classification. -- `libffwd/tokenize.c` supports the checkpoint's `sentencepiece.bpe.model` format - and wraps text with `<s>` and `</s>`. +```sh +python3 download_model.py +make -j8 +./langdetect 'This is an English sentence.' 'Bugün hava çok güzel.' +printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdetect +``` -The checkpoint declares `XLMRobertaForSequenceClassification`. Its safetensors -header contains these F32 classifier tensors: +Run these commands from this directory. The default model path is `./model`; +use `-d DIRECTORY` for a different path. -| Tensor | Shape | -| --- | --- | -| `classifier.dense.weight` | `[768, 768]` | -| `classifier.dense.bias` | `[768]` | -| `classifier.out_proj.weight` | `[20, 768]` | -| `classifier.out_proj.bias` | `[20]` | +The model is loaded once per process. Each text argument, or each stdin line +when there are no text arguments, produces one JSON line in input order: -## Required inference work +- `label` is the predicted language code. +- `score` is its softmax probability. +- `tokens` includes the leading `<s>` and trailing `</s>`. +- `truncated` indicates whether the input exceeded 512 tokens. +- `ms` measures tokenization and inference, excluding model loading and output. -1. Reuse ffwd's CPU encoder, SentencePiece tokenizer, safetensors reader, and - ARM NEON kernels. On macOS ARM64, matrix multiplication uses Apple's Accelerate - framework; Metal and MLX are not required. -2. Load and validate the four classifier tensors, plus `id2label` from - `config.json`. -3. Take the final hidden state of the first token (`<s>`), not the mean of all - tokens. Do not apply L2 normalization. Compute, with column vectors: +`--all` includes a `scores` object with probabilities for every language. +`-t N` selects 1 to 16 ffwd kernel threads; the default is the detected CPU count, +capped at 16. Accelerate manages its own BLAS threads. `-h` prints help. +Use `--` before text that begins with a hyphen. Diagnostics go to stderr. - ```text - h = final_hidden_state[0] # 768 values - z = tanh(classifier.dense.weight @ h + classifier.dense.bias) - logits = classifier.out_proj.weight @ z + classifier.out_proj.bias - probabilities = softmax(logits) # 20 values - language = id2label[argmax(probabilities)] - ``` +## Inference - Dropout is disabled during inference. Stable softmax subtracts the largest - logit before exponentiation. -4. Add a classification runner that returns labels and scores. Keep each input - within 512 tokens, including both special tokens; any truncation must retain - the final `</s>` token. -5. Compare token IDs, logits, and probabilities against Hugging Face Transformers - on CPU, including all 20 supported languages, Unicode, empty input, mixed - sequence lengths, and the length limit. +`langdetect.c` loads and validates the four F32 classifier tensors and the +`id2label` mapping. The encoder uses first-token pooling without L2 normalization. +The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer, +and stable softmax. Dropout is disabled during inference. -This requires additional classification code, not retraining or a new transformer -architecture. A configuration change alone cannot execute the missing head. +The CPU encoder and tokenizer are in `libffwd/`; the JSON parser is in +`deps/yyjson/`. Their license files are included. The build uses only local source +files and the macOS SDK, with no dependency on another repository. + +Inputs are processed individually, not as a parallel batch. Long inputs are +truncated to 512 tokens while preserving the final `</s>`. Empty inputs are +accepted, but their predictions are not useful. The model always chooses among +its 20 labels; it has no unknown-language result, and scores are not a guarantee +of accuracy on ambiguous or unsupported text. ## Download @@ -85,16 +76,13 @@ byte offset. ## Verification status -- The host reports `Darwin arm64`. -- All seven files were downloaded and their sizes and hashes verified. A second - script run verified and reused every file. -- The local safetensors header was checked for the four classifier shapes and - F32 types, and its final data offset matches the file size. -- The classification formula matches Transformers 4.12.5, the version recorded - in the model configuration. -- No ffwd inference code is included or built here. No language predictions or - numerical comparisons against Transformers have been run. Encoder compatibility - is based on source and checkpoint inspection, not a runtime test. - -The remaining work is the classification runner and the numerical verification -listed above. The reference ffwd source is unchanged. +- The executable builds and runs as native macOS ARM64, linked to Accelerate and + the system C library, with no GPU runtime. +- CLI smoke checks pass for one sample in each of the 20 supported languages, + normalized scores, stdin and argument input, empty and whitespace-only input, + single-thread inference, and 512-token truncation. +- Invalid thread counts, a missing model directory, and NUL bytes in stdin + produce errors. +- The downloaded checkpoint files match Hugging Face's sizes and hashes. +- Numerical equivalence with Hugging Face Transformers has not been established. + These smoke checks are not an accuracy benchmark or exhaustive tokenizer test. |
