aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-09-29 12:54:13 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-09-29 12:54:13 +0200
commit91871be6ece9317c5d49353f6474583e7e28168d (patch)
tree8cabf2546322a9e1b724bf8413f32de54378824f /README.md
parent35ab9c12ba4d396e6cba99d6ea947e26becedad4 (diff)
downloadlangdet-91871be6ece9317c5d49353f6474583e7e28168d.tar.gz
langdet-91871be6ece9317c5d49353f6474583e7e28168d.zip
Add native CPU language detection CLI
Diffstat (limited to 'README.md')
-rw-r--r--README.md116
1 files changed, 52 insertions, 64 deletions
diff --git a/README.md b/README.md
index efd292c..2fe792b 100644
--- a/README.md
+++ b/README.md
@@ -1,65 +1,56 @@
-# XLM-R language detection: ffwd compatibility
+# XLM-R language detection on macOS CPU
-`papluca/xlm-roberta-base-language-detection` is **not supported as a language
-classifier by the inspected ffwd implementation**. Its XLM-R encoder is supported,
-but ffwd does not load or execute the classification head. Loading the checkpoint
-as an embedding model would produce 768 numbers, not 20 language scores.
+A native C CLI for `papluca/xlm-roberta-base-language-detection`, using ffwd's
+CPU encoder, ARM NEON kernels, and Apple Accelerate BLAS. It returns language
+predictions for the checkpoint's 20 languages. No MLX, Metal, PyTorch, or Python
+runtime is required for inference.
-## Compatibility findings
+## Build and run
-The reference source is `/Users/ogz/x/ffwd`.
+Use an Apple Silicon Mac with Xcode Command Line Tools installed
+(`xcode-select --install` if needed). Python 3 is required only for downloading
+the model.
-- `README.md` lists XLM-R support for text embeddings.
-- `libffwd/config.c` recognizes `xlm-roberta`, sets the correct position offset,
- and defaults to mean pooling. It does not interpret the classification
- architecture or `id2label`.
-- `libffwd/model.c`, in `load_bert_weights`, accepts the checkpoint's `roberta.`
- tensor prefix and loads the encoder, but not `classifier.*`.
-- `libffwd/forward.c` implements the encoder and embedding pooling, including
- first-token pooling (`FFWD_POOL_CLS`), but no classification head.
-- `libffwd/ffwd.h` exposes embeddings and reranking, not sequence classification.
-- `libffwd/tokenize.c` supports the checkpoint's `sentencepiece.bpe.model` format
- and wraps text with `<s>` and `</s>`.
+```sh
+python3 download_model.py
+make -j8
+./langdetect 'This is an English sentence.' 'Bugün hava çok güzel.'
+printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdetect
+```
-The checkpoint declares `XLMRobertaForSequenceClassification`. Its safetensors
-header contains these F32 classifier tensors:
+Run these commands from this directory. The default model path is `./model`;
+use `-d DIRECTORY` for a different path.
-| Tensor | Shape |
-| --- | --- |
-| `classifier.dense.weight` | `[768, 768]` |
-| `classifier.dense.bias` | `[768]` |
-| `classifier.out_proj.weight` | `[20, 768]` |
-| `classifier.out_proj.bias` | `[20]` |
+The model is loaded once per process. Each text argument, or each stdin line
+when there are no text arguments, produces one JSON line in input order:
-## Required inference work
+- `label` is the predicted language code.
+- `score` is its softmax probability.
+- `tokens` includes the leading `<s>` and trailing `</s>`.
+- `truncated` indicates whether the input exceeded 512 tokens.
+- `ms` measures tokenization and inference, excluding model loading and output.
-1. Reuse ffwd's CPU encoder, SentencePiece tokenizer, safetensors reader, and
- ARM NEON kernels. On macOS ARM64, matrix multiplication uses Apple's Accelerate
- framework; Metal and MLX are not required.
-2. Load and validate the four classifier tensors, plus `id2label` from
- `config.json`.
-3. Take the final hidden state of the first token (`<s>`), not the mean of all
- tokens. Do not apply L2 normalization. Compute, with column vectors:
+`--all` includes a `scores` object with probabilities for every language.
+`-t N` selects 1 to 16 ffwd kernel threads; the default is the detected CPU count,
+capped at 16. Accelerate manages its own BLAS threads. `-h` prints help.
+Use `--` before text that begins with a hyphen. Diagnostics go to stderr.
- ```text
- h = final_hidden_state[0] # 768 values
- z = tanh(classifier.dense.weight @ h + classifier.dense.bias)
- logits = classifier.out_proj.weight @ z + classifier.out_proj.bias
- probabilities = softmax(logits) # 20 values
- language = id2label[argmax(probabilities)]
- ```
+## Inference
- Dropout is disabled during inference. Stable softmax subtracts the largest
- logit before exponentiation.
-4. Add a classification runner that returns labels and scores. Keep each input
- within 512 tokens, including both special tokens; any truncation must retain
- the final `</s>` token.
-5. Compare token IDs, logits, and probabilities against Hugging Face Transformers
- on CPU, including all 20 supported languages, Unicode, empty input, mixed
- sequence lengths, and the length limit.
+`langdetect.c` loads and validates the four F32 classifier tensors and the
+`id2label` mapping. The encoder uses first-token pooling without L2 normalization.
+The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer,
+and stable softmax. Dropout is disabled during inference.
-This requires additional classification code, not retraining or a new transformer
-architecture. A configuration change alone cannot execute the missing head.
+The CPU encoder and tokenizer are in `libffwd/`; the JSON parser is in
+`deps/yyjson/`. Their license files are included. The build uses only local source
+files and the macOS SDK, with no dependency on another repository.
+
+Inputs are processed individually, not as a parallel batch. Long inputs are
+truncated to 512 tokens while preserving the final `</s>`. Empty inputs are
+accepted, but their predictions are not useful. The model always chooses among
+its 20 labels; it has no unknown-language result, and scores are not a guarantee
+of accuracy on ambiguous or unsupported text.
## Download
@@ -85,16 +76,13 @@ byte offset.
## Verification status
-- The host reports `Darwin arm64`.
-- All seven files were downloaded and their sizes and hashes verified. A second
- script run verified and reused every file.
-- The local safetensors header was checked for the four classifier shapes and
- F32 types, and its final data offset matches the file size.
-- The classification formula matches Transformers 4.12.5, the version recorded
- in the model configuration.
-- No ffwd inference code is included or built here. No language predictions or
- numerical comparisons against Transformers have been run. Encoder compatibility
- is based on source and checkpoint inspection, not a runtime test.
-
-The remaining work is the classification runner and the numerical verification
-listed above. The reference ffwd source is unchanged.
+- The executable builds and runs as native macOS ARM64, linked to Accelerate and
+ the system C library, with no GPU runtime.
+- CLI smoke checks pass for one sample in each of the 20 supported languages,
+ normalized scores, stdin and argument input, empty and whitespace-only input,
+ single-thread inference, and 512-token truncation.
+- Invalid thread counts, a missing model directory, and NUL bytes in stdin
+ produce errors.
+- The downloaded checkpoint files match Hugging Face's sizes and hashes.
+- Numerical equivalence with Hugging Face Transformers has not been established.
+ These smoke checks are not an accuracy benchmark or exhaustive tokenizer test.