diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 12:43:49 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 12:43:49 +0200 |
| commit | 35ab9c12ba4d396e6cba99d6ea947e26becedad4 (patch) | |
| tree | b9cbbf9d0ade9ac7421782f2630d7330fba06c07 /README.md | |
| download | langdet-35ab9c12ba4d396e6cba99d6ea947e26becedad4.tar.gz langdet-35ab9c12ba4d396e6cba99d6ea947e26becedad4.zip | |
Init
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 100 |
1 files changed, 100 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..efd292c --- /dev/null +++ b/README.md @@ -0,0 +1,100 @@ +# XLM-R language detection: ffwd compatibility + +`papluca/xlm-roberta-base-language-detection` is **not supported as a language +classifier by the inspected ffwd implementation**. Its XLM-R encoder is supported, +but ffwd does not load or execute the classification head. Loading the checkpoint +as an embedding model would produce 768 numbers, not 20 language scores. + +## Compatibility findings + +The reference source is `/Users/ogz/x/ffwd`. + +- `README.md` lists XLM-R support for text embeddings. +- `libffwd/config.c` recognizes `xlm-roberta`, sets the correct position offset, + and defaults to mean pooling. It does not interpret the classification + architecture or `id2label`. +- `libffwd/model.c`, in `load_bert_weights`, accepts the checkpoint's `roberta.` + tensor prefix and loads the encoder, but not `classifier.*`. +- `libffwd/forward.c` implements the encoder and embedding pooling, including + first-token pooling (`FFWD_POOL_CLS`), but no classification head. +- `libffwd/ffwd.h` exposes embeddings and reranking, not sequence classification. +- `libffwd/tokenize.c` supports the checkpoint's `sentencepiece.bpe.model` format + and wraps text with `<s>` and `</s>`. + +The checkpoint declares `XLMRobertaForSequenceClassification`. Its safetensors +header contains these F32 classifier tensors: + +| Tensor | Shape | +| --- | --- | +| `classifier.dense.weight` | `[768, 768]` | +| `classifier.dense.bias` | `[768]` | +| `classifier.out_proj.weight` | `[20, 768]` | +| `classifier.out_proj.bias` | `[20]` | + +## Required inference work + +1. Reuse ffwd's CPU encoder, SentencePiece tokenizer, safetensors reader, and + ARM NEON kernels. On macOS ARM64, matrix multiplication uses Apple's Accelerate + framework; Metal and MLX are not required. +2. Load and validate the four classifier tensors, plus `id2label` from + `config.json`. +3. Take the final hidden state of the first token (`<s>`), not the mean of all + tokens. Do not apply L2 normalization. Compute, with column vectors: + + ```text + h = final_hidden_state[0] # 768 values + z = tanh(classifier.dense.weight @ h + classifier.dense.bias) + logits = classifier.out_proj.weight @ z + classifier.out_proj.bias + probabilities = softmax(logits) # 20 values + language = id2label[argmax(probabilities)] + ``` + + Dropout is disabled during inference. Stable softmax subtracts the largest + logit before exponentiation. +4. Add a classification runner that returns labels and scores. Keep each input + within 512 tokens, including both special tokens; any truncation must retain + the final `</s>` token. +5. Compare token IDs, logits, and probabilities against Hugging Face Transformers + on CPU, including all 20 supported languages, Unicode, empty input, mixed + sequence lengths, and the length limit. + +This requires additional classification code, not retraining or a new transformer +architecture. A configuration change alone cannot execute the missing head. + +## Download + +Python 3 and internet access are the only requirements. No Hugging Face token or +third-party Python packages are needed for this public checkpoint. + +```sh +python3 download_model.py +# Alternatively, choose a destination: +python3 download_model.py ./models/language-detection +``` + +The default destination is `model/` beside the script. The download uses revision +`9865598389ca9d95637462f743f683b51d75b87b` and includes safetensors weights, +configuration, tokenizer files, and the model card. It excludes duplicate PyTorch +and TensorFlow weights. Allow about 1.13 GB of disk space for a fresh download. + +The script checks each file's size and hash against Hugging Face metadata. Large +LFS files use SHA-256; ordinary files use Git blob SHA-1. Verified files are reused. +Downloads go to `.part` files and become final files only after verification. +Interrupted files restart on the next run; they do not resume from their previous +byte offset. + +## Verification status + +- The host reports `Darwin arm64`. +- All seven files were downloaded and their sizes and hashes verified. A second + script run verified and reused every file. +- The local safetensors header was checked for the four classifier shapes and + F32 types, and its final data offset matches the file size. +- The classification formula matches Transformers 4.12.5, the version recorded + in the model configuration. +- No ffwd inference code is included or built here. No language predictions or + numerical comparisons against Transformers have been run. Encoder compatibility + is based on source and checkpoint inspection, not a runtime test. + +The remaining work is the classification runner and the numerical verification +listed above. The reference ffwd source is unchanged. |
