aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-09-29 14:38:02 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-09-29 14:38:02 +0200
commit91c6aa07597462705a7f30ee5b97e2c7943b5616 (patch)
treeb16f686279a1f1b0691a6c0d65a544084fcc1c1e /README.md
parent6f98fc8cf3f7cc4b9d523d16492a07d3e2aea8c9 (diff)
downloadlangdet-91c6aa07597462705a7f30ee5b97e2c7943b5616.tar.gz
langdet-91c6aa07597462705a7f30ee5b97e2c7943b5616.zip
Remove unused checkpoint, tokenizer, and attention compatibility paths
Diffstat (limited to 'README.md')
-rw-r--r--README.md17
1 files changed, 14 insertions, 3 deletions
diff --git a/README.md b/README.md
index 23a6612..0735571 100644
--- a/README.md
+++ b/README.md
@@ -21,6 +21,9 @@ printf '%s\n' 'Bonjour tout le monde.' 'これは日本語です。' | ./langdet
Run these commands from this directory. The default model path is `./model`;
use `-d DIRECTORY` for a different path.
+The Makefile currently targets macOS ARM64. The source includes Linux/OpenBLAS
+and x86 paths, but Linux still needs build configuration and testing.
+
The model is loaded once per process. Each text argument, or each stdin line
when there are no text arguments, produces one JSON line in input order:
@@ -44,6 +47,10 @@ The encoder uses first-token pooling without L2 normalization.
The classifier applies a 768-to-768 linear layer, `tanh`, a 768-to-20 linear layer,
and stable softmax. Dropout is disabled during inference.
+The native loader requires `config.json`, one `model.safetensors` file, and
+`sentencepiece.bpe.model`. Sharded weights and alternate tokenizer files are not
+supported. The safetensors and configuration readers both use `yyjson`.
+
The CPU encoder and tokenizer are in `liblangdet/`; the JSON parser is in
`deps/yyjson/`. Their license files are included. The build uses only local source
files and the macOS SDK, with no dependency on another repository.
@@ -128,15 +135,19 @@ The target builds and runs:
- `tools/test_api.c`: repeated predictions, score normalization, result ownership,
empty input, truncation, invalid arguments, error buffers, and detector lifecycle.
+- `tools/test_safetensors.c`: single-file tensor loading, metadata, lookup, and
+ rejection of malformed shapes, offsets, and JSON headers.
- `tools/token_ids.c`: a test-only adapter that returns the native token IDs.
- `tools/verify.py`: exact token-ID comparison and all 20 language scores against
- Transformers on 28 inputs. These cover the 20 languages, empty and whitespace
+ Transformers on 31 inputs. These cover the 20 languages, empty and whitespace
input, Unicode normalization, multiple lines, and lengths below, at, and above
- the 512-token limit. Scores must differ by no more than `1e-4` in absolute value,
+ the 128-token BLAS threshold and the 512-token limit. Scores must differ by no more than `1e-4` in absolute value,
and the predicted labels must match, with both one and four CPU kernel threads.
The Python checks also verify stdin behavior, invalid CLI arguments, NUL rejection,
and failures for missing or invalid configuration, weights, and tokenizer files.
+They also check rejection of sharded weights, alternate tokenizer filenames, and
+BERT model configuration.
No Python package is linked into or required by the native library.
## Verification status
@@ -151,6 +162,6 @@ No Python package is linked into or required by the native library.
- Standalone C and C++ applications link using only `langdet.h` and the static
archive. Prediction results, error returns, and detector lifecycle checks pass.
- The downloaded checkpoint files match Hugging Face's sizes and hashes.
-- The reference comparisons pass on the 28 test inputs: token IDs match exactly,
+- The reference comparisons pass on the 31 test inputs: token IDs match exactly,
labels agree, and probability differences stay below the `1e-4` tolerance.
These regression checks are not an accuracy benchmark or exhaustive tokenizer test.