diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 14:04:04 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-09-29 14:04:04 +0200 |
| commit | 6f98fc8cf3f7cc4b9d523d16492a07d3e2aea8c9 (patch) | |
| tree | 427e45b901aec902ceba72428fe5de9689d3001b /README.md | |
| parent | cb1a04331b27cc19445de63cedcd4fd30b8a9016 (diff) | |
| download | langdet-6f98fc8cf3f7cc4b9d523d16492a07d3e2aea8c9.tar.gz langdet-6f98fc8cf3f7cc4b9d523d16492a07d3e2aea8c9.zip | |
Add CPU reference comparison and repeatable API tests
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 40 |
1 files changed, 36 insertions, 4 deletions
@@ -8,8 +8,8 @@ runtime is required for inference. ## Build and run Use an Apple Silicon Mac with Xcode Command Line Tools installed -(`xcode-select --install` if needed). Python 3 is required only for downloading -the model. +(`xcode-select --install` if needed). Python 3 is used for downloading the model +and running the optional reference checks, not for native inference. ```sh python3 download_model.py @@ -108,6 +108,37 @@ Downloads go to `.part` files and become final files only after verification. Interrupted files restart on the next run; they do not resume from their previous byte offset. +## Tests + +With the model downloaded, install `uv` and have Python 3.10 through 3.13 +available, then run: + +```sh +make test +``` + +Use `make test MODEL=/path/to/model PYTHON=python3.12` to select a different +checkpoint directory or Python executable. The first run needs internet access +for the test-only Python packages. `uv` manages the pinned dependencies declared +in `tools/verify.py`; its cache stays in `.uv-cache/` inside this project. +The Transformers reference uses only the local model files and runs offline on +CPU in F32, with dropout disabled and eager attention. + +The target builds and runs: + +- `tools/test_api.c`: repeated predictions, score normalization, result ownership, + empty input, truncation, invalid arguments, error buffers, and detector lifecycle. +- `tools/token_ids.c`: a test-only adapter that returns the native token IDs. +- `tools/verify.py`: exact token-ID comparison and all 20 language scores against + Transformers on 28 inputs. These cover the 20 languages, empty and whitespace + input, Unicode normalization, multiple lines, and lengths below, at, and above + the 512-token limit. Scores must differ by no more than `1e-4` in absolute value, + and the predicted labels must match, with both one and four CPU kernel threads. + +The Python checks also verify stdin behavior, invalid CLI arguments, NUL rejection, +and failures for missing or invalid configuration, weights, and tokenizer files. +No Python package is linked into or required by the native library. + ## Verification status - The executable builds and runs as native macOS ARM64, linked to Accelerate and @@ -120,5 +151,6 @@ byte offset. - Standalone C and C++ applications link using only `langdet.h` and the static archive. Prediction results, error returns, and detector lifecycle checks pass. - The downloaded checkpoint files match Hugging Face's sizes and hashes. -- Numerical equivalence with Hugging Face Transformers has not been established. - These smoke checks are not an accuracy benchmark or exhaustive tokenizer test. +- The reference comparisons pass on the 28 test inputs: token IDs match exactly, + labels agree, and probability differences stay below the `1e-4` tolerance. + These regression checks are not an accuracy benchmark or exhaustive tokenizer test. |
