diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-03-28 23:38:06 +0100 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-03-28 23:38:06 +0100 |
| commit | 353aa5394a11ef8a9c69e73338b95a3f7eac5855 (patch) | |
| tree | 414a8e39d7e19b2704088f8c7d691281625f1394 /README.md | |
| parent | c47e900b94d18cca1bf600e58f10dbcc2e34c1f7 (diff) | |
| download | ffwd-353aa5394a11ef8a9c69e73338b95a3f7eac5855.tar.gz ffwd-353aa5394a11ef8a9c69e73338b95a3f7eac5855.zip | |
Add benchmark script
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 46 |
1 files changed, 46 insertions, 0 deletions
@@ -85,3 +85,49 @@ Options: **Daemon mode** (`--daemon`): reads one text per line from stdin, writes one JSON object per line to stdout. EOF or empty line exits. +## Benchmark results + +### pplx-embed-v1-0.6b + +1024-dim, 28 layers, 2.38 GB + +**Inference latency** (median, 5 runs, Apple M4 Pro 12-core): + +| Sentence | Tokens | PyTorch | C/BLAS | Speedup | C/MLX | Speedup | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| short | 2 | 76.2 ms | 53.9 ms | 1.4x | 15.2 ms | 5.0x | +| medium | 13 | 75.7 ms | 50.7 ms | 1.5x | 15.4 ms | 4.9x | +| long | 63 | 97.4 ms | 58.9 ms | 1.7x | 20.0 ms | 4.9x | +| 256tok | 199 | 193.5 ms | 178.2 ms | 1.1x | 53.9 ms | 3.6x | + +**Load/memory**: + +| | PyTorch | C/BLAS | C/MLX | +| :--- | :--- | :--- | :--- | +| **Load time** | 186 ms | 195 ms | 363 ms | +| **RSS** | ~2163 MB | ~1737 MB | ~4603 MB | + +### pplx-embed-v1-4b + +2560-dim, 36 layers, 16.09 GB + +| Sentence | Tokens | PyTorch | C/BLAS | Speedup | C/MLX | Speedup | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| short | 2 | 462 ms | 402 ms | 1.1x | 97 ms | 4.8x | +| medium | 13 | 307 ms | 234 ms | 1.3x | 101 ms | 3.0x | +| long | 63 | 411 ms | 350 ms | 1.2x | 121 ms | 3.4x | +| 256tok | 199 | 875 ms | 919 ms | 1.0x | 376 ms | 2.3x | + +| | PyTorch | C/BLAS | C/MLX | +| :--- | :--- | :--- | :--- | +| **Load time** | 12094 ms | 1202 ms | 6134 ms | +| **RSS** | ~14253 MB | ~13902 MB | ~15483 MB | + +**Summary**: + +* C/MLX is up to **5x** faster than PyTorch on Apple M4 Pro, and C/BLAS is up to **1.7x** faster. +* C/BLAS uses ~20% less memory than PyTorch (1737 vs 2163 MB for 0.6B) because it mmap's weights directly instead of copying them into Python objects. +* MLX RSS is higher (~4.6 GB for 0.6B) because Metal duplicates weight data into GPU-accessible memory. This is the cost of GPU speed. +* Load times: C/BLAS loads fastest (mmap, no conversion). MLX has a one-time Metal kernel compilation cost. PyTorch's 12-second load for 4B is because the first inference faults in the entire mmap'd file. +* Accuracy is identical (cosine sim = 1.000000, max diff < 2e-6) between PyTorch and C implementations. + |
