diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-06-17 02:55:07 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-06-17 02:55:07 +0200 |
| commit | c45ea7fc7b175a5995a8120f504063388242cda9 (patch) | |
| tree | b9a6a83e6cd6ac82b7a7b2efb96c750280311de2 /README.md | |
| parent | 4f743e18aecda8f5ebf25b400452bd4c3409e4aa (diff) | |
| download | ffwd-c45ea7fc7b175a5995a8120f504063388242cda9.tar.gz ffwd-c45ea7fc7b175a5995a8120f504063388242cda9.zip | |
chore: document XLM-R models, F16 weights, and config-driven pooling
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 56 |
1 files changed, 32 insertions, 24 deletions
@@ -3,31 +3,35 @@ `embed.c` implements inference for several text embedding model families — [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed), [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding), -GTE-Qwen2, and BERT/BGE encoders. It serves them through an -OpenAI/Perplexity-compatible HTTP API and runs on CPU, Apple MLX, and NVIDIA -CUDA. See [Supported models](#supported-models). +GTE-Qwen2, BERT/BGE encoders, and multilingual XLM-R encoders (multilingual-E5, +Snowflake Arctic). It serves them through an OpenAI/Perplexity-compatible HTTP +API and runs on CPU, Apple MLX, and NVIDIA CUDA. See +[Supported models](#supported-models). ## Supported models The server registers these model IDs out of the box; pass any of them as `--model ID=DIR`, loading a Hugging Face safetensors checkpoint from `DIR`. -Every model runs on all three backends, and `dimensions` truncates -Matryoshka-trained vectors (pplx-embed down to 128, Qwen3-Embedding down to 32). +Weights load directly from F32, BF16, or F16 safetensors. Every model runs on +all three backends, and `dimensions` truncates Matryoshka-trained vectors +(pplx-embed down to 128, Qwen3-Embedding down to 32, arctic-embed down to 256). **Pooled embeddings** — one vector per text, via `POST /v1/embeddings`: -| Model ID | Family | Dim | Pooling | Output | -| ----------------------- | --------------- | ---- | ---------- | ---------------------- | -| pplx-embed-v1-0.6b | pplx-embed | 1024 | mean | int8, unnormalized | -| pplx-embed-v1-4b | pplx-embed | 2560 | mean | int8, unnormalized | -| Qwen3-Embedding-0.6B | Qwen3-Embedding | 1024 | last token | float32, L2-normalized | -| Qwen3-Embedding-4B | Qwen3-Embedding | 2560 | last token | float32, L2-normalized | -| Qwen3-Embedding-8B | Qwen3-Embedding | 4096 | last token | float32, L2-normalized | -| gte-Qwen2-1.5B-instruct | GTE-Qwen2 | 1536 | last token | float32, L2-normalized | -| all-MiniLM-L6-v2 | BERT (MiniLM) | 384 | mean | float32, L2-normalized | -| bge-small-en-v1.5 | BERT (BGE) | 384 | CLS | float32, L2-normalized | -| bge-base-en-v1.5 | BERT (BGE) | 768 | CLS | float32, L2-normalized | -| bge-large-en-v1.5 | BERT (BGE) | 1024 | CLS | float32, L2-normalized | +| Model ID | Family | Dim | Pooling | Output | +| ------------------------------ | --------------- | ---- | ---------- | ---------------------- | +| pplx-embed-v1-0.6b | pplx-embed | 1024 | mean | int8, unnormalized | +| pplx-embed-v1-4b | pplx-embed | 2560 | mean | int8, unnormalized | +| Qwen3-Embedding-0.6B | Qwen3-Embedding | 1024 | last token | float32, L2-normalized | +| Qwen3-Embedding-4B | Qwen3-Embedding | 2560 | last token | float32, L2-normalized | +| Qwen3-Embedding-8B | Qwen3-Embedding | 4096 | last token | float32, L2-normalized | +| gte-Qwen2-1.5B-instruct | GTE-Qwen2 | 1536 | last token | float32, L2-normalized | +| all-MiniLM-L6-v2 | BERT (MiniLM) | 384 | mean | float32, L2-normalized | +| bge-small-en-v1.5 | BERT (BGE) | 384 | CLS | float32, L2-normalized | +| bge-base-en-v1.5 | BERT (BGE) | 768 | CLS | float32, L2-normalized | +| bge-large-en-v1.5 | BERT (BGE) | 1024 | CLS | float32, L2-normalized | +| multilingual-e5-large-instruct | XLM-R (E5) | 1024 | mean | float32, L2-normalized | +| snowflake-arctic-embed-l-v2.0 | XLM-R (Arctic) | 1024 | CLS | float32, L2-normalized | **Contextual embeddings** — one vector per document chunk, each embedded with its document as context, via `POST /v1/contextualizedembeddings`: @@ -45,10 +49,14 @@ its document as context, via `POST /v1/contextualizedembeddings`: | pplx-embed-v1-late-0.6b | 128 | MaxSim rerank score | The engine handles three transformer blocks — the Qwen3 block (pplx-embed and -Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT block (MiniLM, BGE) -— and selects the tokenizer from the model files (byte-level BPE for the Qwen -families, WordPiece for BERT). pplx-embed vectors are unnormalized int8 (rank -them by cosine similarity); the other families emit L2-normalized float32. +Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT/RoBERTa block (MiniLM, +BGE, and the XLM-R multilingual encoders E5 and Arctic) — and selects the +tokenizer from the model files: byte-level BPE for the Qwen families, WordPiece +for BERT, and SentencePiece (Unigram) for XLM-R. Pooling and L2-normalization +are read from the checkpoint's Sentence Transformers config +(`1_Pooling/config.json` and `modules.json`), so a model's pooling mode does not +have to be hardcoded. pplx-embed vectors are unnormalized int8 (rank them by +cosine similarity); the other families emit L2-normalized float32. ## Building @@ -185,9 +193,9 @@ Returns a list of embeddings, one per input text: `encoding_format` follows the model family: -- The OpenAI-API models (Qwen3-Embedding, GTE-Qwen2, MiniLM, BGE) default to - `float` (the true float32 vector) and also accept `base64` (base64 of - float32). +- The OpenAI-API models (Qwen3-Embedding, GTE-Qwen2, MiniLM, BGE, + multilingual-E5, Snowflake Arctic) default to `float` (the true float32 vector) + and also accept `base64` (base64 of float32). - pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts `base64_binary` and `float` (the decoded int8 view). |
