aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-14 15:19:12 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-14 15:19:12 +0200
commitf463018cd700690336898d109226d489deca036d (patch)
tree4e1d83f55a5e231af519aa4d647141db5de23af7 /README.md
parentd98ee1568c0bd88304a12e59d12104066e01e8d0 (diff)
downloadffwd-f463018cd700690336898d109226d489deca036d.tar.gz
ffwd-f463018cd700690336898d109226d489deca036d.zip
all: add support for Qwen3-Embedding + causal GQA attention
Diffstat (limited to 'README.md')
-rw-r--r--README.md24
1 files changed, 21 insertions, 3 deletions
diff --git a/README.md b/README.md
index 977576e..f6b01b0 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,11 @@
# embed.c
-`embed.c` implements inference for
-[pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed)
-dense/contextual and late-interaction text embedding models.
+`embed.c` implements inference for:
+
+- [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed)
+ dense, contextual, and late-interaction models
+- [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding)
+ causal, last-token pooled models
Supported platforms are CUDA/cuBLAS on Linux/NVIDIA, Apple Silicon MLX, and
CPU/BLAS.
@@ -34,6 +37,17 @@ matrix:
"Berlin is the capital of Germany."
```
+Qwen3 retrieval queries benefit from an explicit task instruction. Documents are
+embedded directly:
+
+```bash
+./embed -d ./Qwen3-Embedding-0.6B \
+ $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?'
+```
+
+The Qwen3 tokenizer terminal token is appended automatically by the CLI and
+server. Qwen3 embeddings are L2-normalized; pplx embeddings remain unnormalized.
+
Pipe in lines and use `--stream` to get one JSON embedding per line:
```bash
@@ -54,6 +68,7 @@ Serves the HTTP API. Takes one or more `--model ID=DIR` pairs:
```bash
./embed-server \
--model pplx-embed-v1-0.6b=./model \
+ --model Qwen3-Embedding-0.6B=./qwen3-model \
--port 8000
```
@@ -66,6 +81,9 @@ Late-interaction reranking endpoint:
- `POST /v1/rerank` — late-interaction MaxSim reranking
+For `Qwen3-Embedding-0.6B`, `/v1/embeddings` accepts `dimensions` from 32
+to 1024. Truncated Matryoshka embeddings are re-normalized before encoding.
+
Example:
```bash