1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
|
# embed.c
`embed.c` implements inference for:
- [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed)
dense, contextual, and late-interaction models
- [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding)
causal, last-token pooled models
Supported platforms are CUDA/cuBLAS on Linux/NVIDIA, Apple Silicon MLX, and
CPU/BLAS.
## Building
**Prerequisites**:
- On Linux, install `libcjson-dev`, `libcjson1`, and `libopenblas-dev`.
- On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew.
**Makefile targets**:
- Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux).
- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS,
OpenBLAS on Linux).
## Usage
### embed
Pass one text to get its embedding, two or more to get a cosine similarity
matrix:
```bash
./embed -d ./model "What is the capital of France?"
./embed -d ./model "What is the capital of France?" \
"Paris is the capital of France." \
"Berlin is the capital of Germany."
```
Qwen3 retrieval queries benefit from an explicit task instruction. Documents are
embedded directly:
```bash
./embed -d ./Qwen3-Embedding-0.6B \
$'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?'
```
The Qwen3 tokenizer terminal token is appended automatically by the CLI and
server. Qwen3 embeddings are L2-normalized; pplx embeddings remain unnormalized.
Pipe in lines and use `--stream` to get one JSON embedding per line:
```bash
cat texts.txt | ./embed -d ./model --stream -b 8
# {"embedding":[...],"dim":2048,"tokens":7,"ms":12.3,"workspace_bytes":1048576}
```
Without `--stream`, reading from stdin accumulates all lines then prints the
similarity matrix.
Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is
CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output.
### embed-server
Serves the HTTP API. Takes one or more `--model ID=DIR` pairs:
```bash
./embed-server \
--model pplx-embed-v1-0.6b=./model \
--model Qwen3-Embedding-0.6B=./qwen3-model \
--port 8000
```
Perplexity API compatible endpoints:
- `POST /v1/embeddings` — pooled embeddings
- `POST /v1/contextualizedembeddings` — contextual document embeddings
Late-interaction reranking endpoint:
- `POST /v1/rerank` — late-interaction MaxSim reranking
For `Qwen3-Embedding-0.6B`, `/v1/embeddings` accepts `dimensions` from 32
to 1024. Truncated Matryoshka embeddings are re-normalized before encoding.
Example:
```bash
./embed-server --model pplx-embed-v1-late-0.6b=./model
# can be tested with:
curl -s http://127.0.0.1:8000/v1/rerank \
-H 'Content-Type: application/json' \
-d '{
"model":"pplx-embed-v1-late-0.6b",
"query":"scientific curiosity",
"documents":["Scientists explore from curiosity.","SQLite stores data."]
}'
```
## Acknowledgements
- `embed.c` was inspired by and initially forked from
[antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been
mostly rewritten, including a major CUDA upgrade, but the kernels and some
structural ideas are derived from that codebase.
- The server event loop library in `deps/ae` is extracted from
[Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo
(antirez) and Redis contributors.
- The
[pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval)
models were developed by the Perplexity AI research team.
- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part
of the Qwen model family from Alibaba.
|