aboutsummaryrefslogtreecommitdiff
path: root/README.md
blob: 1caf581cac366d428196cb5ba734c81be048f627 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
# embed.c

`embed.c` implements inference for the
[pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) and
[Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding) text
embedding models. It also serves these models via an
OpenAI/Perplexity-compatible HTTP API.

## Building

**Prerequisites**:

- On Linux, install `libcjson-dev`, and `libopenblas-dev`.
- On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew.

**Makefile targets**:

- Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux).
- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS,
  OpenBLAS on Linux).

## embed

Pass one text to get its embedding, two or more to get a cosine similarity
matrix:

```bash
./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?"
./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \
  "Paris is the capital of France." \
  "Berlin is the capital of Germany."
```

With Qwen3, prefix the retrieval query with a task instruction and embed
documents as-is. Pass them together to rank the documents against the query (the
query is the first row of the similarity matrix):

```bash
# arg 1: the query, with Qwen3's instruction prefix
# args 2+: documents
./embed -d ./Qwen3-Embedding-0.6B \
  $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \
  "Beijing is the capital of China." \
  "The Great Wall is a famous landmark."
```

> pplx-embed vectors are unnormalized, so rank them with cosine similarity.
> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot
> product give the same ranking.

Pipe in lines and use `--stream` to get one JSON embedding per line:

```bash
cat texts.txt | ./embed -d ./model --stream -b 8
# {"embedding":[...],"dim":2048,"tokens":7,"ms":12.3,"workspace_bytes":1048576}
```

Without `--stream`, reading from stdin accumulates all lines then prints the
similarity matrix.

Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is
CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output.

## embed-server

Serves the Perplexity/OpenAI-compatible HTTP API. Takes one or more
`--model ID=DIR` pairs:

```bash
./embed-server \
  --model pplx-embed-v1-0.6b=./model \
  --model pplx-embed-context-v1-0.6b=./context-model \
  --model Qwen3-Embedding-0.6B=./qwen3-model \
  --port 8000
```

### API

Every request body takes a `model` parameter and returns results in input order.

| Endpoint                            | Returns                          |
| ----------------------------------- | -------------------------------- |
| `POST /v1/embeddings`               | one pooled embedding per input   |
| `POST /v1/contextualizedembeddings` | one embedding per document chunk |
| `POST /v1/rerank`                   | documents ranked by MaxSim score |

#### POST /v1/embeddings

```bash
curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-Embedding-0.6B",
    "input": [
      "What is the capital of France?",
      "Paris is the capital of France."
    ]
  }'
```

Returns a list of embeddings, one per input text:

```json
{
  "object": "list",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]},
    {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]}
  ],
  "model": "Qwen3-Embedding-0.6B",
  "usage": {"prompt_tokens": 16, "total_tokens": 16}
}
```

| Field             | Type            | Description                                             |
| ----------------- | --------------- | ------------------------------------------------------- |
| `input`           | string or array | One text, or up to 512 texts. Required.                 |
| `dimensions`      | integer         | Truncate the embedding (Matryoshka), 32 up to its size. |
| `encoding_format` | string          | Output encoding (see below).                            |

**Encoding format:**

`encoding_format` follows the model family:

- Qwen3-Embedding (OpenAI-compatible) defaults to `float` (the true float32
  vector) and also accepts `base64` (base64 of float32).
- pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts
  `base64_binary` and `float` (the decoded int8 view).

#### POST /v1/contextualizedembeddings

`input` is a list of documents, each a list of chunk strings. Every chunk is
embedded with its document as context, and the results are grouped the same way:
one list per document, one embedding per chunk.

```bash
curl http://127.0.0.1:8000/v1/contextualizedembeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "pplx-embed-context-v1-0.6b",
    "input": [
      ["Intro paragraph.", "Methods paragraph."],
      ["A single-chunk note."]
    ]
  }'
```

Returns a list of chunk embeddings per document, in order:

```json
{
  "object": "list",
  "data": [
    {
      "object": "list",
      "index": 0,
      "data": [
        { "object": "embedding", "index": 0, "embedding": "<base64>" },
        { "object": "embedding", "index": 1, "embedding": "<base64>" }
      ]
    },
    {
      "object": "list",
      "index": 1,
      "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }]
    }
  ],
  "model": "pplx-embed-context-v1-0.6b",
  "usage": { "prompt_tokens": 12, "total_tokens": 12 }
}
```

#### POST /v1/rerank

Scores each document against the query with token-level MaxSim and returns
documents ranked by raw score.

```bash
curl http://127.0.0.1:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "pplx-embed-v1-late-0.6b",
    "query": "scientific curiosity",
    "documents": [
      "Scientists explore from curiosity.",
      "SQLite stores data."
    ]
  }'
```

Documents ranked by descending relevance score:

```json
{
  "object": "list",
  "model": "pplx-embed-v1-late-0.6b",
  "results": [
    { "index": 0, "relevance_score": 2.14 },
    { "index": 1, "relevance_score": 0.87 }
  ],
  "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 }
}
```

- `documents` takes up to 1000 strings. `top_n` caps the number of results
  (default: all); `return_documents: true` echoes each document text.
- Scores are raw MaxSim values, not probabilities.

#### Limits and errors

- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768
  tokens per item, 120000 tokens total, and a 64 MiB body.
- Invalid input returns `422` with a `detail` list naming the offending field; a
  registered model that is not loaded returns `503`.

#### Deployment

The server runs one inference worker and serves one in-flight request per
connection (HTTP/1.1 keep-alive).

For concurrency and availability, run several processes behind a load balancer
and route by model ID; keep one large model per process. Terminate TLS and
authenticate requests at the proxy.

## Acknowledgements

- `embed.c` was inspired by and initially forked from
  [antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been
  mostly rewritten, including a major CUDA upgrade, but the kernels and some
  structural ideas are derived from that codebase.
- The server event loop library in `deps/ae` is extracted from
  [Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo
  (antirez) and Redis contributors.
- The
  [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval)
  models were developed by the Perplexity AI research team.
- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part
  of the Qwen model family from Alibaba.

## License

MIT