aboutsummaryrefslogtreecommitdiff
path: root/README.md
blob: b9bd023b283ca2b019669496770dfb8a8f97c23d (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
# ffwd

Super fast inference server for transformer-based text embedding models.

It runs on NVIDIA and Apple silicon GPUs, as well as CPU-only hosts with BLAS
acceleration. It's mostly tested on Blackwell, Ada, and Apple M4 GPUs, but
should work with any CUDA or Metal capable device.

Supported models include Qwen3 embedding models such as `Qwen3-Embedding`, the
`pplx-embed-v1` family, Qwen2 models such as `GTE-Qwen2`, and BERT/RoBERTa-style
encoders including `BERT`, `BGE`, `MiniLM`, `XLM-R`, `multilingual-E5`, and
`Snowflake Arctic`.

## Installation

Download the latest build for your platform from the
[releases](https://github.com/tetsuo/ffwd/releases) page.

ffwd supports models in
[safetensors](https://huggingface.co/docs/safetensors/index) format. You can
download compatible models from Hugging Face using the
[Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli).

## Building

To build ffwd from source, first install the dependencies for your platform:

- **Linux:** Install `libopenblas-dev` (required for both CPU and GPU builds).
- **macOS:** Install `mlx` and `mlx-c` via Homebrew if you're targeting
  Metal/GPU. Otherwise, no extra dependencies needed.

Once the dependencies are in place, build the project by running:

```sh
make  # or make gpu for GPU build
```

## Getting started

You can start the server with one or more models loaded, each with a label:

```bash
ffwd-server \
  --model pplx=/path/to/pplx-model:api=perplexity:min_dim=128 \
  --contextual-model pplx-context=/path/to/pplx-context-model:api=perplexity \
  --late-model pplx-late=/path/to/pplx-late-model \
  --model qwen=/path/to/qwen3-model \
  --port 8000
```

Retrieve embeddings with `curl`:

```bash
curl http://127.0.0.1:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen","input":["What is the capital of France?",
                               "Paris is the capital of France."]}'
```

## Loading models

Each load flag binds a `LABEL` to a model directory and enables one endpoint.
The flags are repeatable, so several models can share a server.

| Flag                           | Enables                             |
| ------------------------------ | ----------------------------------- |
| `--model LABEL=DIR`            | `POST /v1/embeddings`               |
| `--contextual-model LABEL=DIR` | `POST /v1/contextualizedembeddings` |
| `--late-model LABEL=DIR`       | `POST /v1/rerank`                   |

Each flag also accepts inline, colon-separated per-model options appended to the
directory:

- `:api=openai|perplexity` - the output-compatibility mode (see below).
- `:min_dim=N` - the Matryoshka truncation floor; requests may ask for any
  `dimensions` from `N` up to the model's full size.
- `:query=STR` - a query-prompt override for models that prepend one.

For example: `--model qwen=/path/to/qwen3-model:api=openai:min_dim=32`.

Loading several labels keeps several complete model runtimes resident at once.
Large models need substantial memory; on a memory-limited host, host one large
model per process and route requests across processes rather than loading
several into one server.

## Usage

Run `ffwd-server --help` for the exact, current list of options. Some important
ones are:

| Flag                          | Default     | Meaning                                                                                       |
| ----------------------------- | ----------- | --------------------------------------------------------------------------------------------- |
| `--host HOST`                 | `127.0.0.1` | Bind address.                                                                                 |
| `--port PORT`                 | -           | Listen port.                                                                                  |
| `-t, --threads N`             | backend     | Worker thread count for CPU inference.                                                        |
| `-b, --batch-size N`          | `32`        | Max texts or contextual documents per inference micro-batch.                                  |
| `--max-batch-tokens N`        | `16384`     | Token budget per batch (packed for CPU, padded dense rows for MLX). Also caps a single input. |
| `--max-concurrent-requests N` | `512`       | In-flight request cap; excess requests are answered `429` immediately.                        |
| `--auto-truncate`             | off         | Truncate inputs that exceed the per-input token cap instead of rejecting them.                |
| `--batch-wait-us N`           | backend     | Micro-batch deadline from the first request's arrival (CPU/MLX `0`, CUDA `1000`).             |
| `--memory-utilization F`      | `0.90`      | Fraction of physical memory the MLX load preflight may plan for.                              |
| `--gpu-quant-bits N`          | off         | MLX: quantize transformer linear weights at load. Opt-in, approximate.                        |
| `--gpu-quant-group-size N`    | `64`        | MLX quantization group size.                                                                  |
| `--gpu-gemm-mode MODE`        | `f32`       | CUDA GEMM compute: `f32` (exact), or `tf32`/`bf16`/`16f` (reduced).                           |
| `--gpu-weight-dtype DTYPE`    | `f32`       | CUDA weight storage: `f32`, or `bf16` (half the weight memory).                               |
| `--api-key TOKEN`             | off         | Require `Authorization: Bearer TOKEN`. `FFWD_API_KEY` is also honored.                        |
| `-v, --verbose`               | off         | Verbose logging.                                                                              |

**Batching:**

- `-b 32` is a balanced default for concurrent short requests.
- Long-document throughput is compute-bound and largely insensitive to `-b`; for
  large models on memory-limited GPUs, prefer a smaller `-b` (for example `8`)
  to bound activation memory and queue latency.

**GPU precision flags:**

- `--gpu-gemm-mode` sets the GEMM compute precision - how the matmuls are
  computed. `f32` (the default) is exact. `bf16`/`16f` run the tensor-core
  reduced-precision matmul (16-bit operands, F32 accumulation). This flag also
  gates the reduced-precision attention/FFN fast paths; with the default `f32`
  they stay off.
- `--gpu-weight-dtype` sets the weight storage precision - how weights are held
  in GPU memory. `bf16`/`f16` roughly halve the weight footprint.

Because these two settings are independent, changing only `--gpu-weight-dtype`
bf16 means your calculations (GEMMs) will still run in full, precise `F32`.
While this saves memory by compressing the weights, it won't actually speed up
your processing.

To get the full speed benefits of reduced precision during serving, you need to
set the compute mode and ensure your storage dtype matches it:

```
# bf16 inference (recommended production GPU default):
ffwd-server --cuda --model my=/path/to/model --gpu-gemm-mode bf16 --gpu-weight-dtype bf16
# fp16 inference:
ffwd-server --cuda --model my=/path/to/model --gpu-gemm-mode 16f  --gpu-weight-dtype f16
```

## API compatibility modes

Each model serves in one of two output modes, chosen per model with
`:api=openai` (the default) or `:api=perplexity`. The mode fixes the whole
output contract for that model's endpoint, not just a default encoding:

| Mode         | Default       | Accepted encodings                      | `float` returns              |
| ------------ | ------------- | --------------------------------------- | ---------------------------- |
| `openai`     | `float`       | `float`, `base64`                       | the true float32 vector      |
| `perplexity` | `base64_int8` | `base64_int8`, `base64_binary`, `float` | the int8 vector, dequantized |

`perplexity` exists because some models (such as pplx-embed) are int8-native:
the embedding _is_ the tanh-quantized int8 vector, so even a `float` response is
that vector dequantized, not a separate full-precision one.

- `base64_int8` - `round(tanh(x) * 127)` after Matryoshka truncation, base64 of
  the signed int8 bytes.
- `base64_binary` - sign bits packed LSB-first; requires `dimensions` divisible
  by 8.
- `float` (perplexity) - the int8 values as floats (`int8 / 128`), lossy.
- `base64` (openai) - the raw little-endian float32 bytes, base64-encoded,
  lossless.

A request whose `encoding_format` is not in the model's accepted set returns
`422`. pplx-embed vectors are unnormalized - always compare with cosine
similarity; the core API exposes `ffwd_l2_normalize()` for clients that need an
inner-product index.

## API

Every request body takes a `model` label and returns results in input order.

| Endpoint                            | Returns                          |
| ----------------------------------- | -------------------------------- |
| `POST /v1/embeddings`               | one pooled embedding per input   |
| `POST /v1/contextualizedembeddings` | one embedding per document chunk |
| `POST /v1/rerank`                   | documents ranked by MaxSim score |

### POST /v1/embeddings

```bash
curl http://127.0.0.1:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen","input":["What is the capital of France?",
                               "Paris is the capital of France."]}'
```

```json
{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.013, -0.021] },
    { "object": "embedding", "index": 1, "embedding": [0.046, 0.009] }
  ],
  "model": "qwen",
  "usage": { "prompt_tokens": 16, "total_tokens": 16 }
}
```

| Field             | Type            | Description                                                           |
| ----------------- | --------------- | --------------------------------------------------------------------- |
| `input`           | string or array | One text, or up to 512 texts. Required.                               |
| `dimensions`      | integer         | Truncate the embedding, from the configured `min_dim` up to its size. |
| `encoding_format` | string          | Output encoding; follows the model's API compatibility mode.          |

### POST /v1/contextualizedembeddings

`input` is a list of documents, each a list of chunk strings. Every chunk is
embedded with its document as context, and results are grouped the same way: one
list per document, one embedding per chunk, in order.

```bash
curl http://127.0.0.1:8000/v1/contextualizedembeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"pplx-context",
       "input":[["Intro paragraph.","Methods paragraph."],
                ["A single-chunk note."]]}'
```

```json
{
  "object": "list",
  "data": [
    {
      "object": "list",
      "index": 0,
      "data": [
        { "object": "embedding", "index": 0, "embedding": "<base64>" },
        { "object": "embedding", "index": 1, "embedding": "<base64>" }
      ]
    },
    {
      "object": "list",
      "index": 1,
      "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }]
    }
  ],
  "model": "pplx-context",
  "usage": { "prompt_tokens": 12, "total_tokens": 12 }
}
```

Documents are concatenated with the model tokenizer's separator for context;
separator tokens are excluded from chunk pooling.

### POST /v1/rerank

Scores each document against the query with token-level MaxSim and returns
documents ranked by descending raw score.

```bash
curl http://127.0.0.1:8000/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{"model":"pplx-late","query":"scientific curiosity",
       "documents":["Scientists explore from curiosity.","SQLite stores data."]}'
```

```json
{
  "object": "list",
  "model": "pplx-late",
  "results": [
    { "index": 0, "relevance_score": 2.14 },
    { "index": 1, "relevance_score": 0.87 }
  ],
  "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 }
}
```

- `documents` takes up to 1000 strings.
- `top_n` (alias `top_k`) caps the number of results (default: all).
- `return_documents: true` echoes each document text.
- Scores are raw MaxSim values, not probabilities.

### Limits and errors

- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 120000
  tokens total, and a 64 MiB body.
- Each input (or packed contextual document, or rerank text) is capped at the
  smallest of: the model's positional capacity (for example 512 for BERT
  encoders), `--max-batch-tokens`, and 32768. Inputs past the cap return `422`
  (`value_error.context_length`, the message states the cap), or are truncated
  to it when the server runs with `--auto-truncate`. Truncation preserves the
  model's trailing special token. The server logs each model's effective cap at
  startup.
- Invalid input returns `422` with a `detail` list naming the offending field.
  Unknown labels, and labels loaded for another endpoint, also return `422`.
- Beyond `--max-concurrent-requests` in-flight requests, new requests return
  `429` immediately instead of queueing; retry with backoff.

## Acknowledgements

- This project started as a fork of
  [antirez/qwen-asr](https://github.com/antirez/qwen-asr). Most of it has been
  rewritten since then, but the early kernels and some of the structure come
  from that codebase.
- The event loop library in `deps/ae` comes from
  [Redis](https://github.com/redis/redis). Thanks to Salvatore Sanfilippo
  (antirez) and the Redis contributors.
- Many thanks to the authors of all the open-weight models that ffwd supports.