# ffwd-server ffwd HTTP server. ## Quickstart ```bash make # use make gpu for GPU ./ffwd-server \ --model pplx=./pplx-model:api=perplexity:min_dim=128 \ --contextual-model pplx-context=./pplx-context-model:api=perplexity \ --late-model pplx-late=./pplx-late-model \ --model qwen=./qwen3-model \ --port 8000 ``` ```bash curl http://127.0.0.1:8000/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{"model":"qwen","input":["What is the capital of France?", "Paris is the capital of France."]}' ``` ## Loading models Each load flag binds a `LABEL` to a model directory and enables one endpoint. The flags are repeatable, so several models can share a server. | Flag | Enables | | ------------------------------ | ----------------------------------- | | `--model LABEL=DIR` | `POST /v1/embeddings` | | `--contextual-model LABEL=DIR` | `POST /v1/contextualizedembeddings` | | `--late-model LABEL=DIR` | `POST /v1/rerank` | Each flag also accepts inline, colon-separated per-model options appended to the directory: - `:api=openai|perplexity` - the output-compatibility mode (see below). - `:min_dim=N` - the Matryoshka truncation floor; requests may ask for any `dimensions` from `N` up to the model's full size. - `:query=STR` - a query-prompt override for models that prepend one. For example: `--model qwen=./qwen3-model:api=openai:min_dim=32`. Loading several labels keeps several complete model runtimes resident at once. Large models need substantial memory; on a memory-limited host, host one large model per process and route requests across processes rather than loading several into one server. ## Options Run `./ffwd-server --help` for the exact, current list. Some important flags: | Flag | Default | Meaning | | -------------------------- | ----------- | --------------------------------------------------------------------------------- | | `--host HOST` | `127.0.0.1` | Bind address. | | `--port PORT` | - | Listen port. | | `-t, --threads N` | backend | Worker thread count for CPU inference. | | `-b, --batch-size N` | `32` | Max texts or contextual documents per inference micro-batch. | | `--max-batch-tokens N` | `16384` | Token budget per batch (packed for CPU, padded dense rows for MLX). | | `--batch-wait-us N` | backend | Micro-batch deadline from the first request's arrival (CPU/MLX `0`, CUDA `1000`). | | `--memory-utilization F` | `0.90` | Fraction of physical memory the MLX load preflight may plan for. | | `--gpu-quant-bits N` | off | MLX: quantize transformer linear weights at load. Opt-in, approximate. | | `--gpu-quant-group-size N` | `64` | MLX quantization group size. | | `--gpu-gemm-mode MODE` | `f32` | CUDA GEMM compute: `f32` (exact), or `tf32`/`bf16`/`16f` (reduced). | | `--gpu-weight-dtype DTYPE` | `f32` | CUDA weight storage: `f32`, or `bf16` (half the weight memory). | | `--api-key TOKEN` | off | Require `Authorization: Bearer TOKEN`. `FFWD_API_KEY` is also honored. | | `-v, --verbose` | off | Verbose logging. | **Batching:** - `-b 32` is a balanced default for concurrent short requests. - Long-document throughput is compute-bound and largely insensitive to `-b`; for large models on memory-limited GPUs, prefer a smaller `-b` (for example `8`) to bound activation memory and queue latency. **GPU precision flags:** - `--gpu-gemm-mode` sets the GEMM compute precision - how the matmuls are computed. `f32` (the default) is exact. `bf16`/`16f` run the tensor-core reduced-precision matmul (16-bit operands, F32 accumulation). This flag also gates the reduced-precision attention/FFN fast paths; with the default `f32` they stay off. - `--gpu-weight-dtype` sets the weight storage precision - how weights are held in GPU memory. `bf16`/`f16` roughly halve the weight footprint. Because these two settings are independent, changing only `--gpu-weight-dtype` bf16 means your calculations (GEMMs) will still run in full, precise `F32`. While this saves memory by compressing the weights, it won't actually speed up your processing. To get the full speed benefits of reduced precision during serving, you need to set the compute mode and ensure your storage dtype matches it: ``` # bf16 inference (recommended production GPU default): ffwd-server --cuda --model my=/path/to/model --gpu-gemm-mode bf16 --gpu-weight-dtype bf16 # fp16 inference: ffwd-server --cuda --model my=/path/to/model --gpu-gemm-mode 16f --gpu-weight-dtype f16 ``` ## API compatibility modes Each model serves in one of two output modes, chosen per model with `:api=openai` (the default) or `:api=perplexity`. The mode fixes the whole output contract for that model's endpoint, not just a default encoding: | Mode | Default | Accepted encodings | `float` returns | | ------------ | ------------- | --------------------------------------- | ---------------------------- | | `openai` | `float` | `float`, `base64` | the true float32 vector | | `perplexity` | `base64_int8` | `base64_int8`, `base64_binary`, `float` | the int8 vector, dequantized | `perplexity` exists because some models (such as pplx-embed) are int8-native: the embedding _is_ the tanh-quantized int8 vector, so even a `float` response is that vector dequantized, not a separate full-precision one. - `base64_int8` - `round(tanh(x) * 127)` after Matryoshka truncation, base64 of the signed int8 bytes. - `base64_binary` - sign bits packed LSB-first; requires `dimensions` divisible by 8. - `float` (perplexity) - the int8 values as floats (`int8 / 128`), lossy. - `base64` (openai) - the raw little-endian float32 bytes, base64-encoded, lossless. A request whose `encoding_format` is not in the model's accepted set returns `422`. pplx-embed vectors are unnormalized - always compare with cosine similarity; the core API exposes `ffwd_l2_normalize()` for clients that need an inner-product index. ## API Every request body takes a `model` label and returns results in input order. | Endpoint | Returns | | ----------------------------------- | -------------------------------- | | `POST /v1/embeddings` | one pooled embedding per input | | `POST /v1/contextualizedembeddings` | one embedding per document chunk | | `POST /v1/rerank` | documents ranked by MaxSim score | ### POST /v1/embeddings ```bash curl http://127.0.0.1:8000/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{"model":"qwen","input":["What is the capital of France?", "Paris is the capital of France."]}' ``` ```json { "object": "list", "data": [ { "object": "embedding", "index": 0, "embedding": [0.013, -0.021] }, { "object": "embedding", "index": 1, "embedding": [0.046, 0.009] } ], "model": "qwen", "usage": { "prompt_tokens": 16, "total_tokens": 16 } } ``` | Field | Type | Description | | ----------------- | --------------- | --------------------------------------------------------------------- | | `input` | string or array | One text, or up to 512 texts. Required. | | `dimensions` | integer | Truncate the embedding, from the configured `min_dim` up to its size. | | `encoding_format` | string | Output encoding; follows the model's API compatibility mode. | ### POST /v1/contextualizedembeddings `input` is a list of documents, each a list of chunk strings. Every chunk is embedded with its document as context, and results are grouped the same way: one list per document, one embedding per chunk, in order. ```bash curl http://127.0.0.1:8000/v1/contextualizedembeddings \ -H 'Content-Type: application/json' \ -d '{"model":"pplx-context", "input":[["Intro paragraph.","Methods paragraph."], ["A single-chunk note."]]}' ``` ```json { "object": "list", "data": [ { "object": "list", "index": 0, "data": [ { "object": "embedding", "index": 0, "embedding": "" }, { "object": "embedding", "index": 1, "embedding": "" } ] }, { "object": "list", "index": 1, "data": [{ "object": "embedding", "index": 0, "embedding": "" }] } ], "model": "pplx-context", "usage": { "prompt_tokens": 12, "total_tokens": 12 } } ``` Documents are concatenated with the model tokenizer's separator for context; separator tokens are excluded from chunk pooling. ### POST /v1/rerank Scores each document against the query with token-level MaxSim and returns documents ranked by descending raw score. ```bash curl http://127.0.0.1:8000/v1/rerank \ -H 'Content-Type: application/json' \ -d '{"model":"pplx-late","query":"scientific curiosity", "documents":["Scientists explore from curiosity.","SQLite stores data."]}' ``` ```json { "object": "list", "model": "pplx-late", "results": [ { "index": 0, "relevance_score": 2.14 }, { "index": 1, "relevance_score": 0.87 } ], "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 } } ``` - `documents` takes up to 1000 strings. - `top_n` (alias `top_k`) caps the number of results (default: all). - `return_documents: true` echoes each document text. - Scores are raw MaxSim values, not probabilities. ### Limits and errors - A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768 tokens per item, 120000 tokens total, and a 64 MiB body. - Invalid input returns `422` with a `detail` list naming the offending field. Unknown labels, and labels loaded for another endpoint, also return `422`.