aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-14 18:54:54 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-15 03:01:04 +0200
commit6aeb68ee1ac276bfa87d5fabbbb33287cdde89d9 (patch)
tree76035d66fa990c3bff6288d1a01d46a8a7ef4a5c
parent5228c61d39a7486b0a67579f4ed18222c5043da0 (diff)
downloadffwd-6aeb68ee1ac276bfa87d5fabbbb33287cdde89d9.tar.gz
ffwd-6aeb68ee1ac276bfa87d5fabbbb33287cdde89d9.zip
server: document HTTP API and add openapi.yaml spec
Diffstat (limited to '')
-rw-r--r--README.md208
-rw-r--r--tools/openapi.yaml270
2 files changed, 434 insertions, 44 deletions
diff --git a/README.md b/README.md
index 715d28d..1caf581 100644
--- a/README.md
+++ b/README.md
@@ -1,20 +1,16 @@
# embed.c
-`embed.c` implements inference for:
-
-- [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed)
- dense, contextual, and late-interaction models
-- [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding)
- causal, last-token pooled models
-
-Supported platforms are CUDA/cuBLAS on Linux/NVIDIA, Apple Silicon MLX, and
-CPU/BLAS.
+`embed.c` implements inference for the
+[pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) and
+[Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding) text
+embedding models. It also serves these models via an
+OpenAI/Perplexity-compatible HTTP API.
## Building
**Prerequisites**:
-- On Linux, install `libcjson-dev`, `libcjson1`, and `libopenblas-dev`.
+- On Linux, install `libcjson-dev`, and `libopenblas-dev`.
- On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew.
**Makefile targets**:
@@ -23,30 +19,34 @@ CPU/BLAS.
- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS,
OpenBLAS on Linux).
-## Usage
-
-### embed
+## embed
Pass one text to get its embedding, two or more to get a cosine similarity
matrix:
```bash
-./embed -d ./model "What is the capital of France?"
-./embed -d ./model "What is the capital of France?" \
+./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?"
+./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \
"Paris is the capital of France." \
"Berlin is the capital of Germany."
```
-Qwen3 retrieval queries benefit from an explicit task instruction. Documents are
-embedded directly:
+With Qwen3, prefix the retrieval query with a task instruction and embed
+documents as-is. Pass them together to rank the documents against the query (the
+query is the first row of the similarity matrix):
```bash
+# arg 1: the query, with Qwen3's instruction prefix
+# args 2+: documents
./embed -d ./Qwen3-Embedding-0.6B \
- $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?'
+ $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \
+ "Beijing is the capital of China." \
+ "The Great Wall is a famous landmark."
```
-The Qwen3 tokenizer terminal token is appended automatically by the CLI and
-server. Qwen3 embeddings are L2-normalized; pplx embeddings remain unnormalized.
+> pplx-embed vectors are unnormalized, so rank them with cosine similarity.
+> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot
+> product give the same ranking.
Pipe in lines and use `--stream` to get one JSON embedding per line:
@@ -61,51 +61,167 @@ similarity matrix.
Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is
CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output.
-### embed-server
+## embed-server
-Serves the HTTP API. Takes one or more `--model ID=DIR` pairs:
+Serves the Perplexity/OpenAI-compatible HTTP API. Takes one or more
+`--model ID=DIR` pairs:
```bash
./embed-server \
--model pplx-embed-v1-0.6b=./model \
+ --model pplx-embed-context-v1-0.6b=./context-model \
--model Qwen3-Embedding-0.6B=./qwen3-model \
--port 8000
```
-Perplexity API compatible endpoints:
+### API
+
+Every request body takes a `model` parameter and returns results in input order.
+
+| Endpoint | Returns |
+| ----------------------------------- | -------------------------------- |
+| `POST /v1/embeddings` | one pooled embedding per input |
+| `POST /v1/contextualizedembeddings` | one embedding per document chunk |
+| `POST /v1/rerank` | documents ranked by MaxSim score |
+
+#### POST /v1/embeddings
+
+```bash
+curl http://127.0.0.1:8000/v1/embeddings \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen3-Embedding-0.6B",
+ "input": [
+ "What is the capital of France?",
+ "Paris is the capital of France."
+ ]
+ }'
+```
+
+Returns a list of embeddings, one per input text:
+
+```json
+{
+ "object": "list",
+ "data": [
+ {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]},
+ {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]}
+ ],
+ "model": "Qwen3-Embedding-0.6B",
+ "usage": {"prompt_tokens": 16, "total_tokens": 16}
+}
+```
-- `POST /v1/embeddings` — pooled embeddings
-- `POST /v1/contextualizedembeddings` — contextual document embeddings
+| Field | Type | Description |
+| ----------------- | --------------- | ------------------------------------------------------- |
+| `input` | string or array | One text, or up to 512 texts. Required. |
+| `dimensions` | integer | Truncate the embedding (Matryoshka), 32 up to its size. |
+| `encoding_format` | string | Output encoding (see below). |
-Late-interaction reranking endpoint:
+**Encoding format:**
-- `POST /v1/rerank` — late-interaction MaxSim reranking
+`encoding_format` follows the model family:
-For Qwen3-Embedding models, `/v1/embeddings` accepts `dimensions` from 32 up to
-the model's output size (1024 for `Qwen3-Embedding-0.6B`, 2560 for
-`Qwen3-Embedding-4B`, 4096 for `Qwen3-Embedding-8B`). Truncated Matryoshka
-embeddings are re-normalized before encoding.
+- Qwen3-Embedding (OpenAI-compatible) defaults to `float` (the true float32
+ vector) and also accepts `base64` (base64 of float32).
+- pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts
+ `base64_binary` and `float` (the decoded int8 view).
-`encoding_format` follows each model's API family. Qwen3-Embedding models are
-OpenAI-compatible: it defaults to `float` (the true float32 vector) and also
-accepts `base64` (base64-encoded float32). pplx-embed models default to
-`base64_int8` and also accept `base64_binary` and `float` (the decoded int8
-view). Always compare embeddings with cosine similarity.
+#### POST /v1/contextualizedembeddings
-Example:
+`input` is a list of documents, each a list of chunk strings. Every chunk is
+embedded with its document as context, and the results are grouped the same way:
+one list per document, one embedding per chunk.
```bash
-./embed-server --model pplx-embed-v1-late-0.6b=./model
-# can be tested with:
-curl -s http://127.0.0.1:8000/v1/rerank \
- -H 'Content-Type: application/json' \
+curl http://127.0.0.1:8000/v1/contextualizedembeddings \
+ -H "Content-Type: application/json" \
-d '{
- "model":"pplx-embed-v1-late-0.6b",
- "query":"scientific curiosity",
- "documents":["Scientists explore from curiosity.","SQLite stores data."]
+ "model": "pplx-embed-context-v1-0.6b",
+ "input": [
+ ["Intro paragraph.", "Methods paragraph."],
+ ["A single-chunk note."]
+ ]
}'
```
+Returns a list of chunk embeddings per document, in order:
+
+```json
+{
+ "object": "list",
+ "data": [
+ {
+ "object": "list",
+ "index": 0,
+ "data": [
+ { "object": "embedding", "index": 0, "embedding": "<base64>" },
+ { "object": "embedding", "index": 1, "embedding": "<base64>" }
+ ]
+ },
+ {
+ "object": "list",
+ "index": 1,
+ "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }]
+ }
+ ],
+ "model": "pplx-embed-context-v1-0.6b",
+ "usage": { "prompt_tokens": 12, "total_tokens": 12 }
+}
+```
+
+#### POST /v1/rerank
+
+Scores each document against the query with token-level MaxSim and returns
+documents ranked by raw score.
+
+```bash
+curl http://127.0.0.1:8000/v1/rerank \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "pplx-embed-v1-late-0.6b",
+ "query": "scientific curiosity",
+ "documents": [
+ "Scientists explore from curiosity.",
+ "SQLite stores data."
+ ]
+ }'
+```
+
+Documents ranked by descending relevance score:
+
+```json
+{
+ "object": "list",
+ "model": "pplx-embed-v1-late-0.6b",
+ "results": [
+ { "index": 0, "relevance_score": 2.14 },
+ { "index": 1, "relevance_score": 0.87 }
+ ],
+ "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 }
+}
+```
+
+- `documents` takes up to 1000 strings. `top_n` caps the number of results
+ (default: all); `return_documents: true` echoes each document text.
+- Scores are raw MaxSim values, not probabilities.
+
+#### Limits and errors
+
+- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768
+ tokens per item, 120000 tokens total, and a 64 MiB body.
+- Invalid input returns `422` with a `detail` list naming the offending field; a
+ registered model that is not loaded returns `503`.
+
+#### Deployment
+
+The server runs one inference worker and serves one in-flight request per
+connection (HTTP/1.1 keep-alive).
+
+For concurrency and availability, run several processes behind a load balancer
+and route by model ID; keep one large model per process. Terminate TLS and
+authenticate requests at the proxy.
+
## Acknowledgements
- `embed.c` was inspired by and initially forked from
@@ -120,3 +236,7 @@ curl -s http://127.0.0.1:8000/v1/rerank \
models were developed by the Perplexity AI research team.
- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part
of the Qwen model family from Alibaba.
+
+## License
+
+MIT
diff --git a/tools/openapi.yaml b/tools/openapi.yaml
new file mode 100644
index 0000000..8a4b96f
--- /dev/null
+++ b/tools/openapi.yaml
@@ -0,0 +1,270 @@
+openapi: 3.0.3
+info:
+ title: embed.c server API
+ version: "1.0.0"
+ description: >
+ HTTP API served by `embed-server` for pplx-embed and Qwen3-Embedding models.
+ The endpoints follow the Perplexity API (pplx-embed) and the Qwen/DashScope
+ OpenAI-compatible API (Qwen3-Embedding). How an embedding is encoded on the
+ wire depends on the model family; see `encoding_format`. Always compare
+ embeddings with cosine similarity.
+ license:
+ name: MIT
+servers:
+ - url: http://127.0.0.1:8000
+# Auth is optional: it applies only when the server is started with --api-key
+# (or EMBED_API_KEY). The empty requirement allows unauthenticated calls.
+security:
+ - {}
+ - bearerAuth: []
+paths:
+ /v1/embeddings:
+ post:
+ summary: Create pooled embeddings
+ operationId: createEmbeddings
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/EmbeddingsRequest" }
+ responses:
+ "200":
+ description: One embedding per input, in request order.
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/EmbeddingsResponse" }
+ "422": { $ref: "#/components/responses/ValidationError" }
+ "503": { $ref: "#/components/responses/ModelNotLoaded" }
+ /v1/contextualizedembeddings:
+ post:
+ summary: Create contextual (late-chunking) document embeddings
+ operationId: createContextualizedEmbeddings
+ description: >
+ Each document is a list of chunk strings. Chunks are embedded with
+ document-level context and returned per document, per chunk, in order.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/ContextualRequest" }
+ responses:
+ "200":
+ description: One list of chunk embeddings per document, in order.
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/ContextualResponse" }
+ "422": { $ref: "#/components/responses/ValidationError" }
+ "503": { $ref: "#/components/responses/ModelNotLoaded" }
+ /v1/rerank:
+ post:
+ summary: Late-interaction (MaxSim) reranking
+ operationId: rerank
+ description: >
+ Scores each document against the query with token-level MaxSim and
+ returns documents ranked by raw score (not a probability).
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/RerankRequest" }
+ responses:
+ "200":
+ description: Documents ranked by descending relevance score.
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/RerankResponse" }
+ "422": { $ref: "#/components/responses/ValidationError" }
+ "503": { $ref: "#/components/responses/ModelNotLoaded" }
+components:
+ securitySchemes:
+ bearerAuth:
+ type: http
+ scheme: bearer
+ description: >
+ Single shared token set with --api-key. Not a real auth system;
+ terminate TLS and authenticate at a reverse proxy in production.
+ schemas:
+ EmbeddingsRequest:
+ type: object
+ required: [model, input]
+ properties:
+ model:
+ type: string
+ example: Qwen3-Embedding-0.6B
+ input:
+ description: A single string or an array of up to 512 strings.
+ oneOf:
+ - type: string
+ - type: array
+ items: { type: string }
+ minItems: 1
+ maxItems: 512
+ encoding_format:
+ type: string
+ description: >
+ Output encoding, validated against the model's API family.
+ Qwen3-Embedding accepts `float` (default; true float32) and `base64`
+ (base64 of little-endian float32). pplx-embed accepts `base64_int8`
+ (default), `base64_binary` (sign bits, needs `dimensions` divisible
+ by 8), and `float` (the int8-decoded view). A format outside the
+ model's family returns 422.
+ enum: [float, base64, base64_int8, base64_binary]
+ dimensions:
+ type: integer
+ description: >
+ Matryoshka truncation, from the model minimum (32 for Qwen3, 128 for
+ pplx) up to the model's native dimension. Truncated vectors are
+ re-normalized.
+ EmbeddingsResponse:
+ type: object
+ properties:
+ object: { type: string, example: list }
+ data:
+ type: array
+ items: { $ref: "#/components/schemas/Embedding" }
+ model: { type: string }
+ usage: { $ref: "#/components/schemas/Usage" }
+ Embedding:
+ type: object
+ properties:
+ object: { type: string, example: embedding }
+ index: { type: integer }
+ embedding:
+ description: >
+ A float array when `encoding_format` is `float`, otherwise a base64
+ string (`base64`, `base64_int8`, or `base64_binary`).
+ oneOf:
+ - type: array
+ items: { type: number, format: float }
+ - type: string
+ ContextualRequest:
+ type: object
+ required: [model, input]
+ properties:
+ model:
+ type: string
+ example: pplx-embed-context-v1-0.6b
+ input:
+ description: Array of documents; each document is an array of chunk strings.
+ type: array
+ items:
+ type: array
+ items: { type: string }
+ minItems: 1
+ encoding_format:
+ type: string
+ enum: [float, base64, base64_int8, base64_binary]
+ dimensions: { type: integer }
+ ContextualResponse:
+ type: object
+ properties:
+ object: { type: string, example: list }
+ data:
+ type: array
+ description: One entry per document, in request order.
+ items:
+ type: object
+ properties:
+ object: { type: string, example: list }
+ index: { type: integer }
+ data:
+ type: array
+ description: One embedding per chunk, in document order.
+ items: { $ref: "#/components/schemas/Embedding" }
+ model: { type: string }
+ usage: { $ref: "#/components/schemas/Usage" }
+ RerankRequest:
+ type: object
+ required: [model, query, documents]
+ properties:
+ model:
+ type: string
+ example: pplx-embed-v1-late-0.6b
+ query: { type: string }
+ documents:
+ type: array
+ items: { type: string }
+ minItems: 1
+ maxItems: 1000
+ top_n:
+ type: integer
+ description: Number of results to return. Defaults to all. `top_k` is an alias.
+ top_k:
+ type: integer
+ return_documents:
+ type: boolean
+ default: false
+ description: Include the source document text in each result.
+ additionalProperties: false
+ RerankResponse:
+ type: object
+ properties:
+ object: { type: string, example: list }
+ model: { type: string }
+ results:
+ type: array
+ items:
+ type: object
+ properties:
+ index:
+ type: integer
+ description: Index of the document in the request array.
+ relevance_score:
+ type: number
+ format: float
+ description: Raw MaxSim score (not a probability).
+ document:
+ type: string
+ description: Present only when `return_documents` is true.
+ usage:
+ type: object
+ properties:
+ query_tokens: { type: integer }
+ document_tokens: { type: integer }
+ total_tokens: { type: integer }
+ Usage:
+ type: object
+ properties:
+ prompt_tokens: { type: integer }
+ total_tokens: { type: integer }
+ cost:
+ type: object
+ description: pplx-embed only; Qwen3-Embedding omits it.
+ properties:
+ input_cost: { type: number }
+ total_cost: { type: number }
+ currency: { type: string, example: USD }
+ ValidationDetail:
+ type: object
+ properties:
+ loc:
+ type: array
+ description: Path to the offending field, e.g. ["body","input",0].
+ items: {}
+ msg: { type: string }
+ type: { type: string }
+ ValidationError:
+ type: object
+ properties:
+ detail:
+ type: array
+ items: { $ref: "#/components/schemas/ValidationDetail" }
+ Error:
+ type: object
+ properties:
+ error:
+ type: object
+ properties:
+ message: { type: string }
+ type: { type: string }
+ responses:
+ ValidationError:
+ description: Request body failed validation.
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/ValidationError" }
+ ModelNotLoaded:
+ description: The model ID is valid for this build but is not loaded.
+ content:
+ application/json:
+ schema: { $ref: "#/components/schemas/Error" }