diff options
| -rw-r--r-- | README.md | 208 | ||||
| -rw-r--r-- | tools/openapi.yaml | 270 |
2 files changed, 434 insertions, 44 deletions
@@ -1,20 +1,16 @@ # embed.c -`embed.c` implements inference for: - -- [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) - dense, contextual, and late-interaction models -- [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding) - causal, last-token pooled models - -Supported platforms are CUDA/cuBLAS on Linux/NVIDIA, Apple Silicon MLX, and -CPU/BLAS. +`embed.c` implements inference for the +[pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) and +[Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding) text +embedding models. It also serves these models via an +OpenAI/Perplexity-compatible HTTP API. ## Building **Prerequisites**: -- On Linux, install `libcjson-dev`, `libcjson1`, and `libopenblas-dev`. +- On Linux, install `libcjson-dev`, and `libopenblas-dev`. - On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew. **Makefile targets**: @@ -23,30 +19,34 @@ CPU/BLAS. - Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS, OpenBLAS on Linux). -## Usage - -### embed +## embed Pass one text to get its embedding, two or more to get a cosine similarity matrix: ```bash -./embed -d ./model "What is the capital of France?" -./embed -d ./model "What is the capital of France?" \ +./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" +./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \ "Paris is the capital of France." \ "Berlin is the capital of Germany." ``` -Qwen3 retrieval queries benefit from an explicit task instruction. Documents are -embedded directly: +With Qwen3, prefix the retrieval query with a task instruction and embed +documents as-is. Pass them together to rank the documents against the query (the +query is the first row of the similarity matrix): ```bash +# arg 1: the query, with Qwen3's instruction prefix +# args 2+: documents ./embed -d ./Qwen3-Embedding-0.6B \ - $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' + $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \ + "Beijing is the capital of China." \ + "The Great Wall is a famous landmark." ``` -The Qwen3 tokenizer terminal token is appended automatically by the CLI and -server. Qwen3 embeddings are L2-normalized; pplx embeddings remain unnormalized. +> pplx-embed vectors are unnormalized, so rank them with cosine similarity. +> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot +> product give the same ranking. Pipe in lines and use `--stream` to get one JSON embedding per line: @@ -61,51 +61,167 @@ similarity matrix. Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output. -### embed-server +## embed-server -Serves the HTTP API. Takes one or more `--model ID=DIR` pairs: +Serves the Perplexity/OpenAI-compatible HTTP API. Takes one or more +`--model ID=DIR` pairs: ```bash ./embed-server \ --model pplx-embed-v1-0.6b=./model \ + --model pplx-embed-context-v1-0.6b=./context-model \ --model Qwen3-Embedding-0.6B=./qwen3-model \ --port 8000 ``` -Perplexity API compatible endpoints: +### API + +Every request body takes a `model` parameter and returns results in input order. + +| Endpoint | Returns | +| ----------------------------------- | -------------------------------- | +| `POST /v1/embeddings` | one pooled embedding per input | +| `POST /v1/contextualizedembeddings` | one embedding per document chunk | +| `POST /v1/rerank` | documents ranked by MaxSim score | + +#### POST /v1/embeddings + +```bash +curl http://127.0.0.1:8000/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3-Embedding-0.6B", + "input": [ + "What is the capital of France?", + "Paris is the capital of France." + ] + }' +``` + +Returns a list of embeddings, one per input text: + +```json +{ + "object": "list", + "data": [ + {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]}, + {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]} + ], + "model": "Qwen3-Embedding-0.6B", + "usage": {"prompt_tokens": 16, "total_tokens": 16} +} +``` -- `POST /v1/embeddings` — pooled embeddings -- `POST /v1/contextualizedembeddings` — contextual document embeddings +| Field | Type | Description | +| ----------------- | --------------- | ------------------------------------------------------- | +| `input` | string or array | One text, or up to 512 texts. Required. | +| `dimensions` | integer | Truncate the embedding (Matryoshka), 32 up to its size. | +| `encoding_format` | string | Output encoding (see below). | -Late-interaction reranking endpoint: +**Encoding format:** -- `POST /v1/rerank` — late-interaction MaxSim reranking +`encoding_format` follows the model family: -For Qwen3-Embedding models, `/v1/embeddings` accepts `dimensions` from 32 up to -the model's output size (1024 for `Qwen3-Embedding-0.6B`, 2560 for -`Qwen3-Embedding-4B`, 4096 for `Qwen3-Embedding-8B`). Truncated Matryoshka -embeddings are re-normalized before encoding. +- Qwen3-Embedding (OpenAI-compatible) defaults to `float` (the true float32 + vector) and also accepts `base64` (base64 of float32). +- pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts + `base64_binary` and `float` (the decoded int8 view). -`encoding_format` follows each model's API family. Qwen3-Embedding models are -OpenAI-compatible: it defaults to `float` (the true float32 vector) and also -accepts `base64` (base64-encoded float32). pplx-embed models default to -`base64_int8` and also accept `base64_binary` and `float` (the decoded int8 -view). Always compare embeddings with cosine similarity. +#### POST /v1/contextualizedembeddings -Example: +`input` is a list of documents, each a list of chunk strings. Every chunk is +embedded with its document as context, and the results are grouped the same way: +one list per document, one embedding per chunk. ```bash -./embed-server --model pplx-embed-v1-late-0.6b=./model -# can be tested with: -curl -s http://127.0.0.1:8000/v1/rerank \ - -H 'Content-Type: application/json' \ +curl http://127.0.0.1:8000/v1/contextualizedembeddings \ + -H "Content-Type: application/json" \ -d '{ - "model":"pplx-embed-v1-late-0.6b", - "query":"scientific curiosity", - "documents":["Scientists explore from curiosity.","SQLite stores data."] + "model": "pplx-embed-context-v1-0.6b", + "input": [ + ["Intro paragraph.", "Methods paragraph."], + ["A single-chunk note."] + ] }' ``` +Returns a list of chunk embeddings per document, in order: + +```json +{ + "object": "list", + "data": [ + { + "object": "list", + "index": 0, + "data": [ + { "object": "embedding", "index": 0, "embedding": "<base64>" }, + { "object": "embedding", "index": 1, "embedding": "<base64>" } + ] + }, + { + "object": "list", + "index": 1, + "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }] + } + ], + "model": "pplx-embed-context-v1-0.6b", + "usage": { "prompt_tokens": 12, "total_tokens": 12 } +} +``` + +#### POST /v1/rerank + +Scores each document against the query with token-level MaxSim and returns +documents ranked by raw score. + +```bash +curl http://127.0.0.1:8000/v1/rerank \ + -H "Content-Type: application/json" \ + -d '{ + "model": "pplx-embed-v1-late-0.6b", + "query": "scientific curiosity", + "documents": [ + "Scientists explore from curiosity.", + "SQLite stores data." + ] + }' +``` + +Documents ranked by descending relevance score: + +```json +{ + "object": "list", + "model": "pplx-embed-v1-late-0.6b", + "results": [ + { "index": 0, "relevance_score": 2.14 }, + { "index": 1, "relevance_score": 0.87 } + ], + "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 } +} +``` + +- `documents` takes up to 1000 strings. `top_n` caps the number of results + (default: all); `return_documents: true` echoes each document text. +- Scores are raw MaxSim values, not probabilities. + +#### Limits and errors + +- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768 + tokens per item, 120000 tokens total, and a 64 MiB body. +- Invalid input returns `422` with a `detail` list naming the offending field; a + registered model that is not loaded returns `503`. + +#### Deployment + +The server runs one inference worker and serves one in-flight request per +connection (HTTP/1.1 keep-alive). + +For concurrency and availability, run several processes behind a load balancer +and route by model ID; keep one large model per process. Terminate TLS and +authenticate requests at the proxy. + ## Acknowledgements - `embed.c` was inspired by and initially forked from @@ -120,3 +236,7 @@ curl -s http://127.0.0.1:8000/v1/rerank \ models were developed by the Perplexity AI research team. - The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part of the Qwen model family from Alibaba. + +## License + +MIT diff --git a/tools/openapi.yaml b/tools/openapi.yaml new file mode 100644 index 0000000..8a4b96f --- /dev/null +++ b/tools/openapi.yaml @@ -0,0 +1,270 @@ +openapi: 3.0.3 +info: + title: embed.c server API + version: "1.0.0" + description: > + HTTP API served by `embed-server` for pplx-embed and Qwen3-Embedding models. + The endpoints follow the Perplexity API (pplx-embed) and the Qwen/DashScope + OpenAI-compatible API (Qwen3-Embedding). How an embedding is encoded on the + wire depends on the model family; see `encoding_format`. Always compare + embeddings with cosine similarity. + license: + name: MIT +servers: + - url: http://127.0.0.1:8000 +# Auth is optional: it applies only when the server is started with --api-key +# (or EMBED_API_KEY). The empty requirement allows unauthenticated calls. +security: + - {} + - bearerAuth: [] +paths: + /v1/embeddings: + post: + summary: Create pooled embeddings + operationId: createEmbeddings + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/EmbeddingsRequest" } + responses: + "200": + description: One embedding per input, in request order. + content: + application/json: + schema: { $ref: "#/components/schemas/EmbeddingsResponse" } + "422": { $ref: "#/components/responses/ValidationError" } + "503": { $ref: "#/components/responses/ModelNotLoaded" } + /v1/contextualizedembeddings: + post: + summary: Create contextual (late-chunking) document embeddings + operationId: createContextualizedEmbeddings + description: > + Each document is a list of chunk strings. Chunks are embedded with + document-level context and returned per document, per chunk, in order. + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/ContextualRequest" } + responses: + "200": + description: One list of chunk embeddings per document, in order. + content: + application/json: + schema: { $ref: "#/components/schemas/ContextualResponse" } + "422": { $ref: "#/components/responses/ValidationError" } + "503": { $ref: "#/components/responses/ModelNotLoaded" } + /v1/rerank: + post: + summary: Late-interaction (MaxSim) reranking + operationId: rerank + description: > + Scores each document against the query with token-level MaxSim and + returns documents ranked by raw score (not a probability). + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/RerankRequest" } + responses: + "200": + description: Documents ranked by descending relevance score. + content: + application/json: + schema: { $ref: "#/components/schemas/RerankResponse" } + "422": { $ref: "#/components/responses/ValidationError" } + "503": { $ref: "#/components/responses/ModelNotLoaded" } +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + description: > + Single shared token set with --api-key. Not a real auth system; + terminate TLS and authenticate at a reverse proxy in production. + schemas: + EmbeddingsRequest: + type: object + required: [model, input] + properties: + model: + type: string + example: Qwen3-Embedding-0.6B + input: + description: A single string or an array of up to 512 strings. + oneOf: + - type: string + - type: array + items: { type: string } + minItems: 1 + maxItems: 512 + encoding_format: + type: string + description: > + Output encoding, validated against the model's API family. + Qwen3-Embedding accepts `float` (default; true float32) and `base64` + (base64 of little-endian float32). pplx-embed accepts `base64_int8` + (default), `base64_binary` (sign bits, needs `dimensions` divisible + by 8), and `float` (the int8-decoded view). A format outside the + model's family returns 422. + enum: [float, base64, base64_int8, base64_binary] + dimensions: + type: integer + description: > + Matryoshka truncation, from the model minimum (32 for Qwen3, 128 for + pplx) up to the model's native dimension. Truncated vectors are + re-normalized. + EmbeddingsResponse: + type: object + properties: + object: { type: string, example: list } + data: + type: array + items: { $ref: "#/components/schemas/Embedding" } + model: { type: string } + usage: { $ref: "#/components/schemas/Usage" } + Embedding: + type: object + properties: + object: { type: string, example: embedding } + index: { type: integer } + embedding: + description: > + A float array when `encoding_format` is `float`, otherwise a base64 + string (`base64`, `base64_int8`, or `base64_binary`). + oneOf: + - type: array + items: { type: number, format: float } + - type: string + ContextualRequest: + type: object + required: [model, input] + properties: + model: + type: string + example: pplx-embed-context-v1-0.6b + input: + description: Array of documents; each document is an array of chunk strings. + type: array + items: + type: array + items: { type: string } + minItems: 1 + encoding_format: + type: string + enum: [float, base64, base64_int8, base64_binary] + dimensions: { type: integer } + ContextualResponse: + type: object + properties: + object: { type: string, example: list } + data: + type: array + description: One entry per document, in request order. + items: + type: object + properties: + object: { type: string, example: list } + index: { type: integer } + data: + type: array + description: One embedding per chunk, in document order. + items: { $ref: "#/components/schemas/Embedding" } + model: { type: string } + usage: { $ref: "#/components/schemas/Usage" } + RerankRequest: + type: object + required: [model, query, documents] + properties: + model: + type: string + example: pplx-embed-v1-late-0.6b + query: { type: string } + documents: + type: array + items: { type: string } + minItems: 1 + maxItems: 1000 + top_n: + type: integer + description: Number of results to return. Defaults to all. `top_k` is an alias. + top_k: + type: integer + return_documents: + type: boolean + default: false + description: Include the source document text in each result. + additionalProperties: false + RerankResponse: + type: object + properties: + object: { type: string, example: list } + model: { type: string } + results: + type: array + items: + type: object + properties: + index: + type: integer + description: Index of the document in the request array. + relevance_score: + type: number + format: float + description: Raw MaxSim score (not a probability). + document: + type: string + description: Present only when `return_documents` is true. + usage: + type: object + properties: + query_tokens: { type: integer } + document_tokens: { type: integer } + total_tokens: { type: integer } + Usage: + type: object + properties: + prompt_tokens: { type: integer } + total_tokens: { type: integer } + cost: + type: object + description: pplx-embed only; Qwen3-Embedding omits it. + properties: + input_cost: { type: number } + total_cost: { type: number } + currency: { type: string, example: USD } + ValidationDetail: + type: object + properties: + loc: + type: array + description: Path to the offending field, e.g. ["body","input",0]. + items: {} + msg: { type: string } + type: { type: string } + ValidationError: + type: object + properties: + detail: + type: array + items: { $ref: "#/components/schemas/ValidationDetail" } + Error: + type: object + properties: + error: + type: object + properties: + message: { type: string } + type: { type: string } + responses: + ValidationError: + description: Request body failed validation. + content: + application/json: + schema: { $ref: "#/components/schemas/ValidationError" } + ModelNotLoaded: + description: The model ID is valid for this build but is not loaded. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } |
