# embed.c `embed.c` implements inference for the [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) and [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding) text embedding models. It also serves these models via an OpenAI/Perplexity-compatible HTTP API. ## Building **Prerequisites**: - On Linux, install `libcjson-dev`, and `libopenblas-dev`. - On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew. **Makefile targets**: - Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux). - Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS, OpenBLAS on Linux). ## embed Pass one text to get its embedding, two or more to get a cosine similarity matrix: ```bash ./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" ./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \ "Paris is the capital of France." \ "Berlin is the capital of Germany." ``` With Qwen3, prefix the retrieval query with a task instruction and embed documents as-is. Pass them together to rank the documents against the query (the query is the first row of the similarity matrix): ```bash # arg 1: the query, with Qwen3's instruction prefix # args 2+: documents ./embed -d ./Qwen3-Embedding-0.6B \ $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \ "Beijing is the capital of China." \ "The Great Wall is a famous landmark." ``` > pplx-embed vectors are unnormalized, so rank them with cosine similarity. > Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot > product give the same ranking. Pipe in lines and use `--stream` to get one JSON embedding per line: ```bash cat texts.txt | ./embed -d ./model --stream -b 8 # {"embedding":[...],"dim":2048,"tokens":7,"ms":12.3,"workspace_bytes":1048576} ``` Without `--stream`, reading from stdin accumulates all lines then prints the similarity matrix. Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output. ## embed-server Serves the Perplexity/OpenAI-compatible HTTP API. Takes one or more `--model ID=DIR` pairs: ```bash ./embed-server \ --model pplx-embed-v1-0.6b=./model \ --model pplx-embed-context-v1-0.6b=./context-model \ --model Qwen3-Embedding-0.6B=./qwen3-model \ --port 8000 ``` ### API Every request body takes a `model` parameter and returns results in input order. | Endpoint | Returns | | ----------------------------------- | -------------------------------- | | `POST /v1/embeddings` | one pooled embedding per input | | `POST /v1/contextualizedembeddings` | one embedding per document chunk | | `POST /v1/rerank` | documents ranked by MaxSim score | #### POST /v1/embeddings ```bash curl http://127.0.0.1:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-Embedding-0.6B", "input": [ "What is the capital of France?", "Paris is the capital of France." ] }' ``` Returns a list of embeddings, one per input text: ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]}, {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]} ], "model": "Qwen3-Embedding-0.6B", "usage": {"prompt_tokens": 16, "total_tokens": 16} } ``` | Field | Type | Description | | ----------------- | --------------- | ------------------------------------------------------- | | `input` | string or array | One text, or up to 512 texts. Required. | | `dimensions` | integer | Truncate the embedding (Matryoshka), 32 up to its size. | | `encoding_format` | string | Output encoding (see below). | **Encoding format:** `encoding_format` follows the model family: - Qwen3-Embedding (OpenAI-compatible) defaults to `float` (the true float32 vector) and also accepts `base64` (base64 of float32). - pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts `base64_binary` and `float` (the decoded int8 view). #### POST /v1/contextualizedembeddings `input` is a list of documents, each a list of chunk strings. Every chunk is embedded with its document as context, and the results are grouped the same way: one list per document, one embedding per chunk. ```bash curl http://127.0.0.1:8000/v1/contextualizedembeddings \ -H "Content-Type: application/json" \ -d '{ "model": "pplx-embed-context-v1-0.6b", "input": [ ["Intro paragraph.", "Methods paragraph."], ["A single-chunk note."] ] }' ``` Returns a list of chunk embeddings per document, in order: ```json { "object": "list", "data": [ { "object": "list", "index": 0, "data": [ { "object": "embedding", "index": 0, "embedding": "" }, { "object": "embedding", "index": 1, "embedding": "" } ] }, { "object": "list", "index": 1, "data": [{ "object": "embedding", "index": 0, "embedding": "" }] } ], "model": "pplx-embed-context-v1-0.6b", "usage": { "prompt_tokens": 12, "total_tokens": 12 } } ``` #### POST /v1/rerank Scores each document against the query with token-level MaxSim and returns documents ranked by raw score. ```bash curl http://127.0.0.1:8000/v1/rerank \ -H "Content-Type: application/json" \ -d '{ "model": "pplx-embed-v1-late-0.6b", "query": "scientific curiosity", "documents": [ "Scientists explore from curiosity.", "SQLite stores data." ] }' ``` Documents ranked by descending relevance score: ```json { "object": "list", "model": "pplx-embed-v1-late-0.6b", "results": [ { "index": 0, "relevance_score": 2.14 }, { "index": 1, "relevance_score": 0.87 } ], "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 } } ``` - `documents` takes up to 1000 strings. `top_n` caps the number of results (default: all); `return_documents: true` echoes each document text. - Scores are raw MaxSim values, not probabilities. #### Limits and errors - A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768 tokens per item, 120000 tokens total, and a 64 MiB body. - Invalid input returns `422` with a `detail` list naming the offending field; a registered model that is not loaded returns `503`. #### Deployment The server runs one inference worker and serves one in-flight request per connection (HTTP/1.1 keep-alive). For concurrency and availability, run several processes behind a load balancer and route by model ID; keep one large model per process. Terminate TLS and authenticate requests at the proxy. ## Acknowledgements - `embed.c` was inspired by and initially forked from [antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been mostly rewritten, including a major CUDA upgrade, but the kernels and some structural ideas are derived from that codebase. - The server event loop library in `deps/ae` is extracted from [Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo (antirez) and Redis contributors. - The [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval) models were developed by the Perplexity AI research team. - The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part of the Qwen model family from Alibaba. ## License MIT