diff options
| -rw-r--r-- | README.md | 307 | ||||
| -rw-r--r-- | tools/cli/README.md | 57 |
2 files changed, 92 insertions, 272 deletions
@@ -1,287 +1,50 @@ -# embed.c +# ffwd -`embed.c` implements high-performance embedding inference for Hugging Face -safetensors checkpoints, with CPU, Apple MLX, and NVIDIA CUDA backends. +ffwd is a library and toolkit for text embedding inference. -Tested model families include pplx-embed, Qwen3-Embedding, GTE-Qwen2, BERT/BGE -encoders, and XLM-R encoders such as multilingual-E5 and Snowflake Arctic. -Weights load directly from F32, BF16, or F16 safetensors. +It works with a variety of transformer-based models and runs natively on Apple +Silicon and NVIDIA GPUs, with CPU fallback on x86/ARM. -The engine handles three transformer blocks — the Qwen3 block (pplx-embed and -Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT/RoBERTa block -(MiniLM, BGE, and the XLM-R multilingual encoders E5 and Arctic) — and selects -the tokenizer from the model files: byte-level BPE for the Qwen families, -WordPiece for BERT, and SentencePiece (Unigram) for XLM-R. Pooling and -L2-normalization are read from the checkpoint's Sentence Transformers config -(`1_Pooling/config.json` and `modules.json`), so a model's pooling mode does not -have to be hardcoded. pplx-embed vectors are unnormalized int8 (rank them by -cosine similarity); the other families emit L2-normalized float32. +## Supported models -## Building - -**Prerequisites**: - -- On Linux, install `libopenblas-dev`. -- On macOS, install `mlx` and `mlx-c` via Homebrew. - -You also need `libcjson-dev`/`cjson` to build the server; the CLI and the shared -library can be built without it. - -**Makefile targets**: - -- Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux). -- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS, - OpenBLAS on Linux). - -## embed - -Pass one text to get its embedding, two or more to get a cosine similarity -matrix: - -```bash -# one text: the embedding as a line of space-separated floats -./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" - -# two or more: a cosine similarity matrix, one row per line, no labels -./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \ - "Paris is the capital of France." \ - "Berlin is the capital of Germany." -# 1.000000 0.719822 0.422054 -# 0.719822 1.000000 0.601722 -# 0.422054 0.601722 1.000000 -``` - -With Qwen3, prefix the retrieval query with a task instruction and embed -documents as-is. Pass them together to rank the documents against the query (the -query is the first row of the similarity matrix): - -```bash -# arg 1: the query, with Qwen3's instruction prefix -# args 2+: documents -./embed -d ./Qwen3-Embedding-0.6B \ - $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \ - "Beijing is the capital of China." \ - "The Great Wall is a famous landmark." -``` - -> pplx-embed vectors are unnormalized, so rank them with cosine similarity. -> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot -> product give the same ranking. - -Pipe in lines and use `--stream` to get one JSON embedding per line: - -```bash -cat texts.txt | ./embed -d ./model --stream -b 8 -# {"embedding":[...],"dim":2560,"tokens":7,"ms":12.3,"workspace_bytes":1048576} -``` - -Without `--stream`, reading from stdin accumulates all lines then prints the -similarity matrix. - -Add `--json` to emit JSON. One text prints an array of floats; two or more print -an array of rows (the matrix): - -```bash -./embed -d ./model --json "cat" "dog" -# [[1.000000,0.380861],[0.380861,1.000000]] -``` - -Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is -CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output. -`-V`/`--version` prints the version; `--build-info` also reports the commit, -build date, target, and compiler. - -## embed-server - -Serves the Perplexity/OpenAI-compatible HTTP API. Each loaded model gets an -operator-chosen label, and requests use that label in the `model` field. The -label is only a routing name; architecture, pooling, normalization, tokenizer, -and dimensionality are read from the model directory. - -Use a separate load flag for each endpoint contract: - -- `--model LABEL=DIR` serves `/v1/embeddings`. -- `--contextual-model LABEL=DIR` serves `/v1/contextualizedembeddings`. -- `--late-model LABEL=DIR` serves `/v1/rerank`. - -Scalar serving options that are not in standard model files can be appended to -the load argument: `:api=openai|perplexity`, `:min_dim=N`, and a final -`:query=TEXT` prompt override. `api` defaults to `openai`; use -`api=perplexity` for pplx-compatible base64-int8 output. `min_dim` defaults to -the model's full embedding size; set it for Matryoshka truncation floors. +ffwd supports models in the Hugging Face +[safetensors](https://huggingface.co/docs/safetensors/index) format. Tested +architectures include: -```bash -./embed-server \ - --model pplx=./pplx-model:api=perplexity:min_dim=128 \ - --contextual-model pplx-context=./pplx-context-model:api=perplexity:min_dim=128 \ - --late-model pplx-late=./pplx-late-model \ - --model qwen=./qwen3-model:min_dim=32 \ - --port 8000 -``` +- Qwen3: Qwen3-Embedding and pplx-embed families +- Qwen2: GTE-Qwen2 +- BERT/RoBERTa: BERT, BGE, MiniLM, and XLM-R multilingual encoders such as + multilingual-E5 and Snowflake Arctic -### API +## Components -Every request body takes a `model` label and returns results in input order. +- [libffwd](./libffwd) is the core embedding inference library with kernels, + tokenizer, and model loading. +- [ffwd-server](./tools/server) is a lightweight HTTP server with + OpenAI/Perplexity-compatible embedding APIs, including late-interaction and + contextualized embeddings. +- [ffwd-cli](./tools/cli) is a basic command-line tool that generates embeddings + and computes cosine similarity. -| Endpoint | Returns | -| ----------------------------------- | -------------------------------- | -| `POST /v1/embeddings` | one pooled embedding per input | -| `POST /v1/contextualizedembeddings` | one embedding per document chunk | -| `POST /v1/rerank` | documents ranked by MaxSim score | - -#### POST /v1/embeddings - -```bash -curl http://127.0.0.1:8000/v1/embeddings \ - -H "Content-Type: application/json" \ - -d '{ - "model": "qwen", - "input": [ - "What is the capital of France?", - "Paris is the capital of France." - ] - }' -``` - -Returns a list of embeddings, one per input text: - -```json -{ - "object": "list", - "data": [ - {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]}, - {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]} - ], - "model": "qwen", - "usage": {"prompt_tokens": 16, "total_tokens": 16} -} -``` - -| Field | Type | Description | -| ----------------- | --------------- | ------------------------------------------------------- | -| `input` | string or array | One text, or up to 512 texts. Required. | -| `dimensions` | integer | Truncate the embedding, from the configured `min_dim` up to its size. | -| `encoding_format` | string | Output encoding (see below). | - -**Encoding format:** - -`encoding_format` follows the serving API selected at load time: - -- `api=openai` defaults to `float` and also accepts `base64` (base64 of - little-endian float32). -- `api=perplexity` defaults to `base64_int8` and also accepts `base64_binary` - and `float` (the decoded int8 view). - -#### POST /v1/contextualizedembeddings - -`input` is a list of documents, each a list of chunk strings. Every chunk is -embedded with its document as context, and the results are grouped the same way: -one list per document, one embedding per chunk. - -```bash -curl http://127.0.0.1:8000/v1/contextualizedembeddings \ - -H "Content-Type: application/json" \ - -d '{ - "model": "pplx-context", - "input": [ - ["Intro paragraph.", "Methods paragraph."], - ["A single-chunk note."] - ] - }' -``` - -Returns a list of chunk embeddings per document, in order: - -```json -{ - "object": "list", - "data": [ - { - "object": "list", - "index": 0, - "data": [ - { "object": "embedding", "index": 0, "embedding": "<base64>" }, - { "object": "embedding", "index": 1, "embedding": "<base64>" } - ] - }, - { - "object": "list", - "index": 1, - "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }] - } - ], - "model": "pplx-context", - "usage": { "prompt_tokens": 12, "total_tokens": 12 } -} -``` - -#### POST /v1/rerank - -Scores each document against the query with token-level MaxSim and returns -documents ranked by raw score. - -```bash -curl http://127.0.0.1:8000/v1/rerank \ - -H "Content-Type: application/json" \ - -d '{ - "model": "pplx-late", - "query": "scientific curiosity", - "documents": [ - "Scientists explore from curiosity.", - "SQLite stores data." - ] - }' -``` - -Documents ranked by descending relevance score: - -```json -{ - "object": "list", - "model": "pplx-late", - "results": [ - { "index": 0, "relevance_score": 2.14 }, - { "index": 1, "relevance_score": 0.87 } - ], - "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 } -} -``` - -- `documents` takes up to 1000 strings. `top_n` caps the number of results - (default: all); `return_documents: true` echoes each document text. -- Scores are raw MaxSim values, not probabilities. - -#### Limits and errors +## Building -- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768 - tokens per item, 120000 tokens total, and a 64 MiB body. -- Invalid input returns `422` with a `detail` list naming the offending field. - Unknown labels and labels loaded for another endpoint also return `422`. +Before building, install the dependencies for your platform: -#### Deployment +- On Linux, install `libopenblas-dev`. +- On macOS, install `mlx` and `mlx-c` through Homebrew. -The server runs one inference worker and serves one in-flight request per -connection (HTTP/1.1 keep-alive). +The server also needs `libcjson-dev`/`cjson`. You can skip it if you only want +`libffwd` and the command-line tool. -For concurrency and availability, run several processes behind a load balancer -and route by model label; keep one large model per process. Terminate TLS and -authenticate requests at the proxy. +Once the dependencies are in place, run `make` to build the libraries and tools. ## Acknowledgements -- `embed.c` was inspired by and initially forked from - [antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been - mostly rewritten, including a major CUDA upgrade, but the kernels and some - structural ideas are derived from that codebase. -- The server event loop library in `deps/ae` is extracted from - [Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo - (antirez) and Redis contributors. -- The - [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval) - models were developed by the Perplexity AI research team. -- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part - of the Qwen model family from Alibaba. - -## License - -MIT +- This project started as a fork of + [antirez/qwen-asr](https://github.com/antirez/qwen-asr). Most of it has been + rewritten since then, but the early kernels and some of the structure come + from that codebase. +- The event loop library in `deps/ae` comes from + [Redis](https://github.com/redis/redis). Thanks to Salvatore Sanfilippo + (antirez) and the Redis contributors. +- Many thanks to the authors of all the open-weight models that ffwd supports. diff --git a/tools/cli/README.md b/tools/cli/README.md new file mode 100644 index 0000000..2ce7a00 --- /dev/null +++ b/tools/cli/README.md @@ -0,0 +1,57 @@ +# ffwd-cli: command-line interface for ffwd + +Pass one text to get its embedding, two or more to get a cosine similarity +matrix: + +```bash +# one text: the embedding as a line of space-separated floats +./ffwd-cli -d ./pplx-embed-v1-0.6b "What is the capital of France?" + +# two or more: a cosine similarity matrix, one row per line, no labels +./ffwd-cli -d ./pplx-embed-v1-0.6b "What is the capital of France?" \ + "Paris is the capital of France." \ + "Berlin is the capital of Germany." +# 1.000000 0.719822 0.422054 +# 0.719822 1.000000 0.601722 +# 0.422054 0.601722 1.000000 +``` + +With Qwen3, prefix the retrieval query with a task instruction and embed +documents as-is. Pass them together to rank the documents against the query (the +query is the first row of the similarity matrix): + +```bash +# arg 1: the query, with Qwen3's instruction prefix +# args 2+: documents +./ffwd-cli -d ./Qwen3-Embedding-0.6B \ + $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \ + "Beijing is the capital of China." \ + "The Great Wall is a famous landmark." +``` + +> pplx-embed vectors are unnormalized, so rank them with cosine similarity. +> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot +> product give the same ranking. + +Pipe in lines and use `--stream` to get one JSON embedding per line: + +```bash +cat texts.txt | ./ffwd-cli -d ./model --stream -b 8 +# {"embedding":[...],"dim":2560,"tokens":7,"ms":12.3,"workspace_bytes":1048576} +``` + +Without `--stream`, reading from stdin accumulates all lines then prints the +similarity matrix. + +Add `--json` to emit JSON. One text prints an array of floats; two or more print +an array of rows (the matrix): + +```bash +./ffwd-cli -d ./model --json "cat" "dog" +# [[1.000000,0.380861],[0.380861,1.000000]] +``` + +You can configure the thread count and batch size using the `-t` and `-b` +options. + +See `./ffwd-cli --help` for the full list of options. |
