aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-21 01:38:38 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-21 06:39:10 +0200
commit7afacec17eda2a615b7a34b89dcf3ebad9a626c4 (patch)
tree9f0a8c185609e006d8431054263dade7271c373e /README.md
parent3a1a22758093ab499bacfe6348cb686989cd6a92 (diff)
downloadffwd-7afacec17eda2a615b7a34b89dcf3ebad9a626c4.tar.gz
ffwd-7afacec17eda2a615b7a34b89dcf3ebad9a626c4.zip
docs: update README.md
Diffstat (limited to 'README.md')
-rw-r--r--README.md307
1 files changed, 35 insertions, 272 deletions
diff --git a/README.md b/README.md
index 9eb54e8..fbc4dc3 100644
--- a/README.md
+++ b/README.md
@@ -1,287 +1,50 @@
-# embed.c
+# ffwd
-`embed.c` implements high-performance embedding inference for Hugging Face
-safetensors checkpoints, with CPU, Apple MLX, and NVIDIA CUDA backends.
+ffwd is a library and toolkit for text embedding inference.
-Tested model families include pplx-embed, Qwen3-Embedding, GTE-Qwen2, BERT/BGE
-encoders, and XLM-R encoders such as multilingual-E5 and Snowflake Arctic.
-Weights load directly from F32, BF16, or F16 safetensors.
+It works with a variety of transformer-based models and runs natively on Apple
+Silicon and NVIDIA GPUs, with CPU fallback on x86/ARM.
-The engine handles three transformer blocks — the Qwen3 block (pplx-embed and
-Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT/RoBERTa block
-(MiniLM, BGE, and the XLM-R multilingual encoders E5 and Arctic) — and selects
-the tokenizer from the model files: byte-level BPE for the Qwen families,
-WordPiece for BERT, and SentencePiece (Unigram) for XLM-R. Pooling and
-L2-normalization are read from the checkpoint's Sentence Transformers config
-(`1_Pooling/config.json` and `modules.json`), so a model's pooling mode does not
-have to be hardcoded. pplx-embed vectors are unnormalized int8 (rank them by
-cosine similarity); the other families emit L2-normalized float32.
+## Supported models
-## Building
-
-**Prerequisites**:
-
-- On Linux, install `libopenblas-dev`.
-- On macOS, install `mlx` and `mlx-c` via Homebrew.
-
-You also need `libcjson-dev`/`cjson` to build the server; the CLI and the shared
-library can be built without it.
-
-**Makefile targets**:
-
-- Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux).
-- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS,
- OpenBLAS on Linux).
-
-## embed
-
-Pass one text to get its embedding, two or more to get a cosine similarity
-matrix:
-
-```bash
-# one text: the embedding as a line of space-separated floats
-./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?"
-
-# two or more: a cosine similarity matrix, one row per line, no labels
-./embed -d ./pplx-embed-v1-0.6b "What is the capital of France?" \
- "Paris is the capital of France." \
- "Berlin is the capital of Germany."
-# 1.000000 0.719822 0.422054
-# 0.719822 1.000000 0.601722
-# 0.422054 0.601722 1.000000
-```
-
-With Qwen3, prefix the retrieval query with a task instruction and embed
-documents as-is. Pass them together to rank the documents against the query (the
-query is the first row of the similarity matrix):
-
-```bash
-# arg 1: the query, with Qwen3's instruction prefix
-# args 2+: documents
-./embed -d ./Qwen3-Embedding-0.6B \
- $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?' \
- "Beijing is the capital of China." \
- "The Great Wall is a famous landmark."
-```
-
-> pplx-embed vectors are unnormalized, so rank them with cosine similarity.
-> Qwen3-Embedding vectors are L2-normalized, so cosine similarity and dot
-> product give the same ranking.
-
-Pipe in lines and use `--stream` to get one JSON embedding per line:
-
-```bash
-cat texts.txt | ./embed -d ./model --stream -b 8
-# {"embedding":[...],"dim":2560,"tokens":7,"ms":12.3,"workspace_bytes":1048576}
-```
-
-Without `--stream`, reading from stdin accumulates all lines then prints the
-similarity matrix.
-
-Add `--json` to emit JSON. One text prints an array of floats; two or more print
-an array of rows (the matrix):
-
-```bash
-./embed -d ./model --json "cat" "dog"
-# [[1.000000,0.380861],[0.380861,1.000000]]
-```
-
-Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is
-CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output.
-`-V`/`--version` prints the version; `--build-info` also reports the commit,
-build date, target, and compiler.
-
-## embed-server
-
-Serves the Perplexity/OpenAI-compatible HTTP API. Each loaded model gets an
-operator-chosen label, and requests use that label in the `model` field. The
-label is only a routing name; architecture, pooling, normalization, tokenizer,
-and dimensionality are read from the model directory.
-
-Use a separate load flag for each endpoint contract:
-
-- `--model LABEL=DIR` serves `/v1/embeddings`.
-- `--contextual-model LABEL=DIR` serves `/v1/contextualizedembeddings`.
-- `--late-model LABEL=DIR` serves `/v1/rerank`.
-
-Scalar serving options that are not in standard model files can be appended to
-the load argument: `:api=openai|perplexity`, `:min_dim=N`, and a final
-`:query=TEXT` prompt override. `api` defaults to `openai`; use
-`api=perplexity` for pplx-compatible base64-int8 output. `min_dim` defaults to
-the model's full embedding size; set it for Matryoshka truncation floors.
+ffwd supports models in the Hugging Face
+[safetensors](https://huggingface.co/docs/safetensors/index) format. Tested
+architectures include:
-```bash
-./embed-server \
- --model pplx=./pplx-model:api=perplexity:min_dim=128 \
- --contextual-model pplx-context=./pplx-context-model:api=perplexity:min_dim=128 \
- --late-model pplx-late=./pplx-late-model \
- --model qwen=./qwen3-model:min_dim=32 \
- --port 8000
-```
+- Qwen3: Qwen3-Embedding and pplx-embed families
+- Qwen2: GTE-Qwen2
+- BERT/RoBERTa: BERT, BGE, MiniLM, and XLM-R multilingual encoders such as
+ multilingual-E5 and Snowflake Arctic
-### API
+## Components
-Every request body takes a `model` label and returns results in input order.
+- [libffwd](./libffwd) is the core embedding inference library with kernels,
+ tokenizer, and model loading.
+- [ffwd-server](./tools/server) is a lightweight HTTP server with
+ OpenAI/Perplexity-compatible embedding APIs, including late-interaction and
+ contextualized embeddings.
+- [ffwd-cli](./tools/cli) is a basic command-line tool that generates embeddings
+ and computes cosine similarity.
-| Endpoint | Returns |
-| ----------------------------------- | -------------------------------- |
-| `POST /v1/embeddings` | one pooled embedding per input |
-| `POST /v1/contextualizedembeddings` | one embedding per document chunk |
-| `POST /v1/rerank` | documents ranked by MaxSim score |
-
-#### POST /v1/embeddings
-
-```bash
-curl http://127.0.0.1:8000/v1/embeddings \
- -H "Content-Type: application/json" \
- -d '{
- "model": "qwen",
- "input": [
- "What is the capital of France?",
- "Paris is the capital of France."
- ]
- }'
-```
-
-Returns a list of embeddings, one per input text:
-
-```json
-{
- "object": "list",
- "data": [
- {"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]},
- {"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]}
- ],
- "model": "qwen",
- "usage": {"prompt_tokens": 16, "total_tokens": 16}
-}
-```
-
-| Field | Type | Description |
-| ----------------- | --------------- | ------------------------------------------------------- |
-| `input` | string or array | One text, or up to 512 texts. Required. |
-| `dimensions` | integer | Truncate the embedding, from the configured `min_dim` up to its size. |
-| `encoding_format` | string | Output encoding (see below). |
-
-**Encoding format:**
-
-`encoding_format` follows the serving API selected at load time:
-
-- `api=openai` defaults to `float` and also accepts `base64` (base64 of
- little-endian float32).
-- `api=perplexity` defaults to `base64_int8` and also accepts `base64_binary`
- and `float` (the decoded int8 view).
-
-#### POST /v1/contextualizedembeddings
-
-`input` is a list of documents, each a list of chunk strings. Every chunk is
-embedded with its document as context, and the results are grouped the same way:
-one list per document, one embedding per chunk.
-
-```bash
-curl http://127.0.0.1:8000/v1/contextualizedembeddings \
- -H "Content-Type: application/json" \
- -d '{
- "model": "pplx-context",
- "input": [
- ["Intro paragraph.", "Methods paragraph."],
- ["A single-chunk note."]
- ]
- }'
-```
-
-Returns a list of chunk embeddings per document, in order:
-
-```json
-{
- "object": "list",
- "data": [
- {
- "object": "list",
- "index": 0,
- "data": [
- { "object": "embedding", "index": 0, "embedding": "<base64>" },
- { "object": "embedding", "index": 1, "embedding": "<base64>" }
- ]
- },
- {
- "object": "list",
- "index": 1,
- "data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }]
- }
- ],
- "model": "pplx-context",
- "usage": { "prompt_tokens": 12, "total_tokens": 12 }
-}
-```
-
-#### POST /v1/rerank
-
-Scores each document against the query with token-level MaxSim and returns
-documents ranked by raw score.
-
-```bash
-curl http://127.0.0.1:8000/v1/rerank \
- -H "Content-Type: application/json" \
- -d '{
- "model": "pplx-late",
- "query": "scientific curiosity",
- "documents": [
- "Scientists explore from curiosity.",
- "SQLite stores data."
- ]
- }'
-```
-
-Documents ranked by descending relevance score:
-
-```json
-{
- "object": "list",
- "model": "pplx-late",
- "results": [
- { "index": 0, "relevance_score": 2.14 },
- { "index": 1, "relevance_score": 0.87 }
- ],
- "usage": { "query_tokens": 3, "document_tokens": 8, "total_tokens": 11 }
-}
-```
-
-- `documents` takes up to 1000 strings. `top_n` caps the number of results
- (default: all); `return_documents: true` echoes each document text.
-- Scores are raw MaxSim values, not probabilities.
-
-#### Limits and errors
+## Building
-- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768
- tokens per item, 120000 tokens total, and a 64 MiB body.
-- Invalid input returns `422` with a `detail` list naming the offending field.
- Unknown labels and labels loaded for another endpoint also return `422`.
+Before building, install the dependencies for your platform:
-#### Deployment
+- On Linux, install `libopenblas-dev`.
+- On macOS, install `mlx` and `mlx-c` through Homebrew.
-The server runs one inference worker and serves one in-flight request per
-connection (HTTP/1.1 keep-alive).
+The server also needs `libcjson-dev`/`cjson`. You can skip it if you only want
+`libffwd` and the command-line tool.
-For concurrency and availability, run several processes behind a load balancer
-and route by model label; keep one large model per process. Terminate TLS and
-authenticate requests at the proxy.
+Once the dependencies are in place, run `make` to build the libraries and tools.
## Acknowledgements
-- `embed.c` was inspired by and initially forked from
- [antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been
- mostly rewritten, including a major CUDA upgrade, but the kernels and some
- structural ideas are derived from that codebase.
-- The server event loop library in `deps/ae` is extracted from
- [Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo
- (antirez) and Redis contributors.
-- The
- [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval)
- models were developed by the Perplexity AI research team.
-- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part
- of the Qwen model family from Alibaba.
-
-## License
-
-MIT
+- This project started as a fork of
+ [antirez/qwen-asr](https://github.com/antirez/qwen-asr). Most of it has been
+ rewritten since then, but the early kernels and some of the structure come
+ from that codebase.
+- The event loop library in `deps/ae` comes from
+ [Redis](https://github.com/redis/redis). Thanks to Salvatore Sanfilippo
+ (antirez) and the Redis contributors.
+- Many thanks to the authors of all the open-weight models that ffwd supports.