From 91351355aa5c0904b1a3fe214da53557f511c5c7 Mon Sep 17 00:00:00 2001 From: Onur Gündüz Date: Sun, 28 Jun 2026 12:57:26 +0200 Subject: docs: update README --- README.md | 42 +++++++++++++++++++++++++++++------------- 1 file changed, 29 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 958eb1d..dea096b 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,29 @@ # ffwd -ffwd is a high-performance text embedding inference library and toolkit for -transformer-based models. +ffwd is an experimental, lightweight inference engine for text embedding models, +written from scratch in C. -It runs natively on Apple silicon (M1 or later) and NVIDIA GPUs, with CPU -support for both x86 and ARM architectures. +It runs on NVIDIA GPUs (CUDA), Apple silicon (via MLX), and x86/arm64 CPUs +(OpenBLAS/Accelerate). -## Supported models +Supported models include: -Tested architectures include: - -- **Qwen3:** `Qwen3-Embedding` and `pplx-embed` families +- **Qwen3:** the `Qwen3-Embedding` family, with full support for `pplx-embed-v1` - **Qwen2:** `GTE-Qwen2` -- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders - (such as `multilingual-E5` and `Snowflake Arctic`) +- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders, + including `multilingual-E5` and `Snowflake Arctic` ## Components ### libffwd [libffwd](./libffwd) is the core embedding inference library. It provides -optimized kernels, tokenizers, and model loading. +CPU/GPU kernels, tokenizers, and model loading. ### ffwd-server -[ffwd-server](./tools/server) is an HTTP server that provides OpenAI- and -Perplexity-compatible embedding APIs, including support for late-interaction and +[ffwd-server](./tools/server) is an HTTP server that provides OpenAI and +Perplexity compatible embedding APIs, including support for late-interaction and contextualized embeddings. ### ffwd-cli @@ -47,6 +45,24 @@ Once the dependencies are in place, build the project by running: make # or make gpu for GPU build ``` +### Performance + +ffwd implements custom FlashAttention-style kernels from scratch. On NVIDIA +Blackwell and Ada GPUs, it approaches the theoretical peak performance of `bf16` +math without relying on libraries such as CUTLASS or cuDNN. However, at longer +context lengths—for example, above 16k tokens for `Qwen3-Embedding`—vendor +kernels begin to take the lead. + +When comparing ffwd with +[TEI](https://github.com/huggingface/text-embeddings-inference), the main factor +is attention speed. ffwd is very fast at short and medium context lengths +because the work around the attention kernel is kept extremely low-overhead. But +once both engines are pushed toward saturation—especially with a single long +context or very large batches—the bottleneck becomes attention-kernel +throughput. Unsurprisingly, this is where TEI, which leverages CuTe-based +[Flash Attention](https://github.com/HazyResearch/flash-attention), began to +outperform ffwd in my benchmarks. + ## Acknowledgements - This project started as a fork of -- cgit v1.3.1