diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-06-28 12:57:26 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-06-28 13:03:30 +0200 |
| commit | 91351355aa5c0904b1a3fe214da53557f511c5c7 (patch) | |
| tree | 4c346c23671dc4b072e9ec56368abc14bb955c99 | |
| parent | 511cb1bf631305e83c9042cf53d805398f42b9bb (diff) | |
| download | ffwd-91351355aa5c0904b1a3fe214da53557f511c5c7.tar.gz ffwd-91351355aa5c0904b1a3fe214da53557f511c5c7.zip | |
docs: update README
| -rw-r--r-- | README.md | 42 |
1 files changed, 29 insertions, 13 deletions
@@ -1,31 +1,29 @@ # ffwd -ffwd is a high-performance text embedding inference library and toolkit for -transformer-based models. +ffwd is an experimental, lightweight inference engine for text embedding models, +written from scratch in C. -It runs natively on Apple silicon (M1 or later) and NVIDIA GPUs, with CPU -support for both x86 and ARM architectures. +It runs on NVIDIA GPUs (CUDA), Apple silicon (via MLX), and x86/arm64 CPUs +(OpenBLAS/Accelerate). -## Supported models +Supported models include: -Tested architectures include: - -- **Qwen3:** `Qwen3-Embedding` and `pplx-embed` families +- **Qwen3:** the `Qwen3-Embedding` family, with full support for `pplx-embed-v1` - **Qwen2:** `GTE-Qwen2` -- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders - (such as `multilingual-E5` and `Snowflake Arctic`) +- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders, + including `multilingual-E5` and `Snowflake Arctic` ## Components ### libffwd [libffwd](./libffwd) is the core embedding inference library. It provides -optimized kernels, tokenizers, and model loading. +CPU/GPU kernels, tokenizers, and model loading. ### ffwd-server -[ffwd-server](./tools/server) is an HTTP server that provides OpenAI- and -Perplexity-compatible embedding APIs, including support for late-interaction and +[ffwd-server](./tools/server) is an HTTP server that provides OpenAI and +Perplexity compatible embedding APIs, including support for late-interaction and contextualized embeddings. ### ffwd-cli @@ -47,6 +45,24 @@ Once the dependencies are in place, build the project by running: make # or make gpu for GPU build ``` +### Performance + +ffwd implements custom FlashAttention-style kernels from scratch. On NVIDIA +Blackwell and Ada GPUs, it approaches the theoretical peak performance of `bf16` +math without relying on libraries such as CUTLASS or cuDNN. However, at longer +context lengths—for example, above 16k tokens for `Qwen3-Embedding`—vendor +kernels begin to take the lead. + +When comparing ffwd with +[TEI](https://github.com/huggingface/text-embeddings-inference), the main factor +is attention speed. ffwd is very fast at short and medium context lengths +because the work around the attention kernel is kept extremely low-overhead. But +once both engines are pushed toward saturation—especially with a single long +context or very large batches—the bottleneck becomes attention-kernel +throughput. Unsurprisingly, this is where TEI, which leverages CuTe-based +[Flash Attention](https://github.com/HazyResearch/flash-attention), began to +outperform ffwd in my benchmarks. + ## Acknowledgements - This project started as a fork of |
