aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-28 12:57:26 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-28 13:03:30 +0200
commit91351355aa5c0904b1a3fe214da53557f511c5c7 (patch)
tree4c346c23671dc4b072e9ec56368abc14bb955c99
parent511cb1bf631305e83c9042cf53d805398f42b9bb (diff)
downloadffwd-91351355aa5c0904b1a3fe214da53557f511c5c7.tar.gz
ffwd-91351355aa5c0904b1a3fe214da53557f511c5c7.zip
docs: update README
-rw-r--r--README.md42
1 files changed, 29 insertions, 13 deletions
diff --git a/README.md b/README.md
index 958eb1d..dea096b 100644
--- a/README.md
+++ b/README.md
@@ -1,31 +1,29 @@
# ffwd
-ffwd is a high-performance text embedding inference library and toolkit for
-transformer-based models.
+ffwd is an experimental, lightweight inference engine for text embedding models,
+written from scratch in C.
-It runs natively on Apple silicon (M1 or later) and NVIDIA GPUs, with CPU
-support for both x86 and ARM architectures.
+It runs on NVIDIA GPUs (CUDA), Apple silicon (via MLX), and x86/arm64 CPUs
+(OpenBLAS/Accelerate).
-## Supported models
+Supported models include:
-Tested architectures include:
-
-- **Qwen3:** `Qwen3-Embedding` and `pplx-embed` families
+- **Qwen3:** the `Qwen3-Embedding` family, with full support for `pplx-embed-v1`
- **Qwen2:** `GTE-Qwen2`
-- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders
- (such as `multilingual-E5` and `Snowflake Arctic`)
+- **BERT/RoBERTa:** `BERT`, `BGE`, `MiniLM`, and `XLM-R` multilingual encoders,
+ including `multilingual-E5` and `Snowflake Arctic`
## Components
### libffwd
[libffwd](./libffwd) is the core embedding inference library. It provides
-optimized kernels, tokenizers, and model loading.
+CPU/GPU kernels, tokenizers, and model loading.
### ffwd-server
-[ffwd-server](./tools/server) is an HTTP server that provides OpenAI- and
-Perplexity-compatible embedding APIs, including support for late-interaction and
+[ffwd-server](./tools/server) is an HTTP server that provides OpenAI and
+Perplexity compatible embedding APIs, including support for late-interaction and
contextualized embeddings.
### ffwd-cli
@@ -47,6 +45,24 @@ Once the dependencies are in place, build the project by running:
make # or make gpu for GPU build
```
+### Performance
+
+ffwd implements custom FlashAttention-style kernels from scratch. On NVIDIA
+Blackwell and Ada GPUs, it approaches the theoretical peak performance of `bf16`
+math without relying on libraries such as CUTLASS or cuDNN. However, at longer
+context lengths—for example, above 16k tokens for `Qwen3-Embedding`—vendor
+kernels begin to take the lead.
+
+When comparing ffwd with
+[TEI](https://github.com/huggingface/text-embeddings-inference), the main factor
+is attention speed. ffwd is very fast at short and medium context lengths
+because the work around the attention kernel is kept extremely low-overhead. But
+once both engines are pushed toward saturation—especially with a single long
+context or very large batches—the bottleneck becomes attention-kernel
+throughput. Unsurprisingly, this is where TEI, which leverages CuTe-based
+[Flash Attention](https://github.com/HazyResearch/flash-attention), began to
+outperform ffwd in my benchmarks.
+
## Acknowledgements
- This project started as a fork of