ffwd
ffwd is an experimental, lightweight inference engine for text embedding models, written from scratch in C.
It runs on NVIDIA GPUs (CUDA), Apple silicon (via MLX), and x86/arm64 CPUs (OpenBLAS/Accelerate).
Supported models include:
- Qwen3: the
Qwen3-Embeddingfamily, with full support forpplx-embed-v1 - Qwen2:
GTE-Qwen2 - BERT/RoBERTa:
BERT,BGE,MiniLM, andXLM-Rmultilingual encoders, includingmultilingual-E5andSnowflake Arctic
Components
libffwd
libffwd is the core embedding inference library. It provides CPU/GPU kernels, tokenizers, and model loading.
ffwd-server
ffwd-server is an HTTP server that provides OpenAI and Perplexity compatible embedding APIs, including support for late-interaction and contextualized embeddings.
ffwd-cli
ffwd-cli is a command-line tool for generating embeddings and computing cosine similarity.
Building
Before building, install the dependencies for your platform:
- Linux: Install
libopenblas-dev(required for both CPU and GPU builds). - macOS: Install
mlxandmlx-cvia Homebrew if you're targeting Metal/GPU. Otherwise, no extra dependencies needed.
Once the dependencies are in place, build the project by running:
make # or make gpu for GPU build
Performance
ffwd implements custom FlashAttention-style kernels from scratch. On NVIDIA
Blackwell and Ada GPUs, it approaches the theoretical peak performance of bf16
math without relying on libraries such as CUTLASS or cuDNN. However, at longer
context lengths—for example, above 16k tokens for Qwen3-Embedding—vendor
kernels begin to take the lead.
When comparing ffwd with TEI, the main factor is attention speed. ffwd is very fast at short and medium context lengths because the work around the attention kernel is kept extremely low-overhead. But once both engines are pushed toward saturation—especially with a single long context or very large batches—the bottleneck becomes attention-kernel throughput. Unsurprisingly, this is where TEI, which leverages CuTe-based Flash Attention, began to outperform ffwd in my benchmarks.
Acknowledgements
- This project started as a fork of antirez/qwen-asr. Most of it has been rewritten since then, but the early kernels and some of the structure come from that codebase.
- The event loop library in
deps/aecomes from Redis. Thanks to Salvatore Sanfilippo (antirez) and the Redis contributors. - Many thanks to the authors of all the open-weight models that ffwd supports.
