aboutsummaryrefslogtreecommitdiff

ffwd

ffwd is an experimental, lightweight inference engine for text embedding models, written from scratch in C.

It runs on NVIDIA GPUs (CUDA), Apple silicon (via MLX), and x86/arm64 CPUs (OpenBLAS/Accelerate).

Supported models include:

  • Qwen3: the Qwen3-Embedding family, with full support for pplx-embed-v1
  • Qwen2: GTE-Qwen2
  • BERT/RoBERTa: BERT, BGE, MiniLM, and XLM-R multilingual encoders, including multilingual-E5 and Snowflake Arctic

Components

libffwd

libffwd is the core embedding inference library. It provides CPU/GPU kernels, tokenizers, and model loading.

ffwd-server

ffwd-server is an HTTP server that provides OpenAI and Perplexity compatible embedding APIs, including support for late-interaction and contextualized embeddings.

ffwd-cli

ffwd-cli is a command-line tool for generating embeddings and computing cosine similarity.

Building

Before building, install the dependencies for your platform:

  • Linux: Install libopenblas-dev (required for both CPU and GPU builds).
  • macOS: Install mlx and mlx-c via Homebrew if you're targeting Metal/GPU. Otherwise, no extra dependencies needed.

Once the dependencies are in place, build the project by running:

make  # or make gpu for GPU build

Performance

ffwd implements custom FlashAttention-style kernels from scratch. On NVIDIA Blackwell and Ada GPUs, it approaches the theoretical peak performance of bf16 math without relying on libraries such as CUTLASS or cuDNN. However, at longer context lengths—for example, above 16k tokens for Qwen3-Embedding—vendor kernels begin to take the lead.

When comparing ffwd with TEI, the main factor is attention speed. ffwd is very fast at short and medium context lengths because the work around the attention kernel is kept extremely low-overhead. But once both engines are pushed toward saturation—especially with a single long context or very large batches—the bottleneck becomes attention-kernel throughput. Unsurprisingly, this is where TEI, which leverages CuTe-based Flash Attention, began to outperform ffwd in my benchmarks.

Acknowledgements

  • This project started as a fork of antirez/qwen-asr. Most of it has been rewritten since then, but the early kernels and some of the structure come from that codebase.
  • The event loop library in deps/ae comes from Redis. Thanks to Salvatore Sanfilippo (antirez) and the Redis contributors.
  • Many thanks to the authors of all the open-weight models that ffwd supports.