diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-07-14 04:01:35 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-07-14 04:01:35 +0200 |
| commit | 4e52efea9ae91c14d87cd6f5c3dc7146c147aeb5 (patch) | |
| tree | 57589d7a5f84b617d0fe25fa7985d6ca0a33b39c | |
| parent | c3fcd21e5848c2c9cc35cda57268497410b3f3c9 (diff) | |
| download | ffwd-0.3.0.tar.gz ffwd-0.3.0.zip | |
Diffstat (limited to '')
| -rw-r--r-- | README.md | 28 |
1 files changed, 9 insertions, 19 deletions
@@ -1,34 +1,24 @@ # ffwd -ffwd is a lightweight inference engine for transformer-based text embedding -models. +Super fast inference server for transformer-based text embedding models. -It runs on NVIDIA and Apple silicon GPUs, as well as x86 and arm64 CPUs with -BLAS acceleration. +It runs on NVIDIA and Apple silicon GPUs, as well as CPU-only hosts with BLAS +acceleration. It's mostly tested on Blackwell, Ada, and Apple M4 GPUs, but +should work with any CUDA or Metal capable device. -ffwd supports Qwen3 embedding models such as `Qwen3-Embedding`, the +Supported models include Qwen3 embedding models such as `Qwen3-Embedding`, the `pplx-embed-v1` family, Qwen2 models such as `GTE-Qwen2`, and BERT/RoBERTa-style encoders including `BERT`, `BGE`, `MiniLM`, `XLM-R`, `multilingual-E5`, and `Snowflake Arctic`. -#### Components - -- [libffwd](./libffwd) is the core embedding inference library. It provides - kernels, tokenizers, and model loading. -- [ffwd-server](./tools/server) is an HTTP server that provides OpenAI and - Perplexity compatible embedding APIs, including support for late-interaction - and contextualized embeddings. -- [ffwd-cli](./tools/cli) is a command-line tool for generating embeddings and - computing cosine similarity. - ## Installation -You can download the latest ffwd build for your platform from the +Download the latest build for your platform from the [releases](https://github.com/tetsuo/ffwd/releases) page. ffwd supports models in -[safetensors](https://huggingface.co/docs/safetensors/index) format. Download -compatible models from Hugging Face using the +[safetensors](https://huggingface.co/docs/safetensors/index) format. You can +download compatible models from Hugging Face using the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli). ## Building @@ -47,7 +37,7 @@ make # or make gpu for GPU build ## Getting started -Start a server with one or more models loaded, each with a label: +You can start the server with one or more models loaded, each with a label: ```bash ffwd-server \ |
