blob: fbc4dc321c1d246c298adec1e60ddf4899640a58 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
# ffwd
ffwd is a library and toolkit for text embedding inference.
It works with a variety of transformer-based models and runs natively on Apple
Silicon and NVIDIA GPUs, with CPU fallback on x86/ARM.
## Supported models
ffwd supports models in the Hugging Face
[safetensors](https://huggingface.co/docs/safetensors/index) format. Tested
architectures include:
- Qwen3: Qwen3-Embedding and pplx-embed families
- Qwen2: GTE-Qwen2
- BERT/RoBERTa: BERT, BGE, MiniLM, and XLM-R multilingual encoders such as
multilingual-E5 and Snowflake Arctic
## Components
- [libffwd](./libffwd) is the core embedding inference library with kernels,
tokenizer, and model loading.
- [ffwd-server](./tools/server) is a lightweight HTTP server with
OpenAI/Perplexity-compatible embedding APIs, including late-interaction and
contextualized embeddings.
- [ffwd-cli](./tools/cli) is a basic command-line tool that generates embeddings
and computes cosine similarity.
## Building
Before building, install the dependencies for your platform:
- On Linux, install `libopenblas-dev`.
- On macOS, install `mlx` and `mlx-c` through Homebrew.
The server also needs `libcjson-dev`/`cjson`. You can skip it if you only want
`libffwd` and the command-line tool.
Once the dependencies are in place, run `make` to build the libraries and tools.
## Acknowledgements
- This project started as a fork of
[antirez/qwen-asr](https://github.com/antirez/qwen-asr). Most of it has been
rewritten since then, but the early kernels and some of the structure come
from that codebase.
- The event loop library in `deps/ae` comes from
[Redis](https://github.com/redis/redis). Thanks to Salvatore Sanfilippo
(antirez) and the Redis contributors.
- Many thanks to the authors of all the open-weight models that ffwd supports.
|