aboutsummaryrefslogtreecommitdiff
path: root/README.md
blob: 59ea6b939a285e423ba7bb5f47b9023308310300 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
# embed.c

`embed.c` implements inference for:

- [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed)
  dense, contextual, and late-interaction models
- [Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding)
  causal, last-token pooled models

Supported platforms are CUDA/cuBLAS on Linux/NVIDIA, Apple Silicon MLX, and
CPU/BLAS.

## Building

**Prerequisites**:

- On Linux, install `libcjson-dev`, `libcjson1`, and `libopenblas-dev`.
- On macOS, install `cjson`, `mlx` and `mlx-c` via Homebrew.

**Makefile targets**:

- Run `make gpu` to auto-select the GPU backend (MLX on macOS, CUDA on Linux).
- Run `make cpu` to build the CPU backend only (BLAS: Accelerate on macOS,
  OpenBLAS on Linux).

## Usage

### embed

Pass one text to get its embedding, two or more to get a cosine similarity
matrix:

```bash
./embed -d ./model "What is the capital of France?"
./embed -d ./model "What is the capital of France?" \
  "Paris is the capital of France." \
  "Berlin is the capital of Germany."
```

Qwen3 retrieval queries benefit from an explicit task instruction. Documents are
embedded directly:

```bash
./embed -d ./Qwen3-Embedding-0.6B \
  $'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?'
```

The Qwen3 tokenizer terminal token is appended automatically by the CLI and
server. Qwen3 embeddings are L2-normalized; pplx embeddings remain unnormalized.

Pipe in lines and use `--stream` to get one JSON embedding per line:

```bash
cat texts.txt | ./embed -d ./model --stream -b 8
# {"embedding":[...],"dim":2048,"tokens":7,"ms":12.3,"workspace_bytes":1048576}
```

Without `--stream`, reading from stdin accumulates all lines then prints the
similarity matrix.

Use `--mlx` or `--cuda` to select a GPU backend (build-dependent). Default is
CPU. `-t` sets thread count, `-b` sets batch size, `-v` enables verbose output.

### embed-server

Serves the HTTP API. Takes one or more `--model ID=DIR` pairs:

```bash
./embed-server \
  --model pplx-embed-v1-0.6b=./model \
  --model Qwen3-Embedding-0.6B=./qwen3-model \
  --port 8000
```

Perplexity API compatible endpoints:

- `POST /v1/embeddings` — pooled embeddings
- `POST /v1/contextualizedembeddings` — contextual document embeddings

Late-interaction reranking endpoint:

- `POST /v1/rerank` — late-interaction MaxSim reranking

For `Qwen3-Embedding-0.6B`, `/v1/embeddings` accepts `dimensions` from 32
to 1024. Truncated Matryoshka embeddings are re-normalized before encoding.

Example:

```bash
./embed-server --model pplx-embed-v1-late-0.6b=./model
# can be tested with:
curl -s http://127.0.0.1:8000/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"pplx-embed-v1-late-0.6b",
    "query":"scientific curiosity",
    "documents":["Scientists explore from curiosity.","SQLite stores data."]
  }'
```

## Acknowledgements

- `embed.c` was inspired by and initially forked from
  [antirez/qwen-asr](https://github.com/antirez/qwen-asr). It has since been
  mostly rewritten, including a major CUDA upgrade, but the kernels and some
  structural ideas are derived from that codebase.
- The server event loop library in `deps/ae` is extracted from
  [Redis](https://github.com/redis/redis), credited to Salvatore Sanfilippo
  (antirez) and Redis contributors.
- The
  [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval)
  models were developed by the Perplexity AI research team.
- The [Qwen3-Embedding](https://qwen.ai/blog?id=qwen3-embedding) models are part
  of the Qwen model family from Alibaba.