diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-03-29 05:06:39 +0200 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-03-29 05:06:39 +0200 |
| commit | 47859fbfdc02329090c59357cbb826b42c946bf7 (patch) | |
| tree | b907737acb2b79c14812d562f6a50a27307ad4a4 /README.md | |
| parent | 353aa5394a11ef8a9c69e73338b95a3f7eac5855 (diff) | |
| download | ffwd-0.1.0.tar.gz ffwd-0.1.0.zip | |
Update README.mdv0.1.0
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 138 |
1 files changed, 103 insertions, 35 deletions
@@ -1,12 +1,12 @@ -# pplx-embed +# pplx-embed.c -Pure C inference for [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval) embedding models by [Perplexity AI](https://www.perplexity.ai/). Supports safetensors and hardware-accelerated execution (BLAS/mlx-c) for generating float32 vectors. +C implementation of the inference pipeline for [pplx-embed](https://research.perplexity.ai/articles/pplx-embed-state-of-the-art-embedding-models-for-web-scale-retrieval) embedding models. Based on math kernels from [antirez](https://github.com/antirez)'s [qwen-asr](https://github.com/antirez/qwen-asr). -## Supported models +### Supported models -All [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed-v1-68126b2e7cc987cd820ebbbd) models are supported. +All [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed) models are supported. | Model | Dims | Params | Size | |---|---|---|---| @@ -15,75 +15,140 @@ All [pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed-v1- | [pplx-embed-context-v1-0.6B](https://huggingface.co/perplexity-ai/pplx-embed-context-v1-0.6B) | 1024 | 0.6B | ~2.5 GB | | [pplx-embed-context-v1-4B](https://huggingface.co/perplexity-ai/pplx-embed-context-v1-4B) | 2560 | 4B | ~16 GB | -## Quickstart +## Building ```bash -make mlx # Apple Silicon (fastest) -# or: make blas # CPU with BLAS (any platform) +make blas # CPU with BLAS (Accelerate on macOS, OpenBLAS on Linux) +make mlx # Apple Silicon GPU via mlx-c (recommended on Apple Silicon) +make debug # Debug build with AddressSanitizer +``` + +* For `make mlx`, install MLX first: `brew install mlx mlx-c` +* For `make blas` on Linux: `sudo apt install libopenblas-dev` + +## Usage + +``` +./pplx_embed -d <model_dir> [options] [text...] + +Options: + -d <dir> Model directory (required) + --mlx Use Apple MLX GPU backend + --daemon Read texts from stdin, write JSON embeddings to stdout + -t <n> CPU thread count (default: all cores) + -e Print raw embedding vectors (with multiple texts) + -v Verbose (-vv for debug) +``` + +**Single text**: prints the embedding vector as space-separated floats. +```bash ./pplx_embed -d /path/to/model "query: what is the capital of France?" # prints 1024 (or 2560) floats to stdout +``` + +**Multiple texts**: prints a cosine similarity matrix. +```bash ./pplx_embed -d /path/to/model \ "query: what is the capital of France?" \ "document: Paris is the capital of France." \ "document: Berlin is the capital of Germany." -# prints cosine similarity matrix +# prints a NxN similarity matrix to stdout ``` -### Daemon mode - -Start the process once and feed texts through stdin. The model stays loaded in memory. +**Daemon mode** (`--daemon`): reads one text per line from stdin, writes one JSON object per line to stdout. EOF or empty line exits. ```bash -# One embedding per line, output as JSON ./pplx_embed -d /path/to/model --daemon --mlx <<'EOF' query: what is the capital of France? document: Paris is the capital of France. document: Berlin is the capital of Germany. -document: Istanbul is the capital of Germany. EOF ``` -Output (one JSON object per line): +Output is a line-delimited stream of JSON objects with the embedding vector, dimension, token count, and latency in milliseconds: -```json +``` {"embedding":[0.0231,-0.0412,...], "dim":1024, "tokens":9, "ms":15.2} {"embedding":[0.0187,-0.0339,...], "dim":1024, "tokens":9, "ms":14.8} ... ``` -## Building +## C API -```bash -make blas # CPU with BLAS (Accelerate on macOS, OpenBLAS on Linux) -make mlx # Apple Silicon GPU via mlx-c (recommended on Apple Silicon) -make debug # Debug build with AddressSanitizer -``` +**Single text:** -For `make mlx`, install MLX first: `brew install mlx mlx-c` +```c +#include "pplx_embed.h" +#include "qwen_asr_tokenizer.h" -For `make blas` on Linux: `sudo apt install libopenblas-dev` +/* Load model */ +pplx_ctx_t *ctx = pplx_load("/path/to/model"); +int dim = ctx->config.hidden_size; /* 1024 or 2560 */ -## Usage +/* Load tokenizer */ +qwen_tokenizer_t *tok = qwen_tokenizer_load("/path/to/model/vocab.json"); + +/* Embed a text */ +int n_tokens; +int *ids = qwen_tokenizer_encode(tok, "query: what is AI?", &n_tokens); +float *emb = pplx_embed(ctx, ids, n_tokens); +/* emb is a unit-length float[dim] vector. Use it, then free it. */ +store_in_database(emb, dim); +free(emb); +free(ids); + +/* Cleanup */ +pplx_free(ctx); +qwen_tokenizer_free(tok); ``` -./pplx_embed -d <model_dir> [options] [text...] -Options: - -d <dir> Model directory (required) - --mlx Use Apple MLX GPU backend - --daemon Read texts from stdin, write JSON embeddings to stdout - -t <n> CPU thread count (default: all cores) - -e Print raw embedding vectors (with multiple texts) - -v Verbose (-vv for debug) +**Processing a batch of texts:** + +```c +pplx_ctx_t *ctx = pplx_load(model_dir); +qwen_tokenizer_t *tok = qwen_tokenizer_load(vocab_path); +int dim = ctx->config.hidden_size; + +const char *texts[] = { + "document: Paris is the capital of France.", + "document: The Eiffel Tower is in Paris.", + "document: Berlin is the capital of Germany.", +}; + +for (int i = 0; i < 3; i++) { + int n; + int *ids = qwen_tokenizer_encode(tok, texts[i], &n); + float *emb = pplx_embed(ctx, ids, n); + + /* emb[0..dim-1] is the L2-normalized embedding */ + printf("text %d: %d tokens, first dim = %f\n", i, n, emb[0]); + + free(emb); + free(ids); +} + +pplx_free(ctx); +qwen_tokenizer_free(tok); ``` -**Single text**: prints the embedding vector as space-separated floats. +**Full per-token output** (for contextual/late-chunking models): -**Multiple texts**: prints a cosine similarity matrix. +```c +float *all = pplx_forward(ctx, ids, n_tokens); +/* all[i * dim .. (i+1) * dim - 1] = hidden state for token i */ +/* Do your own pooling (e.g., mean-pool a chunk range) */ +free(all); +``` -**Daemon mode** (`--daemon`): reads one text per line from stdin, writes one JSON object per line to stdout. EOF or empty line exits. +**Cosine similarity:** + +```c +float sim = pplx_cosine_similarity(emb_a, emb_b, dim); +/* For L2-normalized vectors this is just the dot product */ +``` ## Benchmark results @@ -131,3 +196,6 @@ Options: * Load times: C/BLAS loads fastest (mmap, no conversion). MLX has a one-time Metal kernel compilation cost. PyTorch's 12-second load for 4B is because the first inference faults in the entire mmap'd file. * Accuracy is identical (cosine sim = 1.000000, max diff < 2e-6) between PyTorch and C implementations. +## License + +MIT |
