aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-06-18 06:17:41 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-06-18 06:40:54 +0200
commitd4bdb8dae753462f621507d5fc3582e0e53da35c (patch)
tree2e67f32e02fd6ba739e1601493246086d7e0b81b
parent262804db7a2e9ee2e56a7d7faed646152b1e486b (diff)
downloadffwd-d4bdb8dae753462f621507d5fc3582e0e53da35c.tar.gz
ffwd-d4bdb8dae753462f621507d5fc3582e0e53da35c.zip
server: serve models by explicit labels
-rw-r--r--README.md119
-rw-r--r--src/mlx.c3
-rw-r--r--src/server.c218
-rw-r--r--src/server.h18
-rw-r--r--src/server_encode.c16
-rw-r--r--src/server_handlers.c81
-rw-r--r--src/server_internal.h60
-rw-r--r--src/server_json.c20
-rw-r--r--src/server_models.c230
-rw-r--r--tests/test_server.c213
-rw-r--r--tools/openapi.yaml53
11 files changed, 622 insertions, 409 deletions
diff --git a/README.md b/README.md
index a2cdada..9eb54e8 100644
--- a/README.md
+++ b/README.md
@@ -1,59 +1,18 @@
# embed.c
-`embed.c` implements inference for several text embedding model families —
-[pplx-embed](https://huggingface.co/collections/perplexity-ai/pplx-embed),
-[Qwen3-Embedding](https://huggingface.co/collections/Qwen/qwen3-embedding),
-GTE-Qwen2, BERT/BGE encoders, and multilingual XLM-R encoders (multilingual-E5,
-Snowflake Arctic). It serves them through an OpenAI/Perplexity-compatible HTTP
-API and runs on CPU, Apple MLX, and NVIDIA CUDA. See
-[Supported models](#supported-models).
+`embed.c` implements high-performance embedding inference for Hugging Face
+safetensors checkpoints, with CPU, Apple MLX, and NVIDIA CUDA backends.
-## Supported models
-
-The server registers these model IDs out of the box; pass any of them as
-`--model ID=DIR`, loading a Hugging Face safetensors checkpoint from `DIR`.
-Weights load directly from F32, BF16, or F16 safetensors. Every model runs on
-all three backends, and `dimensions` truncates Matryoshka-trained vectors
-(pplx-embed down to 128, Qwen3-Embedding down to 32, arctic-embed down to 256).
-
-**Pooled embeddings** — one vector per text, via `POST /v1/embeddings`:
-
-| Model ID | Family | Dim | Pooling | Output |
-| ------------------------------ | --------------- | ---- | ---------- | ---------------------- |
-| pplx-embed-v1-0.6b | pplx-embed | 1024 | mean | int8, unnormalized |
-| pplx-embed-v1-4b | pplx-embed | 2560 | mean | int8, unnormalized |
-| Qwen3-Embedding-0.6B | Qwen3-Embedding | 1024 | last token | float32, L2-normalized |
-| Qwen3-Embedding-4B | Qwen3-Embedding | 2560 | last token | float32, L2-normalized |
-| Qwen3-Embedding-8B | Qwen3-Embedding | 4096 | last token | float32, L2-normalized |
-| gte-Qwen2-1.5B-instruct | GTE-Qwen2 | 1536 | last token | float32, L2-normalized |
-| all-MiniLM-L6-v2 | BERT (MiniLM) | 384 | mean | float32, L2-normalized |
-| bge-small-en-v1.5 | BERT (BGE) | 384 | CLS | float32, L2-normalized |
-| bge-base-en-v1.5 | BERT (BGE) | 768 | CLS | float32, L2-normalized |
-| bge-large-en-v1.5 | BERT (BGE) | 1024 | CLS | float32, L2-normalized |
-| multilingual-e5-large-instruct | XLM-R (E5) | 1024 | mean | float32, L2-normalized |
-| snowflake-arctic-embed-l-v2.0 | XLM-R (Arctic) | 1024 | CLS | float32, L2-normalized |
-
-**Contextual embeddings** — one vector per document chunk, each embedded with
-its document as context, via `POST /v1/contextualizedembeddings`:
-
-| Model ID | Dim | Output |
-| -------------------------- | ---- | ------------------ |
-| pplx-embed-context-v1-0.6b | 1024 | int8, unnormalized |
-| pplx-embed-context-v1-4b | 2560 | int8, unnormalized |
-
-**Late interaction** — token-level vectors scored with MaxSim for reranking, via
-`POST /v1/rerank`:
-
-| Model ID | Token dim | Output |
-| ----------------------- | --------- | ------------------- |
-| pplx-embed-v1-late-0.6b | 128 | MaxSim rerank score |
+Tested model families include pplx-embed, Qwen3-Embedding, GTE-Qwen2, BERT/BGE
+encoders, and XLM-R encoders such as multilingual-E5 and Snowflake Arctic.
+Weights load directly from F32, BF16, or F16 safetensors.
The engine handles three transformer blocks — the Qwen3 block (pplx-embed and
-Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT/RoBERTa block (MiniLM,
-BGE, and the XLM-R multilingual encoders E5 and Arctic) — and selects the
-tokenizer from the model files: byte-level BPE for the Qwen families, WordPiece
-for BERT, and SentencePiece (Unigram) for XLM-R. Pooling and L2-normalization
-are read from the checkpoint's Sentence Transformers config
+Qwen3-Embedding), the Qwen2 block (GTE-Qwen2), and the BERT/RoBERTa block
+(MiniLM, BGE, and the XLM-R multilingual encoders E5 and Arctic) — and selects
+the tokenizer from the model files: byte-level BPE for the Qwen families,
+WordPiece for BERT, and SentencePiece (Unigram) for XLM-R. Pooling and
+L2-normalization are read from the checkpoint's Sentence Transformers config
(`1_Pooling/config.json` and `modules.json`), so a model's pooling mode does not
have to be hardcoded. pplx-embed vectors are unnormalized int8 (rank them by
cosine similarity); the other families emit L2-normalized float32.
@@ -134,20 +93,35 @@ build date, target, and compiler.
## embed-server
-Serves the Perplexity/OpenAI-compatible HTTP API. Takes one or more
-`--model ID=DIR` pairs:
+Serves the Perplexity/OpenAI-compatible HTTP API. Each loaded model gets an
+operator-chosen label, and requests use that label in the `model` field. The
+label is only a routing name; architecture, pooling, normalization, tokenizer,
+and dimensionality are read from the model directory.
+
+Use a separate load flag for each endpoint contract:
+
+- `--model LABEL=DIR` serves `/v1/embeddings`.
+- `--contextual-model LABEL=DIR` serves `/v1/contextualizedembeddings`.
+- `--late-model LABEL=DIR` serves `/v1/rerank`.
+
+Scalar serving options that are not in standard model files can be appended to
+the load argument: `:api=openai|perplexity`, `:min_dim=N`, and a final
+`:query=TEXT` prompt override. `api` defaults to `openai`; use
+`api=perplexity` for pplx-compatible base64-int8 output. `min_dim` defaults to
+the model's full embedding size; set it for Matryoshka truncation floors.
```bash
./embed-server \
- --model pplx-embed-v1-0.6b=./model \
- --model pplx-embed-context-v1-0.6b=./context-model \
- --model Qwen3-Embedding-0.6B=./qwen3-model \
+ --model pplx=./pplx-model:api=perplexity:min_dim=128 \
+ --contextual-model pplx-context=./pplx-context-model:api=perplexity:min_dim=128 \
+ --late-model pplx-late=./pplx-late-model \
+ --model qwen=./qwen3-model:min_dim=32 \
--port 8000
```
### API
-Every request body takes a `model` parameter and returns results in input order.
+Every request body takes a `model` label and returns results in input order.
| Endpoint | Returns |
| ----------------------------------- | -------------------------------- |
@@ -161,7 +135,7 @@ Every request body takes a `model` parameter and returns results in input order.
curl http://127.0.0.1:8000/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
- "model": "Qwen3-Embedding-0.6B",
+ "model": "qwen",
"input": [
"What is the capital of France?",
"Paris is the capital of France."
@@ -178,7 +152,7 @@ Returns a list of embeddings, one per input text:
{"object": "embedding", "index": 0, "embedding": [0.013, -0.021, ...]},
{"object": "embedding", "index": 1, "embedding": [0.046, 0.009, ...]}
],
- "model": "Qwen3-Embedding-0.6B",
+ "model": "qwen",
"usage": {"prompt_tokens": 16, "total_tokens": 16}
}
```
@@ -186,18 +160,17 @@ Returns a list of embeddings, one per input text:
| Field | Type | Description |
| ----------------- | --------------- | ------------------------------------------------------- |
| `input` | string or array | One text, or up to 512 texts. Required. |
-| `dimensions` | integer | Truncate the embedding (Matryoshka), 32 up to its size. |
+| `dimensions` | integer | Truncate the embedding, from the configured `min_dim` up to its size. |
| `encoding_format` | string | Output encoding (see below). |
**Encoding format:**
-`encoding_format` follows the model family:
+`encoding_format` follows the serving API selected at load time:
-- The OpenAI-API models (Qwen3-Embedding, GTE-Qwen2, MiniLM, BGE,
- multilingual-E5, Snowflake Arctic) default to `float` (the true float32 vector)
- and also accept `base64` (base64 of float32).
-- pplx-embed (Perplexity-compatible) defaults to `base64_int8` and also accepts
- `base64_binary` and `float` (the decoded int8 view).
+- `api=openai` defaults to `float` and also accepts `base64` (base64 of
+ little-endian float32).
+- `api=perplexity` defaults to `base64_int8` and also accepts `base64_binary`
+ and `float` (the decoded int8 view).
#### POST /v1/contextualizedembeddings
@@ -209,7 +182,7 @@ one list per document, one embedding per chunk.
curl http://127.0.0.1:8000/v1/contextualizedembeddings \
-H "Content-Type: application/json" \
-d '{
- "model": "pplx-embed-context-v1-0.6b",
+ "model": "pplx-context",
"input": [
["Intro paragraph.", "Methods paragraph."],
["A single-chunk note."]
@@ -237,7 +210,7 @@ Returns a list of chunk embeddings per document, in order:
"data": [{ "object": "embedding", "index": 0, "embedding": "<base64>" }]
}
],
- "model": "pplx-embed-context-v1-0.6b",
+ "model": "pplx-context",
"usage": { "prompt_tokens": 12, "total_tokens": 12 }
}
```
@@ -251,7 +224,7 @@ documents ranked by raw score.
curl http://127.0.0.1:8000/v1/rerank \
-H "Content-Type: application/json" \
-d '{
- "model": "pplx-embed-v1-late-0.6b",
+ "model": "pplx-late",
"query": "scientific curiosity",
"documents": [
"Scientists explore from curiosity.",
@@ -265,7 +238,7 @@ Documents ranked by descending relevance score:
```json
{
"object": "list",
- "model": "pplx-embed-v1-late-0.6b",
+ "model": "pplx-late",
"results": [
{ "index": 0, "relevance_score": 2.14 },
{ "index": 1, "relevance_score": 0.87 }
@@ -282,8 +255,8 @@ Documents ranked by descending relevance score:
- A request allows up to 512 inputs (1000 documents for `/v1/rerank`), 32768
tokens per item, 120000 tokens total, and a 64 MiB body.
-- Invalid input returns `422` with a `detail` list naming the offending field; a
- registered model that is not loaded returns `503`.
+- Invalid input returns `422` with a `detail` list naming the offending field.
+ Unknown labels and labels loaded for another endpoint also return `422`.
#### Deployment
@@ -291,7 +264,7 @@ The server runs one inference worker and serves one in-flight request per
connection (HTTP/1.1 keep-alive).
For concurrency and availability, run several processes behind a load balancer
-and route by model ID; keep one large model per process. Terminate TLS and
+and route by model label; keep one large model per process. Terminate TLS and
authenticate requests at the proxy.
## Acknowledgements
diff --git a/src/mlx.c b/src/mlx.c
index 4d5abe6..7d05e91 100644
--- a/src/mlx.c
+++ b/src/mlx.c
@@ -520,6 +520,7 @@ static embed_mlx_ctx_t *mlx_load_range_ex(const char *model_dir,
}
ctx->layer_start = layer_start;
ctx->layer_end = layer_end;
+ int h = c->hidden_size;
ctx->ms = multi_safetensors_open(model_dir);
if (!ctx->ms) {
@@ -541,7 +542,7 @@ static embed_mlx_ctx_t *mlx_load_range_ex(const char *model_dir,
goto fail;
}
- int h = c->hidden_size, qd = c->q_dim, kvd = c->kv_dim;
+ int qd = c->q_dim, kvd = c->kv_dim;
int inter = c->intermediate_size, hd = c->head_dim;
char name[256];
diff --git a/src/server.c b/src/server.c
index cb5a857..808c6f8 100644
--- a/src/server.c
+++ b/src/server.c
@@ -79,7 +79,6 @@ static bool auth_ok(http_server *s, const char *auth) {
!strcmp(auth + sizeof(prefix) - 1, s->api_key);
}
-
void dispatch_request(client *c) {
aeDeleteFileEvent(c->srv->loop, c->fd, AE_READABLE);
@@ -190,10 +189,8 @@ static void *worker_main(void *arg) {
#ifdef USE_MLX
if (s->use_mlx) {
int rc = 0;
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
- if (!m->info)
- continue;
embed_mlx_options_t mlx_opts = {
.quantize_bits = s->mlx_quantize_bits,
.quantize_group_size = s->mlx_quantize_group_size,
@@ -230,7 +227,7 @@ static void *worker_main(void *arg) {
pthread_cond_broadcast(&s->cv);
pthread_mutex_unlock(&s->mu);
if (rc) {
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
if (m->mlx_ctx) {
embed_mlx_free(m->mlx_ctx);
@@ -248,10 +245,8 @@ static void *worker_main(void *arg) {
#ifdef USE_CUDA
if (s->use_cuda) {
int rc = 0;
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
- if (!m->info)
- continue;
if (m->info->kind == MODEL_KIND_LATE)
m->cuda_late_ctx = embed_cuda_late_load(m->path);
else
@@ -284,7 +279,7 @@ static void *worker_main(void *arg) {
pthread_cond_broadcast(&s->cv);
pthread_mutex_unlock(&s->mu);
if (rc) {
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
if (m->cuda_ctx) {
embed_cuda_free(m->cuda_ctx);
@@ -315,7 +310,7 @@ static void *worker_main(void *arg) {
}
#ifdef USE_MLX
if (s->use_mlx) {
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
if (m->mlx_ctx) {
embed_mlx_free(m->mlx_ctx);
@@ -330,7 +325,7 @@ static void *worker_main(void *arg) {
#endif
#ifdef USE_CUDA
if (s->use_cuda) {
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
if (m->cuda_ctx) {
embed_cuda_free(m->cuda_ctx);
@@ -375,23 +370,34 @@ static uint64_t physical_memory_nbytes(void) {
}
static int validate_model_specs(const embed_server_config_t *cfg) {
- int seen[MODEL_COUNT] = {0};
for (int i = 0; i < cfg->n_models; i++) {
const embed_server_model_spec_t *spec = &cfg->models[i];
- model_slot slot = model_slot_for_id(spec->id);
- if (slot == MODEL_UNKNOWN) {
- fprintf(stderr, "embed-server: unknown model id: %s\n", spec->id ? spec->id : "<null>");
+ if (!spec->id || !spec->id[0]) {
+ fprintf(stderr, "embed-server: model label must not be empty\n");
return -1;
}
if (!spec->path || !spec->path[0]) {
fprintf(stderr, "embed-server: model %s has an empty path\n", spec->id);
return -1;
}
- if (seen[slot]) {
- fprintf(stderr, "embed-server: duplicate model id: %s\n", spec->id);
+ if (spec->kind < EMBED_SERVER_MODEL_STANDARD || spec->kind > EMBED_SERVER_MODEL_LATE) {
+ fprintf(stderr, "embed-server: model %s has an invalid kind\n", spec->id);
+ return -1;
+ }
+ if (spec->api < EMBED_SERVER_API_DEFAULT || spec->api > EMBED_SERVER_API_PERPLEXITY) {
+ fprintf(stderr, "embed-server: model %s has an invalid api\n", spec->id);
+ return -1;
+ }
+ if (spec->min_dim < 0) {
+ fprintf(stderr, "embed-server: model %s has an invalid min_dim\n", spec->id);
return -1;
}
- seen[slot] = 1;
+ for (int j = 0; j < i; j++) {
+ if (spec->id && cfg->models[j].id && !strcmp(cfg->models[j].id, spec->id)) {
+ fprintf(stderr, "embed-server: duplicate model label: %s\n", spec->id);
+ return -1;
+ }
+ }
}
return 0;
}
@@ -452,7 +458,7 @@ static int mlx_memory_preflight(const embed_server_config_t *cfg) {
int embed_run_server(const embed_server_config_t *cfg) {
if (!cfg || !cfg->models || cfg->n_models <= 0) {
- fprintf(stderr, "embed-server: at least one --model MODEL_ID=PATH is required\n");
+ fprintf(stderr, "embed-server: at least one model label and path is required\n");
return 1;
}
if (validate_model_specs(cfg) != 0 || mlx_memory_preflight(cfg) != 0)
@@ -485,13 +491,7 @@ int embed_run_server(const embed_server_config_t *cfg) {
pthread_cond_init(&s.cv, NULL);
for (int i = 0; i < cfg->n_models; i++) {
- model_slot slot = model_slot_for_id(cfg->models[i].id);
- if (slot == MODEL_UNKNOWN) {
- fprintf(stderr, "embed-server: unknown model id: %s\n", cfg->models[i].id);
- free_models(&s);
- return 1;
- }
- if (load_one_model(&s, slot, cfg->models[i].path) != 0) {
+ if (load_one_model(&s, &cfg->models[i]) != 0) {
free_models(&s);
return 1;
}
@@ -599,10 +599,17 @@ int embed_run_server(const embed_server_config_t *cfg) {
static void print_usage(const char *prog) {
fprintf(stderr,
- "Usage: %s --model ID=DIR [options]\n"
+ "Usage: %s --model LABEL=DIR [options]\n"
"\n"
"Options:\n"
- " --model ID=DIR Model to serve (repeatable)\n"
+ " --model LABEL=DIR[:key=val]\n"
+ " Standard embedding model (repeatable)\n"
+ " --contextual-model LABEL=DIR[:key=val]\n"
+ " Contextual embedding model (repeatable)\n"
+ " --late-model LABEL=DIR[:key=val]\n"
+ " Late-interaction rerank model (repeatable)\n"
+ " keys: api=openai|perplexity, min_dim=N,\n"
+ " query=TEXT (last key; \\n escapes newline)\n"
#ifdef USE_MLX
" --mlx Use Apple MLX GPU backend\n"
#endif
@@ -641,14 +648,15 @@ static void print_usage(const char *prog) {
" -h, --help Show this help\n"
"\n"
"Examples:\n"
- " %s --model pplx-embed-v1-0.6b=./model --port 8000\n"
+ " %s --model docs=./model --port 8000\n"
+ " %s --model pplx=./model:api=perplexity:min_dim=128 --port 8000\n"
#ifdef USE_MLX
" %s --mlx --mlx-quant-bits 8 \\\n"
- " --model pplx-embed-v1-4b=./model-4b-bf16\n",
- prog, prog, prog);
+ " --model pplx=./model-4b-bf16:api=perplexity:min_dim=128\n",
+ prog, prog, prog, prog);
#else
,
- prog, prog);
+ prog, prog, prog);
#endif
}
@@ -658,10 +666,82 @@ typedef struct {
int cap;
} model_specs_t;
-static int append_model_spec(model_specs_t *specs, const char *arg) {
+static char *dup_range(const char *p, size_t n) {
+ char *s = (char *)malloc(n + 1);
+ if (!s)
+ return NULL;
+ memcpy(s, p, n);
+ s[n] = '\0';
+ return s;
+}
+
+static char *unescape_model_option(const char *s) {
+ size_t n = strlen(s);
+ char *out = (char *)malloc(n + 1);
+ if (!out)
+ return NULL;
+ size_t w = 0;
+ for (size_t r = 0; r < n; r++) {
+ if (s[r] == '\\' && r + 1 < n) {
+ r++;
+ if (s[r] == 'n')
+ out[w++] = '\n';
+ else if (s[r] == 't')
+ out[w++] = '\t';
+ else if (s[r] == 'r')
+ out[w++] = '\r';
+ else
+ out[w++] = s[r];
+ } else {
+ out[w++] = s[r];
+ }
+ }
+ out[w] = '\0';
+ return out;
+}
+
+static int parse_model_option(embed_server_model_spec_t *spec, const char *opt, const char *flag) {
+ const char *eq = strchr(opt, '=');
+ if (!eq || eq == opt || !eq[1]) {
+ fprintf(stderr, "%s option expects key=value: %s\n", flag, opt);
+ return -1;
+ }
+ size_t key_len = (size_t)(eq - opt);
+ const char *val = eq + 1;
+ if (key_len == 3 && strncmp(opt, "api", key_len) == 0) {
+ if (!strcmp(val, "openai")) {
+ spec->api = EMBED_SERVER_API_OPENAI;
+ } else if (!strcmp(val, "perplexity") || !strcmp(val, "pplx")) {
+ spec->api = EMBED_SERVER_API_PERPLEXITY;
+ } else if (!strcmp(val, "default")) {
+ spec->api = EMBED_SERVER_API_DEFAULT;
+ } else {
+ fprintf(stderr, "%s api must be openai or perplexity\n", flag);
+ return -1;
+ }
+ return 0;
+ }
+ if (key_len == 7 && strncmp(opt, "min_dim", key_len) == 0) {
+ char *end = NULL;
+ long v = strtol(val, &end, 10);
+ if (!end || *end || v <= 0 || v > INT_MAX) {
+ fprintf(stderr, "%s min_dim must be a positive integer\n", flag);
+ return -1;
+ }
+ spec->min_dim = (int)v;
+ return 0;
+ }
+ fprintf(stderr, "%s unknown model option: %.*s\n", flag, (int)key_len, opt);
+ return -1;
+}
+
+static int append_model_spec(model_specs_t *specs,
+ const char *arg,
+ embed_server_model_kind_t kind,
+ const char *flag) {
const char *eq = strchr(arg, '=');
if (!eq || eq == arg || !eq[1]) {
- fprintf(stderr, "--model expects MODEL_ID=DIR\n");
+ fprintf(stderr, "%s expects LABEL=DIR\n", flag);
return -1;
}
if (specs->n == specs->cap) {
@@ -673,18 +753,52 @@ static int append_model_spec(model_specs_t *specs, const char *arg) {
specs->cap = new_cap;
}
size_t id_len = (size_t)(eq - arg);
- char *id = (char *)malloc(id_len + 1);
- char *path = strdup(eq + 1);
- if (id) {
- memcpy(id, arg, id_len);
- id[id_len] = '\0';
+ const char *path_start = eq + 1;
+ const char *opts = strchr(path_start, ':');
+ size_t path_len = opts ? (size_t)(opts - path_start) : strlen(path_start);
+ if (path_len == 0) {
+ fprintf(stderr, "%s expects a non-empty DIR\n", flag);
+ return -1;
}
- specs->v[specs->n].id = id;
- specs->v[specs->n].path = path;
- if (!specs->v[specs->n].id || !specs->v[specs->n].path)
+ embed_server_model_spec_t spec;
+ memset(&spec, 0, sizeof(spec));
+ spec.id = dup_range(arg, id_len);
+ spec.path = dup_range(path_start, path_len);
+ spec.kind = kind;
+ if (!spec.id || !spec.path) {
+ free((char *)spec.id);
+ free((char *)spec.path);
return -1;
+ }
+ const char *p = opts ? opts + 1 : NULL;
+ while (p && *p) {
+ if (!strncmp(p, "query=", 6) || !strncmp(p, "query_instruct=", 15)) {
+ const char *val = p + (!strncmp(p, "query=", 6) ? 6 : 15);
+ spec.query_instruct = unescape_model_option(val);
+ if (!spec.query_instruct)
+ goto fail;
+ break;
+ }
+ const char *next = strchr(p, ':');
+ size_t opt_len = next ? (size_t)(next - p) : strlen(p);
+ char *opt = dup_range(p, opt_len);
+ if (!opt)
+ goto fail;
+ int rc = parse_model_option(&spec, opt, flag);
+ free(opt);
+ if (rc != 0)
+ goto fail;
+ p = next ? next + 1 : NULL;
+ }
+ specs->v[specs->n] = spec;
specs->n++;
return 0;
+
+fail:
+ free((char *)spec.id);
+ free((char *)spec.path);
+ free((char *)spec.query_instruct);
+ return -1;
}
static void free_model_specs(model_specs_t *specs) {
@@ -693,6 +807,7 @@ static void free_model_specs(model_specs_t *specs) {
for (int i = 0; i < specs->n; i++) {
free((char *)specs->v[i].id);
free((char *)specs->v[i].path);
+ free((char *)specs->v[i].query_instruct);
}
free(specs->v);
}
@@ -732,7 +847,22 @@ int main(int argc, char *argv[]) {
free_model_specs(&model_specs);
return 0;
} else if (!strcmp(f, "--model")) {
- if (append_model_spec(&model_specs, argv[++arg]) != 0) {
+ if (arg + 1 >= argc ||
+ append_model_spec(&model_specs, argv[++arg], EMBED_SERVER_MODEL_STANDARD, f) !=
+ 0) {
+ free_model_specs(&model_specs);
+ return 1;
+ }
+ } else if (!strcmp(f, "--contextual-model")) {
+ if (arg + 1 >= argc ||
+ append_model_spec(&model_specs, argv[++arg], EMBED_SERVER_MODEL_CONTEXTUAL, f) !=
+ 0) {
+ free_model_specs(&model_specs);
+ return 1;
+ }
+ } else if (!strcmp(f, "--late-model")) {
+ if (arg + 1 >= argc ||
+ append_model_spec(&model_specs, argv[++arg], EMBED_SERVER_MODEL_LATE, f) != 0) {
free_model_specs(&model_specs);
return 1;
}
@@ -822,7 +952,7 @@ int main(int argc, char *argv[]) {
return 1;
}
if (model_specs.n == 0) {
- fprintf(stderr, "at least one --model MODEL_ID=DIR is required\n");
+ fprintf(stderr, "at least one --model LABEL=DIR is required\n");
print_usage(prog);
free_model_specs(&model_specs);
return 1;
diff --git a/src/server.h b/src/server.h
index 2a39b94..5ad2b12 100644
--- a/src/server.h
+++ b/src/server.h
@@ -14,9 +14,25 @@
#include <stdbool.h>
+typedef enum {
+ EMBED_SERVER_MODEL_STANDARD = 0,
+ EMBED_SERVER_MODEL_CONTEXTUAL = 1,
+ EMBED_SERVER_MODEL_LATE = 2,
+} embed_server_model_kind_t;
+
+typedef enum {
+ EMBED_SERVER_API_DEFAULT = 0, /* currently OpenAI-compatible float output */
+ EMBED_SERVER_API_OPENAI = 1,
+ EMBED_SERVER_API_PERPLEXITY = 2,
+} embed_server_embedding_api_t;
+
typedef struct {
- const char *id;
+ const char *id; /* served label; requests match this string exactly */
const char *path;
+ embed_server_model_kind_t kind;
+ embed_server_embedding_api_t api;
+ int min_dim; /* 0 means the loaded model's full hidden size */
+ const char *query_instruct; /* NULL means use config_sentence_transformers.json if present */
} embed_server_model_spec_t;
typedef struct {
diff --git a/src/server_encode.c b/src/server_encode.c
index 7e86a74..ef4e3ec 100644
--- a/src/server_encode.c
+++ b/src/server_encode.c
@@ -11,11 +11,10 @@
/* ========================================================================
* Embedding output encoding
*
- * Turns a pooled float vector into the wire form the model's API family wants:
- * base64 of int8 (tanh-quantized, Perplexity default), base64 of a binary sign
- * bitmap, base64 of little-endian float32, or a plain JSON float array. The
- * render_* functions assemble the full OpenAI-style response object from a
- * prepared request and its computed embeddings.
+ * Turns a pooled float vector into the requested wire form. OpenAI-compatible
+ * models emit true float32 arrays by default. Perplexity-compatible models
+ * emit tanh-quantized int8 by default, and their "float" view is the decoded
+ * int8 value for SDK compatibility.
* ======================================================================== */
signed char quantize_int8_tanh(float x) {
@@ -66,10 +65,6 @@ static void append_embedding_object(sbuf *b, int index, const char *embedding) {
sbuf_puts(b, "\"}");
}
-/* Emit one embedding into the response, following the model's API family.
- * "float" renders a JSON array: the true float32 vector for OpenAI/DashScope
- * (Qwen3), or the int8-decoded view (int8/128) for Perplexity (pplx). Every
- * other encoding renders a base64 string via encode_embedding(). */
void append_embedding_value(
sbuf *b, int index, const float *emb, int dims, const char *encoding, embedding_api_t api) {
if (!strcmp(encoding, "float")) {
@@ -77,7 +72,8 @@ void append_embedding_value(
for (int i = 0; i < dims; i++) {
if (i)
sbuf_putc(b, ',');
- float v = api == EMBED_API_OPENAI ? emb[i] : (float)quantize_int8_tanh(emb[i]) / 128.0f;
+ float v =
+ api == EMBED_API_PERPLEXITY ? (float)quantize_int8_tanh(emb[i]) / 128.0f : emb[i];
sbuf_printf(b, "%.9g", (double)v);
}
sbuf_puts(b, "]}");
diff --git a/src/server_handlers.c b/src/server_handlers.c
index 95df1de..b3476d9 100644
--- a/src/server_handlers.c
+++ b/src/server_handlers.c
@@ -271,37 +271,31 @@ static int validate_common(cJSON *root,
http_server *s) {
cJSON *model_item = cJSON_GetObjectItemCaseSensitive(root, "model");
const char *model_id = NULL;
+ loaded_model *m = NULL;
if (!model_item) {
ve_add(detail, "[\"body\",\"model\"]", "field required", "missing");
} else if (!cJSON_IsString(model_item) || !model_item->valuestring) {
ve_add(detail, "[\"body\",\"model\"]", "model must be a string", "type_error.string");
} else {
model_id = model_item->valuestring;
- model_slot slot = model_slot_for_id(model_id);
- if (slot == MODEL_UNKNOWN || k_models[slot].kind != expected_kind) {
+ m = loaded_model_for_label(s, model_id);
+ if (!m) {
ve_add(detail, "[\"body\",\"model\"]",
"value is not a valid enum member for this endpoint", "enum");
- } else {
- loaded_model *m = &s->models[slot];
- if (!m->info) {
- *out_model = NULL;
- } else {
- *out_model = m;
- }
+ } else if (m->info->kind != expected_kind) {
+ ve_add(detail, "[\"body\",\"model\"]",
+ "model is not available on this endpoint", "enum");
+ m = NULL;
}
}
- model_slot slot = model_id ? model_slot_for_id(model_id) : MODEL_UNKNOWN;
- embedding_api_t api = slot != MODEL_UNKNOWN ? k_models[slot].api : EMBED_API_PERPLEXITY;
- const char *enc = encoding_from_root(root, detail, api);
- int min_dim = slot != MODEL_UNKNOWN ? k_models[slot].min_dim : 128;
- int max_dim = slot != MODEL_UNKNOWN ? k_models[slot].dim : 2560;
+ *out_model = m;
+ const char *enc = encoding_from_root(root, detail, m ? m->info->api : EMBED_API_OPENAI);
+ int min_dim = m ? m->info->min_dim : 128;
+ int max_dim = m ? m->info->dim : 2560;
int dims = dimensions_from_root(root, detail, min_dim, max_dim, enc);
*out_encoding = enc;
*out_dims = dims;
- return model_id && model_slot_for_id(model_id) != MODEL_UNKNOWN &&
- !s->models[model_slot_for_id(model_id)].info
- ? 503
- : 0;
+ return 0;
}
void prepare_embedding_request(job *j, cJSON *root, http_server *s, embedding_request *out) {
@@ -313,22 +307,15 @@ void prepare_embedding_request(job *j, cJSON *root, http_server *s, embedding_re
loaded_model *m = NULL;
int dims = 0;
const char *encoding = NULL;
- int unloaded = validate_common(root, detail, MODEL_KIND_STANDARD, &m, &dims, &encoding, s);
+ validate_common(root, detail, MODEL_KIND_STANDARD, &m, &dims, &encoding, s);
out->model = m;
out->dims = dims;
out->encoding = encoding;
- /* Resolve request behavior from the model registry independent of load
- * state. The query instruction, when requested, applies to every input. */
- cJSON *model_item = cJSON_GetObjectItemCaseSensitive(root, "model");
- model_slot api_slot = (model_item && cJSON_IsString(model_item) && model_item->valuestring)
- ? model_slot_for_id(model_item->valuestring)
- : MODEL_UNKNOWN;
- const char *query_instruct =
- api_slot != MODEL_UNKNOWN &&
- text_type_is_query(root, detail, k_models[api_slot].query_instruct)
- ? k_models[api_slot].query_instruct
- : NULL;
+ /* The query instruction, when requested, applies to every input. */
+ const char *query_instruct = m && text_type_is_query(root, detail, m->info->query_instruct)
+ ? m->info->query_instruct
+ : NULL;
cJSON *input = cJSON_GetObjectItemCaseSensitive(root, "input");
int n_inputs = 0;
@@ -363,12 +350,6 @@ void prepare_embedding_request(job *j, cJSON *root, http_server *s, embedding_re
"type_error");
}
- if (cJSON_GetArraySize(detail) == 0 && unloaded) {
- job_set_error(j, 503, "requested model is valid but not loaded", "model_not_loaded");
- cJSON_Delete(detail);
- return;
- }
-
if (cJSON_GetArraySize(detail) == 0 && m && input) {
out->n_inputs = n_inputs;
out->tokens = xcalloc((size_t)n_inputs, sizeof(*out->tokens));
@@ -436,7 +417,7 @@ void prepare_contextual_request(job *j, cJSON *root, http_server *s, contextual_
loaded_model *m = NULL;
int dims = 0;
const char *encoding = NULL;
- int unloaded = validate_common(root, detail, MODEL_KIND_CONTEXTUAL, &m, &dims, &encoding, s);
+ validate_common(root, detail, MODEL_KIND_CONTEXTUAL, &m, &dims, &encoding, s);
out->model = m;
out->dims = dims;
out->encoding = encoding;
@@ -497,12 +478,6 @@ void prepare_contextual_request(job *j, cJSON *root, http_server *s, contextual_
"value_error.list.max_items");
}
- if (cJSON_GetArraySize(detail) == 0 && unloaded) {
- job_set_error(j, 503, "requested model is valid but not loaded", "model_not_loaded");
- cJSON_Delete(detail);
- return;
- }
-
if (cJSON_GetArraySize(detail) == 0 && m) {
out->n_docs = n_docs;
out->docs = xcalloc((size_t)n_docs, sizeof(*out->docs));
@@ -594,14 +569,18 @@ validate_rerank_model(cJSON *root, cJSON *detail, http_server *s, loaded_model *
ve_add(detail, "[\"body\",\"model\"]", "model must be a string", "type_error.string");
return 0;
}
- model_slot slot = model_slot_for_id(item->valuestring);
- if (slot == MODEL_UNKNOWN || k_models[slot].kind != MODEL_KIND_LATE) {
+ loaded_model *m = loaded_model_for_label(s, item->valuestring);
+ if (!m) {
ve_add(detail, "[\"body\",\"model\"]", "value is not a valid enum member for this endpoint",
"enum");
return 0;
}
- *out_model = s->models[slot].info ? &s->models[slot] : NULL;
- return *out_model ? 0 : 503;
+ if (m->info->kind != MODEL_KIND_LATE) {
+ ve_add(detail, "[\"body\",\"model\"]", "model is not available on this endpoint", "enum");
+ return 0;
+ }
+ *out_model = m;
+ return 0;
}
void prepare_rerank_request(job *j, cJSON *root, http_server *s, rerank_request *out) {
@@ -610,7 +589,7 @@ void prepare_rerank_request(job *j, cJSON *root, http_server *s, rerank_request
out->root = root;
cJSON *detail = cJSON_CreateArray();
- int unloaded = validate_rerank_model(root, detail, s, &out->model);
+ validate_rerank_model(root, detail, s, &out->model);
cJSON *query = cJSON_GetObjectItemCaseSensitive(root, "query");
cJSON *documents = cJSON_GetObjectItemCaseSensitive(root, "documents");
/* query_str is set only once query passes every check; the tokenize step
@@ -682,12 +661,6 @@ void prepare_rerank_request(job *j, cJSON *root, http_server *s, rerank_request
out->return_documents = cJSON_IsTrue(return_documents);
}
- if (cJSON_GetArraySize(detail) == 0 && unloaded) {
- job_set_error(j, 503, "requested model is valid but not loaded", "model_not_loaded");
- cJSON_Delete(detail);
- return;
- }
-
if (cJSON_GetArraySize(detail) == 0 && out->model) {
out->n_documents = n_documents;
out->documents = xcalloc((size_t)n_documents, sizeof(*out->documents));
diff --git a/src/server_internal.h b/src/server_internal.h
index 56bf25b..1f106ca 100644
--- a/src/server_internal.h
+++ b/src/server_internal.h
@@ -8,6 +8,7 @@
* the CLI and library do not include it. */
#include "embed.h"
+#include "server.h"
#include "sbuf.h"
#include "tokenizer_bpe.h"
#include "tokenizer_sentencepiece.h"
@@ -47,18 +48,6 @@
#define EMBED_LATE_MASK_TOKEN_ID 151642
#define EMBED_LATE_QUERY_PREFIX_ID 151669
#define EMBED_LATE_DOCUMENT_PREFIX_ID 151670
-/* `text_type: query` prepends the model's published retrieval instruction.
- * Documents and the default pass through unchanged. Callers needing a custom
- * instruction prepend it themselves and leave text_type at "document". */
-#define EMBED_QWEN3_QUERY_INSTRUCT \
- "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
-#define EMBED_GTE_QWEN2_QUERY_INSTRUCT \
- "Instruct: Given a web search query, retrieve relevant passages that answer the " \
- "query\nQuery: "
-#define EMBED_E5_QUERY_INSTRUCT \
- "Instruct: Given a web search query, retrieve relevant passages that answer the " \
- "query\nQuery: "
-#define EMBED_SNOWFLAKE_QUERY_INSTRUCT "query: "
/* Chosen from the L4 scheduler sweep: -b 32 gained ~24% concurrent
* short-request throughput over 8 with no long-document penalty, while 128
* was no faster and inflated long-document tail latency. */
@@ -74,42 +63,15 @@
#define EMBED_MLX_MEMORY_BUDGET_PERCENT 90
#define EMBED_MLX_RESIDENT_MULTIPLIER 2
-typedef enum {
- MODEL_STD_06,
- MODEL_STD_4,
- MODEL_QWEN3_06,
- MODEL_QWEN3_4,
- MODEL_QWEN3_8,
- MODEL_GTE_QWEN2_15,
- MODEL_MINILM_L6,
- MODEL_BGE_SMALL,
- MODEL_BGE_BASE,
- MODEL_BGE_LARGE,
- MODEL_E5_MULTI_LARGE_INSTRUCT,
- MODEL_SNOWFLAKE_ARCTIC_L_V2,
- MODEL_CTX_06,
- MODEL_CTX_4,
- MODEL_LATE_06,
- MODEL_COUNT,
- MODEL_UNKNOWN
-} model_slot;
-
typedef enum { MODEL_KIND_STANDARD, MODEL_KIND_CONTEXTUAL, MODEL_KIND_LATE } model_kind;
-/* Which embedding API a model speaks. pplx-embed models follow the Perplexity
- * API: the canonical output is the tanh int8 quantization, so the default
- * encoding is base64_int8 and "float" is the int8-decoded view (int8/128).
- * Instruction embedding models follow the OpenAI-compatible API: "float" is
- * the true float32 vector and "base64" is base64 of little-endian float32.
- * int8 quantization underuses the range of L2-normalized vectors, so it is not
- * used for these models. */
typedef enum {
- EMBED_API_PERPLEXITY = 0,
- EMBED_API_OPENAI = 1,
+ EMBED_API_OPENAI = 0,
+ EMBED_API_PERPLEXITY = 1,
} embedding_api_t;
typedef struct {
- const char *id;
+ char *id;
model_kind kind;
int dim;
int min_dim;
@@ -118,11 +80,11 @@ typedef struct {
embed_pooling_mode_t pooling_mode;
int normalize_embeddings;
embedding_api_t api;
- const char *query_instruct;
+ char *query_instruct;
} model_info;
typedef struct {
- const model_info *info;
+ model_info *info;
char *path;
embed_tokenizer_t *tok;
embed_tokenizer_workspace_t *tok_ws;
@@ -172,7 +134,8 @@ typedef struct {
int max_batch_tokens;
int batch_wait_us;
char *api_key;
- loaded_model models[MODEL_COUNT];
+ loaded_model *models;
+ int n_models;
int use_mlx;
int use_cuda;
int mlx_quantize_bits;
@@ -347,9 +310,8 @@ void set_response_from_buf(job *j, sbuf *b);
int render_embedding_response(embedding_request *r, const float *embs);
void render_contextual_response(contextual_request *r, const float *embs);
-/* ---- server_models.c: registry, load/lifecycle, tokenize/inference dispatch ---- */
-extern const model_info k_models[];
-model_slot model_slot_for_id(const char *id);
+/* ---- server_models.c: model load/lifecycle, tokenize/inference dispatch ---- */
+loaded_model *loaded_model_for_label(http_server *s, const char *label);
void free_token_bufs(token_buf *t, int n);
void embedding_request_free(embedding_request *r);
void contextual_request_free(contextual_request *r);
@@ -367,7 +329,7 @@ int model_embed_spans_batch(loaded_model *m,
int inference_batch_accepts_input(
const loaded_model *m, int batch, int packed_tokens, int next_tokens, int max_batch_tokens);
int configure_loaded_model(loaded_model *m, const embed_config_t *config, int token_dim);
-int load_one_model(http_server *s, model_slot slot, const char *path);
+int load_one_model(http_server *s, const embed_server_model_spec_t *spec);
void free_models(http_server *s);
/* ---- server_handlers.c: per-endpoint prepare / batch-group / execute ---- */
diff --git a/src/server_json.c b/src/server_json.c
index e45c2c7..cca8fd3 100644
--- a/src/server_json.c
+++ b/src/server_json.c
@@ -143,10 +143,7 @@ bool cjson_is_integer(cJSON *item) {
}
const char *encoding_from_root(cJSON *root, cJSON *detail, embedding_api_t api) {
- /* Default and accepted encodings follow the model's API family: Perplexity
- * defaults to base64_int8 and accepts {base64_int8, base64_binary, float};
- * OpenAI/DashScope (Qwen3) defaults to float and accepts {float, base64}. */
- const char *dflt = api == EMBED_API_OPENAI ? "float" : "base64_int8";
+ const char *dflt = api == EMBED_API_PERPLEXITY ? "base64_int8" : "float";
cJSON *encoding = cJSON_GetObjectItemCaseSensitive(root, "encoding_format");
if (!encoding)
return dflt;
@@ -156,15 +153,16 @@ const char *encoding_from_root(cJSON *root, cJSON *detail, embedding_api_t api)
return dflt;
}
const char *v = encoding->valuestring;
- int ok = api == EMBED_API_OPENAI ? (!strcmp(v, "float") || !strcmp(v, "base64"))
- : (!strcmp(v, "base64_int8") || !strcmp(v, "base64_binary") ||
- !strcmp(v, "float"));
+ int ok = api == EMBED_API_PERPLEXITY
+ ? (!strcmp(v, "base64_int8") || !strcmp(v, "base64_binary") ||
+ !strcmp(v, "float"))
+ : (!strcmp(v, "float") || !strcmp(v, "base64"));
if (!ok) {
ve_add(detail, "[\"body\",\"encoding_format\"]",
- api == EMBED_API_OPENAI
- ? "value is not a valid enum member; permitted: 'float', 'base64'"
- : "value is not a valid enum member; permitted: "
- "'base64_int8', 'base64_binary', 'float'",
+ api == EMBED_API_PERPLEXITY
+ ? "value is not a valid enum member; permitted: "
+ "'base64_int8', 'base64_binary', 'float'"
+ : "value is not a valid enum member; permitted: 'float', 'base64'",
"enum");
return dflt;
}
diff --git a/src/server_models.c b/src/server_models.c
index 32199d9..320840b 100644
--- a/src/server_models.c
+++ b/src/server_models.c
@@ -1,6 +1,7 @@
#include "server_internal.h"
#include "server_util.h"
+#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
@@ -8,63 +9,137 @@
#include <unistd.h>
/* ========================================================================
- * Model registry, load/lifecycle, and tokenize/inference dispatch
+ * Model load/lifecycle, and tokenize/inference dispatch
*
- * k_models is the static table of every model the server can serve (name,
- * dimensions, attention/pooling mode, API family). load_one_model resolves a
- * served id to a slot, picks the tokenizer by the files present (WordPiece
- * vocab.txt, SentencePiece .model, else byte-level BPE vocab.json), and loads the CPU model
- * (GPU backends load on the worker thread). The tokenize_* and model_embed_*
+ * Each loaded model owns resolved serving metadata for the operator-chosen
+ * label. Transformer behavior comes from the model files through embed_config;
+ * server-only facts absent from those files, such as output API family and
+ * Matryoshka floor, come from the load spec. The tokenize_* and model_embed_*
* helpers are the single dispatch point the request handlers call, hiding the
* CPU/MLX/CUDA backend choice behind one signature.
* ======================================================================== */
-const model_info k_models[] = {
- {"pplx-embed-v1-0.6b", MODEL_KIND_STANDARD, 1024, 128, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_MEAN, 0, EMBED_API_PERPLEXITY, NULL},
- {"pplx-embed-v1-4b", MODEL_KIND_STANDARD, 2560, 128, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_MEAN, 0, EMBED_API_PERPLEXITY, NULL},
- {"Qwen3-Embedding-0.6B", MODEL_KIND_STANDARD, 1024, 32, 0, EMBED_ATTENTION_CAUSAL,
- EMBED_POOL_LAST_TOKEN, 1, EMBED_API_OPENAI, EMBED_QWEN3_QUERY_INSTRUCT},
- {"Qwen3-Embedding-4B", MODEL_KIND_STANDARD, 2560, 32, 0, EMBED_ATTENTION_CAUSAL,
- EMBED_POOL_LAST_TOKEN, 1, EMBED_API_OPENAI, EMBED_QWEN3_QUERY_INSTRUCT},
- {"Qwen3-Embedding-8B", MODEL_KIND_STANDARD, 4096, 32, 0, EMBED_ATTENTION_CAUSAL,
- EMBED_POOL_LAST_TOKEN, 1, EMBED_API_OPENAI, EMBED_QWEN3_QUERY_INSTRUCT},
- {"gte-Qwen2-1.5B-instruct", MODEL_KIND_STANDARD, 1536, 1536, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_LAST_TOKEN, 1, EMBED_API_OPENAI, EMBED_GTE_QWEN2_QUERY_INSTRUCT},
- /* BERT encoders: WordPiece tokenizer, learned positions, no instruction
- * prefix. all-MiniLM-L6-v2 mean-pools; bge-small-en-v1.5 pools the [CLS]
- * token. Both emit L2-normalized float32, so they speak the OpenAI API. */
- {"all-MiniLM-L6-v2", MODEL_KIND_STANDARD, 384, 384, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_MEAN, 1, EMBED_API_OPENAI, NULL},
- {"bge-small-en-v1.5", MODEL_KIND_STANDARD, 384, 384, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_CLS, 1, EMBED_API_OPENAI, NULL},
- {"bge-base-en-v1.5", MODEL_KIND_STANDARD, 768, 768, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_CLS, 1, EMBED_API_OPENAI, NULL},
- {"bge-large-en-v1.5", MODEL_KIND_STANDARD, 1024, 1024, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_CLS, 1, EMBED_API_OPENAI, NULL},
- {"multilingual-e5-large-instruct", MODEL_KIND_STANDARD, 1024, 1024, 0,
- EMBED_ATTENTION_BIDIRECTIONAL, EMBED_POOL_MEAN, 1, EMBED_API_OPENAI,
- EMBED_E5_QUERY_INSTRUCT},
- {"snowflake-arctic-embed-l-v2.0", MODEL_KIND_STANDARD, 1024, 256, 0,
- EMBED_ATTENTION_BIDIRECTIONAL, EMBED_POOL_CLS, 1, EMBED_API_OPENAI,
- EMBED_SNOWFLAKE_QUERY_INSTRUCT},
- {"pplx-embed-context-v1-0.6b", MODEL_KIND_CONTEXTUAL, 1024, 128, 0,
- EMBED_ATTENTION_BIDIRECTIONAL, EMBED_POOL_MEAN, 0, EMBED_API_PERPLEXITY, NULL},
- {"pplx-embed-context-v1-4b", MODEL_KIND_CONTEXTUAL, 2560, 128, 0, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_MEAN, 0, EMBED_API_PERPLEXITY, NULL},
- {"pplx-embed-v1-late-0.6b", MODEL_KIND_LATE, 1024, 128, 128, EMBED_ATTENTION_BIDIRECTIONAL,
- EMBED_POOL_MEAN, 0, EMBED_API_PERPLEXITY, NULL},
-};
+loaded_model *loaded_model_for_label(http_server *s, const char *label) {
+ if (!label)
+ return NULL;
+ for (int i = 0; i < s->n_models; i++) {
+ if (s->models[i].info && !strcmp(label, s->models[i].info->id))
+ return &s->models[i];
+ }
+ return NULL;
+}
+
+static model_kind model_kind_from_public(embed_server_model_kind_t kind) {
+ switch (kind) {
+ case EMBED_SERVER_MODEL_CONTEXTUAL:
+ return MODEL_KIND_CONTEXTUAL;
+ case EMBED_SERVER_MODEL_LATE:
+ return MODEL_KIND_LATE;
+ case EMBED_SERVER_MODEL_STANDARD:
+ default:
+ return MODEL_KIND_STANDARD;
+ }
+}
+
+static embedding_api_t embedding_api_from_public(embed_server_embedding_api_t api) {
+ return api == EMBED_SERVER_API_PERPLEXITY ? EMBED_API_PERPLEXITY : EMBED_API_OPENAI;
+}
+
+static int read_optional_text_file(const char *model_dir, const char *relative, char **out) {
+ char path[1024];
+ int n = snprintf(path, sizeof(path), "%s/%s", model_dir, relative);
+ if (n < 0 || (size_t)n >= sizeof(path)) {
+ server_log("embed-server: model path too long");
+ return -1;
+ }
+ FILE *f = fopen(path, "rb");
+ if (!f && errno == ENOENT)
+ return 0;
+ if (!f) {
+ server_log("embed-server: failed to open %s", path);
+ return -1;
+ }
+ if (fseek(f, 0, SEEK_END) != 0) {
+ fclose(f);
+ return -1;
+ }
+ long sz = ftell(f);
+ if (sz < 0 || (size_t)sz == SIZE_MAX || fseek(f, 0, SEEK_SET) != 0) {
+ fclose(f);
+ return -1;
+ }
+ char *buf = (char *)malloc((size_t)sz + 1);
+ if (!buf) {
+ fclose(f);
+ return -1;
+ }
+ if (fread(buf, 1, (size_t)sz, f) != (size_t)sz) {
+ free(buf);
+ fclose(f);
+ return -1;
+ }
+ fclose(f);
+ buf[sz] = '\0';
+ *out = buf;
+ return 1;
+}
+
+static int read_query_instruct_from_files(const char *model_dir, char **out) {
+ char *buf = NULL;
+ int rc = read_optional_text_file(model_dir, "config_sentence_transformers.json", &buf);
+ if (rc <= 0)
+ return rc;
+ cJSON *root = cJSON_Parse(buf);
+ free(buf);
+ if (!root) {
+ server_log("embed-server: invalid config_sentence_transformers.json: %s", model_dir);
+ return -1;
+ }
+ cJSON *prompts = cJSON_GetObjectItemCaseSensitive(root, "prompts");
+ cJSON *query = prompts ? cJSON_GetObjectItemCaseSensitive(prompts, "query") : NULL;
+ if (query && !cJSON_IsString(query)) {
+ cJSON_Delete(root);
+ server_log("embed-server: prompts.query must be a string: %s", model_dir);
+ return -1;
+ }
+ if (query && query->valuestring && query->valuestring[0]) {
+ *out = xstrdup(query->valuestring);
+ cJSON_Delete(root);
+ return *out ? 1 : -1;
+ }
+ cJSON_Delete(root);
+ return 0;
+}
-model_slot model_slot_for_id(const char *id) {
- if (!id)
- return MODEL_UNKNOWN;
- for (int i = 0; i < MODEL_COUNT; i++) {
- if (!strcmp(id, k_models[i].id))
- return (model_slot)i;
+static int init_model_info(model_info **out, const embed_server_model_spec_t *spec) {
+ model_info *info = xcalloc(1, sizeof(*info));
+ info->id = xstrdup(spec->id);
+ info->kind = model_kind_from_public(spec->kind);
+ info->api = embedding_api_from_public(spec->api);
+ info->min_dim = spec->min_dim;
+ if (spec->query_instruct) {
+ info->query_instruct = xstrdup(spec->query_instruct);
+ } else if (read_query_instruct_from_files(spec->path, &info->query_instruct) < 0) {
+ free(info->id);
+ free(info);
+ return -1;
+ }
+ if (!info->id || (spec->query_instruct && !info->query_instruct)) {
+ free(info->query_instruct);
+ free(info->id);
+ free(info);
+ return -1;
}
- return MODEL_UNKNOWN;
+ *out = info;
+ return 0;
+}
+
+static void free_model_info(model_info *info) {
+ if (!info)
+ return;
+ free(info->id);
+ free(info->query_instruct);
+ free(info);
}
void free_token_bufs(token_buf *t, int n) {
@@ -307,14 +382,35 @@ int inference_batch_accepts_input(
}
int configure_loaded_model(loaded_model *m, const embed_config_t *config, int token_dim) {
- if (!m || !m->info || !config || config->hidden_size != m->info->dim ||
- token_dim != m->info->token_dim || config->attention_mode != m->info->attention_mode ||
- config->pooling_mode != m->info->pooling_mode ||
- config->normalize_embeddings != m->info->normalize_embeddings) {
- if (m && m->info && config)
- server_log("embed-server: model %s has incompatible config", m->info->id);
+ if (!m || !m->info || !config || config->hidden_size <= 0) {
+ if (m && m->info)
+ server_log("embed-server: model %s has invalid config", m->info->id);
+ return -1;
+ }
+ if (m->info->kind == MODEL_KIND_LATE) {
+ if (token_dim <= 0) {
+ server_log("embed-server: late model %s is missing token projection metadata",
+ m->info->id);
+ return -1;
+ }
+ } else if (token_dim != 0) {
+ server_log("embed-server: non-late model %s unexpectedly has token projection metadata",
+ m->info->id);
return -1;
}
+
+ int min_dim = m->info->min_dim > 0 ? m->info->min_dim : config->hidden_size;
+ if (min_dim <= 0 || min_dim > config->hidden_size) {
+ server_log("embed-server: model %s has invalid min_dim=%d for dim=%d", m->info->id,
+ min_dim, config->hidden_size);
+ return -1;
+ }
+ m->info->dim = config->hidden_size;
+ m->info->min_dim = min_dim;
+ m->info->token_dim = token_dim;
+ m->info->attention_mode = config->attention_mode;
+ m->info->pooling_mode = config->pooling_mode;
+ m->info->normalize_embeddings = config->normalize_embeddings;
m->append_terminal_token = config->append_terminal_token;
/* m->terminal_token_id is resolved from the tokenizer in load_one_model. */
m->renormalize_truncated =
@@ -322,9 +418,19 @@ int configure_loaded_model(loaded_model *m, const embed_config_t *config, int to
return 0;
}
-int load_one_model(http_server *s, model_slot slot, const char *path) {
- loaded_model *m = &s->models[slot];
- m->info = &k_models[slot];
+int load_one_model(http_server *s, const embed_server_model_spec_t *spec) {
+ /* Grow the model list by one entry. */
+ loaded_model *nm = realloc(s->models, (size_t)(s->n_models + 1) * sizeof(*s->models));
+ if (!nm)
+ return -1;
+ s->models = nm;
+ loaded_model *m = &s->models[s->n_models];
+ memset(m, 0, sizeof(*m));
+ s->n_models++;
+
+ if (init_model_info(&m->info, spec) != 0)
+ return -1;
+ const char *path = spec->path;
m->path = xstrdup(path);
m->terminal_token_id = -1;
m->cls_id = -1;
@@ -478,8 +584,9 @@ int load_one_model(http_server *s, model_slot slot, const char *path) {
}
void free_models(http_server *s) {
- for (int i = 0; i < MODEL_COUNT; i++) {
+ for (int i = 0; i < s->n_models; i++) {
loaded_model *m = &s->models[i];
+ free_model_info(m->info);
free(m->path);
if (m->tok_ws)
embed_tokenizer_workspace_free(m->tok_ws);
@@ -514,4 +621,7 @@ void free_models(http_server *s) {
embed_cuda_late_free(m->cuda_late_ctx);
#endif
}
+ free(s->models);
+ s->models = NULL;
+ s->n_models = 0;
}
diff --git a/tests/test_server.c b/tests/test_server.c
index 81ed4f2..8799b29 100644
--- a/tests/test_server.c
+++ b/tests/test_server.c
@@ -1,9 +1,9 @@
/* tests/test_server.c - tests for embed_server.c.
- * ds4-style: includes the whole server translation unit with the real main
- * compiled out, so statics are directly testable. Unit-tests the encoding
- * helpers, then boots the real server in-process (tiny 1024-dim hermetic
- * model + byte-vocab tokenizer fixture) and drives the full HTTP request
- * path over loopback sockets. Runs via `make test`. */
+ * Includes the whole server translation unit with the real main compiled out,
+ * so statics are directly testable. Unit-tests the encoding helpers, then boots
+ * the real server in-process (tiny 1024-dim hermetic model + byte-vocab tokenizer
+ * fixture) and drives the full HTTP request path over loopback sockets.
+ * Runs via make test. */
#define EMBED_SERVER_TEST
#include "../src/server.c"
@@ -17,6 +17,12 @@
static int test_failures = 0;
+static const char *TEST_QWEN3_QUERY_INSTRUCT =
+ "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:";
+static const char *TEST_GTE_QUERY_INSTRUCT =
+ "Instruct: Given a web search query, retrieve relevant passages that answer the "
+ "query\nQuery: ";
+
#define TEST_ASSERT(cond) \
do { \
if (!(cond)) { \
@@ -197,13 +203,13 @@ static void test_encode_embedding_base64_float32(void) {
static void test_append_embedding_value_float(void) {
float emb[1] = {0.5f};
- /* OpenAI/DashScope (Qwen3): the true float32 value. */
+ /* OpenAI-compatible models render the true float32 value. */
sbuf b = {0};
append_embedding_value(&b, 0, emb, 1, "float", EMBED_API_OPENAI);
TEST_ASSERT(strstr(b.ptr, "\"embedding\":[0.5]") != NULL);
sbuf_free(&b);
- /* Perplexity: the int8-decoded view, round(tanh(0.5)*127)/128 = 59/128. */
+ /* Perplexity-compatible models render the int8-decoded view. */
sbuf b2 = {0};
append_embedding_value(&b2, 0, emb, 1, "float", EMBED_API_PERPLEXITY);
TEST_ASSERT(strstr(b2.ptr, "0.4609375") != NULL);
@@ -213,7 +219,7 @@ static void test_append_embedding_value_float(void) {
static void test_encoding_from_root_family(void) {
cJSON *detail = cJSON_CreateArray();
- /* Default encoding differs by family when the field is absent. */
+ /* Default encoding follows the explicit serving API. */
cJSON *empty = cJSON_CreateObject();
TEST_ASSERT(strcmp(encoding_from_root(empty, detail, EMBED_API_OPENAI), "float") == 0);
TEST_ASSERT(strcmp(encoding_from_root(empty, detail, EMBED_API_PERPLEXITY), "base64_int8") ==
@@ -234,6 +240,13 @@ static void test_encoding_from_root_family(void) {
TEST_ASSERT(cJSON_GetArraySize(detail) == 2);
cJSON_Delete(p);
+ /* An unknown format is rejected. */
+ cJSON *bad = cJSON_CreateObject();
+ cJSON_AddStringToObject(bad, "encoding_format", "base64_fp16");
+ encoding_from_root(bad, detail, EMBED_API_OPENAI);
+ TEST_ASSERT(cJSON_GetArraySize(detail) == 3);
+ cJSON_Delete(bad);
+
cJSON_Delete(detail);
}
@@ -242,7 +255,7 @@ static void test_text_type(void) {
/* Absent: no-op for both model types, no error. */
cJSON *empty = cJSON_CreateObject();
- TEST_ASSERT(text_type_is_query(empty, detail, EMBED_QWEN3_QUERY_INSTRUCT) == 0);
+ TEST_ASSERT(text_type_is_query(empty, detail, TEST_QWEN3_QUERY_INSTRUCT) == 0);
TEST_ASSERT(text_type_is_query(empty, detail, NULL) == 0);
cJSON_Delete(empty);
TEST_ASSERT(cJSON_GetArraySize(detail) == 0);
@@ -250,22 +263,22 @@ static void test_text_type(void) {
/* Qwen3: query -> 1, document -> 0, both accepted. */
cJSON *q = cJSON_CreateObject();
cJSON_AddStringToObject(q, "text_type", "query");
- TEST_ASSERT(text_type_is_query(q, detail, EMBED_QWEN3_QUERY_INSTRUCT) == 1);
+ TEST_ASSERT(text_type_is_query(q, detail, TEST_QWEN3_QUERY_INSTRUCT) == 1);
cJSON_Delete(q);
cJSON *d = cJSON_CreateObject();
cJSON_AddStringToObject(d, "text_type", "document");
- TEST_ASSERT(text_type_is_query(d, detail, EMBED_QWEN3_QUERY_INSTRUCT) == 0);
+ TEST_ASSERT(text_type_is_query(d, detail, TEST_QWEN3_QUERY_INSTRUCT) == 0);
cJSON_Delete(d);
TEST_ASSERT(cJSON_GetArraySize(detail) == 0);
/* Invalid enum value on Qwen3 is reported. */
cJSON *bad = cJSON_CreateObject();
cJSON_AddStringToObject(bad, "text_type", "passage");
- TEST_ASSERT(text_type_is_query(bad, detail, EMBED_QWEN3_QUERY_INSTRUCT) == 0);
+ TEST_ASSERT(text_type_is_query(bad, detail, TEST_QWEN3_QUERY_INSTRUCT) == 0);
TEST_ASSERT(cJSON_GetArraySize(detail) == 1);
cJSON_Delete(bad);
- /* text_type on a Perplexity-family model is rejected. */
+ /* text_type on a model with no query instruction is rejected. */
cJSON *p = cJSON_CreateObject();
cJSON_AddStringToObject(p, "text_type", "query");
TEST_ASSERT(text_type_is_query(p, detail, NULL) == 0);
@@ -275,17 +288,19 @@ static void test_text_type(void) {
cJSON_Delete(detail);
}
-static void test_model_registry(void) {
- model_slot slot = model_slot_for_id("gte-Qwen2-1.5B-instruct");
- TEST_ASSERT(slot == MODEL_GTE_QWEN2_15);
- if (slot == MODEL_UNKNOWN)
- return;
- TEST_ASSERT(k_models[slot].dim == 1536);
- TEST_ASSERT(k_models[slot].min_dim == 1536);
- TEST_ASSERT(k_models[slot].attention_mode == EMBED_ATTENTION_BIDIRECTIONAL);
- TEST_ASSERT(k_models[slot].pooling_mode == EMBED_POOL_LAST_TOKEN);
- TEST_ASSERT(k_models[slot].normalize_embeddings == 1);
- TEST_ASSERT(strcmp(k_models[slot].query_instruct, EMBED_GTE_QWEN2_QUERY_INSTRUCT) == 0);
+static void test_loaded_model_lookup(void) {
+ model_info info;
+ memset(&info, 0, sizeof(info));
+ info.id = "served-label";
+ loaded_model models[1];
+ memset(models, 0, sizeof(models));
+ models[0].info = &info;
+ http_server s;
+ memset(&s, 0, sizeof(s));
+ s.models = models;
+ s.n_models = 1;
+ TEST_ASSERT(loaded_model_for_label(&s, "served-label") == &models[0]);
+ TEST_ASSERT(loaded_model_for_label(&s, "compiled-name") == NULL);
}
static void test_collect_job_batch_deadline(void) {
@@ -552,7 +567,7 @@ static void test_http_embeddings(int port) {
TEST_ASSERT(http_req(port, "POST", "/nope", TEST_API_KEY, "{}", NULL, NULL) == 404);
/* Auth: required, and the exact key. */
- const char *ok_req = "{\"model\":\"pplx-embed-v1-0.6b\",\"input\":\"hello world\","
+ const char *ok_req = "{\"model\":\"std-main\",\"input\":\"hello world\","
"\"encoding_format\":\"float\"}";
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", NULL, ok_req, NULL, NULL) == 401);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", "wrong-key", ok_req, NULL, NULL) == 401);
@@ -561,42 +576,41 @@ static void test_http_embeddings(int port) {
TEST_ASSERT(http_req(port, NULL, NULL, NULL, NULL, "GARBAGE\r\n\r\n", NULL) == 400);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY, "{nope", NULL, NULL) == 422);
- /* Validation: missing/unknown model, wrong endpoint family, valid but
- * unloaded id, bad encoding enum, dimensions out of range; the
- * API answers validation errors with 422 detail arrays. */
+ /* Validation: missing/unknown model, wrong endpoint family, bad encoding
+ * enum, dimensions out of range; validation errors return 422 detail arrays. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY, "{\"input\":\"hi\"}", NULL,
NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
"{\"model\":\"foo\",\"input\":\"hi\"}", NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-0.6b\","
+ "{\"model\":\"ctx-main\","
"\"input\":\"hi\"}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-4b\",\"input\":\"hi\"}", NULL, NULL) == 503);
+ "{\"model\":\"std-missing\",\"input\":\"hi\"}", NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\",\"input\":\"hi\","
+ "{\"model\":\"std-main\",\"input\":\"hi\","
"\"encoding_format\":\"yaml\"}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\",\"input\":\"hi\","
+ "{\"model\":\"std-main\",\"input\":\"hi\","
"\"dimensions\":64}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":[]}",
NULL, NULL) == 422);
- /* Contextual endpoint: standard id is the wrong enum, the 4B contextual
- * id is valid but not loaded. */
+ /* Contextual endpoint: a standard label is the wrong endpoint contract,
+ * and an unloaded label is unknown. */
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":[[\"a\"]]}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-4b\","
+ "{\"model\":\"ctx-missing\","
"\"input\":[[\"a\"]]}",
- NULL, NULL) == 503);
+ NULL, NULL) == 422);
/* Happy path, float encoding, single string input. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY, ok_req, NULL, &body) == 200);
@@ -628,7 +642,7 @@ static void test_http_embeddings(int port) {
/* Batch order: one embedding per input, indexes 0..n-1, and different
* texts produce different vectors. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":[\"hello\",\"world\",\"held\"],"
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -652,9 +666,9 @@ static void test_http_embeddings(int port) {
free(body);
body = NULL;
- /* Default encoding is base64_int8: a string of 1024 quantized bytes. */
+ /* Perplexity-compatible default encoding is base64_int8. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":\"hello\"}",
NULL, &body) == 200);
root = body ? parse_embeddings_body(body, 1) : NULL;
@@ -668,9 +682,32 @@ static void test_http_embeddings(int port) {
free(body);
body = NULL;
+ /* base64_int8 is also accepted when requested explicitly. */
+ TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
+ "{\"model\":\"std-main\","
+ "\"input\":\"hello\",\"encoding_format\":\"base64_int8\"}",
+ NULL, &body) == 200);
+ root = body ? parse_embeddings_body(body, 1) : NULL;
+ if (root) {
+ cJSON *emb = cJSON_GetObjectItemCaseSensitive(
+ cJSON_GetArrayItem(cJSON_GetObjectItemCaseSensitive(root, "data"), 0), "embedding");
+ TEST_ASSERT(cJSON_IsString(emb));
+ TEST_ASSERT(strlen(emb->valuestring) == ((1024 + 2) / 3) * 4);
+ cJSON_Delete(root);
+ }
+ free(body);
+ body = NULL;
+
+ TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
+ "{\"model\":\"std-main\","
+ "\"input\":\"hello\",\"encoding_format\":\"base64\"}",
+ NULL, &body) == 422);
+ free(body);
+ body = NULL;
+
/* Matryoshka truncation via dimensions. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":\"hello\",\"dimensions\":128,"
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -687,14 +724,14 @@ static void test_http_embeddings(int port) {
/* Qwen3 appends its terminal token, supports 32D MRL output, and
* re-normalizes the truncated prefix. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"Qwen3-Embedding-0.6B\","
+ "{\"model\":\"qwen-search\","
"\"input\":\"hello\",\"encoding_format\":\"float\"}",
NULL, &body) == 200);
cJSON *embed_full = body ? parse_embeddings_body(body, 1) : NULL;
free(body);
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"Qwen3-Embedding-0.6B\","
+ "{\"model\":\"qwen-search\","
"\"input\":\"hello\",\"dimensions\":32,"
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -728,14 +765,14 @@ static void test_http_embeddings(int port) {
/* text_type=query must equal manually prepending the Qwen3 instruction with
* text_type=document: both produce the same tokens, hence the same vector. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"Qwen3-Embedding-0.6B\",\"input\":\"hello\","
+ "{\"model\":\"qwen-search\",\"input\":\"hello\","
"\"text_type\":\"query\",\"encoding_format\":\"float\"}",
NULL, &body) == 200);
cJSON *tt_query = body ? parse_embeddings_body(body, 1) : NULL;
free(body);
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"Qwen3-Embedding-0.6B\",\"input\":\"Instruct: Given a web "
+ "{\"model\":\"qwen-search\",\"input\":\"Instruct: Given a web "
"search query, retrieve relevant passages that answer the "
"query\\nQuery:hello\",\"text_type\":\"document\","
"\"encoding_format\":\"float\"}",
@@ -764,17 +801,17 @@ static void test_http_embeddings(int port) {
cJSON_Delete(tt_query);
cJSON_Delete(tt_manual);
- /* GTE-Qwen2 uses the same retrieval task text but publishes a space after
- * "Query:". Verify the registry-specific prompt through the full path. */
+ /* GTE-Qwen2 uses the same retrieval task text but with a space after
+ * "Query:". Verify the explicit prompt override through the full path. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"gte-Qwen2-1.5B-instruct\",\"input\":\"hello\","
+ "{\"model\":\"gte-search\",\"input\":\"hello\","
"\"text_type\":\"query\",\"encoding_format\":\"float\"}",
NULL, &body) == 200);
cJSON *gte_query = body ? parse_embeddings_body(body, 1) : NULL;
free(body);
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"gte-Qwen2-1.5B-instruct\",\"input\":\"Instruct: Given a "
+ "{\"model\":\"gte-search\",\"input\":\"Instruct: Given a "
"web search query, retrieve relevant passages that answer the "
"query\\nQuery: hello\",\"text_type\":\"document\","
"\"encoding_format\":\"float\"}",
@@ -803,15 +840,15 @@ static void test_http_embeddings(int port) {
cJSON_Delete(gte_query);
cJSON_Delete(gte_manual);
- /* text_type is rejected on a Perplexity-family model, and bad values 422. */
+ /* text_type is rejected on a model with no query instruction, and bad values 422. */
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\",\"input\":\"hello\","
+ "{\"model\":\"std-main\",\"input\":\"hello\","
"\"text_type\":\"query\"}",
NULL, &body) == 422);
free(body);
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"Qwen3-Embedding-0.6B\",\"input\":\"hello\","
+ "{\"model\":\"qwen-search\",\"input\":\"hello\","
"\"text_type\":\"passage\"}",
NULL, &body) == 422);
free(body);
@@ -863,16 +900,37 @@ static double emb_max_absdiff(cJSON *a, cJSON *b) {
return m;
}
+static int write_query_prompt_config(const char *dir, const char *query) {
+ char path[2048];
+ snprintf(path, sizeof(path), "%s/config_sentence_transformers.json", dir);
+ FILE *f = fopen(path, "w");
+ if (!f)
+ return -1;
+ fputs("{\"prompts\":{\"query\":\"", f);
+ for (const char *p = query; *p; p++) {
+ if (*p == '\\' || *p == '"')
+ fputc('\\', f);
+ if (*p == '\n') {
+ fputs("\\n", f);
+ } else {
+ fputc(*p, f);
+ }
+ }
+ fputs("\",\"document\":\"\"}}", f);
+ fclose(f);
+ return 0;
+}
+
static void test_http_contextual(int port) {
char *body = NULL;
/* Input must be an array of chunk arrays; chunks must be non-empty. */
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-0.6b\","
+ "{\"model\":\"ctx-main\","
"\"input\":\"hi\"}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-0.6b\","
+ "{\"model\":\"ctx-main\","
"\"input\":[[\"\"]]}",
NULL, NULL) == 422);
@@ -881,7 +939,7 @@ static void test_http_contextual(int port) {
* token counts: hello=1, world=3, held=2, one separator in doc 0, so
* usage must report 7. */
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-0.6b\","
+ "{\"model\":\"ctx-main\","
"\"input\":[[\"hello\",\"world\"],[\"held\"]],"
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -916,7 +974,7 @@ static void test_http_contextual(int port) {
* the whole sequence, so it must match the standard embedding of the
* same text from the same weights. */
TEST_ASSERT(http_req(port, "POST", "/v1/contextualizedembeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-context-v1-0.6b\","
+ "{\"model\":\"ctx-main\","
"\"input\":[[\"hello\"]],"
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -925,7 +983,7 @@ static void test_http_contextual(int port) {
free(body);
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/embeddings", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"input\":\"hello\","
"\"encoding_format\":\"float\"}",
NULL, &body) == 200);
@@ -953,24 +1011,24 @@ static void test_http_rerank(int port) {
char *body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/rerank", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-0.6b\","
+ "{\"model\":\"std-main\","
"\"query\":\"hello\",\"documents\":[\"world\"]}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/rerank", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-late-0.6b\","
+ "{\"model\":\"late-main\","
"\"query\":\"\",\"documents\":[\"world\"]}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/rerank", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-late-0.6b\","
+ "{\"model\":\"late-main\","
"\"query\":\"hello\",\"documents\":[]}",
NULL, NULL) == 422);
TEST_ASSERT(http_req(port, "POST", "/v1/rerank", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-late-0.6b\","
+ "{\"model\":\"late-main\","
"\"query\":\"hello\",\"documents\":[\"world\"],"
"\"top_n\":1,\"top_k\":1}",
NULL, NULL) == 422);
- const char *request = "{\"model\":\"pplx-embed-v1-late-0.6b\","
+ const char *request = "{\"model\":\"late-main\","
"\"query\":\"hello\",\"documents\":["
"\"hello!\",\"world\",\"quoted \\\"document\\\"\"],"
"\"top_n\":2,\"return_documents\":true}";
@@ -982,8 +1040,7 @@ static void test_http_rerank(int port) {
cJSON *model = cJSON_GetObjectItemCaseSensitive(root, "model");
cJSON *results = cJSON_GetObjectItemCaseSensitive(root, "results");
TEST_ASSERT(cJSON_IsString(object) && strcmp(object->valuestring, "list") == 0);
- TEST_ASSERT(cJSON_IsString(model) &&
- strcmp(model->valuestring, "pplx-embed-v1-late-0.6b") == 0);
+ TEST_ASSERT(cJSON_IsString(model) && strcmp(model->valuestring, "late-main") == 0);
TEST_ASSERT(cJSON_IsArray(results) && cJSON_GetArraySize(results) == 2);
double previous = INFINITY;
int seen[3] = {0};
@@ -1025,7 +1082,7 @@ static void test_http_rerank(int port) {
body = NULL;
TEST_ASSERT(http_req(port, "POST", "/v1/rerank", TEST_API_KEY,
- "{\"model\":\"pplx-embed-v1-late-0.6b\","
+ "{\"model\":\"late-main\","
"\"query\":\"hello\","
"\"documents\":[\"hello\",\"world\"],\"top_k\":1}",
NULL, &body) == 200);
@@ -1081,6 +1138,7 @@ static void test_http_server(void) {
tm_write_late_projection(late_dir, "F32", 128, 1024) != 0 ||
tf_write_vocab(embed_dir) != 0 ||
tm_write_qwen3_model_dims(embed_dir, "F32", &dims, TF_EOT_ID) != 0 ||
+ write_query_prompt_config(embed_dir, TEST_QWEN3_QUERY_INSTRUCT) != 0 ||
tf_write_vocab(gte_dir) != 0 ||
tm_write_qwen2_model_dims(gte_dir, "F32", &gte_dims, TF_EOT_ID, 0) != 0) {
fprintf(stderr, "FAIL: fixture write\n");
@@ -1095,16 +1153,27 @@ static void test_http_server(void) {
memset(&ctx, 0, sizeof(ctx));
/* Both 0.6b slots serve the same tiny weights; the contextual slot
* resolves its separator from the fixture's <|endoftext|>. */
- ctx.spec[0].id = "pplx-embed-v1-0.6b";
+ ctx.spec[0].id = "std-main";
ctx.spec[0].path = dir;
- ctx.spec[1].id = "pplx-embed-context-v1-0.6b";
+ ctx.spec[0].api = EMBED_SERVER_API_PERPLEXITY;
+ ctx.spec[0].min_dim = 128;
+ ctx.spec[1].id = "ctx-main";
ctx.spec[1].path = dir;
- ctx.spec[2].id = "pplx-embed-v1-late-0.6b";
+ ctx.spec[1].kind = EMBED_SERVER_MODEL_CONTEXTUAL;
+ ctx.spec[1].api = EMBED_SERVER_API_PERPLEXITY;
+ ctx.spec[1].min_dim = 128;
+ ctx.spec[2].id = "late-main";
ctx.spec[2].path = late_dir;
- ctx.spec[3].id = "Qwen3-Embedding-0.6B";
+ ctx.spec[2].kind = EMBED_SERVER_MODEL_LATE;
+ ctx.spec[2].api = EMBED_SERVER_API_PERPLEXITY;
+ ctx.spec[2].min_dim = 128;
+ ctx.spec[3].id = "qwen-search";
ctx.spec[3].path = embed_dir;
- ctx.spec[4].id = "gte-Qwen2-1.5B-instruct";
+ ctx.spec[3].min_dim = 32;
+ ctx.spec[4].id = "gte-search";
ctx.spec[4].path = gte_dir;
+ ctx.spec[4].min_dim = 1536;
+ ctx.spec[4].query_instruct = TEST_GTE_QUERY_INSTRUCT;
ctx.cfg.models = ctx.spec;
ctx.cfg.n_models = 5;
ctx.cfg.port = port;
@@ -1154,7 +1223,7 @@ int main(void) {
test_append_embedding_value_float();
test_encoding_from_root_family();
test_text_type();
- test_model_registry();
+ test_loaded_model_lookup();
test_collect_job_batch_deadline();
test_collect_job_batch_zero_wait();
test_http_server();
diff --git a/tools/openapi.yaml b/tools/openapi.yaml
index 657502b..6dd3818 100644
--- a/tools/openapi.yaml
+++ b/tools/openapi.yaml
@@ -3,12 +3,10 @@ info:
title: embed.c server API
version: "1.0.0"
description: >
- HTTP API served by `embed-server` for pplx-embed, Qwen3-Embedding, GTE-Qwen2,
- BERT/BGE, and multilingual XLM-R (E5, Arctic) embedding models. Requests
- follow one of two conventions by model family: the Perplexity API (pplx-embed)
- or the Qwen/DashScope OpenAI-compatible API (every other family). How an
- embedding is encoded on the wire depends on the model family; see
- `encoding_format`. Always compare embeddings with cosine similarity.
+ HTTP API served by `embed-server`. Each loaded model has an operator-chosen
+ label; request `model` fields match that label. Embedding encoding follows
+ the serving API selected at load time (`api=openai` or `api=perplexity`).
+ Always compare embeddings with cosine similarity.
license:
name: MIT
servers:
@@ -35,7 +33,6 @@ paths:
application/json:
schema: { $ref: "#/components/schemas/EmbeddingsResponse" }
"422": { $ref: "#/components/responses/ValidationError" }
- "503": { $ref: "#/components/responses/ModelNotLoaded" }
/v1/contextualizedembeddings:
post:
summary: Create contextual (late-chunking) document embeddings
@@ -55,7 +52,6 @@ paths:
application/json:
schema: { $ref: "#/components/schemas/ContextualResponse" }
"422": { $ref: "#/components/responses/ValidationError" }
- "503": { $ref: "#/components/responses/ModelNotLoaded" }
/v1/rerank:
post:
summary: Late-interaction (MaxSim) reranking
@@ -75,7 +71,6 @@ paths:
application/json:
schema: { $ref: "#/components/schemas/RerankResponse" }
"422": { $ref: "#/components/responses/ValidationError" }
- "503": { $ref: "#/components/responses/ModelNotLoaded" }
components:
securitySchemes:
bearerAuth:
@@ -91,7 +86,8 @@ components:
properties:
model:
type: string
- example: Qwen3-Embedding-0.6B
+ description: Served label chosen when starting `embed-server`.
+ example: qwen
input:
description: A single string or an array of up to 512 strings.
oneOf:
@@ -103,19 +99,19 @@ components:
encoding_format:
type: string
description: >
- Output encoding, validated against the model's API family.
- Qwen3-Embedding accepts `float` (default; true float32) and `base64`
- (base64 of little-endian float32). pplx-embed accepts `base64_int8`
- (default), `base64_binary` (sign bits, needs `dimensions` divisible
- by 8), and `float` (the int8-decoded view). A format outside the
- model's family returns 422.
+ Output encoding, validated against the serving API selected for the
+ loaded label. `api=openai` accepts `float` (default; true float32)
+ and `base64` (base64 of little-endian float32). `api=perplexity`
+ accepts `base64_int8` (default), `base64_binary` (sign bits, needs
+ `dimensions` divisible by 8), and `float` (the int8-decoded view).
+ A format outside the label's serving API returns 422.
enum: [float, base64, base64_int8, base64_binary]
dimensions:
type: integer
description: >
- Matryoshka truncation, from the model minimum (32 for Qwen3, 128 for
- pplx, 256 for arctic-embed) up to the model's native dimension.
- Truncated vectors are re-normalized.
+ Matryoshka truncation, from the configured load-time `min_dim` up to
+ the model's native dimension. Truncated normalized vectors are
+ re-normalized.
EmbeddingsResponse:
type: object
properties:
@@ -144,7 +140,8 @@ components:
properties:
model:
type: string
- example: pplx-embed-context-v1-0.6b
+ description: Served contextual-model label.
+ example: pplx-context
input:
description: Array of documents; each document is an array of chunk strings.
type: array
@@ -180,7 +177,8 @@ components:
properties:
model:
type: string
- example: pplx-embed-v1-late-0.6b
+ description: Served late-model label.
+ example: pplx-late
query: { type: string }
documents:
type: array
@@ -243,22 +241,9 @@ components:
detail:
type: array
items: { $ref: "#/components/schemas/ValidationDetail" }
- Error:
- type: object
- properties:
- error:
- type: object
- properties:
- message: { type: string }
- type: { type: string }
responses:
ValidationError:
description: Request body failed validation.
content:
application/json:
schema: { $ref: "#/components/schemas/ValidationError" }
- ModelNotLoaded:
- description: The model ID is valid for this build but is not loaded.
- content:
- application/json:
- schema: { $ref: "#/components/schemas/Error" }