diff options
| author | Onur Gündüz <ogunduz@gmail.com> | 2026-03-28 21:27:58 +0100 |
|---|---|---|
| committer | Onur Gündüz <ogunduz@gmail.com> | 2026-03-28 23:10:03 +0100 |
| commit | c47e900b94d18cca1bf600e58f10dbcc2e34c1f7 (patch) | |
| tree | bf7a4247c528841abe71bbbd408eda3ff71b1152 /qwen_asr_tokenizer.c | |
| parent | 34e9eefde7870edcc318209738cabb17c71b50ed (diff) | |
| download | ffwd-c47e900b94d18cca1bf600e58f10dbcc2e34c1f7.tar.gz ffwd-c47e900b94d18cca1bf600e58f10dbcc2e34c1f7.zip | |
Refactor and enhance pplx_embed and pplx_embed_mlx for improved model support
- Updated `pplx_embed.h` to support new embedding models: `pplx-embed-v1-4b` and `pplx-embed-context-v1-4b`.
- Simplified model configuration structure and constants for better maintainability.
- Enhanced comments for clarity on model architecture and configuration.
- Modified `pplx_embed_mlx.c` to streamline loading of model weights and configuration from `config.json`.
- Improved memory management and error handling in the MLX backend.
- Updated embedding computation to support dynamic hidden sizes based on model configuration.
- Added `pplx_mlx_config` function to retrieve model configuration details.
- Updated README.md
- Added .gitignore
Diffstat (limited to 'qwen_asr_tokenizer.c')
| -rw-r--r-- | qwen_asr_tokenizer.c | 8 |
1 files changed, 0 insertions, 8 deletions
diff --git a/qwen_asr_tokenizer.c b/qwen_asr_tokenizer.c index 7bbc5db..0e6df38 100644 --- a/qwen_asr_tokenizer.c +++ b/qwen_asr_tokenizer.c @@ -1,11 +1,3 @@ -/* - * qwen_asr_tokenizer.c - Qwen2/Qwen3 BPE tokenizer (GPT-2 byte-level) - * - * Supports: - * - decode token IDs to UTF-8 text (for ASR output assembly) - * - encode UTF-8 text to token IDs using vocab.json + merges.txt - */ - #include "qwen_asr_tokenizer.h" #include "qwen_asr_kernels.h" #include <limits.h> |
