# framesed Frame-level sound event detection using `PretrainedSED/frame_mn10`. ## Build and run ```sh make # libframesed.a and build/framesed make test # model parity, policies, threading, CLI, class metadata build/framesed --weights models/framesed_weights.bin audio.wav ``` A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is `framesed/framesed.h`. ## CLI ```text build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav build/framesed --list-classes ``` Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono. The CLI does not resample. `--threads` defaults to 1 and must be positive. `--track-step` enables score tracks and must be at least 0.04 seconds; the returned step rounds down to a multiple of 0.04 seconds. Default detection uses 59 caption groups from `framesed/framesed_policy.inc`. `--all-classes` detects each of the 447 model classes independently, without caption grouping or suppression. This includes speech and background/context classes, not just sound effects. ```sh build/framesed --all-classes --weights models/framesed_weights.bin audio.wav build/framesed --list-classes # tab-separated index, AudioSet ID, label; no weights needed ``` Class-mode gates are uniform: start score 0.50, release score 0.35, minimum duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated; recognition quality varies by class. Tracks are returned for classes reaching 0.35, even when no event survives the duration gate. Output is one record per line, with times in seconds: ```text event START END SCORE GROUP LABEL track TIME SCORE GROUP LABEL ``` `GROUP` is a caption-group index by default, or a model class index (0–446) with `--all-classes`. Labels may contain spaces and commas. ## API Set `framesed_options.all_classes = 1` for individual-class detection; zero or NULL options select caption groups. `framesed_class_label(index)` and `framesed_class_mid(index)` return model metadata, or NULL for an invalid index. `framesed_group_label(index)` always refers to caption groups. With `keep_planes = 1`, class-mode results contain 447 probability rows in model order, at 0.04 s resolution. `framesed_infer_s16()` returns raw logits for one 10-second window. Free detection results with `framesed_result_free()` and logits with `framesed_logits_free()`. The vocabulary is in [`framesed/classes.csv`](framesed/classes.csv). Its indices follow PretrainedSED's [`as_strong_train_classes`](https://github.com/fschmid56/PretrainedSED/blob/1aa47e482f7e89904cba2338999345025d8b4e36/data_util/audioset_classes.py); AudioSet IDs come from its `hf_dataset_gen/metadata` label tables. Class order must match the weights; editing labels does not teach the model new sounds. `make regenerate-classes` generates the compiled table; `make check-classes` checks that it matches the CSV. Neither command requires PyTorch. ## Recreating the weights `models/framesed_weights.bin` is an inference export of PretrainedSED's [`frame_mn10_strong_1.pt`](https://github.com/fschmid56/PretrainedSED/releases/download/v0.0.1/frame_mn10_strong_1.pt), not a separately trained model. The exporter folds batch normalization into convolution weights, includes the Hann window and Kaldi mel filterbank, and keeps the 447-class strong head (the weak head is omitted). From the project root, in a Python environment with `torch`, `torchaudio`, and `numpy` installed (use matching torch/torchaudio versions): ```sh python3 devtools/export_framesed_weights.py make test ``` The script downloads and verifies the upstream checkpoint before writing `models/framesed_weights.bin`. For offline use or a different output path: ```sh python3 devtools/export_framesed_weights.py \ --checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin ``` The exporter follows upstream revision `1aa47e482f7e89904cba2338999345025d8b4e36`. `devtools/export_framesed_classes.py` exports class metadata. ## Build options - `make debug` / `make profile` / `make release` - `make AUDIO_BLAS=none all test` for portable CPU inference - `make AUDIO_BLAS=openblas all test` for OpenBLAS (requires OpenBLAS; discovered through `pkg-config`) - `make check-sanitize` for AddressSanitizer and UndefinedBehaviorSanitizer - `make clean` removes build outputs, not weights or test data macOS defaults to Accelerate; other systems default to portable CPU. Configuration and header changes automatically trigger rebuilds. `FRAMESED_WEIGHTS` and `FRAMESED_REFERENCE` override test fixture paths. Link `libframesed.a` with the math and pthread libraries, plus Accelerate or OpenBLAS when enabled. The WAV reader is used only by the CLI, not the library. ## License MIT License. The model weights are `PretrainedSED/frame_mn10` by Floarian Schmid; see the [model repository](https://github.com/fschmid56/PretrainedSED) for details.