framesed
Frame-level sound event detection using PretrainedSED/frame_mn10.
Build and run
make # libframesed.a and build/framesed
make test # model parity, policies, threading, CLI, class metadata
build/framesed --weights models/framesed_weights.bin audio.wav
A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is framesed/framesed.h.
CLI
build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav
build/framesed --list-classes
Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono.
The CLI does not resample. --threads defaults to 1 and must be positive.
--track-step enables score tracks and must be at least 0.04 seconds; the
returned step rounds down to a multiple of 0.04 seconds.
Default detection uses 59 caption groups from framesed/framesed_policy.inc.
--all-classes detects each of the 447 model classes independently, without
caption grouping or suppression. This includes speech and background/context
classes, not just sound effects.
build/framesed --all-classes --weights models/framesed_weights.bin audio.wav
build/framesed --list-classes # tab-separated index, AudioSet ID, label; no weights needed
Class-mode gates are uniform: start score 0.50, release score 0.35, minimum duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated; recognition quality varies by class. Tracks are returned for classes reaching 0.35, even when no event survives the duration gate.
Output is one record per line, with times in seconds:
event START END SCORE GROUP LABEL
track TIME SCORE GROUP LABEL
GROUP is a caption-group index by default, or a model class index (0–446)
with --all-classes. Labels may contain spaces and commas.
API
Set framesed_options.all_classes = 1 for individual-class detection; zero or
NULL options select caption groups. framesed_class_label(index) and
framesed_class_mid(index) return model metadata, or NULL for an invalid index.
framesed_group_label(index) always refers to caption groups.
With keep_planes = 1, class-mode results contain 447 probability rows in
model order, at 0.04 s resolution. framesed_infer_s16() returns raw logits for one
10-second window. Free detection results with framesed_result_free() and logits
with framesed_logits_free().
The vocabulary is in framesed/classes.csv. Its indices follow
PretrainedSED's as_strong_train_classes;
AudioSet IDs come from its hf_dataset_gen/metadata label tables. Class order
must match the weights; editing labels does not teach the model new sounds.
make regenerate-classes generates the compiled table; make check-classes
checks that it matches the CSV. Neither command requires PyTorch.
Recreating the weights
models/framesed_weights.bin is an inference export of PretrainedSED's
frame_mn10_strong_1.pt,
not a separately trained model. The exporter folds batch normalization into
convolution weights, includes the Hann window and Kaldi mel filterbank, and
keeps the 447-class strong head (the weak head is omitted).
From the project root, in a Python environment with torch, torchaudio, and
numpy installed (use matching torch/torchaudio versions):
python3 devtools/export_framesed_weights.py
make test
The script downloads and verifies the upstream checkpoint before writing
models/framesed_weights.bin. For offline use or a different output path:
python3 devtools/export_framesed_weights.py \
--checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin
The exporter follows upstream revision 1aa47e482f7e89904cba2338999345025d8b4e36.
devtools/export_framesed_classes.py exports class metadata.
Build options
make debug/make profile/make releasemake AUDIO_BLAS=none all testfor portable CPU inferencemake AUDIO_BLAS=openblas all testfor OpenBLAS (requires OpenBLAS; discovered throughpkg-config)make check-sanitizefor AddressSanitizer and UndefinedBehaviorSanitizermake cleanremoves build outputs, not weights or test data
macOS defaults to Accelerate; other systems default to portable CPU.
Configuration and header changes automatically trigger rebuilds.
FRAMESED_WEIGHTS and FRAMESED_REFERENCE override test fixture paths.
Link libframesed.a with the math and pthread libraries, plus Accelerate or OpenBLAS
when enabled. The WAV reader is used only by the CLI, not the library.
License
MIT License.
The model weights are PretrainedSED/frame_mn10 by Floarian Schmid; see the model repository for details.
