aboutsummaryrefslogtreecommitdiff

framesed

Frame-level sound event detection using PretrainedSED/frame_mn10.

Build and run

make                 # libframesed.a and build/framesed
make test            # model parity, policies, threading, CLI, class metadata
build/framesed --weights models/framesed_weights.bin audio.wav

A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is framesed/framesed.h.

CLI

build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav
build/framesed --list-classes

Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono. The CLI does not resample. --threads defaults to 1 and must be positive. --track-step enables score tracks and must be at least 0.04 seconds; the returned step rounds down to a multiple of 0.04 seconds.

Default detection uses 59 caption groups from framesed/framesed_policy.inc. --all-classes detects each of the 447 model classes independently, without caption grouping or suppression. This includes speech and background/context classes, not just sound effects.

build/framesed --all-classes --weights models/framesed_weights.bin audio.wav
build/framesed --list-classes   # tab-separated index, AudioSet ID, label; no weights needed

Class-mode gates are uniform: start score 0.50, release score 0.35, minimum duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated; recognition quality varies by class. Tracks are returned for classes reaching 0.35, even when no event survives the duration gate.

Output is one record per line, with times in seconds:

event START END SCORE GROUP LABEL
track TIME SCORE GROUP LABEL

GROUP is a caption-group index by default, or a model class index (0–446) with --all-classes. Labels may contain spaces and commas.

API

Set framesed_options.all_classes = 1 for individual-class detection; zero or NULL options select caption groups. framesed_class_label(index) and framesed_class_mid(index) return model metadata, or NULL for an invalid index. framesed_group_label(index) always refers to caption groups.

With keep_planes = 1, class-mode results contain 447 probability rows in model order, at 0.04 s resolution. framesed_infer_s16() returns raw logits for one 10-second window. Free detection results with framesed_result_free() and logits with framesed_logits_free().

The vocabulary is in framesed/classes.csv. Its indices follow PretrainedSED's as_strong_train_classes; AudioSet IDs come from its hf_dataset_gen/metadata label tables. Class order must match the weights; editing labels does not teach the model new sounds. make regenerate-classes generates the compiled table; make check-classes checks that it matches the CSV. Neither command requires PyTorch.

Recreating the weights

models/framesed_weights.bin is an inference export of PretrainedSED's frame_mn10_strong_1.pt, not a separately trained model. The exporter folds batch normalization into convolution weights, includes the Hann window and Kaldi mel filterbank, and keeps the 447-class strong head (the weak head is omitted).

From the project root, in a Python environment with torch, torchaudio, and numpy installed (use matching torch/torchaudio versions):

python3 devtools/export_framesed_weights.py
make test

The script downloads and verifies the upstream checkpoint before writing models/framesed_weights.bin. For offline use or a different output path:

python3 devtools/export_framesed_weights.py \
  --checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin

The exporter follows upstream revision 1aa47e482f7e89904cba2338999345025d8b4e36.

devtools/export_framesed_classes.py exports class metadata.

Build options

  • make debug / make profile / make release
  • make AUDIO_BLAS=none all test for portable CPU inference
  • make AUDIO_BLAS=openblas all test for OpenBLAS (requires OpenBLAS; discovered through pkg-config)
  • make check-sanitize for AddressSanitizer and UndefinedBehaviorSanitizer
  • make clean removes build outputs, not weights or test data

macOS defaults to Accelerate; other systems default to portable CPU. Configuration and header changes automatically trigger rebuilds. FRAMESED_WEIGHTS and FRAMESED_REFERENCE override test fixture paths.

Link libframesed.a with the math and pthread libraries, plus Accelerate or OpenBLAS when enabled. The WAV reader is used only by the CLI, not the library.

License

MIT License.

The model weights are PretrainedSED/frame_mn10 by Floarian Schmid; see the model repository for details.