From a1e355b8ceb12ac66c9d82e7786df4c3c40be479 Mon Sep 17 00:00:00 2001 From: Onur Gündüz Date: Wed, 21 Oct 2026 20:08:53 +0200 Subject: Init --- README.md | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 README.md (limited to 'README.md') diff --git a/README.md b/README.md new file mode 100644 index 0000000..00ae9f4 --- /dev/null +++ b/README.md @@ -0,0 +1,119 @@ +# framesed + +Frame-level sound event detection using `PretrainedSED/frame_mn10`. + +## Build and run + +```sh +make # libframesed.a and build/framesed +make test # model parity, policies, threading, CLI, class metadata +build/framesed --weights models/framesed_weights.bin audio.wav +``` + +A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is `framesed/framesed.h`. + +## CLI + +```text +build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav +build/framesed --list-classes +``` + +Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono. +The CLI does not resample. `--threads` defaults to 1 and must be positive. +`--track-step` enables score tracks and must be at least 0.04 seconds; the +returned step rounds down to a multiple of 0.04 seconds. + +Default detection uses 59 caption groups from `framesed/framesed_policy.inc`. +`--all-classes` detects each of the 447 model classes independently, without +caption grouping or suppression. This includes speech and background/context +classes, not just sound effects. + +```sh +build/framesed --all-classes --weights models/framesed_weights.bin audio.wav +build/framesed --list-classes # tab-separated index, AudioSet ID, label; no weights needed +``` + +Class-mode gates are uniform: start score 0.50, release score 0.35, minimum +duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated; +recognition quality varies by class. Tracks are returned for classes reaching +0.35, even when no event survives the duration gate. + +Output is one record per line, with times in seconds: + +```text +event START END SCORE GROUP LABEL +track TIME SCORE GROUP LABEL +``` + +`GROUP` is a caption-group index by default, or a model class index (0–446) +with `--all-classes`. Labels may contain spaces and commas. + +## API + +Set `framesed_options.all_classes = 1` for individual-class detection; zero or +NULL options select caption groups. `framesed_class_label(index)` and +`framesed_class_mid(index)` return model metadata, or NULL for an invalid index. +`framesed_group_label(index)` always refers to caption groups. + +With `keep_planes = 1`, class-mode results contain 447 probability rows in +model order, at 0.04 s resolution. `framesed_infer_s16()` returns raw logits for one +10-second window. Free detection results with `framesed_result_free()` and logits +with `framesed_logits_free()`. + +The vocabulary is in [`framesed/classes.csv`](framesed/classes.csv). Its indices follow +PretrainedSED's [`as_strong_train_classes`](https://github.com/fschmid56/PretrainedSED/blob/1aa47e482f7e89904cba2338999345025d8b4e36/data_util/audioset_classes.py); +AudioSet IDs come from its `hf_dataset_gen/metadata` label tables. Class order +must match the weights; editing labels does not teach the model new sounds. +`make regenerate-classes` generates the compiled table; `make check-classes` +checks that it matches the CSV. Neither command requires PyTorch. + +## Recreating the weights + +`models/framesed_weights.bin` is an inference export of PretrainedSED's +[`frame_mn10_strong_1.pt`](https://github.com/fschmid56/PretrainedSED/releases/download/v0.0.1/frame_mn10_strong_1.pt), +not a separately trained model. The exporter folds batch normalization into +convolution weights, includes the Hann window and Kaldi mel filterbank, and +keeps the 447-class strong head (the weak head is omitted). + +From the project root, in a Python environment with `torch`, `torchaudio`, and +`numpy` installed (use matching torch/torchaudio versions): + +```sh +python3 devtools/export_framesed_weights.py +make test +``` + +The script downloads and verifies the upstream checkpoint before writing +`models/framesed_weights.bin`. For offline use or a different output path: + +```sh +python3 devtools/export_framesed_weights.py \ + --checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin +``` + +The exporter follows upstream revision `1aa47e482f7e89904cba2338999345025d8b4e36`. + +`devtools/export_framesed_classes.py` exports class metadata. + +## Build options + +- `make debug` / `make profile` / `make release` +- `make AUDIO_BLAS=none all test` for portable CPU inference +- `make AUDIO_BLAS=openblas all test` for OpenBLAS (requires OpenBLAS; + discovered through `pkg-config`) +- `make check-sanitize` for AddressSanitizer and UndefinedBehaviorSanitizer +- `make clean` removes build outputs, not weights or test data + +macOS defaults to Accelerate; other systems default to portable CPU. +Configuration and header changes automatically trigger rebuilds. +`FRAMESED_WEIGHTS` and `FRAMESED_REFERENCE` override test fixture paths. + +Link `libframesed.a` with the math and pthread libraries, plus Accelerate or OpenBLAS +when enabled. The WAV reader is used only by the CLI, not the library. + +## License + +MIT License. + +The model weights are `PretrainedSED/frame_mn10` by Floarian Schmid; see the [model repository](https://github.com/fschmid56/PretrainedSED) for details. -- cgit v1.3.1