aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorOnur Gündüz <ogunduz@gmail.com>2026-10-21 20:08:53 +0200
committerOnur Gündüz <ogunduz@gmail.com>2026-10-21 20:08:53 +0200
commita1e355b8ceb12ac66c9d82e7786df4c3c40be479 (patch)
tree9d69cf196ec00e85009e1037a3a314d703466ac3 /README.md
downloadframesed-a1e355b8ceb12ac66c9d82e7786df4c3c40be479.tar.gz
framesed-a1e355b8ceb12ac66c9d82e7786df4c3c40be479.zip
Diffstat (limited to 'README.md')
-rw-r--r--README.md119
1 files changed, 119 insertions, 0 deletions
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..00ae9f4
--- /dev/null
+++ b/README.md
@@ -0,0 +1,119 @@
+# framesed
+
+Frame-level sound event detection using `PretrainedSED/frame_mn10`.
+
+## Build and run
+
+```sh
+make # libframesed.a and build/framesed
+make test # model parity, policies, threading, CLI, class metadata
+build/framesed --weights models/framesed_weights.bin audio.wav
+```
+
+A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is `framesed/framesed.h`.
+
+## CLI
+
+```text
+build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav
+build/framesed --list-classes
+```
+
+Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono.
+The CLI does not resample. `--threads` defaults to 1 and must be positive.
+`--track-step` enables score tracks and must be at least 0.04 seconds; the
+returned step rounds down to a multiple of 0.04 seconds.
+
+Default detection uses 59 caption groups from `framesed/framesed_policy.inc`.
+`--all-classes` detects each of the 447 model classes independently, without
+caption grouping or suppression. This includes speech and background/context
+classes, not just sound effects.
+
+```sh
+build/framesed --all-classes --weights models/framesed_weights.bin audio.wav
+build/framesed --list-classes # tab-separated index, AudioSet ID, label; no weights needed
+```
+
+Class-mode gates are uniform: start score 0.50, release score 0.35, minimum
+duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated;
+recognition quality varies by class. Tracks are returned for classes reaching
+0.35, even when no event survives the duration gate.
+
+Output is one record per line, with times in seconds:
+
+```text
+event START END SCORE GROUP LABEL
+track TIME SCORE GROUP LABEL
+```
+
+`GROUP` is a caption-group index by default, or a model class index (0–446)
+with `--all-classes`. Labels may contain spaces and commas.
+
+## API
+
+Set `framesed_options.all_classes = 1` for individual-class detection; zero or
+NULL options select caption groups. `framesed_class_label(index)` and
+`framesed_class_mid(index)` return model metadata, or NULL for an invalid index.
+`framesed_group_label(index)` always refers to caption groups.
+
+With `keep_planes = 1`, class-mode results contain 447 probability rows in
+model order, at 0.04 s resolution. `framesed_infer_s16()` returns raw logits for one
+10-second window. Free detection results with `framesed_result_free()` and logits
+with `framesed_logits_free()`.
+
+The vocabulary is in [`framesed/classes.csv`](framesed/classes.csv). Its indices follow
+PretrainedSED's [`as_strong_train_classes`](https://github.com/fschmid56/PretrainedSED/blob/1aa47e482f7e89904cba2338999345025d8b4e36/data_util/audioset_classes.py);
+AudioSet IDs come from its `hf_dataset_gen/metadata` label tables. Class order
+must match the weights; editing labels does not teach the model new sounds.
+`make regenerate-classes` generates the compiled table; `make check-classes`
+checks that it matches the CSV. Neither command requires PyTorch.
+
+## Recreating the weights
+
+`models/framesed_weights.bin` is an inference export of PretrainedSED's
+[`frame_mn10_strong_1.pt`](https://github.com/fschmid56/PretrainedSED/releases/download/v0.0.1/frame_mn10_strong_1.pt),
+not a separately trained model. The exporter folds batch normalization into
+convolution weights, includes the Hann window and Kaldi mel filterbank, and
+keeps the 447-class strong head (the weak head is omitted).
+
+From the project root, in a Python environment with `torch`, `torchaudio`, and
+`numpy` installed (use matching torch/torchaudio versions):
+
+```sh
+python3 devtools/export_framesed_weights.py
+make test
+```
+
+The script downloads and verifies the upstream checkpoint before writing
+`models/framesed_weights.bin`. For offline use or a different output path:
+
+```sh
+python3 devtools/export_framesed_weights.py \
+ --checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin
+```
+
+The exporter follows upstream revision `1aa47e482f7e89904cba2338999345025d8b4e36`.
+
+`devtools/export_framesed_classes.py` exports class metadata.
+
+## Build options
+
+- `make debug` / `make profile` / `make release`
+- `make AUDIO_BLAS=none all test` for portable CPU inference
+- `make AUDIO_BLAS=openblas all test` for OpenBLAS (requires OpenBLAS;
+ discovered through `pkg-config`)
+- `make check-sanitize` for AddressSanitizer and UndefinedBehaviorSanitizer
+- `make clean` removes build outputs, not weights or test data
+
+macOS defaults to Accelerate; other systems default to portable CPU.
+Configuration and header changes automatically trigger rebuilds.
+`FRAMESED_WEIGHTS` and `FRAMESED_REFERENCE` override test fixture paths.
+
+Link `libframesed.a` with the math and pthread libraries, plus Accelerate or OpenBLAS
+when enabled. The WAV reader is used only by the CLI, not the library.
+
+## License
+
+MIT License.
+
+The model weights are `PretrainedSED/frame_mn10` by Floarian Schmid; see the [model repository](https://github.com/fschmid56/PretrainedSED) for details.