1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
# framesed
Frame-level sound event detection using `PretrainedSED/frame_mn10`.
## Build and run
```sh
make # libframesed.a and build/framesed
make test # model parity, policies, threading, CLI, class metadata
build/framesed --weights models/framesed_weights.bin audio.wav
```
A C11 compiler, Make, and POSIX threads are required. Tests and metadata generation also require Python 3 (standard library only). The public API is `framesed/framesed.h`.
## CLI
```text
build/framesed [--all-classes] [--threads N] [--track-step SECONDS] --weights FILE INPUT.wav
build/framesed --list-classes
```
Input is 16-bit PCM WAV at 16 kHz; multiple channels are averaged to mono.
The CLI does not resample. `--threads` defaults to 1 and must be positive.
`--track-step` enables score tracks and must be at least 0.04 seconds; the
returned step rounds down to a multiple of 0.04 seconds.
Default detection uses 59 caption groups from `framesed/framesed_policy.inc`.
`--all-classes` detects each of the 447 model classes independently, without
caption grouping or suppression. This includes speech and background/context
classes, not just sound effects.
```sh
build/framesed --all-classes --weights models/framesed_weights.bin audio.wav
build/framesed --list-classes # tab-separated index, AudioSet ID, label; no weights needed
```
Class-mode gates are uniform: start score 0.50, release score 0.35, minimum
duration 0.24 s, allowed gap 0.32 s. They are not individually calibrated;
recognition quality varies by class. Tracks are returned for classes reaching
0.35, even when no event survives the duration gate.
Output is one record per line, with times in seconds:
```text
event START END SCORE GROUP LABEL
track TIME SCORE GROUP LABEL
```
`GROUP` is a caption-group index by default, or a model class index (0–446)
with `--all-classes`. Labels may contain spaces and commas.
## API
Set `framesed_options.all_classes = 1` for individual-class detection; zero or
NULL options select caption groups. `framesed_class_label(index)` and
`framesed_class_mid(index)` return model metadata, or NULL for an invalid index.
`framesed_group_label(index)` always refers to caption groups.
With `keep_planes = 1`, class-mode results contain 447 probability rows in
model order, at 0.04 s resolution. `framesed_infer_s16()` returns raw logits for one
10-second window. Free detection results with `framesed_result_free()` and logits
with `framesed_logits_free()`.
The vocabulary is in [`framesed/classes.csv`](framesed/classes.csv). Its indices follow
PretrainedSED's [`as_strong_train_classes`](https://github.com/fschmid56/PretrainedSED/blob/1aa47e482f7e89904cba2338999345025d8b4e36/data_util/audioset_classes.py);
AudioSet IDs come from its `hf_dataset_gen/metadata` label tables. Class order
must match the weights; editing labels does not teach the model new sounds.
`make regenerate-classes` generates the compiled table; `make check-classes`
checks that it matches the CSV. Neither command requires PyTorch.
## Recreating the weights
`models/framesed_weights.bin` is an inference export of PretrainedSED's
[`frame_mn10_strong_1.pt`](https://github.com/fschmid56/PretrainedSED/releases/download/v0.0.1/frame_mn10_strong_1.pt),
not a separately trained model. The exporter folds batch normalization into
convolution weights, includes the Hann window and Kaldi mel filterbank, and
keeps the 447-class strong head (the weak head is omitted).
From the project root, in a Python environment with `torch`, `torchaudio`, and
`numpy` installed (use matching torch/torchaudio versions):
```sh
python3 devtools/export_framesed_weights.py
make test
```
The script downloads and verifies the upstream checkpoint before writing
`models/framesed_weights.bin`. For offline use or a different output path:
```sh
python3 devtools/export_framesed_weights.py \
--checkpoint /path/to/frame_mn10_strong_1.pt --output /tmp/framesed_weights.bin
```
The exporter follows upstream revision `1aa47e482f7e89904cba2338999345025d8b4e36`.
`devtools/export_framesed_classes.py` exports class metadata.
## Build options
- `make debug` / `make profile` / `make release`
- `make AUDIO_BLAS=none all test` for portable CPU inference
- `make AUDIO_BLAS=openblas all test` for OpenBLAS (requires OpenBLAS;
discovered through `pkg-config`)
- `make check-sanitize` for AddressSanitizer and UndefinedBehaviorSanitizer
- `make clean` removes build outputs, not weights or test data
macOS defaults to Accelerate; other systems default to portable CPU.
Configuration and header changes automatically trigger rebuilds.
`FRAMESED_WEIGHTS` and `FRAMESED_REFERENCE` override test fixture paths.
Link `libframesed.a` with the math and pthread libraries, plus Accelerate or OpenBLAS
when enabled. The WAV reader is used only by the CLI, not the library.
## License
MIT License.
The model weights are `PretrainedSED/frame_mn10` by Floarian Schmid; see the [model repository](https://github.com/fschmid56/PretrainedSED) for details.
|