aboutsummaryrefslogtreecommitdiff
path: root/README.md
blob: 8407db4247a5b25152bd0e69413dd6dbaa20fa61 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
# pyseg

`pyannote/segmentation-3.0` speaker segmentation in dependency-free C.

The model reads ten seconds of 16 kHz mono audio and returns, for each of 589
output frames, a distribution over seven classes: no speaker, one of three
speakers alone, or one of the three pairs. That is enough for voice activity
detection, overlapped speech detection and per-frame speaker counting over a
recording of any length, all of which this library does.

The library is C11 and libc. The weights are compiled into the binary.
Building the weight blob needs PyTorch and pyannote.audio once.

## Build

```sh
make                    # library and CLI, weights embedded
make test               # all gates (needs the fixtures below)
make bench              # timings
```

Options: `THREADS=0` drops the pthread dependency, `EMBED=0` leaves the weights
out of the binary so it loads `--weights pyseg_weights.bin` instead, `STATIC=1`
links a self-contained executable, `OPT=...` overrides the compiler flags.

Do not add `-ffast-math`. It changes how the reductions contract and moves the
results outside the tolerances above.

## Command line

```
pyseg_cli [options] file.wav

  --mode speech|overlap|count|frames   what to print (default speech)
  --step SECONDS       window hop, default 1.0 (a tenth of the window)
  --threads N          windows to run in parallel, default 1
  --batch N            windows per engine call, default 4
  --weights FILE       weight blob; default is the embedded copy
  --onset X            score above which a region opens, default 0.5
  --offset X           score below which it closes, default 0.5
  --min-on SECONDS     drop regions shorter than this, default 0
  --min-off SECONDS    fill gaps shorter than this, default 0
  --scalar             portable C kernels instead of SIMD
```

Input must be 16 kHz 16-bit PCM WAV; the first channel is used. `speech` and
`overlap` print `start end` pairs in seconds, `count` prints the expected
number of speakers per frame, and `frames` prints all three tracks.

```console
$ pyseg_cli --threads 8 meeting.wav
0.031 0.588
2.427 18.593
18.897 27.149
```

Cost scales with `--step`: the default hop of one second means every part of
the recording is covered by ten windows, which is what pyannote's pipelines are
tuned for. A larger hop costs proportionally less and blurs the aggregate.

## Library

```c
#include "pyseg.h"

pyseg_t *v = pyseg_create();                 /* embedded weights */
float probs[PYSEG_FRAMES * PYSEG_CLASSES];
pyseg_process_f32(v, samples, probs);        /* 160000 samples in */
pyseg_destroy(v);
```

`probs` holds one distribution over the seven powerset classes per frame.
`pyseg_to_multilabel` turns that into per-speaker activity, and frame `f`
covers the 991 samples starting at `f * PYSEG_FRAME_STEP`.

For a whole recording:

```c
pyseg_slide_params p;
pyseg_slide_params_default(&p);
p.threads = 8;

size_t n = pyseg_slide_frames(num_samples, p.step);
float *tracks = malloc(n * PYSEG_TRACKS * sizeof(float));
pyseg_slide_s16(samples, num_samples, &p, tracks);

pyseg_segment segs[256];
size_t count = pyseg_binarize(tracks, n, PYSEG_TRACK_SPEECH, NULL, segs, 256);
```

`pyseg_slide` covers the recording with overlapping windows and averages them
exactly as `pyannote.audio.Inference` does. `pyseg_binarize` turns one track
into segments the way `pyannote.audio.utils.signal.Binarize` does.

No allocation, syscall or lock happens inside `pyseg_process_*`. An engine is
not thread-safe and is not meant to be: `pyseg_slide` gives each thread its
own, and so should any other caller.

## Layout

| path | content |
|---|---|
| `src/pyseg.h` | the public interface |
| `src/pyseg.c` | the forward pass |
| `src/kernels.c` | the one dense primitive, scalar and SIMD |
| `src/weights.c` | weight blob reader |
| `src/slide.c` | sliding window, aggregation, binarization |
| `tools/reference.py` | numpy forward pass, the executable specification |
| `tools/pipeline.py` | the aggregation reference |
| `tools/export_weights.py` | checkpoint to `pyseg_weights.bin` |
| `tools/embed.py` | blob to C source, standard library only |
| `tools/verify_cli.py` | the built binary against pyannote's pipelines |

## Regenerating the fixtures

The parity tests need captured references, which need the Python environment:

```sh
uv venv --python 3.12 .venv
VIRTUAL_ENV=.venv uv pip install "pyannote.audio>=3.1" numpy soundfile
make weights            # only after a layout change; the blob is checked in
make golden             # capture reference outputs
make test
make verify             # end to end against pyannote's own pipelines
```

`make golden` needs a 16 kHz WAV at `testdata_aepyx.wav`; override with
`AUDIO=...`.

## Licence

MIT license.

The model weights are `pyannote/segmentation-3.0`, by Alexis Plaquet and Hervé Bredin; see the
[model card](https://hf.co/pyannote/segmentation-3.0) and the
[paper](https://www.isca-speech.org/archive/interspeech_2023/plaquet23_interspeech.html).