Improving Audio Spectrogram Transformers ForSound Event Detection Through Multi-Stage Training

This technical report describes the CP-JKU team’s submission
for Task 4 Sound Event Detection with Heterogeneous Training
Datasets and Potentially Missing Labels of the DCASE 24 Chal
lenge. We fine-tune three large Audio Spectrogram Transformers,
PaSST, BEATs, and ATST, on the joint DESED and MAESTRO
datasets in a two-stage training procedure. The first stage closely
matches the baseline system setup and trains a CRNN model while
keeping the large pre-trained transformer model frozen. In the sec
ond stage, both CRNN and transformer are fine-tuned using heavily
weighted self-supervised losses. After the second stage, we com
pute strong pseudo-labels for all audio clips in the training set using
an ensemble of all three fine-tuned transformers. Then, in a sec
ond iteration, we repeat the two-stage training process and include a
distillation loss based on the pseudo-labels, boosting single-model
performance substantially. Additionally, we pre-train PaSST and
ATST on the subset of AudioSet that comes with strong temporal
labels, before fine-tuning them on the Task 4 datasets