Distilling Knowledge For Low-Complexity Convolutional Neural Networks From a Patchout Audio Transformer

Florian Schmid; Shahed Masoudian; Khaled Koutini; Gerhard Widmer

Distilling Knowledge For Low-Complexity Convolutional Neural Networks From a Patchout Audio Transformer

Florian Schmid, Shahed Masoudian, Khaled Koutini, Gerhard Widmer

Institute of Computational Perception

Research output: Chapter in Book/Report/Conference proceeding › Conference proceedings

Abstract

In this technical report, we describe the CP-JKU team’s submission for Task 1 Low-Complexity Acoustic Scene Classification of the DCASE 22 challenge. We use Knowledge Distillation to teach low-complexity CNN student models from Patchout Spectrogram Transformer (PaSST) models. We use the pre-trained PaSST models on Audioset and fine-tune them on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. We experiment with using an ensemble of teachers, different receptive fields of the student models, and mixing frequency-wise statistics of spectrograms to enhance generalization to unseen devices. Finally, the student models are quantized in order to perform inference computations using 8 bit integers, simulating the low-complexity constraints of edge devices.

Original language	English
Title of host publication	in Detection and Classification of Acoustic Scenes and Events (DCASE2022 Challenge), Technical Report
Number of pages	5
Publication status	Published - 2022

Fields of science

202002 Audiovisual media
102 Computer Sciences
102001 Artificial intelligence
102003 Image processing
102015 Information systems

JKU Focus areas

Digital Transformation

Cite this

@inproceedings{53f93e4611104feda9ddc672937815dd,

title = "Distilling Knowledge For Low-Complexity Convolutional Neural Networks From a Patchout Audio Transformer",

abstract = "In this technical report, we describe the CP-JKU team{\textquoteright}s submission for Task 1 Low-Complexity Acoustic Scene Classification of the DCASE 22 challenge. We use Knowledge Distillation to teach low-complexity CNN student models from Patchout Spectrogram Transformer (PaSST) models. We use the pre-trained PaSST models on Audioset and fine-tune them on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. We experiment with using an ensemble of teachers, different receptive fields of the student models, and mixing frequency-wise statistics of spectrograms to enhance generalization to unseen devices. Finally, the student models are quantized in order to perform inference computations using 8 bit integers, simulating the low-complexity constraints of edge devices.",

author = "Florian Schmid and Shahed Masoudian and Khaled Koutini and Gerhard Widmer",

year = "2022",

language = "English",

booktitle = "in Detection and Classification of Acoustic Scenes and Events (DCASE2022 Challenge), Technical Report",

}

TY - GEN

T1 - Distilling Knowledge For Low-Complexity Convolutional Neural Networks From a Patchout Audio Transformer

AU - Schmid, Florian

AU - Masoudian, Shahed

AU - Koutini, Khaled

AU - Widmer, Gerhard

PY - 2022

Y1 - 2022

N2 - In this technical report, we describe the CP-JKU team’s submission for Task 1 Low-Complexity Acoustic Scene Classification of the DCASE 22 challenge. We use Knowledge Distillation to teach low-complexity CNN student models from Patchout Spectrogram Transformer (PaSST) models. We use the pre-trained PaSST models on Audioset and fine-tune them on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. We experiment with using an ensemble of teachers, different receptive fields of the student models, and mixing frequency-wise statistics of spectrograms to enhance generalization to unseen devices. Finally, the student models are quantized in order to perform inference computations using 8 bit integers, simulating the low-complexity constraints of edge devices.

AB - In this technical report, we describe the CP-JKU team’s submission for Task 1 Low-Complexity Acoustic Scene Classification of the DCASE 22 challenge. We use Knowledge Distillation to teach low-complexity CNN student models from Patchout Spectrogram Transformer (PaSST) models. We use the pre-trained PaSST models on Audioset and fine-tune them on the TAU Urban Acoustic Scenes 2022 Mobile development dataset. We experiment with using an ensemble of teachers, different receptive fields of the student models, and mixing frequency-wise statistics of spectrograms to enhance generalization to unseen devices. Finally, the student models are quantized in order to perform inference computations using 8 bit integers, simulating the low-complexity constraints of edge devices.

M3 - Conference proceedings

BT - in Detection and Classification of Acoustic Scenes and Events (DCASE2022 Challenge), Technical Report

ER -