Fusing Audio and Metadata Embeddings Improves Language-based Audio

Paul Primus; Gerhard Widmer

Fusing Audio and Metadata Embeddings Improves Language-based Audio

Publikation: Beitrag in Buch/Bericht/Konferenzband › Konferenzbeitrag › Begutachtung

Abstract

Matching raw audio signals with textual descriptionsrequires understanding the audio's content and the description'ssemantics and then drawing connections between the two modalities.This paper investigates a hybrid retrieval system that utilizesaudio metadata as an additional clue to understand the content ofaudio signals before matching them with textual queries. Weexperimented with metadata often attached to audio recordings,such as keywords and natural-language descriptions, and weinvestigated late and mid-level fusion strategies to merge audioand metadata. Our hybrid approach with keyword metadata and latefusion improved the retrieval performance over a content-basedbaseline by 2.36 and 3.69 pp. mAP@10 on the ClothoV2 and AudioCapsbenchmarks, respectively.

Originalsprache	Englisch
Titel	Proceedings of the 32nd European Signal Processing Conference(EUSIPCO), Lyon, France
Seitenumfang	5
Publikationsstatus	Veröffentlicht - 2024

Wissenschaftszweige

202002 Audiovisuelle Medien
102 Informatik
102001 Artificial Intelligence
102003 Bildverarbeitung
102015 Informationssysteme
101019 Stochastik
103029 Statistische Physik
101018 Statistik
101017 Spieltheorie
202017 Embedded Systems
101016 Optimierung
101015 Operations Research
101014 Numerische Mathematik
101029 Mathematische Statistik
101028 Mathematische Modellierung
101026 Zeitreihenanalyse
101024 Wahrscheinlichkeitstheorie
102032 Computational Intelligence
102004 Bioinformatik
102013 Human-Computer Interaction
101027 Dynamische Systeme
305907 Medizinische Statistik
101004 Biomathematik
305905 Medizinische Informatik
101031 Approximationstheorie
102033 Data Mining
305901 Computerunterstützte Diagnose und Therapie
102019 Machine Learning
106007 Biostatistik
102018 Künstliche Neuronale Netze
106005 Bioinformatik
202037 Signalverarbeitung
202036 Sensorik
202035 Robotik

JKU-Schwerpunkte

Digital Transformation

Andere Dateien und Links

https://arxiv.org/abs/2406.15897

Dieses zitieren

@inproceedings{f49f473541e64607a0da2f007f6f9e8e,

title = "Fusing Audio and Metadata Embeddings Improves Language-based Audio",

abstract = "Matching raw audio signals with textual descriptionsrequires understanding the audio's content and the description'ssemantics and then drawing connections between the two modalities.This paper investigates a hybrid retrieval system that utilizesaudio metadata as an additional clue to understand the content ofaudio signals before matching them with textual queries. Weexperimented with metadata often attached to audio recordings,such as keywords and natural-language descriptions, and weinvestigated late and mid-level fusion strategies to merge audioand metadata. Our hybrid approach with keyword metadata and latefusion improved the retrieval performance over a content-basedbaseline by 2.36 and 3.69 pp. mAP@10 on the ClothoV2 and AudioCapsbenchmarks, respectively.",

author = "Paul Primus and Gerhard Widmer",

year = "2024",

language = "English",

booktitle = "Proceedings of the 32nd European Signal Processing Conference(EUSIPCO), Lyon, France",

}

TY - GEN

T1 - Fusing Audio and Metadata Embeddings Improves Language-based Audio

AU - Primus, Paul

AU - Widmer, Gerhard

PY - 2024

Y1 - 2024

N2 - Matching raw audio signals with textual descriptionsrequires understanding the audio's content and the description'ssemantics and then drawing connections between the two modalities.This paper investigates a hybrid retrieval system that utilizesaudio metadata as an additional clue to understand the content ofaudio signals before matching them with textual queries. Weexperimented with metadata often attached to audio recordings,such as keywords and natural-language descriptions, and weinvestigated late and mid-level fusion strategies to merge audioand metadata. Our hybrid approach with keyword metadata and latefusion improved the retrieval performance over a content-basedbaseline by 2.36 and 3.69 pp. mAP@10 on the ClothoV2 and AudioCapsbenchmarks, respectively.

AB - Matching raw audio signals with textual descriptionsrequires understanding the audio's content and the description'ssemantics and then drawing connections between the two modalities.This paper investigates a hybrid retrieval system that utilizesaudio metadata as an additional clue to understand the content ofaudio signals before matching them with textual queries. Weexperimented with metadata often attached to audio recordings,such as keywords and natural-language descriptions, and weinvestigated late and mid-level fusion strategies to merge audioand metadata. Our hybrid approach with keyword metadata and latefusion improved the retrieval performance over a content-basedbaseline by 2.36 and 3.69 pp. mAP@10 on the ClothoV2 and AudioCapsbenchmarks, respectively.

UR - https://arxiv.org/abs/2406.15897

M3 - Conference proceedings

BT - Proceedings of the 32nd European Signal Processing Conference(EUSIPCO), Lyon, France

ER -