Detailed info

WaveTransformer: An Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information

AuthorsAn Tran, Konstantinos Drossos, Tuomas Virtanen
TitleWaveTransformer: An Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information
AbstractAutomated audio captioning (AAC) is a novel task, where a method takes as an input an audio sample and outputs a textual description (i.e. a caption) of its contents. Most AAC methods are adapted from image captioning or machine translation fields. In this work, we present a novel AAC method, explicitly focused on the exploitation of the temporal and time-frequency patterns in audio. We employ three learnable processes for audio encoding, two for extracting the temporal and time-frequency information, and one to merge the output of the previous two processes. To generate the caption, we employ the widely used Transformer decoder. We assess our method utilizing the freely available splits of the Clotho dataset. Our results increase previously reported highest SPIDEr to 17.3, from 16.2 (higher is better).
ISBNTBD
Conference2021 29th European Signal Processing Conference (EUSIPCO)
Date23/08/2021
LocationDublin, Ireland
Year of Publication, Publisher2021
Urlhttps://zenodo.org/record/5723160#.YZ_nCrqxVPZ
DOI10.5281/zenodo.5723159
  
  

Key Facts

  • Project Coordinator: Dr. Sotiris Ioannidis
  • Institution: Foundation for Research and Technology Hellas (FORTH)
  • E-mail: marvel-info@marvel-project.eu 
  • Start: 01.01.2021
  • Duration: 36 months
  • Participating Organisations: 17
  • Number of countries: 12

Get Connected

Funding

eu FLAG

This project has received funding from the European Union’s Horizon 2020 Research and Innovation program under grant agreement No 957337. The website reflects only the view of the author(s) and the Commission is not responsible for any use that may be made of the information it contains.