Empirical Evaluation of Audio Encoders and Multimodal Fusion for Active Speaker Detection

• Suárez Quintana, Álvaro Antonio ^[1] ; Salas Cáceres, José Ignacio ^[1] ; Castrillón Santana, Modesto Fernando ^[1] ; Lorenzo Navarro, José Javier ^[1]
  1. [1] Universidad de Las Palmas de Gran Canaria

     Universidad de Las Palmas de Gran Canaria

     Gran Canaria, España

     
• Localización: Inteligencia artificial: Revista Iberoamericana de Inteligencia Artificial, ISSN-e 1988-3064, ISSN 1137-3601, Vol. 29, Nº. 78, 2026 (Ejemplar dedicado a: Inteligencia Artificial (December 2026)), págs. 202-214
• Idioma: inglés
• DOI: 10.4114/intartif.vol29iss78pp202-214
• Enlaces
  • Texto completo
• Resumen

  • Active Speaker Detection (ASD) aims to identify the speaking individual within a visual scene across consecutive frames. This task has gained significant prominence due to its role in applications like Human-Robot Interaction (HRI). While existing multimodal approaches primarily focus on innovating video encoders, leveraging architectures such as Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), and Graph Neural Networks (GNN), audio encoders have remained almost unaltered, often relying on Mel-frequency cepstral coefficients (MFCCs) processed through standard CNNs. In this paper, we explore the use of alternative audio encoding architectures and fusion mechanisms within Light-ASD framework. We conduct an extensive comparative analysis under in-database and cross-database settings using the AVA and WASD datasets. Furthermore, we investigate the impact of multi-dataset training by merging the training sets of both. Our results demonstrate that under in-database conditions, the MFCCs model achieves the best performance for AVA (93.54% mAP), while Wav2Vec yields the best results for WASD (94.28% mAP). In cross-database evaluations, X-Vectors exhibits the best generalization capabilities when trained on AVA (87.35% mAP, with a -2.76% relative performance drop relative to the in-database setting), whereas Wav2Vec generalizes more effectively when trained on WASD (83.62% mAP, with a -8.03% relative performance drop). Finally, we show that dataset fusion significantly enhances model robustness, delivering performance metrics that are comparable to their in-database setting, surpassing them in certain instances.

• Referencias bibliográficas
  • Siti Nur Aisyah Mohd Robi, Muhammad Atiff Zakwan Mohd Ariffin, Mohd Azri Mohd Izhar, Norul-husna Ahmad, and Hazilah Mad Kaidi. Active speaker...
  • Muhammad Mohsin Kabir, M. F. Mridha, Jungpil Shin, Israt Jahan, and Abu Quwsar Ohi. A survey of speaker recognition: Fundamental theories,...
  • Ville Vestman, Dhananjaya Gowda, Md Sahidullah, Paavo Alku, and Tomi Kinnunen. Speaker recognition from whispered speech: A tutorial survey...
  • Nikhilsingh Parihar, Kanan, Rashmi Chawla, and Giancarlo Fortino. Emotion recognition in human robot collaboration for multimodal approaches,...
  • Aref Farhadipour, Hossein Ranjbar, Masoumeh Chapariniya, Teodora Vukovic, Sarah Ebling, and Volker Dellwo. Multimodal emotion recognition...
  • Samira Rasouli, Garima Gupta, Elizabeth Nilsen, and Kerstin Dautenhahn. Potential applications of social robots in robot-assisted interventions...
  • Tony Belpaeme, James Kennedy, Aditi Ramachandran, Brian Scassellati, and Fumihide Tanaka. Social robots for education: A review. Science Robotics,...
  • Herbert H. Clark and Kerstin Fischer. Social robots as depictions of social agents. Behavioral and Brain Sciences, 46:e21, 2023.
  • Robotics Today. Bandit. [Online]. Available at: https://www.roboticstoday.com/robots/bandit-description [Accessed on: 17/12/2025].
  • SoftBank Robotics America. Meet Pepper: The Robot Built for People. [Online]. Available at: https://us.softbankrobotics.com/pepper [Accessed...
  • R. Cutler and L. Davis. Look who’s talking: speaker detection using video and audio correlation. In 2000 IEEE International Conference on...
  • Pradeep K. Atrey, M. Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S. Kankanhalli. Multi-modal fusion for multimedia analysis: a survey....
  • Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia...
  • Victoria Mingote, Alfonso Ortega, Antonio Miguel, and Eduardo Lleida. Audio-visual speaker diarization: Current databases, approaches and...
  • Juan León Alcázar, Fabian Caba, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbeláez, and Bernard Ghanem. Active speakers in context....
  • Juan León Alcázar, Fabian Caba, Ali K. Thabet, and Bernard Ghanem. Maas: Multi-modal assignation for active speaker detection. In Proceedings...
  • Okan Köpüklü, Maja Taseska, and Gerhard Rigoll. How to design a three-stage architecture for audio-visual active speaker detection in the...
  • Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? Exploring long-term temporal features...
  • Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, and Liangyin Chen. A lightweight model for active speaker detection. In...
  • Hugo Carneiro and Stefan Wermter. Fabulight-ASD: Unveiling speech activity via body language. Neural Computing and Applications, 37(5):3561–3579,...
  • Xizi Wang, Feng Cheng, and Gedas Bertasius. LocoNet: Long-short context network for active speaker detection. In Proceedings of the IEEE/CVF...
  • Guanjun Li, Jiangyan Yi, Zhengqi Wen, Ruibo Fu, Yuwang Wang, and Jianhua Tao. MSSG: Multi-scale speaker graph network for active speaker detection....
  • Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, Liangyin Chen, and Yanru Chen. LR-ASD: Lightweight and robust network...
  • Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In 2019 IEEE/CVF Conference...
  • Ahmed Shahabaz and Sudeep Sarkar. Increasing importance of joint analysis of audio and video in computer vision: A survey. IEEE Access, 12:59399–59430,...
  • Tiago Roxo, Joana Cabral Costa, Pedro R. M. Inácio, and Hugo Proença. WASD: A wilder active speaker detection dataset. IEEE Transactions on...
  • David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust DNN embeddings for speaker recognition....
  • Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. CoRR, abs/1904.05862,...
  • Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations....