Integration of a Sign Language Interface in a Socially Assistive Robot

• Sohaila Chtioui Bouhou ^[1] ; Alberto Tudela ; José Galeas ; Oscar Pons ; Antonio Bandera ; A. Romero Garcés
  1. [1] Universidad de Málaga

     Universidad de Málaga

     Málaga, España

     
• Localización: Proceedings of the XXIV Workshop of Physical Agents: September 5-6, 2024 / coord. por Miguel Cazorla Quevedo , Francisco Gómez Donoso , Félix Escalona Moncholi , 2024, ISBN 978-84-09-63822-2, págs. 158-168
• Idioma: español
• Enlaces
  • Texto completo
• Resumen

  • This work aims to design and implement a Sign Language interface to improve the communication of deaf people in hospital and telemedical environments. Using Mediapipe to detect landmarks and a Random Forest classifier to interpret gestures, a system capable of recognising and translating signs in real time has been developed. The interface includes an intuitive graphical application developed in Tkinter that allows the visualisation of the recognised signs and facilitates interaction with the user. In addition, detection modes for both letters and numbers have been implemented, and mechanisms for remote communication and system updating have been considered. The interface was integrated into a prototype robot to test its functionality in real environments.

• Referencias bibliográficas
  • von Agris, U., Zieren, J., Canzler, U. et al. Recent developments in visual sign language recognition. Univ Access Inf Soc 6, 323–362 (2008). https://doi.org/10.1007/s10209-007-0104-x
  • Köse, H., Uluer, P., Akalın, N. et al. The Effect of Embodiment in Sign Language Tutoring with Assistive Humanoid Robots. Int J of Soc Robotics...
  • Uluer, P., Akalın, N., Köse, H. A New Robotic Platform for Sign Language Tutoring. Int J of Soc Robotics 7, 571–585 (2015) https://doi.org/10.1007/s12369-015-0307-x
  • Ohneberg, C., Stöbich, N., Warmbein, A. et al. Assistive robotic systems in nursing care: a scoping review. BMC Nurs 22, 72 (2023). https://doi.org/10.1186/s12912- 023-01230-y
  • Romero-Garcés, A.; Bandera, J.P.; Marfil, R.; González-García, M.; Bandera, A. CLARA: Building a Socially Assistive Robot to Interact with...
  • Johnny, S., Nirmala, S.J. Sign Language Translator Using Machine Learning. SN COMPUT. SCI. 3, 36 (2022) https://doi.org/10.1007/s42979-021-00896-y
  • Quesada, L., López, G., Guerrero, L. Automatic recognition of the American sign language fingerspelling alphabet to assist people living with...
  • Ewe E.L.R., Lee C.P., Lim K.M., Kwek L.C., Alqahtani A. LAVRF: Sign language recognition via Lightweight Attentive VGG16 with Random Forest....
  • Bustos, P., Manso, L.J., Bandera, A., Bandera, J.P., García-Varea, I., Martínez- Gómez, J. The CORTEX cognitive robotics architecture: Use...
  • Shapiro, M., Preguiça, N., Baquero, C., Zawirski, M. A comprehensive study of Convergent and Commutative Replicated Data Types. https://inria.hal.science/inria-00555588,...