Analysis of gender differences in facial expression recognition based on deep learning using explainable artificial intelligence

• Manresa-Yee, Cristina ^[1] ; Ramis, Silvia ^[1] ; Buades, José M. ^[1]
  1. [1] Universitat de les Illes Balears

     Universitat de les Illes Balears

     Palma de Mallorca, España

     
• Localización: IJIMAI, ISSN-e 1989-1660, Vol. 9, Nº. 1, 2024, págs. 18-27
• Idioma: inglés
• DOI: 10.9781/ijimai.2023.04.003
• Enlaces
  • Texto completo
• Resumen

  • Potential uses of automated Facial Expression Recognition (FER) cover a wide range of applications such as customer behavior analysis, healthcare applications or providing personalized services. Data for machine learning play a fundamental role, therefore, understanding the relevancy of the data in the outcomes is of utmost importance. In this work we present a study on how gender influences the learning of a FER system. We analyze with Explainable Artificial intelligence (XAI) techniques how gender contributes to the learning and assess which facial expressions are more similar regarding face regions that impact on the classification.

    Results show that there exist common regions in some expressions both for females and males with different intensities (e.g. happiness); however, there are other expressions like disgust, where important face regions differ. The insights of this work will help improving FER systems and understand the source of any inequality.

• Referencias bibliográficas
  • D. McDuff, E. Kodra, R. el Kaliouby, and M. LaFrance, “A large-scale analysis of sex differences in facial expressions,” PLOS ONE, vol. 12,...
  • L. Cattaneo, V. Veroni, S. Boria, G. Tassinari, and L. Turella, “Sex Differences in Affective Facial Reactions Are Present in Childhood,” Frontiers...
  • U. Dimberg and L.-O. Lundquist, “Gender differences in facial reactions to facial expressions.,” Biological Psychology, vol. 30, no. 2. Elsevier Science,...
  • M. A. Kraines, L. J. A. Kelberer, and T. T. Wells, “Sex differences in attention to disgust facial expressions,” Cognition and Emotion, vol....
  • K. AM and G. AH., “Sex differences in emotion: expression, experience, and physiology,” J Pers Soc Psychol., vol. 74, no. 3, pp. 686–703,...
  • M. Thunberg and U. Dimberg, “Gender Differences in Facial Reactions to Fear-Relevant Stimuli,” Journal of Nonverbal Behavior, vol. 24, no....
  • G. E. Schwartz, S. ‐L Brown, and G. L. Ahern, “Facial Muscle Patterning and Subjective Experience During Affective Imagery: Sex Differences.,” Psychophysiology,...
  • C. Evers, A. H. Fischer, and A. S. R. Manstead, “Gender and emotion regulation: a social appraisal perspective on anger.,” in Emotion regulation and...
  • P. Ekman, “Universals and cultural differences in facial expressions of emotion,” Nebraska Symposium on Motivation, vol. 19, pp. 207–283,...
  • P. Ekman and W. . Friesen, Facial action coding system: manual. Palo Alto, Calif.: Consulting Psychologists Press. OCLC: 5851545, 1978.
  • Y. Fan, J. C. K. Lam, and V. O. K. Li, “Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial...
  • O. Houstis and S. Kiliaridis, “Gender and age differences in facial expressions,” European Journal of Orthodontics, vol. 31, no. 5, pp. 459–466, 2009.
  • T. S. H. Wingenbach, C. Ashwin, and M. Brosnan, “Sex differences in facial emotion recognition across varying expression intensity levels from...
  • B. Montagne, R. P. C. Kessels, E. Frigerio, E. H. F. de Haan, and D. I. Perrett, “Sex differences in the perception of affective facial expressions: Do...
  • R. Campbell et al., “The classification of ‘fear’ from faces is associated with face recognition skill in women,” Neuropsychologia, vol. 40,...
  • A. K. Vail, J. F. Grafsgaard, K. E. Boyer, E. N. Wiebe, and J. C. Lester, “Gender Differences in Facial Expressions of Affect During Learning,” in...
  • J. C. Borod, E. Koff, and B. White, “Facial asymmetry in posed and spontaneous expressions of emotion,” Brain and Cognition, vol. 2, no. 2, pp....
  • W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: review and insights,” Procedia Computer Science, vol. 175, pp. 689–694,...
  • A. Domnich and G. Anbarjafari, “Responsible AI: Gender bias assessment in emotion recognition,” arXiv, pp. 1–19, 2021.
  • M. Deramgozin, S. Jovanovic, H. Rabah, and N. Ramzan, A Hybrid Explainable AI Framework Applied to Global and Local Facial Expression Recognition....
  • T. Xu, J. White, S. Kalkan, and H. Gunes, “Investigating Bias and Fairness in Facial Expression Recognition,” in ECCV Workshops 2020, 2020.
  • Z. Wang et al., “Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation,” in IEEE/CVF Conference on Computer Vision...
  • M. Lyons, M. Kamachi, and J. Gyoba, “The Japanese Female Facial Expression (JAFFE) Dataset.” Zenodo, 1998.
  • A. Heimerl, K. Weitz, T. Baur, and E. Andre, “Unraveling ML Models of Emotion with NOVA: Multi-Level Explainable AI for Non-Experts,” IEEE Transactions...
  • D. Schiller, T. Huber, M. Dietz, and E. André, “Relevance-Based Data Masking: A Model-Agnostic Transfer Learning Approach for Facial Expression...
  • P. Prajod, D. Schiller, T. Huber, and E. Andr’e, “Do Deep Neural Networks Forget Facial Action Units? - Exploring the Effects of Transfer...
  • K. Weitz, T. Hassan, U. Schmid, and J.-U. Garbas, “Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions...
  • G. del Castillo Torres, M. F. Roig-Maimó, M. Mascaró-Oliver, E. Amengual- Alcover, and R. Mas-Sansó, “Understanding How CNNs Recognize Facial Expressions:...
  • M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,” in Proceedings of the...
  • M. Alber et al., “iNNvestigate Neural Networks!,” Journal of Machine Learning Research, vol. 20, no. 93, pp. 1–8, 2019.
  • S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On Pixel-Wise Explanations for Non-Linear Classifier Decisions by...
  • C. Manresa-Yee and S. Ramis, “Assessing Gender Bias in Predictive Algorithms Using EXplainable AI,” in Proceedings of the XXI International Conference...
  • K. Grabowski et al., “Emotional expression in psychiatric conditions: New technology for clinicians,” Psychiatry and Clinical Neurosciences,...
  • A. M. Barreto, “Application of facial expression studies on the field of marketing,” Emotional expression: the brain and the face, vol. 9,...
  • S. Medjden, N. Ahmed, and M. Lataifeh, “Adaptive user interface design and analysis using emotion recognition through facial expressions and...
  • A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,”...
  • P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992.
  • S. Li and W. Deng, “Reliable Crowdsourcing and Deep Locality- Preserving Learning for Unconstrained Facial Expression Recognition,” IEEE Transactions...
  • World Health Organization, “Gender,” Available online: https://www.who. int/europe/health-topics/gender. .
  • Y. Chen and J. Joo, “Understanding and Mitigating Annotation Bias in Facial Expression Recognition,” in ICCV 2021, 2021.
  • J.-L. Lisani, S. Ramis, and F. Perales, “A Contrario Detection of Faces: A Case Example,” SIAM Journal on Imaging Sciences, vol. 10, pp. 2091–2118, Jan....
  • C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 Faces in- the-Wild Challenge: The First Facial Landmark Localization Challenge,” in...
  • S. Ramis, J. . Buades, F. J. Perales, and C. Manresa-Yee, “A Novel Approach to Cross dataset studies in Facial Expression Recognition,” Multimedia Tools...
  • A. Barredo Arrieta et al., “Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible...
  • J. Ward, “Hierarchical Grouping to Optimize an Objective Function,” Journal of the American Statistical Association, vol. 58, no. 301, pp....
  • O. Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252,...
  • C. Szegedy et al., “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.
  • M. Lin, C. Qiang, and Y. Shuicheng, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.
  • G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning with Applications in R. Springer, New York, 2013.
  • D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR...
  • K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” Proceedings of the...
  • Q. T. Ngo and S. Yoon, “Facial expression recognition based on weighted- cluster loss and deep transfer learning using a highly imbalanced...
  • C.-T. Yen and K.-H. Li, “Discussions of Different Deep Transfer Learning Models for Emotion Recognitions,” IEEE Access, vol. 10, pp. 102860– 102875,...
  • H. G. Wallbott, “Big girls don’t frown, big boys don’t cry—Gender differences of professional actors in communicating emotion via facial expression.,”...
  • N. K. Benamara, E. Zigh, T. B. Stambouli, and M. Keche, “Towards a Robust Thermal-Visible Heterogeneous Face Recognition Approach Based on...
  • M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino, and J. Torresen, “A facial expression recognition system using robust face features...
  • A. Alcaide, M. A. Patricio, A. Berlanga, A. Arroyo, and J. J. Cuadrado- Gallego, “LIPSNN: A Light Intrusion-Proving Siamese Neural Network Model...