Sentiment Analysis and Classification of Hotel Opinions in Twitter With the Transformer Architecture

• Sergio Arroni ^[1] ; Yeray Galán ^[1] ; Xiomarah Guzmán-Guzmán ^[1] ; Edward Rolando NúñezValdez ^[1] ; Alberto Gómez ^[1]
  1. [1] Universidad de Oviedo

     Universidad de Oviedo

     Oviedo, España

     
• Localización: IJIMAI, ISSN-e 1989-1660, Vol. 8, Nº. 1, 2023 (Ejemplar dedicado a: Special Issue on AI-driven Algorithms and Applications in the Dynamic and Evolving Environments), págs. 53-63
• Idioma: inglés
• DOI: 10.9781/ijimai.2023.02.005
• Enlaces
  • Texto completo (pdf)
• Resumen

  • Sentiment analysis is of great importance to parties who are interested is analyzing the public opinion in social networks. In recent years, deep learning, and particularly, the attention-based architecture, has taken over the field, to the point where most research in Natural Language Processing (NLP) has been shifted towards the development of bigger and bigger attention-based transformer models. However, those models are developed to be all-purpose NLP models, so for a concrete smaller problem, a reduced and specifically studied model can perform better. We propose a simpler attention-based model that makes use of the transformer architecture to predict the sentiment expressed in tweets about hotels in Las Vegas. With their relative predicted performance, we compare the similarity of our ranking to the actual ranking in TripAdvisor to those obtained by more rudimentary sentiment analysis approaches, outperforming them with a 0.64121 Spearman correlation coefficient. We also compare our performance to DistilBERT, obtaining faster and more accurate results and proving that a model designed for a particular problem can perform better than models with several millions of trainable parameters.

• Referencias bibliográficas
  • K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,”...
  • D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp....
  • D. W. Otter, J. R. Medina, and J. K. Kalita, “A Survey of the Usages of Deep Learning for Natural Language Processing,” IEEE Transactions...
  • A. Vaswani et al., “Attention is All you Need,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  • J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” 2019...
  • V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint...
  • S. Smith et al., “Using DeepSpeed and Megatron to Train MegatronTuring NLG 530B, A Large-Scale Generative Language Model,” arXiv preprint...
  • K. Philander and Y. Y. Zhong, “Twitter sentiment analysis: Capturing sentiment from integrated resort tweets,” International Journal of Hospitality...
  • S. Barke, R. Kunkel, N. Polikarpova, E. Meinhardt, E. Baković, and L. Bergen, “Constraint-based Learning of Phonological Processes,” 2019...
  • O. Güngör, T. Güngör, and S. Uskudarli, “EXSEQREG: Explaining sequence-based NLP tasks with regions with a case study using morphological...
  • E. M. Ponti, A. Korhonen, R. Reichart, and I. Vulić, “Isomorphic transfer of syntactic structures in cross-lingual NLP,” 56th Annual Meeting...
  • C. Hutto, E. G.-P. of the international A. conference on, and undefined 2014, “Vader: A parsimonious rule-based model for sentiment analysis...
  • P. Chikersal, S. Poria and E. Cambria, “SeNTU: Sentiment Analysis of Tweets by Combining a Rule-based Classifier with Supervised Learning,”...
  • F. Wunderlich and D. Memmert, “Innovative Approaches in Sports Science—Lexicon-Based Sentiment Analysis as a Tool to Analyze Sports Related...
  • S. Lai, L. Xu, K. Liu and J. Zhao, “Recurrent convolutional neural networks for text classification,” Proceedings of the AAAI conference on...
  • H. Kim and Y. S. Jeong, “Sentiment Classification Using Convolutional Neural Networks,” Applied Sciences, vol. 9, no. 11, p. 2347, Jun. 2019,...
  • X. Fang and J. Zhan, “Sentiment analysis using product review data,” Journal of Big Data, vol. 2, no. 1, pp. 1–14, Dec. 2015, doi: 10.1186/S40537-...
  • M. Imran, P. Mitra, and C. Castillo, “Twitter as a Lifeline: Humanannotated Twitter Corpora for NLP of Crisis-related Messages,” Proceedings...
  • X. Liu, H. Shin, and A. C. Burns, “Examining the impact of luxury brand’s social media marketing on customer engagement: Using big data analytics...
  • F. Z. Xing, E. Cambria, and R. E. Welsch, “Natural language based financial forecasting: a survey,” Artificial Intelligence Review, vol....
  • M. G. Huddar, S. S. Sannakki, and V. S. Rajpurohit, “Attention-based multi-modal sentiment analysis and emotion detection in conversation...
  • P. Dcunha, “Aspect Based Sentiment Analysis and Feedback Ratings using Natural Language Processing on European Hotels,” Doctoral thesis....
  • T. Ghorpade and L. Ragha, “Featured based sentiment classification for hotel reviews using NLP and Bayesian classification,” Proceedings...
  • B.-Ş. Posedaru, T.-M. Georgescu, and F.-V. Pantelimon, “Natural Learning Processing based on Machine Learning Model for automatic analysis...
  • W. Medhat, A. Hassan, and H. Korashy, “Sentiment analysis algorithms and applications: A survey,” Ain Shams Engineering Journal, vol. 5,...
  • L. C. Yu, J. L. Wu, P. C. Chang, and H. S. Chu, “Using a contextual entropy model to expand emotion words and their intensity for the sentiment...
  • M. Hagenau, M. Liebmann, and D. Neumann, “Automated news reading: Stock price prediction based on financial news using context-capturing...
  • I. Maks and P. Vossen, “A lexicon model for deep sentiment analysis and opinion mining applications,” Decision Support System, vol. 53, no....
  • J. Wang et al., “Systematic Evaluation of Research Progress on Natural Language Processing in Medicine Over the Past 20 Years: Bibliometric...
  • A. Alsudais, G. Leroy, and A. Corso, “We know where you are tweeting from: Assigning a type of place to tweets using natural language processing...
  • Y. Goldberg and M. E. Ben, “splitSVM: Fast, Space-Efficient, nonHeuristic, Polynomial Kernel Computation for NLP Applications,” Association...
  • C. Bartz, T. Herold, H. Yang, and C. Meinel, “Language Identification Using Deep Convolutional Recurrent Neural Networks,” Lecture Notes...
  • Y. LeCun, K. Kavukcuoglu, and C. Farabet, “Convolutional networks and applications in vision,” 2010 IEEE International Symposium on Circuits...
  • A. Conneau, H. Schwenk, Y. le Cun, and L. Lo¨ıc Barrault, “Very Deep Convolutional Networks for Text Classification,” Nature, pp. 1–11, Jun....
  • S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/ NECO.1997.9.8.1735....
  • T. Wang, P. Chen, K. Amaral, and J. Qiang, “An Experimental Study of LSTM Encoder-Decoder Model for Text Simplification,” arXiv preprint...
  • K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” 8th...
  • Z. Shaheen, G. Wohlgenannt, and E. Filtz, “Large Scale Legal Text Classification Using Transformer Models,” Computer Science ArXiV, vol....
  • D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” 3rd International Conference...
  • T. Shao, Y. Guo, H. Chen, and Z. Hao, “Transformer-Based Neural Network for Answer Selection in Question Answering,” IEEE Access, vol. 7,...
  • U. Khandelwal, K. Clark, D. Jurafsky, and Ł. Kaiser, “Sample Efficient Text Summarization Using a Single Pre-Trained Transformer,” arXiv...
  • T. Wang, X. Wan, and H. Jin, “Amr-to-text generation with graph transformer,” Transactions of the Association for Computational Linguistics,...
  • S. Hochreiter, “The vanishing gradient problem during learning recurrent neural nets and problem solutions,” International Journal of Uncertainty,...
  • A. Gulati et al., “Conformer: Convolution-augmented Transformer for Speech Recognition,” Proceedings of the Annual Conference of the International...
  • A. Rives et al., “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,” Proceedings...
  • T. Lin, Y. Wang, X. Liu, and X. Qiu, “A Survey of Transformers,” OpenAI, Jun. 2021.
  • T. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems, 2020, vol. 33, pp. 1877–1901....
  • “Hotel Reviews - dataset by datafiniti | data.world.” https://data.world/ datafiniti/hotel-reviews (accessed Feb. 11, 2022).
  • “amazon_reviews_multi · Datasets at Hugging Face.” https://huggingface. co/datasets/amazon_reviews_multi (accessed Feb. 11, 2022).
  • “Hotel Reviews - dataset by datafiniti.” https://data.world/datafiniti/ hotel-reviews (accessed Mar. 23, 2022).
  • A. Joshi, S. Kale, S. Chandel, and D. K. Pal, “Likert Scale: Explored and Explained,” British Journal of Applied Science & Technology,...
  • François Chollet, “Keras: the Python deep learning API,” Astrophysics Source Code Library, 2018.
  • A. Belhadi, Y. Djenouri, J. C. W. Lin, and A. Cano, “A data-driven approach for twitter hashtag recommendation,” IEEE Access, vol. 8, pp....
  • K. Philander and Y. Y. Zhong, “Twitter sentiment analysis: Capturing sentiment from integrated resort tweets,” International Journal of Hospitality...
  • “Natural Language Processing with Transformers .” https://www. oreilly.com/library/view/natural-language-processing/9781098103231/ (accessed...
  • A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018.
  • A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” OpenAI, vol. 1,...