PhrasIS: Phrase Inference and Similarity Benchmark

• I. Lopez-Gazpio ^[1] ; J. Gaviria de la Puerta ^[1] ; P. García ^[1] ; H. Sanjurjo-González ^[1] ; B. Sanz ^[1] ; M. Maritxalar ; E. Agirre
  1. [1] Universidad de Deusto

     Universidad de Deusto

     Bilbao, España

     
• Localización: 16th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2021) / Hugo Sanjurjo González (ed. lit.), Iker Pastor López (ed. lit.) , Héctor Quintián Bringas (ed. lit.), Emilio Santiago Corchado Rodríguez (ed. lit.) , 2022, ISBN 978-3-030-87868-9, págs. 261-272
• Idioma: inglés
• Texto completo no disponible (Saber más ...)
• Resumen

  • We present PhrasIS, a dataset of Phrase pairs with Inference and Similarity annotations for the evaluation of semantic representations. This dataset fills the gap between word and sentence-level datasets, allowing to evaluate compositional models at a finer granularity than sentences. Contrary to other datasets, the phrase pairs are extracted from naturally occurring text in image captions and news, and were annotated by experts. We analyze the dataset, showing the relation between inference labels and similarity scores, and evaluated several well-known techniques obtaining satisfactory performance. The gap with respect to annotator agreement shows that there is plenty of room for improvement. In addition, we introduce the use of similarity and relatedness inference relations, showing that they are useful for inference. With 10K phrase pairs split in development and test, the dataset is an excellent benchmark for testing meaning representation systems.