Analyzing the CoNLL--X Shared Task from a Sentence Accuracy Perspective

• Autores: Miguel Ballesteros, Jesús Herrera, Virginia Francisco, Pablo Gervás Gómez-Navarro
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 48, 2012, págs. 29-34
• Idioma: inglés
• Títulos paralelos:
  • Analizando la CoNLL–X Shared Task con medidas basadas en precisión por frase
• Enlaces
  • Texto completo
• Resumen
  • español

    Hoy en día, dada la relevancia de las CoNLL shared tasks para Análisis de Dependencias, las medidas más usadas son las que allí se computaron. Esas medidas, están basadas en calcular globalmente la precisión palabra por palabra (o token por token) para todo el conjunto de frases. En nuestra opinión el usuario final de un analizador de dependencias podría esperar una precisión local basada en evaluar la precisión frase a frase. En estos casos, unas medidas diferentes pueden añadir algo de información que podría ser relevante acerca de que analizador devuelve un mejor resultado. Es por ello que presentamos el estudio de este artículo con la intención de enriquecer la descripción del comportamiento de los analizadores de dependencias.

  • English

    Nowadays, because of the relevance of the CoNLL shared tasks on Dependency Parsing, the most used evaluation measures are the ones computed in them. These measures, which are token–based, are computed globally for a whole big set of texts considering token by token. But a final user of a dependency parser would expect a high and stable accuracy for every parsed piece of text (usually one sentence). In this cases sentence–based measures add some information that could be relevant. This is why we developed the present study, which is addressed to get a richer description of the performance of dependency parsers.

• Referencias bibliográficas
  • Alena Böhmová, Jan Hajic, Eva Hajicová, and Barbora Hladká. 2001. The PDT: a 3-level annotation scenario. chapter 7. Kluwer Academic Publishers.
  • Eckhard Bick. 2006. Lingpars, a linguistically inspired, language-independent machine learner for dependency treebanks. In CoNLL-X.
  • Giuseppe Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. In CoNLL-X.
  • Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In IWPT. Deniz Yuret. 2006. Dependency...
  • Jan Hajic and Petr Zemánek. 2004. Prague arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on...
  • Jens Nilsson, Johan Hall, and Joakim Nivre. 2005. MAMBA meets TIGER: Reconstructing a Swedish treebank from antiquity. In NODALIDA Special...
  • Joakim Nivre, Johan Hall, Jens Nilsson, G. Eryigit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector...
  • Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. The CoNLL 2007 shared task...
  • Keh-Jiann Chen, Chi-Ching Luo, Ming-Chung Chang, Feng-Yi Chen, Chao-Jan Chen, Chu-Ren Huang, and Zhao-Ming Gao. 2003. Sinica treebank: Design...
  • Kemal Oflazer, Bilge Say, Dilek Zeynep Hakkani-Tür, and Gökhan Tür. 2003. Building a Turkish treebank.
  • Kiril Simov, Petya Osenova, Alexander Simov, and Milen Kouylekov. 2005. Design and implementation of the Bulgarian HPSG-based treebank. Journal...
  • Leonoor van der Beek, Gosse Bouma, Robert Malouf, and Gertjan van Noord. 2002. The Alpino dependency treebank. In Computational Linguistics...
  • Manuel Palomar, Montserrat Civit, Arantza Díaz, Lidia Moreno, Empar Bisbal, Marí J. Aranzabe, Alicia M. Ageno, Mª Antonia Martí MartÍ, and...
  • Markus Dreyer, David A. Smith, and Noah A. Smith. 2006. Vine parsing and minimum risk reranking for speed and precision. In CoNLL-X.
  • Matthias T. Kromann. 2003. The Danish dependency treebank and the underlying linguistic theory. In Joakim Nivre and Erhard Hinrichs, editors,...
  • Michael Schiehlen and Kristina Spranger. 2006. Language independent probabilistic context-free parsing bolstered by machine learning. In CoNLL-X.
  • Ming Wei Chang, Quang Do, and Dan Roth. 2006. A pipeline model for bottom-up dependency parsing. In CoNLL-X.
  • Nobuyuki Shimizu. 2006. Maximum spanning tree algorithm for non-projective labeled dependency parsing. In CoNLL-X.
  • Richard Johansson and Pierre Nugues. 2006. Investigating Multilingual Dependency Parsing. In CoNLL-X.
  • Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. Computational Linguistics.
  • Ryan McDonald, K. Lerman, and Fernando Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In CoNLL-X.
  • Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The tiger treebank. In TLT.
  • Sabine Buchholz and Erwin Marsi. 2006. Conll-x shared task on multilingual dependency parsing. In CoNLL-X.
  • Sander Canisius, Toine Bogers, Antal Van Den Bosch, and Jeroen Geertzen. 2006. Dependency parsing by inference over high-recall dependency...
  • Saso Dzeroski, Tomaz Erjavec, Nina Ledinek, Petr Pajas, Zdenek Zabokrtsky, and Andreja Zele. 2006. Towards a Slovene dependency treebank....
  • Sebastian Riedel, Ruket Çakici, and Ivan Meza-Ruiz. 2006. Multi-lingual dependency parsing with incremental integer linear programming. In...
  • Simon Corston-Oliver and Anthony Aue. 2006. Dependency parsing with reference to slovene, spanish and swedish. In CoNLL-X.
  • Susana Afonso, Eckhard Bick, Renato Haber, and Diana Santos. 2002. Floresta sintá(c)tica: A treebank for Portuguese. In LREC.
  • Ting Liu, Jinshan Ma, Huijia Zhu, and Sheng Li. 2006. Dependency parsing based on dynamic local optimization. In CoNLL-X.
  • Xavier Carreras, Mihai Surdeanu, and Lluís Márquez. 2006. Projective dependency parsing with perceptron. In CoNLL-X.
  • Yasuhiro Kawata and Julia Bartels. 2000. Stylebook for the Japanese treebank in VERBMOBIL. Verbmobil-Report 240, Seminar für Sprachwissenschaft,...
  • Yoav Goldberg and Michael Elhadad. 2010. An eficient algorithm for easy-first nondirectional dependency parsing. In NAACL.
  • Yu-Chieh Wu, Yue-Shi Lee, and Jie-Chi Yang. 2006. The Exploration of Deterministic and Eficient Dependency Parsing. In CoNLL-X.
  • Yuchang Cheng, Masayuki Asahara, and Yuji Matsumoto. 2006. Multi-lingual dependency parsing at NAIST. In CoNLL-X.