tailieunhanh - Báo cáo khoa học: "Task-oriented Evaluation of Syntactic Parsers and Their Representations"

This paper presents a comparative evaluation of several state-of-the-art English parsers based on different frameworks. Our approach is to measure the impact of each parser when it is used as a component of an information extraction system that performs protein-protein interaction (PPI) identification in biomedical papers. We evaluate eight parsers (based on dependency parsing, phrase structure parsing, or deep parsing) using five different parse representations. We run a PPI system with several combinations of parser and parse representation, and examine their impact on PPI identification accuracy. . | Task-oriented Evaluation of Syntactic Parsers and Their Representations Yusuke Miyao Rune Sa tre Kenji Sagae Takuya Matsuzakil Jun ichi Tsujiitt Department of Computer Science University of Tokyo Japan School of Computer Science University of Manchester UK National Center for Text Mining UK yusuke sagae matuzaki tsujii @ Abstract This paper presents a comparative evaluation of several state-of-the-art English parsers based on different frameworks. Our approach is to measure the impact of each parser when it is used as a component of an information extraction system that performs protein-protein interaction PPI identification in biomedical papers. We evaluate eight parsers based on dependency parsing phrase structure parsing or deep parsing using five different parse representations. We run a PPI system with several combinations of parser and parse representation and examine their impact on PPI identification accuracy. Our experiments show that the levels of accuracy obtained with these different parsers are similar but that accuracy improvements vary when the parsers are retrained with domain-specific data. 1 Introduction Parsing technologies have improved considerably in the past few years and high-performance syntactic parsers are no longer limited to PCFG-based frameworks Charniak 2000 Klein and Manning 2003 Charniak and Johnson 2005 Petrov and Klein 2007 but also include dependency parsers McDonald and Pereira 2006 Nivre and Nilsson 2005 Sagae and Tsujii 2007 and deep parsers Kaplan et al. 2004 Clark and Curran 2004 Miyao and Tsujii 2008 . However efforts to perform extensive comparisons of syntactic parsers based on different frameworks have been limited. The most popular method for parser comparison involves the direct measurement of the parser output accuracy in terms of metrics such as bracketing precision and recall or dependency accuracy. This assumes the existence of a gold-standard test corpus such as the Penn Treebank .