tailieunhanh - Báo cáo khoa học: "Cross-Framework Evaluation for Statistical Parsing"

A serious bottleneck of comparative parser evaluation is the fact that different parsers subscribe to different formal frameworks and theoretical assumptions. Converting outputs from one framework to another is less than optimal as it easily introduces noise into the process. Here we present a principled protocol for evaluating parsing results across frameworks based on function trees, tree generalization and edit distance metrics. This extends a previously proposed framework for cross-theory evaluation and allows us to compare a wider class of parsers. . | Cross-Framework Evaluation for Statistical Parsing Reut Tsarfaty Joakim Nivre Evelina Andersson Uppsala University Box 635 75126 Uppsala Sweden tsarfaty@ @ Abstract A serious bottleneck of comparative parser evaluation is the fact that different parsers subscribe to different formal frameworks and theoretical assumptions. Converting outputs from one framework to another is less than optimal as it easily introduces noise into the process. Here we present a principled protocol for evaluating parsing results across frameworks based on function trees tree generalization and edit distance metrics. This extends a previously proposed framework for cross-theory evaluation and allows us to compare a wider class of parsers. We demonstrate the usefulness and language independence of our procedure by evaluating constituency and dependency parsers on English and Swedish. 1 Introduction The goal of statistical parsers is to recover a formal representation of the grammatical relations that constitute the argument structure of natural language sentences. The argument structure encompasses grammatical relationships between elements such as subject predicate object etc. which are useful for further . semantic processing. The parses yielded by different parsing frameworks typically obey different formal and theoretical assumptions concerning how to represent the grammatical relationships in the data Rambow 2010 . For example grammatical relations may be encoded on top of dependency arcs in a dependency tree Mel cuk 1988 they may decorate nodes in a phrase-structure tree Marcus et al. 1993 Maamouri et al. 2004 Sima an et al. 2001 or they may be read off of positions in a phrase-structure tree using hard-coded conversion procedures de Marneffe et al. 2006 . This diversity poses a challenge to cross-experimental parser evaluation namely How can we evaluate the performance of these different parsers relative to one another .