tailieunhanh - Báo cáo khoa học: "USING BRACKETED PARSES TO EVALUATE A GRAMMAR CHECKING APPLICATION"

We describe a method for evaluating a grammar checking application with hand-bracketed parses. A randomly-selected set of sentences was submitted to a grammar checker in both bracketed and unbracketed formats. A comparison of the resulting error reports illuminates the relationship between the underlying performance of the parsergrammar system and the error critiques presented to the user. INTRODUCTION The recent development o f broad-coverage natural language processing systems has stimulated work on the evaluation of the syntactic component of such systems, for purposes of basic evaluation and improvement o f system performance. . | USING BRACKETED PARSES TO EVALUATE A GRAMMAR CHECKING APPLICATION Richard H. Wojcik Philip Harrison John Bremer Boeing Computer Services Research and Technology Division . Box 24346 MS 7L-43 Seattle WA 98124-2964 Internet rwojcik@ pharrison@ jbremer@ Abstract We describe a method for evaluating a grammar checking application with hand-bracketed parses. A randomly-selected set of sentences was submitted to a grammar checker in both bracketed and unbracketed formats. A comparison of the resulting error reports illuminates the relationship between the underlying performance of the parsergrammar system and the eưor critiques presented to the user. INTRODUCTION The recent development of broad-coverage natural language processing systems has stimulated work on the evaluation of the syntactic component of such systems for purposes of basic evaluation and improvement of system performance. Methods utilizing hand-bracketed corpora such as the University of Pennsylvania Treebank as a basis for evaluation metrics have been discussed in Black et al. 1991 Harrison et al. 1991 and Black et al. 1992 . Three metrics discussed in those works were the Crossing Parenthesis Score a count of the number of phrases in the machine produced parse which cross with one or more phrases in the hand parse Recall the percentage of phrases in the hand parse that are also in die machine parse and Precision the percentage of phrases in the machine parse that are in the hand parse . We have developed a methodology for using hand-bracketed parses to examine both the internal and external performance of a grammar checker. The internal performance refers to the behavior of the underlying system . the toke-nizer parser lexicon and grammar. The external performance refers to the error critiques generated by the Our evaluation methodology relies on three separate eưor reports generated from a corpus of randomly selected sentences 1 a report based on unbracketed