Đang chuẩn bị liên kết để tải về tài liệu:
Báo cáo khoa học: "Dependency-based Evaluation for Automatic Summaries"

This paper presents DEPEVAL(summ), a dependency-based metric for automatic evaluation of summaries. Using a reranking parser and a Lexical-Functional Grammar (LFG) annotation, we produce a set of dependency triples for each summary. The dependency set for each candidate summary is then automatically compared against dependencies generated from model summaries. We examine a number of variations of the method, including the addition of WordNet, partial matching, or removing relation labels from the dependencies. . | DEPEVAL summ Dependency-based Evaluation for Automatic Summaries Karolina Owczarzak Information Access Division National Institute of Standards and Technology Gaithersburg MD 20899 karolina.owczarzak@nist.gov Abstract This paper presents DEPEVAL summ a dependency-based metric for automatic evaluation of summaries. Using a reranking parser and a Lexical-Functional Grammar LFG annotation we produce a set of dependency triples for each summary. The dependency set for each candidate summary is then automatically compared against dependencies generated from model summaries. We examine a number of variations of the method including the addition of WordNet partial matching or removing relation labels from the dependencies. In a test on TAC 2008 and DUC 2007 data DE-PEVAL summ achieves comparable or higher correlations with human judgments than the popular evaluation metrics ROUGE and Basic Elements BE . 1 Introduction Evaluation is a crucial component in the area of automatic summarization it is used both to rank multiple participant systems in shared summarization tasks such as the Summarization track at Text Analysis Conference TAC 2008 and its Document Understanding Conference DUC predecessors and to provide feedback to developers whose goal is to improve their summarization systems. However manual evaluation of a large number of documents necessary for a relatively unbiased view is often unfeasible especially in the contexts where repeated evaluations are needed. Therefore there is a great need for reliable automatic metrics that can perform evaluation in a fast and consistent manner. In this paper we explore one such evaluation metric DEPEVAL summ based on the comparison of Lexical-Functional Grammar LFG dependencies between a candidate summary and one or more model reference summaries. The method is similar in nature to Basic Elements Hovy et al. 2005 in that it extends beyond a simple string comparison of word sequences reaching instead to a deeper linguistic .