tailieunhanh - Báo cáo khoa học: "Evaluation challenges in large-scale document summarization"

We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than 10,000 manual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. | Evaluation challenges in large-scale document summarization Dragomir R. Radev U. of Michigan radev@ Wai Lam Chinese U. of Hong Kong wlam@ Arda Celebi USC ISI ardax@ Simone Teufel U. of Cambridge John Blitzer U. of Pennsylvania blitzer@ Danyu Liu U. of Alabama liudy@ Horacio Saggion U. of Sheffield Hong Qi U. of Michigan hqi@ Elliott Drabek Johns Hopkins U. edrabek@ Abstract We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of a 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese b more than 10 000 manual abstracts and extracts and c 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and drawbacks of all evaluation methods and how they rank the different summarizers. 1 Introduction Automatic document summarization is a field that has seen increasing attention from the NLP community in recent years. In part this is because summarization incorporates many important aspects of both natural language understanding and natural language generation. In part it is because effective automatic summarization would be useful in a variety of areas. Unfortunately evaluating automatic summarization in a standard and inexpensive way is a difficult task Mani et al. 2001 . Traditional large-scale evaluations are either too simplistic using measures like precision recall and percent agreement which 1 don t take chance agreement into account and 2 don t account for the fact that human judges don t agree which sentences should be in a summary or too expensive an approach using manual judgements can scale up to a few hundred summaries but not to tens or hundreds of thousands . In this .