tailieunhanh - Báo cáo khoa học: "Automatic Assessment of Coverage Quality in Intelligence Reports"

Common approaches to assessing document quality look at shallow aspects, such as grammar and vocabulary. For many real-world applications, deeper notions of quality are needed. This work represents a first step in a project aimed at developing computational methods for deep assessment of quality in the domain of intelligence reports. We present an automated system for ranking intelligence reports with regard to coverage of relevant material. | Automatic Assessment of Coverage Quality in Intelligence Reports Samuel Brody School of Communication and Information Rutgers University sdbrody@ Paul Kantor School of Communication and Information Rutgers University Abstract Common approaches to assessing document quality look at shallow aspects such as grammar and vocabulary. For many real-world applications deeper notions of quality are needed. This work represents a first step in a project aimed at developing computational methods for deep assessment of quality in the domain of intelligence reports. We present an automated system for ranking intelligence reports with regard to coverage of relevant material. The system employs methodologies from the field of automatic summarization and achieves performance on a par with human judges even in the absence of the underlying information sources. 1 Introduction Distinguishing between high- and low-quality documents is an important skill for humans and a challenging task for machines. The majority of previous research on the subject has focused on low-level measures of quality such as spelling vocabulary and grammar. However in many real-world situations it is necessary to employ deeper criteria which look at the content of the document and the structure of argumentation. One example where such criteria are essential is decision-making in the intelligence community. This is also a domain where computational methods can play an important role. In a typical situation an intelligence officer faced with an important decision receives reports from a team of analysts on a specific topic of interest. Each decision may involve several areas of interest resulting in several collections of reports. Addi- 491 tionally the officer may be engaged in many decision processes within a small window of time. Given the nature of the task it is vital that the limited time be used effectively . that the highest-quality information be handled first. Our .