tailieunhanh - Báo cáo khoa học: "Automatic Evaluation of Linguistic Quality in Multi-Document Summarization"

To date, few attempts have been made to develop and validate methods for automatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture various aspects of well-written text. We train and test linguistic quality models on consecutive years of NIST evaluation data in order to show the generality of results. For grammaticality, the best results come from a set of syntactic features. | Automatic Evaluation of Linguistic Quality in Multi-Document Summarization Emily Pitler Annie Louis Ani Nenkova Computer and Information Science University of Pennsylvania Philadelphia PA 19104 USA epitler lannie nenkova@ Abstract To date few attempts have been made to develop and validate methods for automatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture various aspects of well-written text. We train and test linguistic quality models on consecutive years of NIST evaluation data in order to show the generality of results. For grammaticality the best results come from a set of syntactic features. Focus coherence and referential clarity are best evaluated by a class of features measuring local coherence on the basis of cosine similarity between sentences coreference information and summarization specific features. Our best results are 90 accuracy for pairwise comparisons of competing systems over a test set of several inputs and 70 for ranking summaries of a specific input. 1 Introduction Efforts for the development of automatic text sum-marizers have focused almost exclusively on improving content selection capabilities of systems ignoring the linguistic quality of the system output. Part of the reason for this imbalance is the existence of ROUGE Lin and Hovy 2003 Lin 2004 the system for automatic evaluation of content selection which allows for frequent evaluation during system development and for reporting results of experiments performed outside of the annual NIST-led evaluations the Document Understanding Conference DUC 1 and the Text Analysis Conference TAC 2. Few metrics however have been proposed for evaluating linguistic 1http 2http tac quality and none have been validated on data from NIST evaluations. In their pioneering work on automatic evaluation of summary coherence Lapata and Barzilay 2005 provide a .

• Báo cáo khoa học
• Automatic Evaluation
• Linguistic Quality
• Multi Document Summarization
• báo cáo khoa học
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên
• Journal of Science and Technology
• Evaluation of the effectiveness
• Automatic sprinkling system for shallot cultivation
• Soc Trang province
• Automatic sprinkling system
• Automatic irrigation systema
• An Automatic Method
• Summary Evaluation Using Multiple Evaluation Results
• Manual Method
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Automatic Evaluation of Chinese Translation Output
• Word Level
• Character Level
• Automatic Evaluation Method
• Machine Translation
• Noun Phrase Chunking
• Dependency based Evaluation
• Automatic Summaries
• Karolina Owczarzak
• The Contribution
• Linguistic Features
• Automatic Machine Translation Evaluation
• Collecting a Why question corpus
• development and evaluation
• automatic QA system
• Automatic Evaluation of Sentence Level
• Fluency Andrew Mutton
• Mark Dras
• A Re examination
• Features in Regression
• Approach to Automatic MT Evaluation
• Correlating Human
• German Surface Realiser
• The Back translation Score
• Automatic MT Evaluation at the Sentence Level
• Reference Translations
• Towards Web Based
• Evaluation of Automatic Natural Language
• Phrase Generation
• Automatic Evaluation of Machine Translation Quality
• Longest Common Subsequence
• Skip Bigram Statistics
• A Unified Framework
• N gram Co Occurrence Statistics
• Feedback Cleaning of Machine Translation Rules
• Kenji Imamura
• Eiichiro Sumita
• A machine learning
• the automatic evaluation
• Extracting Noun Phrases
• Large Scale Texts
• A Hybrid Approach
• Its Automatic Evaluation
• a Method for Automatic Evaluation
• Long Papers
• A Quantitative Evaluation of Linguistic Tests
• the Automatic Prediction
• Semantic Markedness
• Applications of Automatic Evaluation Methods
• Measuring a Capability
• Speech Translation System
• scientific reports
• model language
• process natural language
• Word Sense Induction
• Triplet Based Clustering
• Comparing Automatic
• Human Evaluation
• NLG Systems
• Dehusking capacity
• De husking efficiency
• Semi automatic coconut de husking machine
• Coast tall variety of coconut
• Performance evaluation
• Journal of technology and science education
• Automatic evaluation of practices
• Moodle for Self Learning in Engineering
• Moodle for Self Learning
• Exercises outside the classroom