tailieunhanh - Báo cáo khoa học: "Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text"

Sentence fluency is an important component of overall text readability but few studies in natural language processing have sought to understand the factors that define it. We report the results of an initial study into the predictive power of surface syntactic statistics for the task; we use fluency assessments done for the purpose of evaluating machine translation. We find that these features are weakly but significantly correlated with fluency. Machine and human translations can be distinguished with accuracy over 80%. The performance of pairwise comparison of fluency is also very high—over 90% for a multi-layer perceptron classifier. . | Predicting the fluency of text with shallow structural features case studies of machine translation and human-written text Jieun Chae University of Pennsylvania chaeji@ Ani Nenkova University of Pennsylvania nenkova@ Abstract Sentence fluency is an important component of overall text readability but few studies in natural language processing have sought to understand the factors that define it. We report the results of an initial study into the predictive power of surface syntactic statistics for the task we use fluency assessments done for the purpose of evaluating machine translation. We find that these features are weakly but significantly correlated with fluency. Machine and human translations can be distinguished with accuracy over 80 . The performance of pairwise comparison of fluency is also very high over 90 for a multi-layer perceptron classifier. We also test the hypothesis that the learned models capture general fluency properties applicable to human-written text. The results do not support this hypothesis prediction accuracy on the new data is only 57 . This finding suggests that developing a dedicated task-independent corpus of fluency judgments will be beneficial for further investigations of the problem. 1 Introduction Numerous natural language applications involve the task of producing fluent text. This is a core problem for surface realization in natural language generation Langkilde and Knight 1998 Bangalore and Rambow 2000 as well as an important step in machine translation. Considerations of sentence fluency are also key in sentence simplification Siddharthan 2003 sentence compression Jing 2000 Knight and Marcu 2002 Clarke and Lapata 2006 McDonald 2006 Turner and Charniak 2005 Galley and McKeown 2007 text re-generation for summarization Daume III and Marcu 2004 Barzilay and McKeown 2005 Wan et al. 2005 and headline generation Banko et al. 2000 Zajic et al. 2007 Soricut and Marcu 2007 . Despite its importance for these