tailieunhanh - Báo cáo khoa học: "Correlating Human and Automatic Evaluation of a German Surface Realiser"

We examine correlations between native speaker judgements on automatically generated German text against automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task, most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast, on a naturalness judgement task, the General Text Matcher (GTM) tool correlates best overall, although in general, correlation between the human judgements and the automatic metrics was quite weak. and fluency on automatically generated English paraphrases. . | Correlating Human and Automatic Evaluation of a German Surface Realiser Aoife Cahill Institut fur Maschinelle Sprachverarbeitung IMS University of Stuttgart 70174 Stuttgart Germany Abstract We examine correlations between native speaker judgements on automatically generated German text against automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast on a naturalness judgement task the General Text Matcher GTM tool correlates best overall although in general correlation between the human judgements and the automatic metrics was quite weak. 1 Introduction During the development of a surface realisation system it is important to be able to quickly and automatically evaluate its performance. The evaluation of a string realisation system usually involves string comparisons between the output of the system and some gold standard set of strings. Typically automatic metrics from the fields of Machine Translation . BLEU or Summarisation . ROUGE are used but it is not clear how successful or even appropriate these are. Belz and Reiter 2006 and Reiter and Belz 2009 describe comparison experiments between the automatic evaluation of system output and human expert and non-expert evaluation of the same data English weather forecasts . Their findings show that the NIST metric correlates best with the human judgements and all automatic metrics favour systems that generate based on frequency. They conclude that automatic evaluations should be accompanied by human evaluations where possible. Stent et al. 2005 investigate a number of automatic evaluation methods for generation in terms of adequacy and fluency on automatically generated English paraphrases. They find that the automatic metrics are reasonably good at measuring adequacy but not .

• Báo cáo khoa học
• Correlating Human
• Automatic Evaluation
• German Surface Realiser
• báo cáo khoa học
• báo cáo ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học