tailieunhanh - Báo cáo khoa học: "Tackling Sparse Data Issue in Machine Translation Evaluation ∗"

We illustrate and explain problems of n-grams-based machine translation (MT) metrics (. BLEU) when applied to morphologically rich languages such as Czech. A novel metric SemPOS based on the deep-syntactic representation of the sentence tackles the issue and retains the performance for translation to English as well. | Tackling Sparse Data Issue in Machine Translation Evaluation Ondrej Bojar Kamil Kos and David MareCek Charles University in Prague Institute of Formal and Applied Linguistics bojar marecek @ kamilkos@ Abstract We illustrate and explain problems of n-grams-based machine translation MT metrics . BLEU when applied to morphologically rich languages such as Czech. A novel metric SemPOS based on the deep-syntactic representation of the sentence tackles the issue and retains the performance for translation to English as well. 1 Introduction Automatic metrics of machine translation MT quality are vital for research progress at a fast pace. Many automatic metrics of MT quality have been proposed and evaluated in terms of correlation with human judgments while various techniques of manual judging are being examined as well see . MetricsMATR08 Przybocki et al. 2008 1 wMt08 and WMT09 Callison-Burch et al. 2008 Callison-Burch et al. 2009 1 2. The contribution of this paper is twofold. Section 2 illustrates and explains severe problems of a widely used BLEU metric Papineni et al. 2002 when applied to Czech as a representative of languages with rich morphology. We see this as an instance of the sparse data problem well known for MT itself too much detail in the formal representation leading to low coverage of . a translation dictionary. In MT evaluation too much detail leads to the lack of comparable parts of the hypothesis and the reference. This work has been supported by the grants EuroMatrixPlus FP7-ICT-2007-3-231720 of the eU and 7E09003 of the Czech Republic FP7-ICT-2009-4-247762 Faust GA201 09 H057 GAUK1163 2010 and MSM 0021620838. We are grateful to the anonymous reviewers for further research suggestions. 1 http speech tests metricsmatr 2008 results 2http wmt0 8 and wmt0 9 Rank -- pctrans eurotranxp google cu-bojar .cu-tectomt uedin -----1----------1-----------1---------1----- BLEU