tailieunhanh - Báo cáo khoa học: "A Graphical Interface for MT Evaluation and Error Analysis"

Error analysis in machine translation is a necessary step in order to investigate the strengths and weaknesses of the MT systems under development and allow fair comparisons among them. This work presents an application that shows how a set of heterogeneous automatic metrics can be used to evaluate a test bed of automatic translations. To do so, we have set up an online graphical interface for the A SIYA toolkit, a rich repository of evaluation measures working at different linguistic levels. . | A Graphical Interface for MT Evaluation and Error Analysis Meritxell Gonzalez and Jesus Gimenez and Lluis Marquez TALP Research Center Universitat Politecnica de Catalunya mgonzalez jgimenez lluism @ Abstract Error analysis in machine translation is a necessary step in order to investigate the strengths and weaknesses of the MT systems under development and allow fair comparisons among them. This work presents an application that shows how a set of heterogeneous automatic metrics can be used to evaluate a test bed of automatic translations. To do so we have set up an online graphical interface for the Asiya toolkit a rich repository of evaluation measures working at different linguistic levels. The current implementation of the interface shows constituency and dependency trees as well as shallow syntactic and semantic annotations and word alignments. The intelligent visualization of the linguistic structures used by the metrics as well as a set of navigational functionalities may lead towards advanced methods for automatic error analysis. 1 Introduction Evaluation methods are a key ingredient in the development cycle of machine translation MT systems. As illustrated in Figure 1 they are used to identify and analyze the system weak points error analysis to introduce new improvements and adjust the internal system parameters system refinement and to measure the system performance in comparison to other systems or previous versions of the same system evaluation . We focus here on the processes involved in the error analysis stage in which MT developers need to understand the output of their systems and to assess the improvements introduced. 139 Automatic detection and classification of the errors produced by MT systems is a challenging problem. The cause of such errors may depend not only on the translation paradigm adopted but also on the language pairs the availability of enough linguistic resources and the performance of the linguistic processors among .