tailieunhanh - Báo cáo khoa học: "Combination of Arabic Preprocessing Schemes for Statistical Machine Translation"

Statistical machine translation is quite robust when it comes to the choice of input representation. It only requires consistency between training and testing. As a result, there is a wide range of possible preprocessing choices for data used in statistical machine translation. This is even more so for morphologically rich languages such as Arabic. In this paper, we study the effect of different word-level preprocessing schemes for Arabic on the quality of phrase-based statistical machine translation. . | Combination of Arabic Preprocessing Schemes for Statistical Machine Translation Fatiha Sadat Nizar Habash Institute for Information Technology Center for Computational Learning Systems National Research Council of Canada Columbia University habash@ Abstract Statistical machine translation is quite robust when it comes to the choice of input representation. It only requires consistency between training and testing. As a result there is a wide range of possible preprocessing choices for data used in statistical machine translation. This is even more so for morphologically rich languages such as Arabic. In this paper we study the effect of different word-level preprocessing schemes for Arabic on the quality of phrase-based statistical machine translation. We also present and evaluate different methods for combining preprocessing schemes resulting in improved translation quality. 1 Introduction Statistical machine translation SMT is quite robust when it comes to the choice of input representation. It only requires consistency between training and testing. As a result there is a wide range of possible preprocessing choices for data used in SMT. This is even more so for morphologically rich languages such as Arabic. We use the term preprocessing to describe various input modifications applied to raw training and testing texts for SMT. Preprocessing includes different kinds of to-kenization stemming part-of-speech POS tagging and lemmatization. The ultimate goal of preprocessing is to improve the quality of the SMT output by addressing issues such as sparsity in training data. We refer to a specific kind of preprocessing as a scheme and differentiate it from the technique used to obtain it. In a previous publication we presented results describing six pre processing schemes for Arabic Habash and Sadat 2006 . These schemes were evaluated against three different techniques that vary in linguistic complexity and across a learning .

• Báo cáo khoa học
• Combination of Arabic
• Preprocessing Schemes
• Statistical Machine Translation
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học