tailieunhanh - Báo cáo khoa học: "A Unified Tagging Approach to Text Normalization"

This paper addresses the issue of text normalization, an important yet often overlooked problem in natural language processing. By text normalization, we mean converting ‘informally inputted’ text into the canonical form, by eliminating ‘noises’ in the text and detecting paragraph and sentence boundaries in the text. Previously, text normalization issues were often undertaken in an ad-hoc fashion or studied separately. This paper first gives a formalization of the entire problem. | A Unified Tagging Approach to Text Normalization Conghui Zhu Jie Tang Hang Li Harbin Institute of Technology Department of Computer Science Microsoft Research Asia Harbin China Tsinghua University China Beijing China chzhu@ jietang@ hangli@ Hwee Tou Ng Department of Computer Science National University of Singapore Singapore nght@ Tiejun Zhao Harbin Institute of Technology Harbin China tjzhao@ Abstract This paper addresses the issue of text normalization an important yet often overlooked problem in natural language processing. By text normalization we mean converting informally inputted text into the canonical form by eliminating noises in the text and detecting paragraph and sentence boundaries in the text. Previously text normalization issues were often undertaken in an ad-hoc fashion or studied separately. This paper first gives a formalization of the entire problem. It then proposes a unified tagging approach to perform the task using Conditional Random Fields CRF . The paper shows that with the introduction of a small set of tags most of the text normalization tasks can be performed within the approach. The accuracy of the proposed method is high because the subtasks of normalization are interdependent and should be performed together. Experimental results on email data cleaning show that the proposed method significantly outperforms the approach of using cascaded models and that of employing independent models. 1 Introduction More and more informally inputted text data becomes available to natural language processing 688 such as raw text data in emails newsgroups forums and blogs. Consequently how to effectively process the data and make it suitable for natural language processing becomes a challenging issue. This is because informally inputted text data is usually very noisy and is not properly segmented. For example it may contain extra line breaks extra spaces and extra punctuation

• Báo cáo khoa học
• A Unified Tagging
• Approach to Text Normalization
• Conghui Zhu Jie Tang Hang Li Harbin
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học