tailieunhanh - Báo cáo khoa học: "Rethinking Chinese Word Segmentation: Tokenization, Character Classification, or Wordbreak Identification"

This paper addresses two remaining challenges in Chinese word segmentation. The challenge in HLT is to find a robust segmentation method that requires no prior lexical knowledge and no extensive training to adapt to new types of data. The challenge in modelling human cognition and acquisition it to segment words efficiently without using knowledge of wordhood. We propose a radical method of word segmentation to meet both challenges. | Rethinking Chinese Word Segmentation Tokenization Character Classification or Wordbreak Identification Chu-Ren Huang Petr Simon Institute of Linguistics Institute of Linguistics Academia Sinica Taiwan Academia Sinica Taiwan churen@ sim@ Shu-Kai Hsieh Laurent Prevot DoFLAL CLLE-ERSS CNRS NIU Taiwan Universite de Toulouse France shukai@ prevot@ Abstract This paper addresses two remaining challenges in Chinese word segmentation. The challenge in HLT is to find a robust segmentation method that requires no prior lexical knowledge and no extensive training to adapt to new types of data. The challenge in modelling human cognition and acquisition it to segment words efficiently without using knowledge of wordhood. We propose a radical method of word segmentation to meet both challenges. The most critical concept that we introduce is that Chinese word segmentation is the classification of a string of character-boundaries CB s into either word-boundaries WB s and non-word-boundaries. In Chinese CB s are delimited and distributed in between two characters. Hence we can use the distributional properties of CB among the background character strings to predict which CB s are WB s. 1 Introduction modeling and theoretical challenges The fact that word segmentation remains a main research topic in the field of Chinese language processing indicates that there maybe unresolved theoretical and processing issues. In terms of processing the fact is that none of exiting algorithms is robust enough to reliably segment unfamiliar types of texts before fine-tuning with massive training data. It is true that performance of participating teams have steadily improved since the first SigHAN Chinese segmentation bakeoff Sproat and Emerson 2004 . Bakeoff 3 in 2006 produced best f-scores at 95 and higher. However these can only be achieved after training with the pre-segmented training dataset. This is still very far away from real-world .

• Báo cáo khoa học
• Tokenization
• Character Classification
• ordbreak Identification
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Returning to a Long Solved Problem
• Contrastive Experiment
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên
• Simultaneous Tokenization
• Part of Speech Tagging for Arabic
• Morphological Analyzer
• Arabic Tokenization
• Part of Speech Tagging
• Morphological Disambiguation in One Fell Swoop
• One Tokenization per Source
• Long Papers
• Workshops
• Lecture Information retrieval techniques
• Bài giảng Kỹ thuật truy xuất thông tin
• Information retrieval techniques
• Parsing a document
• Tokenization language issues
• Basic indexing pipeline
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học