tailieunhanh - Báo cáo khoa học: "A Phonetic-Based Approach to Chinese Chat Text Normalization"

Chatting is a popular communication media on the Internet via ICQ, chat rooms, etc. Chat language is different from natural language due to its anomalous and dynamic natures, which renders conventional NLP tools inapplicable. The dynamic problem is enormously troublesome because it makes static chat language corpus outdated quickly in representing contemporary chat language. To address the dynamic problem, we propose the phonetic mapping models to present mappings between chat terms and standard words via phonetic transcription, . Chinese Pinyin in our case. . | A Phonetic-Based Approach to Chinese Chat Text Normalization Yunqing Xia Kam-Fai Wong Department of . The Chinese University of Hong Kong Shatin Hong Kong yqxia kfwong @ Wenjie Li Department of Computing The Hong Kong Polytechnic University Kowloon Hong Kong cswjli@ Abstract Chatting is a popular communication media on the Internet via ICQ chat rooms etc. Chat language is different from natural language due to its anomalous and dynamic natures which renders conventional NLP tools inapplicable. The dynamic problem is enormously troublesome because it makes static chat language corpus outdated quickly in representing contemporary chat language. To address the dynamic problem we propose the phonetic mapping models to present mappings between chat terms and standard words via phonetic transcription . Chinese Pinyin in our case. Different from character mappings the phonetic mappings can be constructed from available standard Chinese corpus. To perform the task of dynamic chat language term normalization we extend the source channel model by incorporating the phonetic mapping models. Experimental results show that this method is effective and stable in normalizing dynamic chat language terms. 1 Introduction Internet facilitates online chatting by providing ICQ chat rooms BBS email blogs etc. Chat language becomes ubiquitous due to the rapid proliferation of Internet applications. Chat language text appears frequently in chat logs of online education Heard-White 2004 customer relationship management Gianforte 2003 etc. On the other hand wed-based chat rooms and BBS systems are often abused by solicitors of terrorism pornography and crime McCullagh 2004 . Thus there is a social urgency to understand online chat language text. Chat language is anomalous and dynamic. Many words in chat text are anomalous to natural language. Chat text comprises of ill-edited terms and anomalous writing styles. We refer chat terms to the anomalous .