tailieunhanh - Báo cáo khoa học: "FROM N-GRAMS TO COLLOCATIONS AN EVALUATION OF XTRACT"

In previous papers we presented methods for retrieving collocations from large samples of texts. We described a tool, X t r a c t , that implements these methods and able to retrieve a wide range of collocations in a two stage process. These methods well as other related methods however have some limitations. Mainly, the produced collocations do not include any kind of functional information and many of them are invalid. In this paper we introduce methods that address these issues. These methods are implemented in an added third stage to X t r a c t. | FROM N-GRAMS TO COLLOCATIONS AN EVALUATION OF XTRACT Frank A. Smadja Department of Computer Science Columbia University New York NY 10027 Abstract In previous papers we presented methods for retrieving collocations from large samples of texts. We described a tool Xtract that implements these methods and able to retrieve a wide range of collocations in a two stage process. These methods as well as other related methods however have some limitations. Mainly the produced collocations do not include any kind of functional information and many of them are invalid. In this paper we introduce methods that address these issues. These methods are implemented in an added third stage to Xtract that examines the set of collocations retrieved during the previous two stages to both filter out a number of invalid collocations and add useful syntactic information to the retained ones. By combining parsing and statistical techniques the addition of this third stage has raised the overall precision level of xtract from 40 to 80 with a precision of 94 . In the paper we describe the methods and the evaluation experiments. 1 INTRODUCTION In the past several approaches have been proposed to retrieve various types of collocations from the analysis of large samples of textual data. Pairwise associations bigrams or 2-grams . Smadja 1988 Church and Hanks 1989 as well as n-word n 2 associations or n-grams . Choueka et al. 1983 Smadja and McKeown 1990 were retrieved. These techniques automatically produced large numbers of collocations along with statistical figures intended to reflect their relevance. However none of these techniques provides functional information along with the collocation. Also the results produced often contained improper word associations reflecting some spurious aspect of the training corpus that did not stand for true collocations. This paper addresses these two problems. Previous papers . Smadja and McKeown 1990 introduced a set of techniques and a tool. .

• Báo cáo khoa học
• FROM N GRAMS TO COLLOCATIONS
• EVALUATION OF XTRACT
• Frank A
• Smadja
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học