tailieunhanh - Báo cáo khoa học: "Crowdsourcing Inference-Rule Evaluation"

The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However, evaluating such resources has turned out to be a non-trivial task, slowing progress in the field. In this paper, we suggest a framework for evaluating inference-rule resources. | Crowdsourcing Inference-Rule Evaluation Naomi Zeichner Bar-Ilan University Ramat-Gan Israel Jonathan Berant Tel-Aviv University Tel-Aviv Israel jonatha6@ Ido Dagan Bar-Ilan University Ramat-Gan Israel dagan@ Abstract The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However evaluating such resources has turned out to be a non-trivial task slowing progress in the field. In this paper we suggest a framework for evaluating inference-rule resources. Our framework simplifies a previously proposed instance-based evaluation method that involved substantial annotator training making it suitable for crowdsourcing. We show that our method produces a large amount of annotations with high inter-annotator agreement for a low cost at a short period of time without requiring training expert annotators. 1 Introduction Inference rules are an important component in semantic applications such as Question Answering QA Ravichandran and Hovy 2002 and Information Extraction IE Shinyama and Sekine 2006 describing a directional inference relation between two text patterns with variables. For example to answer the question Where was Reagan raised a QA system can use the rule X brought up in Y X raised in Y to extract the answer from Reagan was brought up in Dixon . Similarly an IE system can use the rule X work as Y X hired as Y to extract the PERSON and ROLE entities in the hiring event from Bob worked as an analyst for Dell . The significance of inference rules has led to substantial effort into developing algorithms that automatically learn inference rules Lin and Pantel 2001 Sekine 2005 Schoenmackers et al. 2010 156 and generate knowledge resources for inference systems. However despite their potential utilization of inference rule resources is currently somewhat limited. This is largely due to the fact that these