tailieunhanh - Báo cáo khoa học: "a Visual Tool for Validating Sense Annotations"

In this paper we present Valido, a tool that supports the difficult task of validating sense choices produced by a set of annotators. The validator can analyse the semantic graphs resulting from each sense choice and decide which sense is more coherent with respect to the structure of the adopted lexicon. We describe the interface and report an evaluation of the tool in the validation of manual sense annotations. | Valido a Visual Tool for Validating Sense Annotations Roberto Navigli Dipartimento di Informatica Universita di Roma La Sapienza Roma Italy navigli@ Abstract In this paper we present Valido a tool that supports the difficult task of validating sense choices produced by a set of annotators. The validator can analyse the semantic graphs resulting from each sense choice and decide which sense is more coherent with respect to the structure of the adopted lexicon. We describe the interface and report an evaluation of the tool in the validation of manual sense annotations. 1 Introduction The task of sense annotation consists in the assignment of the appropriate senses to words in context. For each word the senses are chosen with respect to a sense inventory encoded by a reference dictionary. The free availability and as a result the massive adoption of WordNet Fellbaum 1998 largely contributed to its status of de facto standard in the NLP community. Unfortunately WordNet is a fine-grained resource which encodes possibly subtle sense distictions. Several studies report an inter-annotator agreement around 70 when using WordNet as a reference sense inventory. For instance the agreement in the Open Mind Word Expert project Chklovski and Mihalcea 2002 was . Such a low agreement is only in part due to the inexperience of sense annotators . volunteers on the web . Rather to a large part it is due to the difficulty in making clear which are the real distinctions between close word senses in the WordNet inventory. Adjudicating sense choices . the task of validating word senses is therefore critical in building a high-quality data set. The validation task can be defined as follows let w be a word in a sentence Ơ previously annotated by a set of annotators A a1 a2 an each providing a sense for w and let Sa 81 82 8m c Senses w be the set of senses chosen for w by the annotators in A where Senses w is the set of senses of w in the reference inventory . .