tailieunhanh - Báo cáo khoa học: "Dialogue Segmentation with Large Numbers of Volunteer Internet Annotators"

This paper shows the results of an experiment in dialogue segmentation. In this experiment, segmentation was done on a level of analysis similar to adjacency pairs. The method of annotation was somewhat novel: volunteers were invited to participate over the Web, and their responses were aggregated using a simple voting method. Though volunteers received a minimum of training, the aggregated responses of the group showed very high agreement with expert opinion. | Dialogue Segmentation with Large Numbers of Volunteer Internet Annotators T. Daniel Midgley Discipline of Linguistics School of Computer Science and Software Engineering University of Western Australia Perth Australia Abstract This paper shows the results of an experiment in dialogue segmentation. In this experiment segmentation was done on a level of analysis similar to adjacency pairs. The method of annotation was somewhat novel volunteers were invited to participate over the Web and their responses were aggregated using a simple voting method. Though volunteers received a minimum of training the aggregated responses of the group showed very high agreement with expert opinion. The group as a unit performed at the top of the list of annotators and in many cases performed as well as or better than the best annotator. 1 Introduction Aggregated human behaviour is a valuable source of information. The Internet shows us many examples of collaboration as a means of resource creation. Wikipedia reviews and Yahoo Answers are just some examples of large repositories of information powered by individuals who voluntarily contribute their time and talents. Some NLP projects are now using this idea notably the ESP Game von Ahn 2004 a data collection effort presented as a game in which players label images from the Web. This paper presents an extension of this collaborative volunteer ethic in the area of dialogue annotation. For dialogue researchers the prospect of using volunteer annotators from the Web can be an attractive option. The task of training annotators can be time-consuming expensive and if inter-annotator agreement turns out to be poor risky. Getting Internet volunteers for annotation has its own pitfalls. Dialogue annotation is often not very interesting so it can be difficult to attract willing participants. Experimenters will have little control over the conditions of the annotation and the skill of the annotators. Training .