tailieunhanh - Báo cáo khoa học: "Contradictions and Justifications: Extensions to the Textual Entailment Task"

The third PASCAL Recognizing Textual Entailment Challenge (RTE-3) contained an optional task that extended the main entailment task by requiring a system to make three-way entailment decisions (entails, contradicts, neither) and to justify its response. Contradiction was rare in the RTE-3 test set, occurring in only about 10% of the cases, and systems found accurately detecting it difficult. Subsequent analysis of the results shows a test set must contain many more entailment pairs for the three-way decision task than the traditional two-way task to have equal confidence in system comparisons. . | Contradictions and Justifications Extensions to the Textual Entailment Task Ellen M. Voorhees National Institute of Standards and Technology Gaithersburg MD 20899-8940 USA Abstract The third PASCAL Recognizing Textual Entailment Challenge RTE-3 contained an optional task that extended the main entailment task by requiring a system to make three-way entailment decisions entails contradicts neither and to justify its response. Contradiction was rare in the RTE-3 test set occurring in only about 10 of the cases and systems found accurately detecting it difficult. Subsequent analysis of the results shows a test set must contain many more entailment pairs for the three-way decision task than the traditional two-way task to have equal confidence in system comparisons. Each of six human judges representing eventual end users rated the quality of a justification by assigning understandability and correctness scores. Ratings of the same justification across judges differed significantly signaling the need for a better characterization of the justification task. 1 Introduction The PASCAL Recognizing Textual Entailment RTE workshop series see . org Challenges RTE3 has been a catalyst for recent research in developing systems that are able to detect when the content of one piece of text necessarily follows from the content of another piece of text Dagan et al. 2006 Giampiccolo et al. 2007 . This ability is seen as a fundamental component in the solutions for a variety of natural language problems such as question answering summarization and information extraction. In addition to the main entailment task the most recent Challenge RTE-3 contained a second optional task that extended the main task in two ways. The rst extension was to require systems to make three-way entailment decisions the second extension was for systems to return a justi cation or explanation of how its decision was reached. In the main RTE entailment task systems .