tailieunhanh - Báo cáo khoa học: "EVALUATION OF NATURAL LANGUAGE INTERFACES TO DATA BASE SYSTEMS"

IV Familiar with the database and the interaction language Ill Familiar with the contents of database II Familiar with the domain of application I Passing knowledge of the domain of application Of course, as users gain experience with a system, they will continually attempt to adapt to its quirks. If the purpose of the evaluation is to demonstrate that the natural language understanding system is merely useable. | EVALUATION OF NATURAL LANGUAGE INTERFACES TO DATA BASE SYSTEMS Bozena Henisz Thompson California Institute of Technology INTRODUCTION Is evaluation like beauty in the eye of the beholder The answer is far from simple because it depends on who is considered to be the proper beholder. Evaluators may range from casual users to society as a whole with system builders sophisticated users linguists grant providers system buyers and others in between. The members of this panel are system builders and linguists or rather the two fused into one but I believe interested in all or almost all actual or potential bodies of evaluators. One of our colleagues expressed a forceful opinion while being a member of a similar panel at last year s ACL conference Those of US on this panel and other researchers in the field simply don t have the right to determine whether a system is practical. Only the users of such a system can make that determination. Only a user can decide whether the NL natural language capability constitutes sufficient added value to be deemed practical Only a user can decide if the system s frequency of inappropriate response is sufficiently low to be deemed practical. Only a user can decide whether the overall NL interaction taken in toto offers enough benefits over alternative formal interactions to be deemed practical 1 . It is hard for me to disagree since I argued as forcefully on the basis of my study of users evaluation of machine translation 2 a study which was prompted by the evaluations of the quality of machine translation as viewed by linguists and users ranging from 35Z acceptable for the former to 90Z for the latter. What the study also showed was that the practicality of the output could indeed only be judged by the users since even incomplete and stylistically very inelegant translations were found quite useful in practice because they on the one hand provided however crudely the information sought by the users and on the other hand the users .