Table 3 Evaluation measures based on the interaction parameters gathered from the dialogues of simulated and recruited users
From: Predicting user mental states in spoken dialogue systems
Dialogue success |
|---|
Dialogue success rate (%success). The percentage of successfully completed tasks. In each scenario, the user has to obtain one or several pieces of information, and the dialogue success depends on whether the system provides the correct data (according to the aims of the scenario) or incorrect data to the user |
Average number of corrected errors per dialogue (n CE). The average of errors detected and corrected by the dialogue manager. We have considered only the errors that modify the values of the attributes and that could cause dialogue failure |
Average number of uncorrected errors per dialogue (n NCE). The average of errors not corrected by the dialogue manager. Again, only errors that modify the values of the attributes are considered |
Error correction rate (%ECR) is the percentage of corrected errors, computed as n CE/(n CE + n NCE) |
High-level dialogue features |
Average number of turns per dialogue (avg#turns/dial) |
Percentage of different dialogues (%diff) |
Number of repetitions of the most seen dialogue (#repMS) |
Number of turns of the most seen dialogue (#turnsMS) |
Number of turns of the shortest dialogue (#turnsSh) |
Number of turns of the longest dialogue (#turnsLo) |
Ratio users vs. system actions (us/sysAct) |
Dialogue style/cooperativeness measures |
System dialogue acts: confirmation of concepts and attributes, questions to require information and answers generated after a database query |
Confirmation rate (%confirm) was computed as the ratio between the number of explicit confirmations turns (n CT) and the number of turns in the dialogue (n CT/n T) |
User dialogue acts: request to the system, provide information, confirmation, yes/no answers and other answers |
Goal directed actions vs. grounding actions: Goal directed actions are requesting and providing information, while grounding actions are explicit and implicit confirmations, dialogue formalities (greetings, instructions, etc.) and unrecognized actions |