Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur
Article Dans Une Revue Journal of Medical Systems Année : 2021

Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System

Résumé

Simulated consultations through virtual patients allow medical students to practice history-taking skills. Ideally, applications should provide interactions in natural language and be multi-case, multi-specialty. Nevertheless, few systems handle or are tested on a large variety of cases. We present a virtual patient dialogue system in which a medical trainer types new cases and these are processed without human intervention. To develop it, we designed a patient record model, a knowledge model for the history-taking task, and a termino-ontological model for term variation and out-of-vocabulary words. We evaluated whether this system provided quality dialogue across medical specialities (n = 18), and with unseen cases (n = 29) compared to the cases used for development (n = 6). Medical evaluators (students, residents, practitioners, and researchers) conducted simulated history-taking with the system and assessed its performance through Likert-scale questionnaires. We analysed interaction logs and evaluated system correctness. The mean user evaluation score for the 29 unseen cases was 4.06 out of 5 (very good). The evaluation of correctness determined that, on average, 74.3% (sd = 9.5) of replies were correct, 14.9% (sd = 6.3) incorrect, and in 10.7% the system behaved cautiously by deferring a reply. In the user evaluation, all aspects scored higher in the 29 unseen cases than in the 6 seen cases. Although such a multi-case system has its limits, the evaluation showed that creating it is feasible; that it performs adequately; and that it is judged usable. We discuss some lessons learned and pivotal design choices affecting its performance and the end-users, who are primarily medical students.
Fichier principal
Vignette du fichier
Lessons-Learned-from-the-Usability-Evaluation-of-a-Simulated-Patient-Dialogue-System.pdf (2.56 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03452553 , version 1 (27-11-2021)

Identifiants

Citer

Leonardo Campillos-Llanos, Catherine Thomas, Éric Bilinski, Antoine Neuraz, Sophie Rosset, et al.. Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System. Journal of Medical Systems, 2021, 45 (7), pp.69. ⟨10.1007/s10916-021-01737-4⟩. ⟨hal-03452553⟩
285 Consultations
273 Téléchargements

Altmetric

Partager

More