Accéder directement au contenu Accéder directement à la navigation
Nouvelle interface
Article dans une revue

Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System

Leonardo Campillos-Llanos 1, 2 Catherine Thomas 2 Éric Bilinski 3, 4 Antoine Neuraz Sophie Rosset 3 Pierre Zweigenbaum 3 
2 ILES - Information, Langue Ecrite et Signée
LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur
3 ILES - Information, Langue Ecrite et Signée
LISN - Laboratoire Interdisciplinaire des Sciences du Numérique, STL - Sciences et Technologies des Langues
4 ASARD - Accompagnement et Soutien aux Activités de Recherche & Développement
LISN - Laboratoire Interdisciplinaire des Sciences du Numérique
Abstract : Simulated consultations through virtual patients allow medical students to practice history-taking skills. Ideally, applications should provide interactions in natural language and be multi-case, multi-specialty. Nevertheless, few systems handle or are tested on a large variety of cases. We present a virtual patient dialogue system in which a medical trainer types new cases and these are processed without human intervention. To develop it, we designed a patient record model, a knowledge model for the history-taking task, and a termino-ontological model for term variation and out-of-vocabulary words. We evaluated whether this system provided quality dialogue across medical specialities (n = 18), and with unseen cases (n = 29) compared to the cases used for development (n = 6). Medical evaluators (students, residents, practitioners, and researchers) conducted simulated history-taking with the system and assessed its performance through Likert-scale questionnaires. We analysed interaction logs and evaluated system correctness. The mean user evaluation score for the 29 unseen cases was 4.06 out of 5 (very good). The evaluation of correctness determined that, on average, 74.3% (sd = 9.5) of replies were correct, 14.9% (sd = 6.3) incorrect, and in 10.7% the system behaved cautiously by deferring a reply. In the user evaluation, all aspects scored higher in the 29 unseen cases than in the 6 seen cases. Although such a multi-case system has its limits, the evaluation showed that creating it is feasible; that it performs adequately; and that it is judged usable. We discuss some lessons learned and pivotal design choices affecting its performance and the end-users, who are primarily medical students.
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-03452553
Contributeur : Pierre Zweigenbaum Connectez-vous pour contacter le contributeur
Soumis le : samedi 27 novembre 2021 - 03:06:57
Dernière modification le : mardi 25 octobre 2022 - 15:45:56
Archivage à long terme le : : lundi 28 février 2022 - 18:17:40

Fichier

Lessons-Learned-from-the-Usabi...
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Leonardo Campillos-Llanos, Catherine Thomas, Éric Bilinski, Antoine Neuraz, Sophie Rosset, et al.. Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System. Journal of Medical Systems, 2021, 45 (7), ⟨10.1007/s10916-021-01737-4⟩. ⟨hal-03452553⟩

Partager

Métriques

Consultations de la notice

104

Téléchargements de fichiers

83