Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.

Esther E Bron (1) , Marion Smits (2, 3) , Wiesje M van Der Flier (4) , Hugo Vrenken (4) , Frederik Barkhof (4) , Philip Scheltens (4) , Janne M Papma (2) , Rebecca M E Steketee (2) , Carolina Méndez Orellana (2) , Rozanna Meijboom (2) , Madalena Pinto (5) , Joana R Meireles (6) , Carolina Garrett (7) , António J Bastos-Leite (7) , Ahmed Abdulkadir (8, 9) , Olaf Ronneberger (10, 9) , Nicola Amoroso (11, 12) , Roberto Bellotti (11, 12) , David Cárdenas-Peña (13) , Andrés M Álvarez-Meza (13) , Chester V Dolph (14) , Khan M Iftekharuddin (14) , Simon F Eskildsen (15) , Pierrick Coupé (16) , Vladimir S Fonov (17) , Katja Franke (18) , Christian Gaser (18) , Christian Ledig (19) , Ricardo Guerrero (19) , Tong Tong (19) , Katherine R Gray (19) , Elaheh Moradi (20) , Jussi Tohka (20) , Alexandre Routier (21) , Stanley Durrleman (21) , Alessia Sarica (22) , Giuseppe Di Fatta (23) , Francesco Sensi (24) , Andrea Chincarini (24) , Garry M Smith (25, 23) , Zhivko V Stoyanov (23, 25) , Lauge Sørensen (26) , Mads Nielsen (26) , Sabina Tangaro (11) , Paolo Inglese (11) , Christian Wachinger (27) , Martin Reuter (28, 29) , John C van Swieten (2) , Wiro J Niessen (30, 2) , Stefan Klein (1)

1 LERMA - Laboratoire d'Etude du Rayonnement et de la Matière en Astrophysique
2 Erasmus MC - Erasmus University Medical Center [Rotterdam]
3 WUR - Wageningen University and Research [Wageningen]
4 VU University Medical Center [Amsterdam]
5 IFSTTAR/COSYS/LEPSIS - Laboratoire Exploitation, Perception, Simulateurs et Simulations
6 Hospital de São João [Porto]
7 FMUP - Faculdade de Medicina da Universidade do Porto
8 Albert-Ludwigs-Universität Freiburg = University of Freiburg
9 Department of Computer Science [Freiburg]
10 BIOSS - Centre for Biological Signaling Studies [Freiburg]
11 INFN, sezione di Bari - Istituto Nazionale di Fisica Nucleare, sezione di Bari
12 UNIBA - Università degli studi di Bari Aldo Moro = University of Bari Aldo Moro
13 UNAL - Universidad Nacional de Colombia [Bogotà]
14 ODU - Old Dominion University [Norfolk]
15 Aarhus University [Aarhus]
16 LaBRI - Laboratoire Bordelais de Recherche en Informatique
17 MNI - McConnell Brain Imaging Centre
18 Jena University Hospital [Jena]
19 BioMedIA - Biomedical Image Analysis Group [London]
20 Department of Signal Processing [Tampere]
21 ARAMIS - Algorithms, models and methods for images and signals of the human brain = Algorithmes, modèles et méthodes pour les images et les signaux du cerveau humain [ICM Paris]
22 UMG - Università degli Studi "Magna Graecia" di Catanzaro = University of Catanzaro
23 School of Systems Engineering [Reading]
24 UniGe - Università degli studi di Genova = University of Genoa
25 CINN - Centre for Integrative Neuroscience and Neurodynamics [Reading]
26 DIKU - Department of Computer Science [Copenhagen]
27 CAMPAR - Computer Aided Medical Procedures & Augmented Reality
28 Massachusetts General Hospital [Boston]
29 CSAIL - Computer Science and Artificial Intelligence Laboratory [Cambridge]
30 TU Delft - Delft University of Technology

Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.

Mots clés: Structural MRI; Challenge; Alzheimer's disease; Classification; Computer-aided diagnosis; Mild cognitive impairment

Domaines: Imagerie médicale

Esther E Bron, Marion Smits, Wiesje M van Der Flier, Hugo Vrenken, Frederik Barkhof, et al.. Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.. NeuroImage, 2015, 111, pp.562-79. ⟨10.1016/j.neuroimage.2015.01.048⟩. ⟨hal-01220123⟩