Usability Evaluation of Health Technologies / Medical Devices – Useval-DM

Work Package 1 - Identification of potential use errors
The objective is to generate outcomes that will constitute the main observable variables for WP3-6. Triangulation of methods has been used to ensure the best possible (most exhaustive) collection of potential use errors for each device.

Work Package 2 - Definitions of the final experimental designs
The objective is to specify the characteristics of the usability validation environments for each device. With this goal in mind several tasks were performed.
Two literature reviews were performed to identify (i) the relevant dimensions of ecological validity for usability testing, and especially for usability validation (i.e. summative evaluation) of MD; and (ii) the relevant sample size for participants in usability validation of MD along with the definition of dataset required for economic analysis. Based on the results, the various contexts of usability validation were specified for each MD.

Work Packages 3 - 4 - 5 - Usability validation studies for each medical device
The objective is to run the usability validation studies for each device. To do so, the validation plans will be finalized, the suability tests runned and the data collected for the statistical and economical analyses.

Work Package 6 - Statistical analyses and economical evaluation
The objective is here to run the statistical and economical analyses in order to specify the expected value of each usability validation environment as regards to its characteristics (ecologial validity, number of participants). Based on these results, illustrative guidelines will be provided.

In progress

In progress

J Rochat, T Wheinwold, C Lovis, J Schiro, R Marcilly, S Pelayo. A Literature Review to Define Concepts and Dimensions of Ecological Validity/Fidelity for Usability Validation. Communication submitted to the International conference of the European Federation for Medical Informatics, May 2017, Manchester, UK.

R Marcilly, J Schiro, B Dervaux, A Duhamel, C Lovis, MC Beuscart-Zephir, S Pelayo. Toward Scientific Evidence Regarding the Medical Devices Usability Validation Stage: The Useval-DM project. Poster presented at the International conference of the European Federation for Medical Informatics, May 2016, Munich, Germany.

T Weinhold, M Del Zotto, J Rochat, C Lovis, J Schiro, S PELAYO, R Marcilly. Literature review of potential use errors of adrenaline auto-injection pens. Communication submitted to the International conference of the European Federation for Medical Informatics, May 2017, Manchester, UK.

M Del Zotto, PF Gautier, T Weinhold, J Schiro, J Rochat, C Lovis, S PELAYO, R Marcilly. Potential use errors of ANI Monitor to evaluate patient pain and discomfort. Communication submitted to the International conference of the European Federation for Medical Informatics, May 2017, Manchester, UK.

J Schiro, S Pelayo, T Weinhold, M Del Zotto, R Marcilly. Adherence to Guidelines for Identifying Known Use-errors for the Medical Devices Usability Validation: lessons learned. Communication submitted to the International conference Building capacity for the health informatics in the future (ITCH), February 2017, Victoria, Canada

The Useval-DM project refers primarily to challenge 4 “Life Health Wellbeing” and its chapter 13 “Health technologies”. It refers secondarily to the challenge 7 “Information and communication society” and its chapter 7 “Human-Machine Interaction “ (HMI).

Chapter 13 TECSAN of challenge 4 explicitly mentions “hopes” and “promises” provided by innovative health technologies, but also highlights the “concerns” about “risks related to […] the effects on human performances”. These are exactly the risks that qualitative usability validation aims at preventing and this is the core of the Useval-DM project. This project targets (as mentioned in the call): the “implementation of a proof of concept that is sound from an industrial or medical point of view, and also the exchange of knowledge between industrial and academic partners[…] in order to strengthen the French competiveness in the biomedical sector.”
Innovative health technologies and their extensive use of computerized data require a thorough analysis of their human-machine interface (challenge 7 chapter 7). The European regulation on medical devices (MD) imposes a usability engineering approach (in particular for high-tech devices) to handle this problem. This approach allows, as mentioned in the call “to integrate the user […] and take into account the interdisciplinary dimension of human-machine-interaction”.

Despite the obligation to apply usability engineering for MDs since 2010, the method is little known and poorly applied. Standards harmonized to the ergonomics essential requirement impose a final usability evaluation of MD, namely "usability validation".

Usability validation is difficult to implement because:
• it requires multidisciplinary competencies and testing equipment, preferably installed in a health institution like a hospital
• it requires to carry out usability tests in simulated environments based on “worst-case scenarios”. The principal benefit of this procedure is scientific reproducibility. Yet this comes with the downside of a poor ecological validity.
• the number of participants to those tests is a key factor that may introduce large cost variances and may impact the validity of the results. Harmonized standards provide no guidance and there is no scientific evidence to that question.
• the usability validation is costly for manufacturers. It becomes urgent to properly conduct an economic evaluation of these methods in order to better support and guide the manufacturers in its planning and application.

The MAIN OBJECTIVE of the Useval-DM project is to establish scientifically, including from an economic perspective, the critical methodological choices for the usability validation of MDs, focusing on the added value of (i) the variations of ecological validity (laboratory, simulator, work context) and of the cultural specificities (France, Switzerland) (ii) the variations in numbers of participants in the test. On the basis of the gathered results, the project will deliver guidelines and illustrative protocols to the stakeholders (manufacturers, health authorities, certification bodies, standard committees) in order to carry out effectively usability validations of MDs.

Two European usability centers for health technologies (CHU Lille and HU Genève) will be in charge of designing scientific protocols of usability validation and implement their methodological variations. Experts in statistics and health economy (CHU Lille) will assess those protocols. They will cooperate with three manufacturers of innovative medical devices. The three MDs that are included in the study differ in the targeted user group (health care professional or patient) and in the complexity of human-machine interaction. Thus, an estimation of a possible generalization of the results can be given. Subsequently, an extended body of scientific evidence for a wide range of medical technologies could be established, possibly on a European level.