Active Involvement of End-Users in an EHR Procurement Process: a Usability Walkthrough Feasibility Case Study

Leadership at a private, non-profit, 1000-bed teaching hospital in Lille, France, issued a request for proposals (RFP) and organized a procurement process to select a replacement for their current commercial EHR. The process included three steps: (1) a vendor demonstration; (2) a usability walkthrough with each candidate EHR; and (3) technical and economic comparisons of EHRs selected during the walkthrough. The comparisons focused on “back-end” functionality (e.g., data interoperability). In this manuscript, we describe the second step of the process (i.e., usability walkthrough). To conserve resources and adhere to a timeline, it was necessary to quickly thin the pool of EHR candidates for later technical evaluation. We did not set out to exhaustively safety test products or generate summative statistics during the second phase. Therefore, we limited the number of users recruited. The project manager (AP) and four usability experts (RM, SG, JS, SP) designed a usability instructional session and a usability workshop that included structured exercises and evaluation instruments. We planned to conduct two instructional sessions and five workshops over three weeks from September to October 2020 (Fig. 1 ). Evaluation process. We hosted two face-to-face instructional sessions. We held the first session two weeks before the workshops and a refresher on the day of the first workshop. In both sessions, we explained the dimensions of usability, demonstrated the walkthrough method, reviewed usability assessment criteria (adapted from Scapin and Bastien) , and introduced a usability issue severity rating scale. We answered questions and furnished the participants with a written summary of all content. We received proposals from five vendors and scheduled five 3-hour workshops over one week: one per candidate (Fig. 1 ). During each workshop, the vendor presented their EHR to stakeholders. We then excused vendors from the proceedings; they were not permitted to interact with end-users during any other evaluation stage—including the usability walkthrough and end-user debriefing session (outlined below). A multidisciplinary team of clinical representatives, the procurement manager, and a usability expert (JS) identified common inpatient EHR use scenarios and concerned end-users. We designed seven CLIPS simulating 59 EHR tasks (Appendix A ), targeting frequently used or critical functionalities (Table 1 ). We also created a clinical dataset for the EHR. We sent the CLIPS and dataset to the EHR vendors two weeks before the usability sessions. We recruited nine end-users: three physicians (an emergency physician from the emergency unit, a cardiologist from the cardiology unit, and a neurologist from the geriatrics unit), three nurses (from emergency, cardiology, and geriatrics units), one pharmacist (from the central pharmacy), one medical clerk, and one admission officer (i.e., non-clinical staff member trained to manage administrative and logistic duties). Participants volunteered for the workshops; they were not compensated for their participation. None had been trained to use the candidate EHRs. We organized participants into four evaluation groups—each supervised by a usability expert. Each group completed CLIPS at a computer workstation. Three groups included a physician and a nurse, whereas the fourth included the pharmacist, admission officer, and clerk. The usability experts facilitated the walkthrough, tracked time, and gathered field observations. We first gave each group written CLIPS and a patient summary. We then instructed groups to use the EHR to complete tasks, describe issues encountered, and assign each issue a usability criterion (i.e., “guidance,” “workload,” “compatibility,” “significance of codes,” “adaptability,” “error management,” “consistency,” or “explicit control”). Participants also assigned a severity level (i.e., “light,” “minor,” or “major”) to each issue. Please refer to Appendices B and C for criterion and severity definitions. We audio-recorded comments. The usability experts documented direct quotes during the sessions, issues reported by end-users, and criteria and severity scores. After each task, end-users completed a 4-item questionnaire adapted from Schumacher et al. with 5-point Likert-type items evaluating EHR features availability, completeness, ease of use, and efficiency. We published our questionnaire findings in a companion article. The walkthrough ended once participants completed all CLIPS or after two hours had elapsed. Afterward, users completed the System Usability Scale (SUS) —a validated 10-item questionnaire with Likert-type statements and performance benchmarks. We published the results of this questionnaire in a companion article. We organized end-users into groups according to professional roles. A usability expert then debriefed each group using a semi-structured interview script exploring EHR strengths and weaknesses (Appendix D ). Our usability experts first read problems identified by the participants and excluded (1) those unrelated to specific EHR characteristics (e.g., opinions without descriptions), (2) those concerned with the technology platform (e.g., connection failures), or (3) those rooted in data upload problems. They then combined multiple descriptions of the same problem (i.e., deduplication) to reach a final list of usability problems. Next, we created a usability expert “comparison set”. We combined lists of problems identified by participants and problems detected by usability experts. For each problem, we assigned a usability criterion and a severity level. Experts independently categorized problems using our a priori usability criteria and severity levels. Disagreements were resolved through consensus. We then compared end-users’ lists and assignments to the “comparison set.” We calculated concordance between end-users and experts using percent agreement and Krippendorf’s α. We also calculated the average issue detection rate within user profiles when there were multiple representatives (i.e., nurses and physicians). Two usability experts screened problem descriptions to identify those requiring clinical expertise to detect. Since these represented new types of problems only clinicians recognized, our usability experts categorized each problem inductively. After the last walkthrough, we asked participants to provide feedback on the method. To measure user satisfaction, we developed an eight-item questionnaire with 5-point Likert scales anchored by strongly disagree and strongly agree (Table 5 ). Participants also answered open-ended questions about the strengths and weaknesses of the method. We compared each rating to 3 (i.e., the median value) using the Wilcoxon sample signed-rank test with a significance threshold of 0.05. Two usability experts analyzed the qualitative data inductively to identify important or recurrent themes. We recorded all deviations from the workshop agenda and evaluation protocol. Two usability experts categorized each deviation according to the root cause. In this section, we report on the feasibility and utility of our usability evaluation strategy. We reported EHR usability findings in a companion article. Two vendors withdrew their applications during the procurement process, leaving only three candidate EHRs for analysis. Participants reported 361 usability problems. We excluded 21 issues (5.82%) using our eligibility criteria. After deduplication, the final list consisted of 265 problems: 258 detected by end-users (97.36%) and 7 detected only by usability experts (2.64%) (Table 2 ). Each end-user within a professional role detected between 26.82 and 70.37% of all problems identified by that group (mean = 42.92%; SD = 14.11). On average, 59.83% of the problems were detected by one participant, 23.73% by two, and 12.75% by three. End-users assigned a criterion to 218 of the 258 problems (84.49%); 157 matched those assigned by usability experts (72%; Krippendorff’s α = 0.66 [0.59; 0.74], Table 3 ). The largest discrepancies were in three categories: “guidance,” “workload,” and “compatibility.” Participants assigned a severity level to 217 of the 258 problems identified (84.10%): 165 matched those assigned by experts (76%; Krippendorff’s α = 0.75 [0.68; 0.82], Table 4 ). End-users more often rated problems as “light” or "major,” whereas usability experts more often rated problems as “minor.” Thirty-two of the 258 problems (12%) required clinical expertise to detect (Appendix E ). In one instance, a pharmacist using the medication review module could not determine how to make a drug substitution, and in another, a physician did not receive an error message after entering the same treatment order twice. We include a complete list of issues in Appendix E . We classified these problems into 5 novel categories: patient identifiers ( n =4), information availability and visibility ( n =12), EHR configurations ( n =9), access to EHR functions by users’ roles ( n =2), and work and cognitive load problems ( n= 5). There were instances when patient identifiers were not visible while performing tasks, increasing the risk of wrong patient selection errors. End-users encountered screens missing critical information, including orders, decision support, care plan actions, and medication changes. There were readability issues associated with typography and inappropriate “hard stops” (e.g., health insurance information queries blocking access to other functions). We found a mismatch between user EHR permissions and real-world scope of practice (e.g., nurses were granted access to prescribe when not permitted in clinical practice). Finally, the EHRs tended to increase workload and cognitive load. For example, some features increased the number of actions per task, failed to provide the user with feedback, or made selections confusing (e.g., a mismatch between medication type and dosing units). Overall, end-users valued the usability walkthrough. The average score for each questionnaire item was at least four on a five-point scale (Table 5 ). Participants, however, struggled to assign usability criteria ( n = 4 of 9), and all reported some challenges learning the scoring and categorization system. One respondent said, “ understanding the criteria takes time, ” and another would have appreciated “ even more training upstream of the evaluations .” Nevertheless, most respondents ( n = 8 of 9) said the method was “ easy to learn .” Most participants liked the ability to quantify subjective impressions of the technology and indicated they would be willing to use this method again ( n =8 of 9). They all agreed that the method “ clearly distinguished [EHRs] strengths and weaknesses, ” and the data permitted a “ detailed comparison of EHRs .” Two vendors withdrew their applications. One had concerns about their product’s usability; the other did not disclose a reason. The remaining vendors were not prepared for the evaluation. We could not complete all CLIPS due to technical issues (e.g., features not working) and database “locks” that prevented multiple users from opening the same test record simultaneously (Table 6 ). In some cases, the EHRs were missing patient data, whereas, in others, the data that participants were expected to enter were already in the chart.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.