Implementation of a Web-Based Chatbot to Guide Hospital Employees in Returning to Work During the COVID-19 Pandemic: Development and Before-and-After Evaluation
Bibliographic record
Abstract
BACKGROUND: Throughout the COVID-19 pandemic, multiple policies and guidelines were issued and updated for health care personnel (HCP) for COVID-19 testing and returning to work after reporting symptoms, exposures, or infection. The high frequency of changes and complexity of the policies made it difficult for HCP to understand when they needed testing and were eligible to return to work (RTW), which increased calls to Occupational Health Services (OHS), creating a need for other tools to guide HCP. Chatbots have been used as novel tools to facilitate immediate responses to patients' and employees' queries about COVID-19, assess symptoms, and guide individuals to appropriate care resources. OBJECTIVE: This study aims to describe the development of an RTW chatbot and report its impact on demand for OHS support services during the first Omicron variant surge. METHODS: This study was conducted at Mass General Brigham, an integrated health care system with over 80,000 employees. The RTW chatbot was developed using an agile design methodology. We mapped the RTW policy into a unified flow diagram that included all required questions and recommendations, then built and tested the chatbot using the Microsoft Azure Healthbot Framework. Using chatbot data and OHS call data from December 10, 2021, to February 17, 2022, we compared OHS resource use before and after the deployment of the RTW chatbot, including the number of calls to the OHS hotline, wait times, call length, and time OHS hotline staff spent on the phone. We also assessed Centers for Disease Control and Prevention data for COVID-19 case trends during the study period. RESULTS: In the 5 weeks post deployment, 5575 users used the RTW chatbot with a mean interaction time of 1 minute and 17 seconds. The highest engagement was on January 25, 2022, with 368 users, which was 2 weeks after the peak of the first Omicron surge in Massachusetts. Among users who completed all the chatbot questions, 461 (71.6%) met the RTW criteria. During the 10 weeks, the median (IQR) number of daily calls that OHS received before and after deployment of the chatbot were 633 (251-934) and 115 (62-167), respectively (U=163; P<.001). The median time from dialing the OHS phone number to hanging up decreased from 28 minutes and 22 seconds (IQR 25:14-31:05) to 6 minutes and 25 seconds (IQR 5:32-7:08) after chatbot deployment (U=169; P<.001). Over the 10 weeks, the median time OHS hotline staff spent on the phone declined from 3 hours and 11 minutes (IQR 2:32-4:15) per day to 47 (IQR 42-54) minutes (U=193; P<.001), saving approximately 16.8 hours per OHS staff member per week. CONCLUSIONS: Using the agile methodology, a chatbot can be rapidly designed and deployed for employees to efficiently receive guidance regarding RTW that complies with the complex and shifting RTW policies, which may reduce use of OHS resources.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.045 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".