Deployment and Evaluation of a Motivational-Interviewing Chatbot for Moving Smokers Towards the Decision to Quit and Distillation of Large Foundational Language Models for Generating MI Reflections

Motivational Interviewing (MI) is a validated therapeutic approach that has been shown to help ambivalent people struggling with addiction move toward change. MI has been applied to many behaviours, including smoking cessation. If conversational agents could effectively apply MI, they may provide a scalable way to help more people access this therapy. Previous attempts to provide MI therapy through text-based conversational agents have typically employ scripted responses to client statements, but such non-specific responses have been shown to reduce effectiveness. A key technique in MI is to ask open-ended questions and then provide a reflection of the response to evoke contemplation in the client. Recent advances in Natural Language Processing provide a new way to create responses that are specific to client's statements, using a Transformer-based Language Model. We present the design, evolution and impact assessment of a chatbot that makes use of generated reflections, whose goal is to guide ambivalent smokers toward the decision to quit. Through four trials of 349 participants, we show that the chatbot significantly increases participants' confidence to quit smoking one week after the conversation compared to before the conversation (\textit{P}=.001). As a key part of the chatbot is the language model that produces reflection, and it is difficult and sometimes impractical to run a clinical bot with a corporation's proprietary model, we explore methods of model distillation to train smaller, more practical language models to generate MI-adherent reflections. We present a method for distilling the specific tasks of generating MI reflections from a Large Foundational Language Model (GPT-4) into smaller models. We show that GPT-4 can generate MI-adherent reflections near 100\% success and use output generated from that model to fine-tune the much smaller GPT-2 family as form of knowledge distillation. We also use GPT-4 as a zero-shot evaluator to classify the quality distilled student model outputs and validate that classifier with a triple human-review. We show that the GPT-2 small achieves an 83\% success rate on a hold-out test set and the GPT-2 XL achieves 90\% success. In addition, our GPT-4 zero-shot prompt evaluator achieves significantly high inter-rater reliability (.61 Cohen-Kappa) with triple-human review.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.