Messages from patients to care teams via patient portals are increasing¹ and may be associated with burnout.² Health systems are deploying large language models (LLMs) to address this issue by drafting replies to patient messages.^3,4,5

In contemporary team-based care, how useful are LLM’s to different team members? What iterative changes in an LLM prompt might improve the usefulness of the draft reply?

This prospective quality improvement (QI) study was determined to be exempt from review and the requirement of informed consent by the University of Colorado institutional review board. The QI study was conducted from September 2023 to March 2024 within a large academic health system (UCHealth), following the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guideline. We deployed an LLM, GPT-4 (Generative Pretrained Transformer 4 [OpenAI]), via the electronic health record (Epic Systems). Internally dubbed PAM Chat (Patient Advice Message Chatbot reply), the LLM drafted a reply to incoming messages from the patient portal in 9 clinics (6 primary care and 3 specialty), for nurses, medical assistants (MAs), and clinicians (physicians and advance practice clinicians [APCs]).

Based on feedback during the project, we iteratively improved prompts (see the eMethods in Supplement 1 for prompt versions and rationale). For example, we included the assessment and plan (A/P) of the last progress note written by the recipient clinician, emphasizing conciseness. We disclosed that content was computer generated before editing by the clinician to patients.

We surveyed all users 2 weeks after all clinics were live with PAM Chat, including the net promoter score (NPS) and unvalidated de novo items assessing perceptions of the tool. Analysis of variance was used to test the null hypothesis that attitudes toward PAM Chat would not vary by professional role. Data analysis was conducted using JMP Pro software version 17.0.0 (SAS Institute) and took place from April to June 2024. The threshold for significance was a 1-sided P < .05.

PAM Chat generated 21 323 drafts, and 2596 drafts were used (12%) across all 166 users (12 nurses, 14 MAs, and 93 clinicians). Nurses had more favorable views of PAM Chat (Table 1). Nurses were more likely to agree that PAM Chat reduced the need to forward messages to physicians and APCs and helped them to stay within their scope of practice. Of all nurses, 11 (92%) believed that PAM Chat helped improve efficiency, empathy, and tone. Nurses were most likely to recommend PAM Chat to others (NPS, 58). MAs, physicians, and APCs were less favorable.

Comments from colleagues about PAM Chat were mixed: “Sometimes the answers are astoundingly helpful, other times they are not helpful at all.” Including the A/P from the last note made some replies useful. Illustrative comments are in Table 2.

In this QI study, nurses liked PAM Chat more than MAs and physicians or APCs, with more than 90% believing that it improved efficiency, empathy, and tone. Our results diverge from other reports^3,4,5 that describe primary care nurses’ negative views of LLMs. This finding may be because the specific messages each group sees are different. Physicians and APCs may preferentially receive complex messages, which are more difficult for the LLM; MAs may view messages containing clinical information as out of scope. This role-based difference and our overall use rate of 12% suggests that, moving forward, LLMs may need to be tuned to recognize who will receive the message (MA, nurse, or physician or APC) and create a reply accordingly.

Our findings also suggest that implementing PAM Chat with ethics principles (eg, transparency) and including the last A/P note can improve utility. Interestingly, in response to a message about a gift, the LLM generated a response that suggested that gift-receiving is not permitted—an AI oversimplification given that the American Medical Association Code of Ethics permits gifts in limited circumstances.⁶

This evaluation was conducted in 9 clinics in 1 health system and may not be generalizable to all health systems. We hope sharing our prompts and results will support ongoing improvement of LLMs in the pursuit of high-quality, safe, and effective patient care while reducing care team burden.