4511 ajpmfocus AJPM Focus AJPM Focus Elsevier PMC11600769 11600769 11600769 39605784 10.1016/j.focus.2024.100280 Randomized Pilot of a Clinical Decision Support Tool to Increase Suicide Screening for at-Risk Primary Care Patients With Opioid Use Disorder Rossom Rebecca C MD 1 ⁎ Crain A Lauren PhD 1 Bart Gavin MD 2 Richards Julie E PhD 3 Boggs Jennifer M PhD 4 O'Connor Patrick J MD 1 Borgert-Spaniol Caitlin MA 1 Kane Sheryl MS 1 Hooker Stephanie A PhD 1 1 HealthPartners Institute, Minneapolis, Minnesota 2 Hennepin Healthcare Research Institute, Minneapolis, Minnesota 3 Kaiser Permanente Washington Health Research Institute, Seattle, Washington 4 Kaiser Permanente Colorado Institute for Health Research, Aurora, Colorado ⁎ Address correspondence to: Rebecca Rossom, MD, MS, HealthPartners Institute, 8170 33 rd Avenue South, Minneapolis, MN 55425. rebecca.c.rossom@healthpartners.com 2 10 2024 3 6 100280 100280 27 11 2024 © 2024 The Authors This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Abstract Introduction Individuals with opioid use disorder are at elevated suicide risk, but systematic screening in this population is rarely done. This study assessed the effects of targeted clinical decision support prompts on structured suicide risk assessment completion. Methods The study used a cluster-randomized controlled pragmatic pilot design. Adult primary care patients (aged 18–75 years) with or at risk for opioid use disorder or opioid overdose and suicide were eligible. Patients sought care from 15 Midwestern primary care clinics between July, 31, 2021 and July, 31, 2022. Data were analyzed between March and June 2023. Clinicians in intervention and control clinics received a printout from rooming staff, prompted by a clinical decision support-generated electronic health record alert, suggesting clinicians talk with patients about opioid risks. Intervention clinician handouts also alerted them to patients estimated to be at increased suicide risk and recommended completion of a Columbia Suicide Severity Rating Scale to further evaluate suicide risk. The handouts for control clinicians did not include suicide risk alerts. The main outcome measured the completion of the Columbia Suicide Severity Rating Scale in the 14 days following a visit. Results A total of 115 eligible patients (69 intervention, 46 control) made at least 1 visit to a randomized clinic. Patients mean age was 39 years, and 57% were women; 48% of patients had a high risk of opioid use disorder or opioid overdose, 39% had an opioid use disorder diagnosis, 12% had an opioid use disorder in remission diagnosis, and 5% had a recent opioid overdose. Over a mean follow-up of 249 days, 20.3% of intervention patients and 17.4% of control patients had at least 1 Columbia Suicide Severity Rating Scale completed in the next 14 days ( p =0.70). Most (71%–75%) Columbia Suicide Severity Rating Scale scores were 0, indicating no risk. Conclusions This pilot study did not increase the uptake of structured suicide risk assessments in primary care for patients at elevated risk for opioid use disorder and suicide. More robust interventions are likely needed to promote suicide risk assessment in primary care. Graphical abstract Image, graphical abstract HIGHLIGHTS • Individuals with opioid use disorder are at elevated risk for suicide. • This clinic-randomized trial tested the impact of clinical decision support prompts. • The outcome was structured suicide risk assessments in primary care. • Suicide assessment rates were similar in intervention and control clinics. status released display-pdf yes is-olf no is-manuscript no is-preprint no is-journal-matter no is-scanned no is-retracted no Collection date 2024 Dec. INTRODUCTION Suicide and opioid use disorder (OUD) are public health crises of paramount importance with a substantial overlap, and individuals with OUD are at elevated risk for suicide. 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 More than 48,000 people died of suicide and more than 80,000 people died of opioid overdoses in 2021, and over 40% of suicide and overdose deaths involve opioids. 11 , 12 , 13 A significant proportion of fatal opioid overdoses are intentional, that is, suicide deaths, and people with OUD are 13 times more likely to die by suicide than the general population. 14 , 15 , 16 Nearly half of patients who die by suicide make a healthcare visit in the previous month, with most visits occurring in primary care, providing primary care clinicians (PCCs) with critical opportunities to assess and address suicide risk. 17 Despite this, systematic screening for suicide risk for people with OUD is rarely done. 18 A commentary in the New England Journal of Medicine asserted that improved screening for suicide risk accompanied by rapid access to treatment are critical to prevent opioid-related suicide deaths for patients with OUD. 19 Although the U.S. Preventative Services Task Force does not recommend universal screening for suicide risk, this is not intended to discourage screening at-risk populations, such as those with OUD. 20 Ultimately, people with OUD should be targeted for both suicide and opioid overdose prevention, recognizing that comorbid mental health conditions, which are more prevalent for people with OUD, further increase risk. 8 , 21 The ability of machine-learning models using electronic health record (EHR) data to identify individuals at risk for suicide and those at risk for OUD is rapidly improving. One such model for suicide risk, developed and validated by the NIMH-funded Mental Health Research Network (MHRN), uses EHR and administrative data to generate risk models that achieve C-statistics of 0.83 to 0.86, outperforming similar previous models. 22 A separate model for estimating OUD and opioid overdose risk was developed and validated by the Epic Systems Corporation and reported C-statistics of 0.83 to 0.88 (Verona, WI; unpublished). Use of these models can help primary care practices target screening activities to patients most likely to be at risk rather than using a universal screening approach, which is often unrealistic given the many competing demands of primary care. The objective of this cluster-randomized pragmatic pilot study was to test an EHR-integrated clinical decision support (CDS) tool's ability to promote the use of a structured suicide assessment tool by PCCs for patients with OUD or estimated elevated OUD risk and with elevated suicide risk. This study was supplemental to a larger trial focused on the implementation and testing of a CDS tool to increase diagnosis and treatment of OUD in primary care. 23 This paper reports the implementation and outcomes of the pilot study. METHODS Study Sample This clinic-randomized pragmatic pilot took place at HealthPartners in Minnesota, the largest consumer-governed nonprofit U.S. health care system. HealthPartners cares for over 1.2 million patients insured by a mix of insurance types and is one of three sites in a larger clinical trial evaluating the effectiveness of a CDS intervention on OUD diagnosis and treatment for patients with or at-risk for OUD or opioid overdose. 23 The 15 HealthPartners clinics that were randomized to the OUD-CDS intervention group in the main trial were rerandomized 1:1 by the study statistician (ALC) to either add the suicide prevention intervention to the OUD-CDS intervention (Intervention; n =8) or continue the OUD-CDS intervention only (Control; n =7) using simple randomization (PROC PLAN in SAS version 9.4). As this was a supplemental study to an ongoing pragmatic clinical trial, the starting sample of patients were those eligible for the parent trial, that is., people with or at elevated risk for OUD or opioid overdose. The authors then identified patients estimated to be in the top 5% of suicide risk using the MHRN suicide risk models for this pilot intervention, as described below. Eligible patients were (1) aged 18–75 years, inclusive; (2) seen by a PCC in a randomized clinic; and (3) had a diagnosis of OUD or OUD in remission, a recent opioid overdose, or high estimated risk of OUD or opioid overdose (using Epic's recommended cutoff score of ≥55 on a 100-point scale). Opioid overdose was identified using ICD-10 codes of T40.1 × 3S-T40.41X, T40.601-T40.604S, T40.691-T40.694S, T40.0 × 1-T40.494S, or an order for emergency department-administered naloxone on a day without documentation of facility-administrated opioids (to avoid misclassifying naloxone administered following procedures as overdoses) in the previous 6 months. Patients who had stage 4 or equivalent cancer diagnoses, received parenteral chemotherapy in the last year, or were enrolled in hospice or palliative care programs were excluded and not exposed to the intervention. Patients who requested nonparticipation in research studies were excluded from analyses. Patients eligible for the main trial were also eligible for this pilot study if they scored in the highest 5% of risk for suicide using the MHRN primary care risk models for fatal and nonfatal suicide attempts. 22 Patients were ineligible and not flagged for suicide risk if structured suicide risk scores completed in the previous 60 days indicated no or low risk (i.e., a score of 0–2 on the Columbia Suicide Severity Rating Scale (CSSRS); see Measures below). 22 Each patient's index visit was the first visit at which they met all eligibility criteria, and patients were assigned to the clinic at which their index visit occurred. Patients were accrued into the study between July 31, 2021, and April 30, 2022, and were followed through July 31, 2022, or their date of death. This allowed for at least 3 months of data collection for the last accrued patient. Measures The primary outcome was the completion of the Columbia Suicide Severity Rating Scale (CSSRS) in the 14 days following a primary care visit. The CSSRS is an EHR-embedded structured assessment that asks about suicidal ideation, means, plan, intent, and previous suicide attempts. 24 PCCs in both intervention and control clinics were able to use the CSSRS in the EHR to assess suicide risk, but only PCCs in intervention clinics received alerts about patients’ elevated estimated suicide risk. The CSSRS score was calculated in the EHR, which categorized risk into four categories: no risk (score=0); low risk (score=1–2); moderate risk (score=3); and high risk (score=4–6). The Patient Health Questionnaire (PHQ9) is a structured assessment of depression symptoms. 25 Scores range from 0–27, with scores ≥10 indicating active depression. The ninth item of the PHQ9 asks about frequency of thoughts of being better off dead or hurting oneself in the previous two weeks, with possible scores of 0 (not at all), 1 (several days), 2 (more than half the days), or 3 (nearly every day). Suicide risk at outpatient visits was considered high if (a) a CSSRS score indicated moderate or high risk, or (b) there was no CSSRS score and item 9 on the PHQ9 was >0. Suicide risk was categorized as low if (a) a CSSRS score indicated low or no risk, or (b) there was no CSSRS score and item 9 of the PHQ9 was 0. If there were no CSSRS or PHQ9 assessments, suicide risk was assumed to be high, because this was the more conservative approach in the absence of information about a given patient's risk. The number of outpatient mental health specialty visits and primary care visits associated with a mental health diagnosis during the intervention period were counted, regardless of whether they were initiated by an intervention-prompted referral. Mental health engagement was considered adequate following an outpatient visit if (1) suicide risk was assessed as high or moderate and the next mental health visit occurred in 1–30 days following the visit, or (2) suicide risk was assessed as low or no risk. Medically-attended suicide attempts were identified using a validated ICD-10 code-based algorithm for EHR and claims data. 26 Suicide deaths were identified using ICD-10 codes indicating definite (X60 to X84) or possible (Y10 to Y34) self-inflicted injury or using state mortality data or the National Death Index classifying a death as a suicide. Opioid overdose was identified using a validated ICD-10 code-based algorithm for EHR and claims data. 27 For eligible patients, rooming staff received an EHR alert, prompting them to print one handout for the patient and another for the PCC. These handouts suggested clinicians and patients discuss the risks of opioids and overdoses in all 15 randomized clinics (intervention and control). In the 8 intervention clinics, the handouts also alerted PCCs when patients were estimated to be at increased risk for suicide, and the OUD-CDS tool prompted PCCs to complete the CSSRS ( Appendix Figure 1 , available online). Depending on the subsequent CSSRS score, recommendations for suicide prevention on the EHR-embedded CSSRS form ranged from care as usual (no or low risk) to referral to behavioral health for evaluation and safety planning 28 (moderate to high risk) to urgent evaluation by behavioral health or emergency department clinicians (very high risk). Before the go-live, training on the intervention, including the use and interpretation of the suicide risk models and the CSSRS, was provided to all intervention PCCs as part of standing primary care meetings, accompanied by handouts with instructions and screenshots of the tool. Control PCCs were trained on the use of the CSSRS but not on the MHRN suicide risk models or the intervention. Statistical Analysis Data were analyzed between March 2023 and June 2023. Inferential comparisons across treatment groups were made for the a priori study outcomes and safety events. Comparisons for outcomes with a single observation per patient (i.e., index visit characteristics, CSSRS completion likelihood) used unadjusted linear mixed models with outcome-appropriate link functions (i.e., identity, logit) and a random clinic intercept to account for potentially correlated outcomes among patients within randomized clinics. Comparisons for outcomes with multiple observations per patient (i.e., mental health engagement) used comparable mixed models but with an added random patient intercept. Generalized estimating equations (GEE) compared post-index to pre-index event rates across treatment groups (i.e., treatment by time). The duration of the post-index observation period (i.e., index visit through the end of the observation period) varied across patients, whereas the pre-index observation period was fixed at 1 year. The GEE models predicted the number of events observed per time period per person (e.g., overdoses, inpatient stays) using a Poisson error distribution (log link) from time period and treatment group, the specified clinic as the unit of analysis, and offset the model with the log of the duration in years of the pre- and post-index periods. Measures of association (e.g., OR, relative rate ratio [RRR]) and the associated 95% CIs and p -values are presented for each of these significance tests. HealthPartners IRB granted a waiver of written informed consent for patient participation because the study was deemed minimal risk compared to the risk associated with primary care encounters and because the study could not practically have been conducted had written informed consent been required. RESULTS Across 8 intervention and 7 control clinics, 115 patients (69 intervention, 46 control) met the inclusion criteria and made at least 1 visit ( Figure 1 ; Table 1 ). The mean age was 39.3 years (SD 12.1), and women comprised just over half of the sample (57%). Overall, 81% of patients self-identified as White, 8% with multiple racial or ethnic groups, 7% as Black, 4% as Asian, 4% as Hispanic, and 1% as Native American or Alaskan Native. Most patients (83%) were insured by Medicaid. The most common reasons for eligibility for the opioid-related requirement were a current diagnosis of OUD (39%) or high estimated OUD/opioid overdose risk (48%), whereas 12% had OUD in remission and 5% had a recent opioid overdose. By design, all patients were estimated to be in the top 5% of risk for suicide attempt in the next 90 days, with approximately equal numbers in each tier of absolute risk for fatal or nonfatal suicide attempt in the next 90 days (2.5–4%, 4–<10% risk, and ≥10%). Patients were followed for an average of 249 days (SD 80). Figure 1 CONSORT diagram. Figure 1: Table 1 Sample Characteristics and Eligibility Criteria at Index Visit by Treatment Group Table 1 All Control Intervention Patient characteristics n (%) M (SD) n (%) M (SD) n (%) M (SD) p Patients 115 — 46 — 69 — — Age — 39.3 (12.1) — 37.6 (12.0) — 40.4 (12.2) 0.25 Women 65 (56.5) — 27 (58.7) — 38 (55.1) — 0.78 Race/ethnicity 0.41 Asian 4 (3.5) — 2 (4.3) — 2 (2.9) — — Black 8 (7.0) — 2 (4.3) — 6 (8.7) — — Native American/Alaskan Native 1 (0.9) — 0 — 1 (1.4) — — Multiple 9 (7.8) — 3 (6.5) — 6 (8.7) — — White 93 (80.9) — 39 (84.8) — 54 (78.3) — — Hispanic 5 (4.3) — 1 (2.2) — 4 (5.8) — — Insurance: Medicaid 95 (82.6) — 38 (82.6) — 57 (82.6) — 0.98 Reason for main study eligibility Opioid use disorder diagnosis 45 (39.1) — 23 (50.0) 22 (31.9) — 0.11 Opioid use disorder in remission diagnosis 14 (12.2) — 4 (8.7) 10 (14.5) — 0.47 Recent opioid overdose 6 (5.2) — 2 (4.3) 4 (5.8) — 0.74 Estimated high risk of opioid use disorder or opioid overdose 55 (47.8) — 19 (41.3) 36 (52.2) — 0.28 Maximum estimated absolute risk of fatal or nonfatal suicide attempt in the next 90 days 2.5% to <4% 40 (34.8) — 12 (26.1) 28 (40.6) — 0.14 4% to <10% 36 (31.3) — 21 (45.7) 15 (21.7) — 0.02 ≥10% 39 (33.9) — 13 (28.3) 26 (37.7) — 0.32 Days from index visit to end of study — 249 (80) — 254 (73) — 246 (85) 0.59 M, mean; SD, standard deviation. Regarding the primary outcome, only 17.4% of control and 20.3% of intervention patients had a CSSRS completed at a visit in the 14 days after an eligible outpatient visit, with no significant difference between groups ( Table 2 ; OR=1.22, 95% CI=0.41, 3.65). Seventy-three percent of patients’ first-observed CSSRS scores were 0, indicating no suicide risk. Of the patients completing the CSSRS, most (55%) completed only 1, but 27% completed 3 or more. Table 2 Patients With CSSRS Completions Within 14 Days of an Outpatient Visits Table 2 Measure Control Intervention n (%) n (%) OR (95% CI) Patients 46 69 — At least one CSSRS completed in the 14 days after outpatient visit 8 (17.4) 14 (20.3) 1.22 (0.41, 3.65) First post-index CSSRS score in 14 days after outpatient visit — — — 0 (no risk) 6 (75.0) 10 (71.4) — 1–2 (low risk) 1 (12.5) 0 — 3 (moderate risk) 1 (12.5) 2 (14.3) — 4–6 (high risk) 0 2 (14.3) — Number of CSSRS completions per patient in 14 days after outpatient visit — — — 0 38 (82.6) 55 (79.7) — 1 3 (6.5) 9 (13.0) — 2 3 (6.5) 1 (1.4) — 3 or more 2 (4.3) 4 (5.8) — CSSRS, Columbia Suicide Severity Risk Scale; PHQ 9, Patient Health Questionnaire Item 9. Table 3 presents data determining patients’ estimated level of suicide risk using available CSSRS and PHQ9 item 9 scores and then assesses the adequacy of MH engagement following visits given those scores. During the observation period, control patients made 554 primary care visits, and intervention patients made 911 visits. In the 0–30 days before those visits, 60 (10.8%) visits in control clinics and 48 (5.3%) visits in intervention clinics were associated with a completed CSSRS, whereas 247 (44.6%) visits in control clinics and 397 (43.6%) visits in intervention clinics were associated with a completed PHQ9. Most completed CSSRS (57.4%) and PHQ9 item 9 (60.9%) measures had scores of 0. When then taking both CSSRS scores and PHQ9 item 9 scores into account (or assuming high or moderate risk when both were missing), 13% of control clinic visits and 22% of intervention clinic visits were associated with high or moderate suicide risk. In addition, 52.5% of control clinic visits and 54.3% of intervention clinic visits had no completed measures, and patients were assumed to have high risk. The vast majority of visits (88.3% in control clinics and 86.6% in intervention clinics) were followed by subsequent visits in the recommended timeframe (within 30 days for high or moderate risk or assumed high risk; no expected timeframe for no or low risk). The mean number of days between visits was 19 (SD 30) in control clinics and 16 (SD 26) in intervention clinics. Table 3 Adequacy of mental health engagement following outpatient visits determined by CSSRS and PHQ9 item 9 results Table 3 Measures of mental health engagement Control Intervention n (%) M (SD) n (%) M (SD) OR (95% CI) Outpatient visits, index to 30 days before study end 554 — 911 — — CSSRS score in the 0–30 days before a primary care PC visit — — — — — 4-6, High risk 12 (2.2) — 5 (0.5) — — 3, Moderate risk 11 (2.0) — 9 (1.0) — — 1–2, Low risk 4 (0.7) — 5 (0.5) — — 0, No risk 33 (6.0) — 29 (3.2) — — No CSSRS completed 494 (89.2) — 863 (94.7) — — PHQ item 9 score in the 0–30 days before a primary care visit — — — — — 1–3, Thoughts of suicide 58 (10.5) — 194 (21.3) — — 0, No thoughts of suicide 189 (34.1) — 203 (22.3) — — No PHQ9 completed 307 (55.4) — 514 (56.4) — — Suicide risk as assessed by CSSRS, or if not completed, by PHQ9 item 9 score — — — — — High or moderate risk a 72 (13.0) — 203 (22.3) — — Low or no risk b 191 (34.5) — 213 (23.4) — — Missing and assumed high risk c 291 (52.5) — 495 (54.3) — — Adequate MH engagement after outpatient visit a 489 (88.3) — 789 (86.6) — 0.87 (0.35, 2.15) Days to next outpatient mental health visit — 19 (30) — 16 (26) — a Mental health engagement was adequate if (a) suicide risk was assessed as high or moderate via the CSSRS or item 9 of the PHQ9 or not assessed and the next mental health visit was 1-30 days after the outpatient visit; or (b) suicide risk was assessed via the CSSRS or PHQ9 as low or no risk. b Mental health engagement was adequate if suicide risk was assessed via the CSSRS or PHQ9 as low or no risk. c Suicide risk was assumed high if there was no CSSRS or PHQ9 item 9 scores in past 30 days. CSSRS, Columbia Suicide Severity Risk Scale; M, mean; MH, mental health; PHQ 9, Patient Health Questionnaire Item 9. Safety data regarding self-harm and overdose outcomes, emergency department and inpatient use and patient deaths are presented in Table 4 . Self-harm event rates per patient-year decreased from 0.65 pre-index to 0 post-index for control patients and from 0.43 pre-index to 0.13 post-index for intervention patients. The absolute magnitude of the pre- or/post-change in self-harm events was larger among control patients than among intervention patients, but a significance test of the pre- or post-change by treatment group was not calculable. Overdose and opioid overdose rates declined from pre-index to post-index in both control and intervention groups, but tests of treatment-by-time interactions were not significant (overdose RRR=1.27, 95% CI=0.41, 3.94, p =0.68; opioid overdose RRR=0.42, 95% CI=0.07, 2.71; p =0.36). Similarly, the pre- to post-index declines in both emergency department visit rates and inpatient hospitalization rates were similar across treatment groups (ED RRR=0.91, 95% CI=0.46, 1.77, p =0.77; IP RRR=1.20, 95% CI=0.43, 3.32, p =0.73). No control patients and three intervention patients died during the study period; state mortality records indicate that 1 death was because of natural causes and 2 deaths were because of accidental overdoses of fentanyl and of fentanyl and methamphetamine. Table 4 Self-Harm, Overdose, Emergency Department and Inpatient Stays Before and After Index Visit Table 4 Safety events. Control Intervention n (%) patients w 1+ event n events rate: events / patient-year n (%) patients w 1+ event n events Rate: events/ patient-year Relative rate ratio (95% CI) Patient-years — — — — — — — Pre-index 46 — — 69 — — — Post-index 32.0 — — 46.4 — — — Self-harm — — — — — — — Pre-index 15 (32.6) 30 0.65 19 (27.5) 30 0.43 a Post-index 0 0 0.00 3 (4.3) 6 0.13 — Overdose — — — — — — — Pre-index 20 (43.5) 55 1.20 28 (40.6) 56 0.81 1.27 (0.41, 3.94) Post-index 3 (6.5) 8 0.25 6 (8.7) 10 0.21 — Opioid overdose — — — — — — — Pre-index 7 (15.2) 15 0.33 8 (11.6) 21 0.30 0.42 (0.07, 2.71) Post-index 5 (10.9) 7 0.22 2 (2.9) 4 0.09 — Emergency department visits — Pre-index 33 (71.7) 160 3.48 56 (81.2) 262 3.80 0.91 (0.46, 1.77) Post-index 22 (47.8) 76 2.37 42 (60.9) 109 2.35 — Inpatient stays — — — — — — — Pre-index 18 (39.1) 46 1.00 27 (39.1) 55 0.80 1.20 (0.43, 3.32) Post-index 5 (10.9) 13 0.41 14 (20.3) 18 0.39 — Deaths b 0 — — 3 (4.3) — — — a Although the absolute magnitude of the pre-post change in self-harm events was larger among control patients than among intervention patients, a significance test of the pre-post change by treatment group was not calculable . b State mortality records indicate that 1 patient died of natural causes and 2 patients had accidental overdoses, 1 with fentanyl and 1 with fentanyl and methamphetamine. DISCUSSION This intervention, aimed at increasing uptake of structured suicide risk assessments in primary care clinics for patients identified at elevated risk of OUD/overdose and suicide, had no effect on process measures. CSSRS completion rates were low at 17%–20%, with no significant difference in rates between intervention and control clinics. Clinicians have many competing priorities in busy primary care practices, and the involvement of other clinical staff to assess suicide risk, such as registered nurses or social workers, or sending suicide risk assessments to patients directly through an EHR patient portal, may be more successful in increasing suicide risk assessment. Alternatively, more intensive clinician education about suicide risk and structured assessment of risk may be helpful. Despite the low completion rates of structured suicide risk assessments, about 87% of patients had return visits within recommended timeframes for the given estimated (when CSSRS or PHQ9 was completed) or assumed (when CSSRS and PHQ9 were missing) levels of suicide risk. It is notable that in this pilot study of 115 participants, there were 2 deaths classified as accidental overdoses involving opioids. The fact that these deaths were classified as accidental rather than intentional suicides is not reassuring. Historically, most opioid overdoses have been considered unintentional or undetermined, but more recent evidence suggests that a significant proportion—estimated at 20% to 30%—are intentional. 29 In addition, distinguishing intentional from unintentional overdoses is complicated and imprecise. 30 , 31 Ultimately, intentionality of overdose is likely dimensional rather than categorical, with many overdoses not fully intentional or unintentional. 32 Regardless of intentionality, the 3% death rate in our study population during the 3–12 months of follow-up is a stark reminder of the high risk of mortality in this population with elevated OUD and suicide risk. The vast majority of completed CSSRS measures indicated low suicide risk, with 80–83% of CSSRS scores indicating no suicide risk. There are several potential reasons for this finding. One explanation could be that the MHRN suicide risk models do not perform as well for patients with OUD or opioid overdose. However, the C-statistics for this model range from 0.83 to 0.86, and adding opioid-specific data elements did not improve model accuracy. 22 , 33 Another explanation could be that CSSRS performance may be different for populations of people with or at high risk for OUD; however, the authors do not have clinical reason to suspect this and are unaware of studies examining this possibility. A third explanation is that some patients may not have wanted to potentially jeopardize their opioid prescriptions by reporting suicide ideation, as many clinicians may understandably not feel comfortable prescribing possibly fatal medications to patients with suicidal ideation. In previous work with patients with mental health conditions, it was found that of patients who went on to attempt suicide in the next 30 days, 39% had denied suicidal ideation on the PHQ9. 34 In related work, a qualitative study of patients who had denied suicidal ideation on the PHQ9 before a suicide attempt in the next 60 days, some patients reported that they had truthfully reported no suicidal ideation on the PHQ9, whereas others reported they had been experiencing suicidal ideation but feared potential consequences of disclosure, including stigma, overreaction, and loss of autonomy. 35 Taken together, these studies suggest that suicidal ideation is complex, patients may have disincentives to report suicidal ideation, particularly if they are prescribed opioids, and—should interventions be effective in increasing structured suicide risk assessments—efforts above and beyond this may likely still be needed. During the course of this study, the authors observed a significant time trend, with lower rates of self-harm, overdose, opioid overdose, emergency department visits, and inpatient stays during the post-implementation period compared to the pre-implementation period for both intervention and control. This may in part reflect a decrease in opioid prescribing rates that occurred in our health system and elsewhere during this time period. However, the authors know that most opioid overdoses are no longer related to prescribed opioids but rather to illicitly-obtained fentanyl and other opioids. 36 The authors also know that opioid overdose rates both locally and nationally continued to rise during this time period. 37 It is possible that the parent trial may have led to increased OUD screening and prescription of medications to treat OUD, but this would not explain findings in the control clinics. Therefore, the reasons behind the decreasing trends over the course of this study for these patient outcomes are unclear. Limitations This study was completed in an integrated health system, and findings may not be generalizable to other settings. Only patients who attended primary care visits were included, and findings may not be generalizable to patients who do not make primary care visits. The study did not have measures of PCC attitudes or experiences in this pilot study. This intervention did not influence the uptake of structured assessments for suicide risk, but rates of mental healthcare engagement were good, perhaps limiting the use of the suicide assessments in this context. Future implementation strategies should consider interruptive clinician alerts, more intensive, robust education strategies about suicide risk and assessment, or involving non-clinician staff to complete structured suicide risk assessments. In addition, as populations of patients with OUD have been shown to have elevated suicide risk, 29 , 38 it would be reasonable for health systems to implement initiatives to assess all patients with OUD for suicide risk rather than programming and implementing a suicide risk model to identify those thought to be at the highest tier of risk. CONCLUSIONS This pilot study did not increase the uptake of structured suicide risk assessments in primary care for patients at elevated risk for OUD and suicide. More robust interventions are likely needed to facilitate meaningful uptake of these process measures in primary care. Acknowledgments ACKNOWLEDGMENTS Disclaimers: The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the NIH Helping to End Addiction Long-term Initiative. The study sponsor had no role in study design, collection, analysis, or interpretation of data; writing the report; or the decision to submit the report for publication. Funding: The organizations employing all authors received funding through the NIH Helping to End Addiction Long-term Initiative under award number (UG1DA04031606S3) to support this work. Declarations of interest: No financial disclosures have been reported by Drs. Rossom, A Crain, Boggs, O'Connor, Borgert-Spaniol, Kane and Hooker. Dr. Bart receives honoraria for content expertise from the Substance Abuse and Mental Health Services Administration (SAMHSA)-funded Providers Clinical Support System Exchange and the SAMHSA-funded Opioid Response Network. Julie Richards has received consulting fees from NowMattersNow.org, a non-profit suicide prevention organization. CRediT AUTHOR STATEMENT Rebecca C. Rossom: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Visualization, Roles/Writing – original draft, Writing – review and editing. A. Lauren Crain: Conceptualization, Data curation, Formal analysis, Funding acquisition, Validation, Visualization, Roles/Writing – original draft, Writing – review and editing. Gavin Bart: Funding acquisition, Writing – review and editing. Julie E. Richards: Methodology, Writing – review and editing. Jennifer M. Boggs: Methodology, Writing – review and editing. Patrick J. O'Connor: Investigation, Methodology, Writing – review and editing. Caitlin Borgert-Spaniol: Project administration, Writing – review & editing. Sheryl Kane: Data curation, Writing – review & editing. Stephanie A. Hooker: Conceptualization, Investigation, Methodology, Writing – review and editing Footnotes Supplementary material associated with this article can be found in the online version at doi:10.1016/j.focus.2024.100280 . Appendix. Supplementary materials Image, image 1 REFERENCES 1. Ilgen MA, Kleinberg F, Ignacio RV, et al. Noncancer pain conditions and risk of suicide. JAMA Psychiatry. 2013;70(7):692–697. doi: 10.1001/jamapsychiatry.2013.908. 2. Ratcliffe GE, Enns MW, Belik SL, Sareen J. Chronic pain conditions and suicidal ideation and suicide attempts: an epidemiologic perspective. Clin J Pain. 2008;24(3):204–210. doi: 10.1097/AJP.0b013e31815ca2a3. 3. Ilgen MA, Zivin K, McCammon RJ, Valenstein M. Pain and suicidal thoughts, plans and attempts in the United States. Gen Hosp Psychiatry. 2008;30(6):521–527. doi: 10.1016/j.genhosppsych.2008.09.003. 4. Bohnert ASB, Valenstein M, Bair MJ, et al. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA. 2011;305(13):1315–1321. doi: 10.1001/jama.2011.370. 5. Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med. 2010;152(2):85–92. doi: 10.7326/0003-4819-152-2-201001190-00006. 6. Bair MJ, Bohnert AS. Overdoses in patients on opioids: risks associated with mental health conditions and their treatment. J Gen Intern Med. 2015;30(8):1051–1053. doi: 10.1007/s11606-015-3332-4. 7. Park TW, Lin LA, Hosanagar A, Kogowski A, Paige K, Bohnert ASB. Understanding risk factors for opioid overdose in clinical populations to inform treatment and policy. J Addict Med. 2016;10(6):369–381. doi: 10.1097/ADM.0000000000000245. 8. Bohnert ASB, Ilgen MA, Ignacio RV, McCarthy JF, Valenstein M, Blow FC. Risk of death from accidental overdose associated with psychiatric and substance use disorders. Am J Psychiatry. 2012;169(1):64–70. doi: 10.1176/appi.ajp.2011.10101476. 9. Brady JE, Giglio R, Keyes KM, DiMaggio C, Li G. Risk markers for fatal and non-fatal prescription drug overdose: a meta-analysis. Inj Epidemiol. 2017;4(1):24. doi: 10.1186/s40621-017-0118-7. 10. Rizk MM, Herzog S, Dugad S, Stanley B. Suicide risk and addiction: the impact of alcohol and opioid use disorders. Curr Addict Rep. 2021;8(2):194–207. doi: 10.1007/s40429-021-00361-z. 11. Hedegaard HM, Miniño AM, Warner M. Drug overdose deaths in the United States, 1999–2018. NCHS Data Brief. 2020;356(356):1–8. https://stacks.cdc.gov/view/cdc/84647 Accessed October 26, 2023. 12. Centers for Disease Control and Prevention. Suicide Data and Statistics. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. https://www.cdc.gov/suicide/facts/data.html?CDC_AAref_Val=https://www.cdc.gov/suicide/suicide-data-statistics.html . Updated July 18, 2024. Accessed October 22, 2024. 13. Spencer MR, Garnett MF, Miniño AM. Drug Overdose Deaths in the United States, 2002–2022. NCHS Data Brief 2024 Issue 491. https://www.cdc.gov/nchs/data/databriefs/db491.pdf . Accessed October 26, 2023. 14. Wilcox HC, Conner KR, Caine ED. Association of alcohol and drug use disorders and completed suicide: an empirical review of cohort studies. Drug Alcohol Depend. 2004;76(suppl):S11–S19. doi: 10.1016/j.drugalcdep.2004.08.003. 15. Oquendo MA, Volkow ND. Suicide: a silent contributor to opioid-overdose deaths. N Engl J Med. 2018;378(17):1567–1569. doi: 10.1056/NEJMp1801417. 16. Cherkis J. Top mental health researcher suggests link between opioid overdoses and suicides. Huffpost. 2017 https://www.huffpost.com/entry/opioids-suicide-depression-public-health_n_59d55486e4b0380b6c9a14e1 Accessed February 9, 2024. 17. Ahmedani BK, Simon GE, Stewart C, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–877. doi: 10.1007/s11606-014-2767-3. 18. O’Connor E, Gaynes B, Burda BU, Williams C, Whitlock EP. Agency for Healthcare Research and Quality (U.S.); Rockville, MD: Published 2013. Screening for suicide risk in primary care: a systematic evidence review for the U.S. Preventive Services Task Force. https://www.ncbi.nlm.nih.gov/books/NBK137737/pdf/Bookshelf_NBK137737.pdf Accessed February 8, 2024. 19. Maruti S, Desjardins I, Bagge CL, Althoff RR. Commentary: opioid use disorder and suicide: an important opportunity to address two significant public health epidemics. Prev Med. 2019;128 doi: 10.1016/j.ypmed.2019.105854. 20. United States Preventive Services Taskforce . United States Preventive Services Taskforce; Rockville, MD: Published May 20, 2014. Suicide risk in adolescents, adults and older adults: screening. https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/suicide-risk-in-adolescents-adults-and-older-adults-screening Accessed February 6, 2024. 21. Bohnert KM, Ilgen MA, Louzon S, McCarthy JF, Katz IR. Substance use disorders and the risk of suicide mortality among men and women in the US Veterans Health Administration. Addiction. 2017;112(7):1193–1201. doi: 10.1111/add.13774. 22. Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951–960. doi: 10.1176/appi.ajp.2018.17101167. 23. Rossom RC, Crain AL, O'Connor PJ, et al. Design of a pragmatic clinical trial to improve screening and treatment for opioid use disorder in primary care. Contemp Clin Trials. 2023;124 doi: 10.1016/j.cct.2022.107012. 24. Posner K, Brown GK, Stanley B, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266–1277. doi: 10.1176/appi.ajp.2011.10111704. 25. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. 26. Simon GE, Shortreed SM, Boggs JM, et al. Accuracy of ICD-10-CM encounter diagnoses from health records for identifying self-harm events. J Am Med Inform Assoc. 2022;29(12):2023–2031. doi: 10.1093/jamia/ocac144. 27. Green CA, Perrin NA, Hazlehurst B, et al. Identifying and classifying opioid-related overdoses: a validation study. Pharmacoepidemiol Drug Saf. 2019;28(8):1127–1137. doi: 10.1002/pds.4772. 28. Stanley BBGK, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract. 2012;19(2):256–264. doi: 10.1016/j.cbpra.2011.01.001. 29. Oquendo MA, Volkow ND. Suicide: a silent contributor to opioid-overdose deaths. N Engl J Med. 2018;378(17):1567–1569. doi: 10.1056/NEJMp1801417. 30. Gray D, Coon H, McGlade E, et al. Comparative analysis of suicide, accidental, and undetermined cause of death classification. Suicide Life Threat Behav. 2014;44(3):304–316. doi: 10.1111/sltb.12079. 31. Rockett IRH, Caine ED, Connery HS, et al. Discerning suicide in drug intoxication deaths: paucity and primacy of suicide notes and psychiatric history. PLoS One. 2018;13(1) doi: 10.1371/journal.pone.0190200. 32. Bohnert ASB, Ilgen MA. Understanding links among opioid use, overdose, and suicide. N Engl J Med. 2019;380(1):71–79. doi: 10.1056/NEJMra1802148. 33. Yarborough BJH, Stumbo SP, Rosales AG, et al. Opioid-related variables did not improve suicide risk prediction models in samples with mental health diagnoses. J Affect Disord Rep. 2022;8 doi: 10.1016/j.jadr.2022.100346. 34. Simon GE, Coleman KJ, Rossom RC, et al. Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77(2):221–227. doi: 10.4088/JCP.15m09776. 35. Richards JE, Whiteside U, Ludman EJ, et al. Understanding why patients may not report suicidal ideation at a health care visit prior to a suicide attempt: a qualitative study. Psychiatr Serv. 2019;70(1):40–45. doi: 10.1176/appi.ps.201800342. 36. Centers for Disease Control and Prevention (CDC) Vital signs: overdoses of prescription opioid pain relievers—United States, 1999–2008. MMWR Morb Mortal Wkly Rep. 2011;60(43):1487–1492. 37. Jackson K. As Twin Cities opioid overdoses surge, officials lean into drug prevention, education. Star Tribune. October 9, 2022 https://www.startribune.com/twin-cities-officials-lean-into-drug-prevention-and-education-as-opioid-overdoses-surge/600214343/ Accessed February 9, 2024. 38. Ali MM, Dubenitz J. Suicidal behavior, opioid use disorder, and behavioral health treatment: prevalence and correlates among adults in the United States 2015–2018. J Subst Abuse Treat. 2021;130 doi: 10.1016/j.jsat.2021.108413. Associated Data Supplementary Materials Image, image 1