Evaluation of Deep Ensemble for the Recognition of Malignancy (DERM) Overview In England, 60% of dermatology referrals are made through the urgent suspected skin cancer pathway, but only 6% are converted to a confirmed case of skin cancer. This inefficiency could be addressed by safely triaging some referrals out of the suspected skin cancer pathway. DERM is a UKCA class IIa AI technology, developed by Skin Analytics. This evaluation was commissioned by the Department of Health and Social Care (DHSC) as part of the Phase IV of the AI Awards in Health and Social Care. Skin Analytics have implemented DERM in six sites as part of the AI Awards, four of which were included in the evaluation, aiding the detection of skin cancers through triage of referrals. The sites and their associated patient numbers under relevant care models (blue for secondary care, green for community hub) were: Chelsea and University Ashford and St Birmingham PCNs / Westminster NHS Hospitals Bristol Peter’s Hospitals University Hospitals Foundation Trust and Weston NHS NHS Foundation Birmingham NHS (CW) Foundation Trust Trust (ASPH) Foundation Trust (UHBW) (UHB) 4,010 2,868 2,234 537 Method Quantitative Qualitative Machine learning Health economic insights insights principles review modelling Performance analysis of A mixed-method Assessing whether best Three models were DERM, subgroup analysis approach using a patient practice standards in developed: a cost-utility (health inequalities), and survey and interviews with machine learning were analysis (CUA), a cost- comparative analysis. patients and staff. There followed in developing benefit analysis (CBA), There was a focus on was a focus on safety, the DERM algorithm. A and a budget impact safety, accuracy, effectiveness, focus on effectiveness, model (BIM). There was a effectiveness, and acceptability, and safety, accuracy, and focus on value, sustainability of DERM in sustainability. sustainability. effectiveness, and deployment settings. sustainability. Secondary care model Quantitative study Sensitivity Secondary care In secondary care, DERM assessed 5,678 lesions out of secondary care: conversion rate: 4,159 referred, yielding high pathway sensitivities for melanoma across all sites (higher than 90%), achieving its target rates and showing that DERM was effective in 97.4% 12.6% channelling high risk lesions to the appropriate management outcome. Community hub Community hub In the community hub, DERM was correctly assigning conversion rate: discharge rate: pathways for lesions based on risk, with high pathway sensitivities, but with a wider confidence interval than seen in secondary care due to the smaller sample size. 18.2% 60.1% Second read impact The second read, involving the review (by a dermatologist consultant) 36% of cases classified as low risk by DERM, is risk averse in its decision making. It overturned 36% (n = 754) of potential DERM discharges, of overturn rate which 41% (n = 307) were discharged by trust dermatologists. Across the evaluation, seven cancers were identified by the second read (out of 754 overturned discharge cases). The majority of these 41% were low risk basal cell carcinomas (BCC). overturned cases discharged following trust review There was no evidence of incorrect discharge by the second read, through patients returning within six months. Subgroup analysis While exploratory, results show that pathway sensitivities were similar between subgroups such as age and Fitzpatrick skin type. There was no indication of performance varying with respect to the likely distribution of cases according to deprivation. Secondary carestudy Qualitative model Patient survey results Interview results Findings suggest that patient perceptions of AI- The importance of appropriate referrals, high quality enabled teledermatology services were largely pictures, and a supported trained workforce were positive. noted as key factors for a successful deployment. rated AI-enabled teledermatology Staff reported that teledermatology services using DERM had a 85% services as good or very good, suggesting a high level of acceptability 'transformational' effect on capacity. It was unclear whether this could be attributed specifically to DERM rather were uncomfortable about the AI- than teledermatology alone. 13% enabled teledermatology being used to help determine their diagnosis Both staff and patients were reassured by the use of the second read, some acknowledged the value of using AI to staff would not be in favour of speed up getting an appointment 67% rather than waiting to be seen by a removing it at this stage. doctor Machine learning principles review The review reported DERM performing binary The existing teledermatology infrastructure was classification per lesion and achieved strong results, highlighted as an enabler to DERM deployment. from data handling, image lesion distribution, sensitivity and perturbation analyses, and corner case There was a need to ensure that cultural factors are classification. appropriately considered. Algorithm development was deemed reusable and Dermatologists required for a national scale-up of reproducible, which is encouraging when considering DERM with the second read should be considered, implementation in future sites. given the national workforce shortages. Secondary care model Cost-benefit analysis The cost-benefit analysis modelled the impact of teledermatology with and without DERM on skin cancer referrals and compared each against a non- 2,600 42% Dermatologist Less time spent digital face-to-face pathway. All instances appointments saved by reviewing images with highlighted savings for the NHS; DERM saved more DERM each year in an DERM than traditional average* NHS Trust teledermatology dermatologist time than traditional teledermatology. (*) Based on 4,320 suspected skin cancer referrals per year 2.5 1.7 1.1 Teledermatology DERM in secondary care DERM in primary care Potential implementation: Actual implementation: Actual implementation: £2.5 returned for every £1 £1.7 returned for every £1 £1.1 returned for every £1 spent. spent. Could increase to £3.7 spent. Could increase to £2.0 if Could decrease to £1.0 if the in more mature sites. the second read is removed. second read is removed. Secondary careanalysis Cost-utility model Probabilistic sensitivity analysis Results for each site were generated separately. showing variation in outputs They included the pathway with the second read, and with no second read (NSR). The CUA showed varied outputs between sites, with UHBW and CW showing a higher cost and worse health effect, and UHB and ASPH showing lower cost and better health effect. Removal of the second read resulted in slightly better cost and health effects at all sites. The study remained broadly inconclusive due to inconsistencies in the data, overshadowing the relatively small quality of life impact of DERM that could be quantified. Secondary care model Recommendations Collection and obtention of more Repeating the subset analyses with a complete site-level baseline data to larger dataset, and obtaining data on the improve robustness and conclusiveness performance of current pathways to of results. treat patients with darker skin tones. Future rollout studies to evaluate in Further analysis regarding adherence, detail how the technology is applied in focusing on those patients who declined different settings, to identify what the teledermatology. most effective model would be. Further qualitative work regarding Investigation of the implementational removal of the second read to and operational factors enabling DERM understand perceptions of patients and deployment, including infrastructure, staff in relation to clinical confidence workforce, and usability. and decision-making. Limitations Baseline data Data quality Data availability Observable cases Direct comparative data Reliability of the There was limited data Relatively few at a granular level could aggregated comparative available for some cases/lesions were not be obtained, making data skewed the measures for example, observed in the limited inferences about the performance of traditional impact of missed deployment period at effect of DERM over and teledermatology models cancers, and quality of UHB, meaning the above existing towards best-in-class life data for outcomes assessment of the pathways difficult to sites, likely inflating the such as anxiety. DERM community hub answer. return on investment. model has been limited. Conclusion Findings suggest DERM performed accurately and safely across secondary care and community hub models, though the latter requires a larger sample size for validation. Unnecessary referrals were reduced and consultant time released due to DERM implementation. The qualitative study found most participants suggested that the DERM service was very good or good. Staff interviews highlighted the platform's user-friendliness, efficiency, and the ability to discharge patients with benign lesions at the triage step as the main benefits of DERM. Both staff and patients were reassured by the use of the second read and some staff suggested they would not be in favour of removing it at this stage. The CBA revealed cost savings with DERM over face-to-face pathways in secondary care, with modest savings in the community hub model. Findings from the CUA showed varied and mostly inconclusive results.