Real-World evaluation of an AI triaging system for chest X-rays: A prospective clinical study
Srinath Sridharan, Alicia Seah Xin Hui, Narayan Venkataraman, Prasanna Sivanath Tirukonda, Ram Pratab Jeyaratnam, Sindhu John, Saraswathy Suresh Babu, Perry Liew, Joe Francis, Tsai Koh Tzan, Wong Kang Min, Goh Min Liong, Charlene Liew Jin Ye
European Journal of Radiology, 2024
Abstract
Chest X-rays (CXRs) are crucial for diagnosing and managing lung conditions. While CXR is a common and cost-effective diagnostic tool, interpreting the high volume of CXRs is challenging due to workforce limitations. Artificial intelligence (AI) offers promise in enhancing efficiency and accuracy. However, real-world applicability and generalizability across diverse patient cohorts remain areas of concerns. In our study, the LUNIT INSIGHT CXR Triage software was evaluated in a diverse patient cohort. Forty-three radiologists, blinded to AI results, assessed CXRs categorized into normal, non-urgent, and urgent using a 3-tier classification system. Performance metrics and turnaround times were analyzed.
The AI system demonstrated sensitivity of 89% for normal CXRs, specificity of 93%, PPV of 83%, and NPV of 95%, with an F1 score of 0.86 and an AUC of 0.91. For non-urgent CXRs, sensitivity and specificity were 93% and 91%, with PPV and NPV at 94% and 89%, respectively, and an F1 score of 0.94 and an AUC of 0.92. In the urgent category, sensitivity was 82%, specificity 99%, PPV 90%, and NPV 98%. Subgroup analysis revealed consistently high accuracy across various age groups (Young, Adult, Senior), genders, and ethnicities (Chinese, Malay, Indian, Others), with sensitivity, specificity, and AUC consistently above 84%. The AI system also significantly reduced turnaround times across all subgroups, indicating its robust performance and generalizability in diverse healthcare settings.