The setting
Ting and colleagues’ 2017 JAMA study evaluated a deep-learning system using retinal photographs from Singapore and external cohorts. It provides a documented diagnostic-validation case, rather than a claim that every later product version improves patient outcomes.
The decision context
Assess the published study at its stated endpoint and threshold. Any proposed service expansion, staffing arrangement or follow-up evaluation is AIPEX analysis, not a deployment reported by this paper.
The evidence to examine
Reported evidence
Keep the denominator straight
The primary validation dataset contained 71,896 images from 14,880 patients. Images, eyes and patients are different units; the study also examined external datasets.
Source: Ting et al. / JAMA, Development and Validation of a Deep Learning System for Diabetic Retinopathy (2017) · Abstract; Table 1: primary validation dataset ↓Reported evidence
Threshold and trade-off
For referable diabetic retinopathy, the primary validation reported sensitivity of 90.5% (95% CI 87.3–93.0) and specificity of 91.6% (91.0–92.2). Trained graders had specificity of 99.3%.
Source: Ting et al. / JAMA, Development and Validation of a Deep Learning System for Diabetic Retinopathy (2017) · Table 4: referable diabetic retinopathy ↓Reported evidence
The authors’ boundary
The authors called for further research on healthcare applicability and improved vision outcomes. Reference standards varied across external datasets; several authors disclosed a related patent interest.
Source: Ting et al. / JAMA, Development and Validation of a Deep Learning System for Diabetic Retinopathy (2017) · Limitations; Conclusions; Conflict of Interest Disclosures ↓What this does not establish
Diagnostic validation is not a trial showing improved vision or a general clinical recommendation. Referral workload depends on prevalence, threshold and downstream capacity. Do not reverse-engineer exact classifier counts from rounded percentages or treat images as independent patients.
Try the decision first
Your task.
What additional evidence would justify moving from diagnostic performance to a claim of improved screening outcomes?
- Record target condition, operating threshold, reference standard, unit of analysis and uncertainty. Compare sensitivity and specificity, rather than summarising both as accuracy.
- Assess implications for missed cases and extra referrals in the intended population. Request actual counts and patient-level workflow data; do not fabricate a matrix from rounded results.
- Specify a comparison of the full screening pathway: timely referral, attendance, clinical outcomes, workload and access. Identify what the validation study supports and what remains to test.
Read one defensible approach
AIPEX analysis for discussion. Other conclusions can be defensible if their evidence, assumptions and trade-offs are explicit.
- Preserve the evidence for diagnostic performance while respecting its endpoint. A useful classifier can still have uncertain benefits once referral and treatment capacity are considered.
- Compare errors at an appropriate threshold and prevalence. Similar sensitivity alongside lower specificity can change the burden on later assessment; the final service design matters.
- Recommend an evaluation of the intended pathway with clinical and operational expertise. Record patient-level outcomes and who is excluded, rather than translate a laboratory metric directly into a rollout decision.
Read the evidence / Reviewed 18 October 2026
Sources & scope.
AIPEX’s prompts and worked approaches are our own synthesis. Use the linked material to examine the method and its limits.
- Ting et al. / JAMA
Development and Validation of a Deep Learning System for Diabetic Retinopathy (2017) ↗
Abstract; Table 4 (primary validation); Limitations; Conclusions; doi:10.1001/jama.2017.18152
Primary diagnostic-validation study using Singapore and external datasets. Retinal images, eyes and patients have different denominators. This is not a randomised trial of patient outcomes or evidence about every later SELENA/SELENA+ product version. Several authors disclosed a related patent interest.
- Mitchell et al.
Model Cards for Model Reporting (2019) ↗
Intended use, evaluation data, metrics and subgroup performance
A reporting proposal, useful for asking what an evaluation covers and leaves out. A model card is not independent validation.
- UK Government
Guidance on the impact evaluation of AI interventions ↗
Evaluating AI interventions in their operational setting
Helps frame evaluation of the intervention and its workflow, alongside technical model testing.