Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A systematic evaluation of human expert agreement on optical coherence tomography biomarkers using multiple devices

Eye volume 37, pages 2573–2579 (2023)Cite this article

Subjects

Abstract

Objectives

To assess the agreement in evaluating optical coherence tomography (OCT) variables in the leading macular diseases such as neovascular age-related macular degeneration (nAMD), diabetic macular oedema (DMO) and retinal vein occlusion (RVO) among OCT-certified graders.

Methods

SD-OCT volume scans of 356 eyes were graded by seven graders. The grading included presence of intra- and subretinal fluid (IRF, SRF), pigment epithelial detachment (PED), epiretinal membrane (ERM), conditions of the vitreomacular interface (VMI), central retinal thickness (CRT) at the foveal centre-point (CP) and central millimetre (CMM), as well as height and location of IRF/SRF/PED. Kappa statistics (κ) and intraclass correlation coefficient (ICC) were used to report categorical grading and measurement agreement.

Results

The overall agreement on the presence of IRF/SRF/PED was κ = 0.82/0.85/0.81; κ of VMI condition was 0.77, that of ERM presence 0.37. ICC for CRT measurements at CP and CMM was excellent with an ICC of 1.00. Height measurements of IRF/SRF/PED showed robust consistency with ICC = 0.85–0.93. There was substantial to almost perfect agreement in locating IRF/SRF/PED with κ = 0.67–0.86. Between diseases, κ of IRF/SRF presence was 0.69/0.80 for nAMD, 0.64/0.83 for DMO and 0.86/0.89 for RVO.

Conclusion

Even in the optimized setting, featuring certified graders, standardized image acquisition and the use of a professional reading platform, there is a disease dependent variability in biomarker evaluation that is most pronounced for IRF in nAMD as well as DMO. Our findings highlight the variability in the performance of human expert OCT grading and the need for AI-based automated feature analyses.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: An ETDRS grid was centred on the foveal centre-point (CP) by the reading centre for comparability of feature localization among all graders.
Fig. 2: Example images of cases leading to disagreement between graders.

Similar content being viewed by others

Data availability

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available.

References

  1. Toth CA, Decroos FC, Ying GS, Stinnett SS, Heydary CS, Burns R, et al. Identification of Fluid on Optical Coherence Tomography by Treating Ophthalmologists Versus a Reading Center in the Comparison of Age-Related Macular Degeneration Treatments Trials. Retina. 2015;35:1303–14.

    Article  PubMed  PubMed Central  Google Scholar 

  2. DeCroos FC, Toth CA, Stinnett SS, Heydary CS, Burns R, Jaffe GJ, et al. Optical coherence tomography grading reproducibility during the Comparison of Age-related Macular Degeneration Treatments Trials. Ophthalmology. 2012;119:2549–57.

    Article  PubMed  Google Scholar 

  3. Mitchell P, Bandello F, Schmidt-Erfurth U, Lang GE, Massin P, Schlingemann RO, et al. The RESTORE study: ranibizumab monotherapy or combined with laser versus laser monotherapy for diabetic macular edema. Ophthalmology. 2011;118:615–25.

    Article  PubMed  Google Scholar 

  4. Nguyen QD, Brown DM, Marcus DM, Boyer DS, Patel S, Feiner L, et al. Ranibizumab for diabetic macular edema: results from 2 phase III randomized trials: RISE and RIDE. Ophthalmology. 2012;119:789–801.

    Article  PubMed  Google Scholar 

  5. Guymer RH, Markey CM, McAllister IL, Gillies MC, Hunyor AP, Arnold JJ, et al. Tolerating Subretinal Fluid in Neovascular Age-Related Macular Degeneration Treated with Ranibizumab Using a Treat-and-Extend Regimen: FLUID Study 24-Month Results. Ophthalmology. 2019;126:723–34.

    Article  PubMed  Google Scholar 

  6. Reiter GS, Grechenig C, Vogl WD, Guymer RH, Arnold JJ, Bogunovic H, et al. Analysis of fluid volume and its impact on visual acuity in the FLUID study as quantified with deep learning. Retina. 2021;41:1318–28.

    CAS  PubMed  Google Scholar 

  7. Keenan TDL, Clemons TE, Domalpally A, Elman MJ, Havilio M, Agron E, et al. Retinal Specialist versus Artificial Intelligence Detection of Retinal Fluid from OCT: Age-Related Eye Disease Study 2: 10-Year Follow-On Study. Ophthalmology. 2021;128:100–9.

    Article  PubMed  Google Scholar 

  8. De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24:1342–50.

    Article  PubMed  Google Scholar 

  9. Muller PL, Liefers B, Treis T, Rodrigues FG, Olvera-Barrios A, Paul B, et al. Reliability of Retinal Pathology Quantification in Age-Related Macular Degeneration: Implications for Clinical Trials and Machine Learning Applications. Transl Vis Sci Technol. 2021;10:4.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Folgar FA, Jaffe GJ, Ying GS, Maguire MG, Toth CA, Comparison of Age-Related Macular Degeneration Treatments Trials Research G. Comparison of optical coherence tomography assessments in the comparison of age-related macular degeneration treatments trials. Ophthalmology. 2014;121:1956–65.

    Article  PubMed  Google Scholar 

  11. Joeres S, Tsong JW, Updike PG, Collins AT, Dustin L, Walsh AC, et al. Reproducibility of quantitative optical coherence tomography subanalysis in neovascular age-related macular degeneration. Invest Ophthalmol Vis Sci. 2007;48:4300–7.

    Article  PubMed  Google Scholar 

  12. Ritter M, Elledge J, Simader C, Deak GG, Benesch T, Blodi BA, et al. Evaluation of optical coherence tomography findings in age-related macular degeneration: a reproducibility study of two independent reading centres. Br J Ophthalmol. 2011;95:381–5.

    Article  PubMed  Google Scholar 

  13. Zhang N, Hoffmeyer GC, Young ES, Burns RE, Winter KP, Stinnett SS, et al. Optical coherence tomography reader agreement in neovascular age-related macular degeneration. Am J Ophthalmol. 2007;144:37–44.

    Article  PubMed  Google Scholar 

  14. Sayegh RG, Simader C, Scheschy U, Montuoro A, Kiss C, Sacu S, et al. A systematic comparison of spectral-domain optical coherence tomography and fundus autofluorescence in patients with geographic atrophy. Ophthalmology. 2011;118:1844–51.

    Article  PubMed  Google Scholar 

  15. Sala-Puigdollers A, Figueras-Roca M, Hereu M, Hernandez T, Morato M, Adan A, et al. Repeatability and reproducibility of retinal and choroidal thickness measurements in Diabetic Macular Edema using Swept-source Optical Coherence Tomography. PLoS One. 2018;13:e0200819.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Glassman AR, Beck RW, Browning DJ, Danis RP, Kollman C. Diabetic Retinopathy Clinical Research Network Study G. Comparison of optical coherence tomography in diabetic macular edema, with and without reading center manual grading from a clinical trials perspective. Invest Ophthalmol Vis Sci. 2009;50:560–6.

    Article  PubMed  Google Scholar 

  17. Munk MR, Lincke J, Giannakaki-Zimmermann H, Ebneter A, Wolf S, Zinkernagel MS. Comparison of 55 degrees Wide-Field Spectral Domain Optical Coherence Tomography and Conventional 30 degrees Optical Coherence Tomography for the Assessment of Diabetic Macular Edema. Ophthalmologica. 2017;237:145–52.

    Article  PubMed  Google Scholar 

  18. Hatef E, Khwaja A, Rentiya Z, Ibrahim M, Shulman M, Turkcuoglu P, et al. Comparison of time domain and spectral domain optical coherence tomography in measurement of macular thickness in macular edema secondary to diabetic retinopathy and retinal vein occlusion. J Ophthalmol. 2012;2012:354783.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Domalpally A, Blodi BA, Scott IU, Ip MS, Oden NL, Lauer AK, et al. The Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) study system for evaluation of optical coherence tomograms: SCORE study report 4. Arch Ophthalmol. 2009;127:1461–7.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Decroos FC, Stinnett SS, Heydary CS, Burns RE, Jaffe GJ. Reading Center Characterization of Central Retinal Vein Occlusion Using Optical Coherence Tomography During the COPERNICUS Trial. Transl Vis Sci Technol. 2013;2:7.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Schmidt-Erfurth U, Reiter GS, Riedl S, Seebock P, Vogl WD, Blodi BA, et al. AI-based monitoring of retinal fluid in disease activity and under therapy. Prog Retin Eye Res. 2021;86:100972

  22. Jill Hopkins J, Keane PA, Balaskas K. Delivering personalized medicine in retinal care: from artificial intelligence algorithms to clinical application. Curr Opin Ophthalmol. 2020;31:329–36.

    Article  CAS  PubMed  Google Scholar 

  23. Simader C, Montuoro A, Waldstein S, Gerendas B, Lammer J, Heiling U, et al. Retinal Thickness Measurements with Spectral Domain Optical Coherence Devices from Different Manufacturers in a Reading Center Environment. Investigative Ophthalmol Vis Sci. 2012;53:4067.

    Google Scholar 

  24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  CAS  PubMed  Google Scholar 

  25. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15:155–63.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Michl M, Liu X, Kaider A, Sadeghipour A, Gerendas BS, Schmidt-Erfurth U. The impact of structural optical coherence tomography changes on visual function in retinal vein occlusion. Acta Ophthalmol. 2021;99:418–26.

    Article  PubMed  Google Scholar 

  27. Liakopoulos S, Ongchin S, Bansal A, Msutta S, Walsh AC, Updike PG, et al. Quantitative Optical Coherence Tomography Findings in Various Subtypes of Neovascular Age-Related Macular Degeneration. Investigative Ophthalmol Vis Sci. 2008;49:5048–54.

    Article  Google Scholar 

  28. Heng LZ, Pefkianaki M, Hykin P, Patel PJ. Interobserver agreement in detecting spectral-domain optical coherence tomography features of diabetic macular edema. PLoS One. 2015;10:e0126557.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Bressler NM, Odia I, Maguire M, Glassman AR, Jampol LM, MacCumber MW, et al. Association Between Change in Visual Acuity and Change in Central Subfield Thickness During Treatment of Diabetic Macular Edema in Participants Randomized to Aflibercept, Bevacizumab, or Ranibizumab: A Post Hoc Analysis of the Protocol T Randomized Clinical Trial. JAMA Ophthalmol. 2019;137:977–85.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Jaffe GJ, Martin DF, Toth CA, Daniel E, Maguire MG, Ying GS, et al. Macular morphology and visual acuity in the comparison of age-related macular degeneration treatments trials. Ophthalmology. 2013;120:1860–70.

    Article  PubMed  Google Scholar 

  31. Deák GG, Schmidt-Erfurth UM, Jampol LM. Correlation of Central Retinal Thickness and Visual Acuity in Diabetic Macular Edema. JAMA Ophthalmol. 2018;136:1215–6.

    Article  PubMed  Google Scholar 

  32. Pawloff M, Bogunovic H, Gruber A, Michl M, Riedl S, Schmidt-Erfurth U. Systematic correlation of central subfield thickness with retinal fluid volumes quantified by deep learning in the major exudative macular diseases. Retina. 2022;42:831–41.

    Article  CAS  PubMed  Google Scholar 

  33. Gerendas BS, Sadeghipour A, Michl M, Goldbach F, Mylonas G, Gruber A, et al. Validation of an Automated Fluid Algorithm on Real-World Data of Neovascular Age-Related Macular Degeneration over Five Years. Retina. 2022;42:1673–82.

    Article  CAS  PubMed  Google Scholar 

  34. Schmidt-Erfurth U, Mulyukov Z, Gerendas BS, Reiter GS, Lorand D, Weissgerber G, et al. Therapeutic response in the HAWK and HARRIER trials using deep learning in retinal fluid volume and compartment analysis. Eye (Lond). (2022). https://doi.org/10.1038/s41433-022-02077-4. Online ahead of print.

Download references

Acknowledgements

We thank the graders from the Vienna Reading Center for their valuable contribution.

Author information

Author notes

  1. These authors contributed equally: Martin Michl, Martina Neschi.

Authors and Affiliations

  1. Department of Ophthalmology, Vienna Reading Center, Medical University of Vienna, Vienna, Austria

    Martin Michl, Katja Hatz, Gabor Deak, Bianca S. Gerendas & Ursula Schmidt-Erfurth

  2. RetInSight, Vienna, Austria

    Martina Neschi

  3. Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria

    Alexandra Kaider

  4. Faculty of Medicine, University of Basel, Basel, Switzerland

    Katja Hatz

  5. Vista Augenklinik Binningen, Binningen, Switzerland

    Katja Hatz

Authors
  1. Martin Michl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martina Neschi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandra Kaider
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katja Hatz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gabor Deak
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bianca S. Gerendas
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ursula Schmidt-Erfurth
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors confirm contribution to the paper as follows: study conception and design: MN, GD, BSG, USE; data collection: MN, KH; analysis and interpretation of results: MM, AK, GD, BSG, USE; draft manuscript preparation: MM, BSG, USE; All authors reviewed the results and approved the final version of the manuscript.

Corresponding author

Correspondence to Bianca S. Gerendas.

Ethics declarations

Competing interests

BSG: Bayer, Zeiss, Novartis (C), IDx (F); USE: Apellis, Novartis, Kodiak, Roche, Boehringer, RetInSight (C). Other authors declare no competing interests.

Ethics approval

Approved by the Ethics Committee of the Medical University of Vienna, #1246/2016.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Michl, M., Neschi, M., Kaider, A. et al. A systematic evaluation of human expert agreement on optical coherence tomography biomarkers using multiple devices. Eye 37, 2573–2579 (2023). https://doi.org/10.1038/s41433-022-02376-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41433-022-02376-w

Search

Advanced search

Quick links