Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination

Eye volume 37, pages 3694–3695 (2023)Cite this article

Subjects

OpenAI recently created the large language model ChatGPT [1], which utilises autoregressive language model GPT-3.5 and is trained on Internet data to possess deductive capabilities [1]. In March 2023, OpenAI introduced GPT-4, a successor to GPT-3.5 [2].

GPT-4, and sometimes GPT-3.5, achieve passing scores on United States Medical Licensing Examination Step exams [3]. Both pass neurosurgery practice board exams at rates comparable to neurosurgery residents [2]. However, GPT-3 could not consistently and accurately answer most mock ophthalmology exam questions [4]. Hence, we compared the performance of GPT-3.5, GPT-4, and human users on a practice ophthalmology written examination.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388:1233–39. https://doi.org/10.1056/NEJMsr2214184.

    Article  PubMed  Google Scholar 

  2. Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, et al. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. medRxiv. 2023;2:e0000198. https://doi.org/10.1101/2023.03.25.23287743.

  3. Nori H, King N, McKinney SM, Carignan D, Horvitzet E. Capabilities of GPT-4 on medical challenge problems. arXiv. 2023. https://doi.org/10.48550/arXiv.2303.13375.

  4. Fares A. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. medRxiv. 2023:2023.01.22.23284882. https://doi.org/10.1101/2023.01.22.23284882.

  5. American Academy of Ophthalmology. Board prep resources for ophthalmology residents San Francisco, CA: American Academy of Ophthalmology; 2023. Available from: https://www.aao.org/education/board-prep-resources accessed April, 2023.

Download references

Acknowledgements

The authors thank Rohaid Ali, MD of Brown University and Ian Connolly, MD, MS of Massachusetts General Hospital for their contributions to this study’s design.

Funding

John Lin was awarded departmental funding from Brown University for expenses related to this study.

Author information

Authors and Affiliations

  1. The Warren Alpert Medical School, Brown University, Providence, RI, USA

    John C. Lin & Oliver Y. Tang

  2. Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    John C. Lin

  3. Feinberg School of Medicine, Northwestern University, Chicago, IL, USA

    David N. Younessi

  4. Departments of Ophthalmology and Public Health Sciences, Penn State College of Medicine, Hershey, PA, USA

    Sai S. Kurapati & Ingrid U. Scott

  5. Department of Neurosurgery, University of Pittsburgh, Pittsburgh, PA, USA

    Oliver Y. Tang

Authors
  1. John C. Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David N. Younessi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sai S. Kurapati
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oliver Y. Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ingrid U. Scott
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors were responsible for conceptualization and research design; JCL, DNY, and SSK were involved in data acquisition and research execution; JCL, DNY, and OYT conducted the data analysis; all authors worked on data interpretation and manuscript preparation.

Corresponding author

Correspondence to Ingrid U. Scott.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, J.C., Younessi, D.N., Kurapati, S.S. et al. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye 37, 3694–3695 (2023). https://doi.org/10.1038/s41433-023-02564-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41433-023-02564-2

This article is cited by

Search

Advanced search

Quick links