Tech ChatGPT scores 55% on Japanese national medical exam

[thomas]

Yudai Kaneda, an aspiring doctor, tested the artificial intelligence ChatGPT to see how it would fare on this year's national licensing exam for doctors held in February. To his surprise, it correctly answered 55 per cent of the questions. That was short of a passing grade on Japan's National Examination for Medical Practitioners but is higher than his own score!

The test was conducted with GPT-4, the latest version of chatGPT.

The exam is a national licensing test for doctors, which generally requires test-takers to be at least sixth-year medical students at a university. It is required to have completed a university's medical program to take the exam and Japanese universities offer six-year programs. Exam takers can take the question sheets home after the test. In addition, the health ministry posts the exam's questions on its website after the test day every year. Kaneda manually entered all 400 questions and answer choices from the February exam into ChatGPT. He could do so because a student who is senior to him took the test and brought home the question sheets.

AI ChatGPT chalks up a 55% score on national exam for doctors | The Asahi Shimbun: Breaking News, Japan News and Analysis

 

[thomas]

This is not strictly Japan-related but a harbinger of the things to come.

A recent study found that version 3.5 of ChatGPT answered patients' questions better and with significantly more empathy than human physicians. ChatGPT 4, released on 23 March, can answer medical exam questions properly 90% of the time and is already considerably better than the 22 November version of ChatGPT, which responded about 60% correctly, so things are accelerating quickly in the field of health care.

The study:

Comparing Physician and Chatbot Responses to Patient Questions

This cross-sectional study evaluates the ability of an artificial intelligence chatbot to provide quality and empathetic responses to patient questions.

jamanetwork.com