The UK-based healthcare startup’s announcement that their AI had achieved medical test scores on par with human doctors, including on relevant sections of the MRCGP, was met with concern from doctors over the research behind the study.
According to a press release from Babylon Health, the average pass mark for the MRCGP over the past five years was 72 per cent. On the first try, Babylon’s AI scored 81 per cent.
Babylon Health‘s research paper on artificial intelligence and human doctors for triage and diagnosis goes as far to claim “the triage advice recommended by the AI System was, on average, safer than that of human doctors, when compared to the ranges of acceptable triage provided by independent expert judges, with only a minimal reduction in appropriateness.”
your paper (non peer reviewed), non independent, used doctors to pretend to be patients – not real patients. because they thought real patients might be too confusing. pic.twitter.com/KDZQxWFrI6
— Margaret McCartney (@mgtmccartney) June 28, 2018
The research paper is not peer reviewed and has drawn a significant amount of criticism from those in the healthcare industry.
This is a very polite dissection of the 'paper' that @babylonhealth published on the performance of its AI chatbot. The science is so poor it is almost funny, but note the URL – marketing-assets- this is about making money, and patient safety will suffer June 28, 2018
To further test the AI’s diagnostic abilities, according to the company, it was tested alongside seven primary care doctors using 100 medical vignettes. On this Babylon’s AI scored 80 per cent for accuracy. When tested against conditions seen frequently in primary care, according to the company, the AI had 98 per cent accuracy.
Why are you not immediately publishing methodology and raw data in the public domain for anyone to scrutinise? Has this “paper” been independently peer reviewed? Has it been scrutinised by MHRA regulators? #BabylonAITest
— Dmitri Nepogodiev (@dnepo) June 27, 2018
Healthcare professionals are questioning the methodology of Babylon’s research and how accurate the AI is at actually diagnosing.
Dr Murphy responds to the claim by @babylonhealth that their #AI #Chatbot technology “achieves equivalent accuracy with human doctors”
Please watch this new #Cardiac #ChestPain triage….#eHealth #PatientSafety #FlawedAI @MHRAdevices @mgtmccartney @GPatHand @CareQualityComm pic.twitter.com/zN4rJXuOW9
— Dr Murphy (@DrMurphy11) June 28, 2018
Going forward, Bablyon Health founder and CEO Dr. Ali Parsa said that their mission it to use the AI to relieve the shortage of doctors worldwide.
“Even in the richest nations, primary care is becoming increasingly unaffordable and inconvenient, often with waiting times that make it not readily accessible,” he said in the release. “Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians.”
Time will tell if they get there.
