The covid-19 pandemic is stretching hospital resources to the breaking point in many countries in the world. It is no surprise that many people hope AI could speed up patient screening and ease the strain on clinical staff. But a study from Google Health—the first to look at the impact of a deep-learning tool in real clinical settings—reveals that even the most accurate AIs can actually make things worse if not tailored to the clinical environments in which they will work.
Existing rules for deploying AI in clinical settings, such as the standards for FDA clearance in the US or a CE mark in Europe, focus primarily on accuracy. There are no explicit requirements that an AI must improve the outcome for patients, largely because such trials have not yet run. But that needs to change, says Emma Beede, a UX researcher at Google Health: “We have to understand how AI tools are going to work for people in context—especially in health care—before they’re widely deployed.”
Google’s first opportunity to test the tool in a real setting came from Thailand. The country’s ministry of health has set an annual goal to screen 60% of people with diabetes for diabetic retinopathy, which can cause blindness if not caught early. But with around 4.5 million patients to only 200 retinal specialists—roughly double the ratio in the US—clinics are struggling to meet the target. Google has CE mark clearance, which covers Thailand, but it is still waiting for FDA approval. So to see if AI could help, Beede and her colleagues outfitted 11 clinics across the country with a deep-learning system trained to spot signs of eye disease in patients with diabetes.
In the system Thailand had been using, nurses take photos of patients’ eyes during check-ups and send them off to be looked at by a specialist elsewhere—a process that can take up to 10 weeks. The AI developed by Google Health can identify signs of diabetic retinopathy from an eye scan with more than 90% accuracy—which the team calls “human specialist level”—and, in principle, give a result in less than 10 minutes. The system analyzes images for telltale indicators of the condition, such as blocked or leaking blood vessels.
Sounds impressive. But an accuracy assessment from a lab goes only so far. It says nothing of how the AI will perform in the chaos of a real-world environment, and this is what the Google Health team wanted to find out. Over several months they observed nurses conducting eye scans and interviewed them about their experiences using the new system. The feedback wasn’t entirely positive.
When it worked well, the AI did speed things up. But it sometimes failed to give a result at all. Like most image recognition systems, the deep-learning model had been trained on high-quality scans; to ensure accuracy, it was designed to reject images that fell below a certain threshold of quality. With nurses scanning dozens of patients an hour and often taking the photos in poor lighting conditions, more than a fifth of the images were rejected.
Patients whose images were kicked out of the system were told they would have to visit a specialist at another clinic on another day. If they found it hard to take time off work or did not have a car, this was obviously inconvenient. Nurses felt frustrated, especially when they believed the rejected scans showed no signs of disease and the follow-up appointments were unnecessary. They sometimes wasted time trying to retake or edit an image that the AI had rejected.
Because the system had to upload images to the cloud for processing, poor internet connections in several clinics also caused delays. “Patients like the instant results, but the internet is slow and patients then complain,” said one nurse. “They’ve been waiting here since 6 a.m., and for the first two hours we could only screen 10 patients.”
The Google Health team is now working with local medical staff to design new workflows. For example, nurses could be trained to use their own judgment in borderline cases. The model itself could also be tweaked to handle imperfect images better.
Risking a backlash
“This is a crucial study for anybody interested in getting their hands dirty and actually implementing AI solutions in real-world settings,” says Hamid Tizhoosh at the University of Waterloo in Canada, who works on AI for medical imaging. Tizhoosh is very critical of what he sees as a rush to announce new AI tools in response to covid-19. In some cases tools are developed and models released by teams with no health-care expertise, he says. He sees the Google study as a timely reminder that establishing accuracy in a lab is just the first step.
Michael Abramoff, an eye doctor and computer scientist at the University of Iowa Hospitals and Clinics, has been developing an AI for diagnosing retinal disease for several years and is CEO of a spinoff startup called IDx Technologies, which has collaborated with IBM Watson. Abramoff has been a cheerleader for health-care AI in the past, but he also cautions against a rush, warning of a backlash if people have bad experiences with AI. “I’m so glad that Google shows they’re willing to look into the actual workflow in clinics,” he says. “There is much more to health care than algorithms.”
Abramoff also questions the usefulness of comparing AI tools with human specialists when it comes to accuracy. Of course, we don’t want an AI to make a bad call. But human doctors disagree all the time, he says—and that’s fine. An AI system needs to fit into a process where sources of uncertainty are discussed rather than simply rejected.
Get it right and the benefits could be huge. When it worked well, Beede and her colleagues saw how the AI made people who were good at their jobs even better. “There was one nurse that screened 1,000 patients on her own, and with this tool she’s unstoppable,” she says. “The patients didn’t really care that it was an AI rather than a human reading their images. They cared more about what their experience was going to be.”
Original post: https://www.technologyreview.com/2020/04/27/1000658/google-medical-ai-accurate-lab-real-life-clinic-covid-diabetes-retina-disease/
I’m the owner of JustCBD Store label (justcbdstore.com) and am seeking to expand my wholesale side of company. I really hope that someone at targetdomain is able to provide some guidance ! I thought that the very best way to accomplish this would be to connect to vape shops and cbd retail stores. I was hoping if anyone could suggest a dependable site where I can get Vape Shop B2B Data List I am already reviewing creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not exactly sure which one would be the best option and would appreciate any guidance on this. Or would it be easier for me to scrape my own leads? Suggestions?
I’m the co-founder of JustCBD Store company (justcbdstore.com) and I am currently planning to expand my wholesale side of business. I am hoping someone at targetdomain give me some advice ! I thought that the most suitable way to do this would be to reach out to vape stores and cbd retailers. I was hoping if someone could suggest a trusted site where I can purchase CBD Shops Business Data I am currently looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. On the fence which one would be the most ideal selection and would appreciate any guidance on this. Or would it be simpler for me to scrape my own leads? Suggestions?
Hello there! This blog post couldn’t be written any better! Looking at this article reminds me of my previous roommate! He continually kept preaching about this. I’ll forward this information to him. Pretty sure he will have a great read. Many thanks for sharing!
Spot on with this write-up, I honestly believe this site needs a lot more attention. I’ll probably be back again to see more, thanks for the info!
When I initially commented I appear to have clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I recieve 4 emails with the exact same comment. Perhaps there is a means you are able to remove me from that service? Many thanks!
An outstanding share! I’ve just forwarded this onto a colleague who had been conducting a little research on this. And he in fact bought me dinner due to the fact that I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to discuss this subject here on your website.
I couldn’t resist commenting. Perfectly written!
You are so cool! I don’t think I have read through anything like that before. So wonderful to find someone with unique thoughts on this subject. Really.. thank you for starting this up. This website is something that’s needed on the internet, someone with a little originality!
This is the perfect website for anyone who hopes to understand this topic. You understand a whole lot its almost hard to argue with you (not that I actually will need to…HaHa). You certainly put a new spin on a topic which has been written about for ages. Wonderful stuff, just excellent!
I wanted to thank you for this very good read!! I certainly loved every little bit of it. I have you bookmarked to look at new stuff you post…
I’m very pleased to discover this website. I wanted to thank you for your time for this particularly wonderful read!! I definitely savored every bit of it and i also have you bookmarked to check out new information in your web site.
I’m amazed, I have to admit. Seldom do I encounter a blog that’s both educative and amusing, and let me tell you, you have hit the nail on the head. The issue is something too few folks are speaking intelligently about. Now i’m very happy that I came across this during my hunt for something concerning this.
Hello there! I could have sworn I’ve been to this website before but after going through many of the posts I realized it’s new to me. Nonetheless, I’m definitely delighted I found it and I’ll be bookmarking it and checking back regularly!
Right here is the perfect web site for anybody who would like to understand this topic. You understand so much its almost hard to argue with you (not that I actually will need to…HaHa). You certainly put a fresh spin on a subject that has been written about for many years. Wonderful stuff, just great!
May I simply say what a comfort to discover somebody who really knows what they are talking about on the net. You definitely know how to bring an issue to light and make it important. More people need to check this out and understand this side of your story. I was surprised that you’re not more popular because you definitely have the gift.
Saved as a favorite, I like your blog!
Achieving your fitness goals does not have to require a certified personal trainer or an expensive gym membership, especially if you have the budget and the space to consider practically every workout machine on the market.
I’m extremely pleased to discover this website. I wanted to thank you for ones time for this wonderful read!! I definitely liked every part of it and i also have you saved to fav to look at new stuff on your website.
I really love your site.. Very nice colors & theme. Did you build this amazing site yourself? Please reply back as I’m hoping to create my very own site and would like to find out where you got this from or just what the theme is called. Kudos!
This is a topic that’s near to my heart… Thank you! Exactly where are your contact details though?
Great blog you’ve got here.. It’s difficult to find excellent writing like yours nowadays. I honestly appreciate individuals like you! Take care!!
This site was… how do I say it? Relevant!! Finally I’ve found something that helped me. Appreciate it!
I’m amazed, I have to admit. Seldom do I encounter a blog that’s both educative and amusing, and without a doubt, you’ve hit the nail on the head. The problem is something that too few folks are speaking intelligently about. I am very happy I came across this in my hunt for something concerning this.
I blog quite often and I really thank you for your content. The article has truly peaked my interest. I will book mark your website and keep checking for new details about once a week. I opted in for your Feed too.
Nice post. I learn something new and challenging on sites I stumbleupon every day. It’s always helpful to read articles from other authors and use a little something from other web sites.
A fascinating discussion is definitely worth comment. I do think that you need to publish more on this topic, it might not be a taboo subject but usually people do not speak about such subjects. To the next! Many thanks!!
This website was… how do you say it? Relevant!! Finally I have found something which helped me. Thanks a lot!
There’s definately a lot to know about this issue. I like all the points you have made.
After looking over a handful of the blog articles on your web page, I really appreciate your way of writing a blog. I bookmarked it to my bookmark site list and will be checking back soon. Please check out my website too and let me know your opinion.
Great info. Lucky me I found your blog by accident (stumbleupon). I’ve book marked it for later!
I blog often and I seriously appreciate your content. This great article has truly peaked my interest. I am going to book mark your blog and keep checking for new information about once a week. I opted in for your RSS feed too.
Aw, this was a very good post. Taking a few minutes and actual effort to generate a great article… but what can I say… I put things off a lot and don’t seem to get anything done.
Pretty! This was a really wonderful article. Thank you for supplying these details.
There is definately a great deal to know about this subject. I really like all of the points you’ve made.