Health Care and the Promise of Explainable Artificial Intelligence (XAI)


By Tina Wallman,
Sr. Director of Strategic Initiatives, Optum

There has recently been a lot of talk around explainable AI (XAI). How can we open up the ‘black box’ around AI to better understand what decisions it is making, what data it is using to make those decisions and how can we have clinical resources get more confident in AI’s ability to make decisions?

One area where (XAI) holds a lot of promise is in health care. The question becomes, is the promise of (XAI) going to follow a similar hype cycle as Artificial Intelligence (AI)? AI has been around for over 50 years, with at least 3 time periods where it was going to take over the world, in the 1960’s, in the 1990’s, and now. Will it take 5 years, 10 years, or longer for explainable AI (XAI) to be adopted?

Health care has to change and explainable AI (XAI) might just be the push

The areas around pathology, radiology and dermatology have all seen advancements in AI. There have been studies that show great strides, with AI models; where AI is better at detecting melanomas than dermatologist. Just because a model is better at detecting something, doesn’t that mean that it is ready to make decisions autonomously.

These are business and clinical challenges that need to be reviewed and understood on a case by case basis. Not every model that gets created will need additional rigor about how it came to its decision. You need to fully understand a system before you can allow it to make decisions without human involvement if that is the goal.

This is where the breakdown starts to happen. Technical teams love experimenting with different technologies and generally look to fully automate their AI models. Business and clinical owners aren’t looking for full automation of AI models because they won’t blindly trust a model. They don’t have confidence in how the AI model is working and transparency is absolute. The regulatory constraints that healthcare has to operate within, provide an opportunity for explainable AI.

This is where business, clinical and technical stakeholders have to all be on the same page. Business and clinical owners have to be involved in working with their technical resources to define what parts of the organization they are comfortable with AI systems making decisions autonomously. What areas should AI systems be augmented and verified by stakeholders? Do business owners, clinical owners and technical owners understand what the risks are around the AI systems that are being created?

The healthcare ecosystem could be greatly advanced with  (XAI). With many strides that academic institutions like the University of Toronto are making and technology companies like Google continuing to do research around (XAI), how can healthcare leverage these advancements to transform itself? Technology, Business and Clinical owners will all need to start by working in smaller and less complex areas to test their hypothesis. Business and Clinical owners will need to push their technical teams from the beginning to design AI models that can be explained. Business owners will need to start asking more questions around the security, regulatory, and financial risks of the AI models.

The hope of (XAI) is that future models will be able to share the features that it used to make the decision. There are two open source projects 1) Local Interpretable Model-agnostic Explanations (LIME) which looks to help explain what AI models are doing and 2) SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. Both of these projects have repositories on GitHub that can be utilized by technical teams to start creating explainable AI (XAI) models.

Health care has to change and explainable AI (XAI) might just be the push the ecosystem needs to transform itself.