By Daniela Hernandez
In 2012, the University of Texas MD Anderson Cancer Center launched an ambitious project with International Business Machines Corp.'s Watson program that promised to transform cancer care with the help of artificial intelligence.
Almost five years and more than $62 million later, the sprawling Houston-based public institution has little to show for it, according to a special review conducted by the University of Texas System Audit Office that details a number of stumbles in the progress and management of the project.
The audit doesn't evaluate IBM Watson's scientific capabilities or whether the technology works for this purpose. But instead, the report details the management and technology challenges at MD Anderson that make it hard to integrate artificial-intelligence software into complicated health-care settings.
The Watson-based program isn't in clinical use, according to the audit.
The plan for a pilot focused on leukemia was "suspended mid-project" and the focus shifted to lung cancer to try to speed up progress, according to audit documents. Clinical trial and drug-protocol data in the system are outdated, and the pilot program doesn't work with the hospital's current electronic health records, according to the audit.
Oncology Expert Advisor, or OEA, is the Watson-based intelligent system IBM was building for MD Anderson. Watson, the Jeopardy-playing artificial-intelligence software, learns patterns from data and uses that knowledge to make suggestions. The new system was supposed to read and learn from MD Anderson's electronic medical records, the academic literature, research data and treatment options to improve cancer care at the renowned cancer center.
An IBM spokeswoman said the "pilot was a success, and likely could have been taken forward had MD Anderson chosen to do so." The report noted that the system "is not ready" for testing on humans or for "clinical use" because the data used to run it is "outdated."
Lynda Chin, the former chair of genomic medicine at MD Anderson, approached IBM about using Watson to upgrade cancer care at the hospital, according to the audit. In June 2012, the two organizations signed their first contract for $2.4 million. Since then, the contract has been renewed a dozen times, totaling $39 million in fees paid by MD Anderson, according to the audit.
The lung-cancer system pilot within MD Anderson achieved an accuracy of 90% in its predictions, according to staff member interviews cited in the audit. The figure refers to the system's success rate at suggesting the same treatment plan as MD Anderson physicians on select cases, based on its reading of relevant information in patient charts, said Rob Merkel, general manager for oncology and genomics at IBM Watson Health, an IBM division focused on healthcare.
The system was tested in 2015 and supported "treatment recommendations as well as clinical-trial matching," according to an IBM spokeswoman. The vision was also to make the tool widely available to physicians at partner hospitals outside MD Anderson, according to the audit.
University of Texas Chancellor William McRaven ordered an audit of the project after MD Anderson's audit and compliances offices expressed concerns in May last year about whether the project was delivering on contracted services, such as piloting OEA at partner hospitals and whether project leaders followed appropriate procurement requirements, according to university spokeswoman Karen Adler and the audit.
MD Anderson hired an external firm to conduct a separate audit to evaluate the market for and value of the artificial intelligence system, Ms. Adler said.
Mr. Merkel of IBM Watson Health, in response to the concerns raised in the audit, said the focus was switched from leukemia because the project at the time was in an "evaluation phase." He also added IBM "wasn't responsible" for integrating Watson with MD Anderson's new electronic medical records.
MD Anderson didn't specify who was responsible for the integration.
In addition to the management and technology issues, the audit found the program was never piloted with partner hospitals as stipulated in a contract with PricewaterhouseCoopers, which was also hired to craft a business plan for the system. MD Anderson paid PwC $23 million in total for various services associated with the project.
IBM stopped supporting the technology in September, according to the report. IBM declined to comment on the status of the project. PwC also declined to comment.
In the audit, Dr. Chin cited cybersecurity reviews and "lack of engagement or interest" by partner hospitals as factors preventing testing the technology outside MD Anderson, as stipulated in the PwC contract. A spokeswoman said Dr. Chin wasn't available for an interview.
Medical institutions often struggle to bring all data on to the same platform, said Peter Szolovits, the head of the Clinical Decision-Making Group at the MIT Computer Science and Artificial Intelligence Laboratory.
The way medical information is stored and labeled can differ widely, even between departments at the same institution, he said. For instance, "there's no standard way to record a heart rate, a blood-glucose value or temperature measured at the bedside," he said. If the way data is stored or labeled changes, often the artificial-intelligence software must be retrained, he said.
At MD Anderson, the lung-cancer pilot, for instance, was conducted on the hospital's old electronic records system. It needs to be integrated with the new system before the program can be tested further, the audit said.
John Kelly III, senior vice president of cognitive solutions and IBM Research, the tech company's research-and-development unit, said the Watson system was "all set to go." He added that it would only take "months to get it up and running" if MD Anderson gave IBM "the green light."
MD Anderson hasn't given up on its original intent. In January, MD Anderson sent out a request for proposals for another software system to use artificial intelligence in cancer, according to documents on the hospital's public bids page. Previous vendors can also bid, the hospital said.
IBM declined to say if it had bid on the project, the deadline for which ended last month.
Write to Daniela Hernandez at firstname.lastname@example.org
(END) Dow Jones Newswires
March 08, 2017 05:44 ET (10:44 GMT)Copyright (c) 2017 Dow Jones & Company, Inc.