By Allison Proffitt
August 5, 2019 | Using simulated data to train highly accurate deep convolutional neural networks (CNNs) to classify breast tumors, researchers from the University of Southern California, the Indian Institute of Technology, Hunter College, the University of Texas, Austin, and Rensselaer Polytechnic Institute demonstrate how simulated data may empower machine learning in data-poor fields.
"A lot of these machine learning algorithms are quite data hungry," explains Assad Oberai, head of the computation and data driven discovery group at the Viterbi School at the University of Southern California, and senior author on the paper. Being able to train a highly-accurate CNN with only images generated on a computer model opens up new possibilities in fields that don't have well-calibrated image data to rely on.
Perhaps medical imaging as a field doesn't immediately register as "data-poor", but Oberai argues that it is. "Unlike pictures of dogs and cats on the internet, there aren't a whole lot of images of different types of organs and diseases to go with them," he explained. "One would imagine that there should be a really big database somewhere that we could use to train different machine learning algorithms with different types of diseases, but if you dig in a little big deeper, you find [variation in] how the imaging was done: What plane was the radiologist looking at? How much x-ray dose was given? For ultrasound, what was the center frequency?… Contrary to other applications in machine learning, there is a scarcity of data with which to train the types of machine learning algorithms," he said.
Oberai is part of an NIH grant exploring elastography—a medical imaging modality that maps the elastic properties and stiffness of soft tissue. He and colleagues at Boston University, Mayo Clinic and the University at Wisconsin have been collaborating on studies of elastic heterogeneity and nonlinear elasticity in breast tumors.
Both are physical measurements of tumor stiffness used to distinguish benign and malignant tumors. We generally understand that tumors that are more heterogenous in elasticity measures are more likely to be malignant; masses with nonlinear elasticity—elasticity changes nonlinearly as force increases—are also more likely to be malignant.
But measuring the elasticity of a tumor with only ultrasound or MRI images is a feat. Start with a series of medical images and use image cross-correlation to determine the change in displacements inside the tissue—the difference in tissue displacement from image A to B. Those displacement data along with a good dose of physics are used to solve an inverse problem to infer whether there is heterogeneity within the tumor or whether it is nonlinearly elastic. Using those tumor features, and based on a priori knowledge of breast tumors, we can classify tumors as malignant or benign.
The inverse problem part of the workflow, is uniquely challenging, explains Oberai. Inverse problems are solved with computational algorithms that require an expert user. "It won't be just plug and play: here, give me a displacement and out pops an image of the stiffness," he explains. "It isn't that way… You need to be an expert user of those algorithms to get something out of it. They're not as turnkey as a machine learning algorithm."
When the team first submitted the grant to NIH about six years ago, "We had no inkling of trying to use machine learning for all of this," Oberai said. But along the way, the researchers wondered if you could computationally skip the inverse problem. Could you move from measured displacements straight to diagnosis?
"Considering that deep learning algorithms are good at extracting useful features from data and then using these for a given task, we ask whether it is possible to leverage the capabilities of these data-based modules in the standard physics-based workflow," the authors write in the paper.
They chose convolutional neural networks for the task because CNNs are specifically designed to learn different levels of abstraction in an efficient manner. The goal was to train a CNN to identify the mechanical features of a tumor—and thus arrive at a diagnosis—using only the displacement data as input.
The team created two CNNs, one for each feature—elastic heterogeneity and nonlinear elasticity. For the elastic heterogeneity CNN, the researchers generated a dataset of 8,000 displacement images labeled malignant and benign, adding Gaussian white noise to the images to test robustness. The displacement data and classification were used to train a 5-layer CNN in batches of 1,000 images. To train a CNN to look for nonlinear elasticity, the group used 4,000 displacement images, again adding Gaussian noise, and training the 4-layer network in 500-image batches.
But why start with displacement images? Why not begin with ultrasound or MRI images? "Good question!" Oberai said. Displacement data are physical quantities, which should be consistent regardless of methods or image input, he explained. "The same in not true for ultrasound or any medical images. What we look at is almost never a physical quantity," he said.
Using synthetic displacement images with corresponding benign/malignant classifiers, both networks performed well, achieving 99.7%-99.9% accuracies across all levels of noise in testing. The nonlinear elasticity CNN was then used to evaluate displacement data from real images of breast lesions from ten human subjects. The CNN correctly classified eight of the ten lesions—80% accuracy.
80% accuracy is certainly much lower than the 99% accuracy the network achieved during testing, but Oberai wasn't discouraged.
The differences in accuracies reflect not a weakness in the CNN, but a hole in our understanding of the biology of tumors. A convolutional neural network can only supply answers for the questions it has been trained to answer. In this case, the network learned to assess nonlinear elasticity and then, based on the findings, classify tumors as benign or malignant.
"Our original hypothesis that tumors have to be elastically nonlinear is maybe correct in some settings, and maybe for some tumors that is not the case," Oberai said. "The 99% is telling us that the CNN has learned whatever we wanted it to learn. The 80% is telling us that the world is not as binary as we would like it to be."
The logical next step is to apply the CNN to larger patient groups, and as part of the NIH-funded elastography research, Mayo Clinic is imaging a large cohort of patients. "We are actively analyzing that data," Oberai said.
But he emphasizes that this work is less novel for its diagnostic applications—"AI in image diagnosis: I think we've reached a point where that is pretty routine,"—than for the concept of training a CNN using simulated images and a priori knowledge.
"Let's say you have a given disease for which the radiologist already knows to watch out for something, because it's a pretty good indicator of whether something might be benign or not. If that information is known a priori, how do you train a machine learning algorithm to make use of that information?"
The authors view their approach as a type of data augmentation, where a given set of data is transformed to generate a larger set and then used to train the CNN. Often that means taking a set of existing images and "rotating, translating, cropping or scaling the input images to create many more copies to train the network," the authors write. Generating appropriate images computationally and using those to train the network represents a "physically-consistent and extreme data augmentation," they argue.
"By applying a network that is trained solely using simulated data to a real-world classification problem," the authors write, "we have demonstrated how physics-based modeling can facilitate transfer learning in data-scarce applications."