by Summary: Deep learning text-to-image generation models like OpenAI’s DALL-E 2 may be a promising new tool for image augmentation, generation and manipulation in healthcare.
Source: JMIR Publications
A new publication in the Journal of Medical Internet Research describes how generative models such as DALL-E 2, a novel deep learning model for text-to-image generation, could represent a promising future tool for image generation, augmentation and manipulation in healthcare.
Do generative models have sufficient medical expertise to provide accurate and useful results? dr Lisa C. Adams and colleagues explore this issue in their latest viewpoint, entitled “What does DALL-E 2 know about radiology?”
First introduced by OpenAI in April 2022, DALL-E 2 is an artificial intelligence (AI) tool that has gained popularity for generation novel photorealistic images or artwork based on text input. DALL-E 2’s generative capabilities are powerful because it has been trained on billions of existing text-image pairs from the Internet.
To understand whether these capabilities can be transferred to the medical field to create or augment data, researchers from Germany and the United States examined DALL-E 2’s radiological knowledge in the creation and manipulation of X-rays, computed tomography (CT ), magnetic resonance imaging (MRI) and ultrasound images.
The study authors found that DALL-E learned 2 relevant representations from X-ray images and shows promising potential for text-to-image generation. In particular, DALL-E 2 was able to create realistic X-ray images based on short text prompts, but it didn’t work very well when given specific CT, MRI, or ultrasound image prompts. It was also able to adequately reconstruct missing aspects within a radiological image.
It could do a lot more – for example, create a full body X-ray using just an image of the knee as a starting point. However, DALL-E 2 was limited in its ability to generate images with pathological abnormalities.
Synthetic data generated by DALL-E 2 could significantly accelerate the development of new deep learning tools for radiology and address privacy concerns related to data sharing between institutions. The authors of the study point out that generated images should be subject to quality control by domain experts to reduce the risk of incorrect information entering a generated dataset.
They also emphasize the need for further research to fine-tune these models to medical data and to incorporate medical terminology to create powerful models for data generation and augmentation in radiology research. Although DALL-E 2 is not available to the public for fine-tuning, others like generative models Stable diffusion are that could be adjusted to produce a variety of medical images.
Overall, this viewpoint is published by JMIR Publications offers a promising outlook on the future of AI imaging in radiology. Further research and development in this area could lead to exciting new tools for radiologists and medical professionals.
While there are limitations that need to be addressed, the potential benefits of using tools like DALL-E 2 and ChatGPT in research and medical education and training are significant. To this end, JMIR Medical Education is now Invitation to Submission for a new e-collection on the use of generative language models in medical education, as announced in a recent Editing by Dr. Gunther Eysenbach.
About this AI and DALL-E 2 research news
Author: Ryan James Jessup Jd/MPA
Source: JMIR Publications
Contact: Ryan James Jessup Jd/MPA – JMIR Publications
Picture: Image is credited to Microsoft Designer (based on DALL-E 2); Copyright: The Authors × DALL E 2; License: Creative Commons Attribution (CC-BY)
Original research: Closed access.
“What does DALL-E 2 know about radiology?” by Lisa C. Adams et al. Journal of Medical Internet Research
Abstract
What does DALL-E 2 know about radiology?
Generative models like DALL-E 2 (OpenAI) could represent promising future tools for image generation, augmentation and manipulation for artificial intelligence research in radiology, provided these models have sufficient medical domain knowledge.
Here we show that DALL-E learned 2 relevant representations of X-ray images, with promising abilities in terms of text-to-image generation of new images without shot, continuation of an image beyond its original boundaries, and element removal; however, its capabilities to generate images of pathological abnormalities (e.g., tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited.
The use of generative models for the enrichment and generation of radiological data thus appears feasible, even if further fine-tuning and adaptation of these models to their respective domains is required.
