Machine Learning for Multimodal Medical Image Reconstruction – MultiRecon

MultiRecon is centred around generative models, a class of machine learning techniques capable of learning how real medical images “should” look. Once trained, they can generate clean, plausible images even from noisy or incomplete input.

Three types of models were studied:

• Variational autoencoders (VAEs): These compress images into a shared latent space and decode them back. They are ideal for joint reconstruction, using a single code to represent several images, allowing PET and CT scans, for instance, to “communicate” and reinforce each other.

• Generative adversarial networks (GANs): GANs learn to generate realistic images through a competition between a generator and a discriminator. While good at realism, their unstable training and non-smooth latent space limit their use for reconstruction.

• Diffusion models (DMs): A popular approach where images are generated by reversing a gradual noise process. These models are highly accurate and particularly effective in low-dose or sparse-view scenarios. To make these models usable for full-body scans, we combined them with wavelet decomposition to reduce memory demands.

To make these models clinically usable, we integrated wavelet decomposition to

reduce memory usage.

We also explored convolutional neural network (CNN)-based denoisers, dictionary learning (DiL) methods, and PnP tools adapted to PET/MRI and spectral CT. In particular, the aim of the work on plug-and-play (PnP) methods was to propose hybrid iterative reconstruction approaches (using AI and a physical model of the acquisition) that are numerically efficient (to handle high-dimensional data), more frugal to take into account small training databases as encountered in PET research protocols, and with convergence guarantees of the iterative scheme for greater robustness and reliability in these hybrid approaches. Learning a regularisation operator adapted to reconstruction by learning from fixed-point conditions has also led to improved reconstruction

performance.

All techniques were designed for simultaneous multichannel reconstruction, using a shared latent representation enabling the model to let multiple modalities “talk to each other”.

MultiRecon successfully demonstrated that AI-based generative models and convergent PnP methods can reconstruct high-quality medical images from significantly reduced scan data, across several imaging modalities. The project focused on PET/CT, PET/MRI, and spectral CT, with an emphasis on simultaneous, multimodal image reconstruction.

Key outcomes include:

• Noise reduction: Both VAEs and DMs delivered substantial improvements in image quality in low-dose or sparse-view conditions. In particular, DMs outperformed standard techniques in preserving structural detail while reducing noise.

• Multimodal synergy: The simultaneous reconstruction of paired modalities—such as PET with CT or MRI—led to improved image quality compared to reconstructing each modality independently. For example, anatomical data helped guide the reconstruction of more accurate functional PET images.

• Cross-modality communication: The use of a shared latent representation enabled different modalities to “talk to each other”, allowing complementary information to be leveraged during reconstruction.

• Robustness to varying conditions: Generative models proved adaptable to different noise levels and scanner types without the need for retraining, which is a major limitation of traditional deep learning methods.

• Scalability to three-dimensional (3-D) reconstruction: While early work was limited to two-dimensional (2-D) slices, optimisations enabled full 3-D reconstruction during the final project phase, making the techniques more suitable for clinical use.

• Integration into the CASToR open-source reconstruction platform: an ADMM PnP method using deep neural networks has been implemented and will be available to the community in a forthcoming code release.

Although parametric imaging and motion-compensated reconstruction were not directly addressed, the handling of low-dose, multimodal data pave the way for these applications. In particular, the ability of generative models to reconstruct high-quality images from sparse or noisy inputs is a crucial first step toward enabling these tasks.

Quantitative results showed noise decrease and qualitative assessments confirmed that the reconstructed images were sharper, cleaner, and more clinically interpretable than those produced by conventional reconstruction or AI-based post-processing.

Together, these findings validate the central hypothesis of MultiRecon: that simultaneous, AI-powered reconstruction across modalities can improve both image quality and patient safety.

The most distinctive contribution of MultiRecon is its use of joint generative models and learned constrained regularization operator for convergent PnP reconstruction schemes that enable multidimensional images to be reconstructed simultaneously. Instead of treating each scan individually, our approach allows different imaging modalities to be represented within a shared latent space in order to let the images “talk to each other” and reinforce each other’s strengths.

This simultaneous reconstruction strategy leads to:

• Improved robustness: When one modality is noisy or incomplete, the shared representation helps to stabilise and enhance the final image.

• Dose and time reduction: Fusing data across channels means high-quality results can be achieved with less radiation exposure and shorter scans.

• Generalisability and scalability: The techniques developed are adaptable across scanner types and noise levels, making them suitable for widespread clinical deployment.

While MultiRecon did not directly address motion compensation or parametric imaging during this project, our research provide the basis for these future applications. In particular:

• Parametric imaging, which aims to reconstruct dynamic physiological parameters from short PET acquisitions, could benefit from generative models or learned PnP regularising operators trained directly on kinetic images.

• Motion-compensated imaging, such as for respiratory and cardiac PET, could leverage the ability of generative models to reconstruct useful images from noisy or gated data, which is currently a bottleneck in clinical imaging.

Through this project, we demonstrated that image quality no longer has to

be limited by conventional trade-offs between dose, resolution, or speed.

Multimodal medical imaging (e.g., PET/CT, PET/MRI) plays an important role for diagnosis and research. Multimodal machine learning (MML) aims at develop methods that can process information from multimodal imaging, thus allowing “to learn” the dependencies between the modalities. Image reconstruction with MML can take advantage of the dependencies between the images and reduce the image noise, thus allowing for patient dose reduction while preserving image quality. However, conventional dictionary learning techniques are memory consuming and cannot be applied to multimodal image reconstruction. The objective of the project MultiRecon are: (i) to develop new mathematical techniques for less memory-consuming convolutional dictionary learning (CDL) techniques for multimodal image joint reconstruction in PET/CT and PET/MRI, (ii) to further extend these methodologies for dynamic imaging (motion estimation/compensation and kinetics), and (iii) to disseminate our research by implementing the developed methodologies on the open-source reconstruction platform CASToR.