publications | Ruiming Cao

2024

Parsing Dynamics
Neural space–time model for dynamic multi-shot imaging

Ruiming Cao, Nikita S Divekar, James K Nuñez, and 2 more authors

Nature Methods, 2024

Abs Bib PDF Blog Code Website

Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space-time model (NSTM) that jointly estimates the scene and its motion dynamics. Hence, we can both remove motion artifacts and resolve sample dynamics. We demonstrate NSTM in three computational imaging systems: differential phase contrast microscopy, 3D structured illumination microscopy, and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.
@article{cao2024neural, title = {Neural space--time model for dynamic multi-shot imaging}, author = {Cao, Ruiming and Divekar, Nikita S and Nu{\~n}ez, James K and Upadhyayula, Srigokul and Waller, Laura}, journal = {Nature Methods}, pages = {1--6}, year = {2024}, publisher = {Nature Publishing Group}, doi = {10.1038/s41592-024-02417-0}, }
Novel Signal
Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras

Ruiming Cao^*, Dekel Galor^*, Amit Kohli, and 2 more authors

arXiv preprint arXiv:2404.01298, 2024

Abs Bib PDF

Event cameras capture changes of intensity over time as a stream of ’events’ and generally cannot measure intensity itself; hence, they are only used for imaging dynamic scenes. However, fluctuations due to random photon arrival inevitably trigger noise events, even for static scenes. While previous efforts have been focused on filtering out these undesirable noise events to improve signal quality, we find that, in the photon-noise regime, these noise events are correlated with the static scene intensity. We analyze the noise event generation and model its relationship to illuminance. Based on this understanding, we propose a method, called Noise2Image, to leverage the illuminance-dependent noise characteristics to recover the static parts of a scene, which are otherwise invisible to event cameras. We experimentally collect a dataset of noise events on static scenes to train and validate Noise2Image. Our results show that Noise2Image can robustly recover intensity images solely from noise events, providing a novel approach for capturing static scenes in event cameras, without additional hardware.
@article{cao2024noise2image, title = {Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras}, author = {Cao, Ruiming and Galor, Dekel and Kohli, Amit and Yates, Jacob L and Waller, Laura}, journal = {arXiv preprint arXiv:2404.01298}, year = {2024}, }
Parsing Dynamics
Space-time reconstruction for lensless imaging using implicit neural representations

Tiffany Chien, Ruiming Cao, Fanglin Linda Liu, and 2 more authors

Optics Express, 2024

Abs Bib PDF

Many computational imaging inverse problems are challenged by noise, model mismatch, and other imperfections that decrease reconstruction quality. For data taken sequentially in time, instead of reconstructing each frame independently, space-time algorithms simultaneously reconstruct multiple frames, thereby taking advantage of temporal redundancy through space-time priors. This helps with denoising and provides improved reconstruction quality, but often requires significant computational and memory resources. Designing effective but flexible temporal priors is also challenging. Here, we propose using an implicit neural representation to model dynamics and act as a computationally tractable and flexible space-time prior. We demonstrate this approach on video captured with a lensless imager, DiffuserCam, and show improved reconstruction results and robustness to noise compared to frame-by-frame methods.
@article{chien2024space, title = {Space-time reconstruction for lensless imaging using implicit neural representations}, author = {Chien, Tiffany and Cao, Ruiming and Liu, Fanglin Linda and Kabuli, Leyla A and Waller, Laura}, journal = {Optics Express}, volume = {32}, number = {20}, pages = {35725--35732}, year = {2024}, publisher = {Optica Publishing Group}, }
Medical Imaging
Multi-modal deformable image registration using untrained neural networks

Quang Luong Nhat Nguyen, Ruiming Cao, and Laura Waller

arXiv preprint arXiv:2411.02672, 2024

Abs Bib PDF

Image registration techniques usually assume that the images to be registered are of a certain type (e.g. single- vs. multi-modal, 2D vs. 3D, rigid vs. deformable) and there lacks a general method that can work for data under all conditions. We propose a registration method that utilizes neural networks for image representation. Our method uses untrained networks with limited representation capacity as an implicit prior to guide for a good registration. Unlike previous approaches that are specialized for specific data types, our method handles both rigid and non-rigid, as well as single- and multi-modal registration, without requiring changes to the model or objective function. We have performed a comprehensive evaluation study using a variety of datasets and demonstrated promising performance.
@article{nguyen2024multi, title = {Multi-modal deformable image registration using untrained neural networks}, author = {Nguyen, Quang Luong Nhat and Cao, Ruiming and Waller, Laura}, journal = {arXiv preprint arXiv:2411.02672}, year = {2024}, }
Novel Signal
Fast quantitative phase imaging with an LED array microscope using a perturbative approach

Kuan-Chen Shen, Ruiming Cao, Michael Unser, and 2 more authors

In Unconventional Optical Imaging IV, 2024

Abs Bib Video

Quantitative Phase Imaging is the method of choice to observe unlabelled biological cells, enabling the exploration of their intricate structures and dynamics. Differential Phase Contrast (DPC) is a fast and flexible approach but is limited to brightfield measurements, which limits the achievable resolution. In this study, we revisit DPC as a perturbative approach for high-resolution phase imaging with a minimal number of images. Our methodology combines both a perturbative phase retrieval algorithm and ring-shaped darkfield illumination patterns. The proposed approach leverages the advantages of DPC, such as its minimal image requirement, while addressing its inability to extend to darkfield imaging. By reducing the number of acquired images, our technique offers a rapid solution for examining dynamic systems.
@inproceedings{dong2024fast, title = {Fast quantitative phase imaging with an LED array microscope using a perturbative approach}, author = {Shen, Kuan-Chen and Cao, Ruiming and Unser, Michael and Waller, Laura and Dong, Jonathan}, booktitle = {Unconventional Optical Imaging IV}, pages = {PC129960D}, year = {2024}, organization = {SPIE}, }

2023

Parsing Dynamics
Speckle flow structured illumination microscopy for dynamic super-resolution imaging

Ruiming Cao, Fanglin Linda Liu, Li-Hao Yeh, and 2 more authors

In High-Speed Biomedical Imaging and Spectroscopy VIII, 2023

Best Presentation Award Abs Bib Video

Hitachi High-Tech Best Presentation Award, SPIE Photonics West High-Speed Biomedical Imaging and Spectroscopy conference

We propose a new method, Speckle Flow SIM, to super-resolve every frame of a dynamic scene from a sequence of diffraction-limited images at different acquisition timepoints. Speckle Flow SIM uses a fixed speckle-structured illumination to encode the super-resolved information into measurements and a neural space-time model to exploit the temporal redundancy for the dynamic scene reconstruction. We built a simple, inexpensive setup for Speckle Flow SIM and experimentally demonstrated 1.88× super-resolution for a dynamic scene.
@inproceedings{cao2023speckle_pw, title = {Speckle flow structured illumination microscopy for dynamic super-resolution imaging}, author = {Cao, Ruiming and Liu, Fanglin Linda and Yeh, Li-Hao and Meng, Guanghan and Waller, Laura}, booktitle = {High-Speed Biomedical Imaging and Spectroscopy VIII}, volume = {12390}, pages = {1239002}, year = {2023}, organization = {SPIE}, }
Parsing Dynamics
Speckle Structured Illumination of Dynamic Samples with a Neural Space-time Model

Ruiming Cao, Guanghan Meng, and Laura Waller

In Computational Optical Sensing and Imaging, 2023

Best Student Paper Award Abs Bib PDF Video

Best Student Paper Award at Computational Optical Sensing and Imaging conference

We propose a structured illumination microscopy (SIM) method, Speckle Flow SIM, that uses static speckle illumination with a dynamic sample and reconstructs super-resolved images via a neural space-time model, without sacrificing temporal resolution.
@inproceedings{cao2023speckle_cosi, title = {Speckle Structured Illumination of Dynamic Samples with a Neural Space-time Model}, author = {Cao, Ruiming and Meng, Guanghan and Waller, Laura}, booktitle = {Computational Optical Sensing and Imaging}, pages = {CW4D--2}, year = {2023}, organization = {Optica Publishing Group}, }

2022

Medical Imaging

Systems, methods and media for automatically segmenting and diagnosing prostate lesions using multi-parametric magnetic resonance imaging data

Kyung Hyun Sung, and Ruiming Cao

May 2022

US Patent 11,340,324

Bib

@misc{sung2022systems,
  title = {Systems, methods and media for automatically segmenting and diagnosing prostate lesions using multi-parametric magnetic resonance imaging data},
  author = {Sung, Kyung Hyun and Cao, Ruiming},
  year = {2022},
  month = may,
  note = {US Patent 11,340,324},
  url = {https://patents.google.com/patent/US11340324B2/en},
}

Novel Signal
Self-calibrated 3D differential phase contrast microscopy with optimized illumination

Ruiming Cao, Michael Kellman, David Ren, and 2 more authors

Biomedical Optics Express, May 2022

Abs Bib PDF

3D phase imaging recovers an object’s volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope’s focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.
@article{cao2022self, title = {Self-calibrated 3D differential phase contrast microscopy with optimized illumination}, author = {Cao, Ruiming and Kellman, Michael and Ren, David and Eckert, Regina and Waller, Laura}, journal = {Biomedical Optics Express}, volume = {13}, number = {3}, pages = {1671--1684}, year = {2022}, publisher = {Optica Publishing Group}, doi = {10.1364/BOE.450838} }
Parsing Dynamics
Dynamic Structured Illumination Microscopy with a Neural Space-time Model

Ruiming Cao, Fanglin Linda Liu, Li-Hao Yeh, and 1 more author

In IEEE International Conference on Computational Photography (ICCP), May 2022

Abs Bib PDF Video Code

Structured illumination microscopy (SIM) reconstructs a super-resolved image from multiple raw images captured with different illumination patterns; hence, acquisition speed is limited, making it unsuitable for dynamic scenes. We propose a new method, Speckle Flow SIM, that uses static patterned illumination with moving samples and models the sample motion during data capture in order to reconstruct the dynamic scene with super-resolution. Speckle Flow SIM relies on sample motion to capture a sequence of raw images. The spatio-temporal relationship of the dynamic scene is modeled using a neural space-time model with coordinate-based multi-layer perceptrons (MLPs), and the motion dynamics and the super-resolved scene are jointly recovered. We validate Speckle Flow SIM for coherent imaging in simulation and build a simple, inexpensive experimental setup with off-the-shelf components. We demonstrate that Speckle Flow SIM can reconstruct a dynamic scene with deformable motion and 1.88x the diffraction-limited resolution in experiment.
@inproceedings{cao2022dynamic, title = {Dynamic Structured Illumination Microscopy with a Neural Space-time Model}, author = {Cao, Ruiming and Liu, Fanglin Linda and Yeh, Li-Hao and Waller, Laura}, booktitle = {IEEE International Conference on Computational Photography (ICCP)}, year = {2022}, organization = {IEEE}, doi = {10.1109/ICCP54855.2022.9887701} }

2021

Medical Imaging

Performance of Deep Learning and Genitourinary Radiologists in Detection of Prostate Cancer Using 3-T Multiparametric Magnetic Resonance Imaging

Ruiming Cao, Xinran Zhong, Sohrab Afshari, and 7 more authors

Journal of Magnetic Resonance Imaging, May 2021

Bib PDF

@article{cao2021performance,
  title = {Performance of Deep Learning and Genitourinary Radiologists in Detection of Prostate Cancer Using 3-T Multiparametric Magnetic Resonance Imaging},
  author = {Cao, Ruiming and Zhong, Xinran and Afshari, Sohrab and Felker, Ely and Suvannarerg, Voraparee and Tubtawee, Teeravut and Vangala, Sitaram and Scalzo, Fabien and Raman, Steven and Sung, Kyunghyun},
  journal = {Journal of Magnetic Resonance Imaging},
  volume = {54},
  number = {2},
  pages = {474--483},
  year = {2021},
  publisher = {John Wiley \& Sons, Inc. Hoboken, USA},
  doi = {10.1002/jmri.27595},
}

2020

Interpretability

Mining interpretable AOG representations from convolutional networks via active question answering

Quanshi Zhang, Jie Ren, Ge Huang, and 3 more authors

IEEE transactions on pattern analysis and machine intelligence, May 2020

Bib

@article{zhang2020mining,
  title = {Mining interpretable AOG representations from convolutional networks via active question answering},
  author = {Zhang, Quanshi and Ren, Jie and Huang, Ge and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  journal = {IEEE transactions on pattern analysis and machine intelligence},
  volume = {43},
  number = {11},
  pages = {3949--3963},
  year = {2020},
  publisher = {IEEE},
}

Interpretability

Extraction of an Explanatory Graph to Interpret a CNN

Quanshi Zhang, Xin Wang, Ruiming Cao, and 3 more authors

IEEE transactions on pattern analysis and machine intelligence, May 2020

Bib

@article{zhang2020extraction,
  title = {Extraction of an Explanatory Graph to Interpret a CNN},
  author = {Zhang, Quanshi and Wang, Xin and Cao, Ruiming and Wu, Ying Nian and Shi, Feng and Zhu, Song-Chun},
  journal = {IEEE transactions on pattern analysis and machine intelligence},
  volume = {43},
  number = {11},
  pages = {3863--3877},
  year = {2020},
  publisher = {IEEE},
}

2019

Medical Imaging

Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI

Xinran Zhong, Ruiming Cao, Sepideh Shakeri, and 6 more authors

Abdominal Radiology, May 2019

Bib

@article{zhong2019deep,
  title = {Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI},
  author = {Zhong, Xinran and Cao, Ruiming and Shakeri, Sepideh and Scalzo, Fabien and Lee, Yeejin and Enzmann, Dieter R and Wu, Holden H and Raman, Steven S and Sung, Kyunghyun},
  journal = {Abdominal Radiology},
  volume = {44},
  number = {6},
  pages = {2030--2039},
  year = {2019},
  publisher = {Springer US},
}

Medical Imaging
Prostate Cancer Detection and Segmentation in Multi-parametric MRI via CNN and Conditional Random Field

Ruiming Cao, Xinran Zhong, Sepideh Shakeri, and 5 more authors

In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), May 2019

Best Paper Award Abs Bib PDF

Best Paper 2nd place, IEEE International Symposium on Biomedical Imaging

Multi-parametric MRI (mp-MRI) is a powerful diagnostic tool for prostate cancer (PCa). However, interpreting prostate mp-MRI requires high-level expertise, causing significant inter-reader variations. Convolutional neural networks (CNNs) have recently shown great promise for various tasks. In this study, we propose an improved CNN to jointly detect PCa lesions and segment for accurate lesions contours. Specifically, we adapt focal loss to overcome the imbalance between cancerous and non-cancerous areas for improved lesion detection and design selective dense conditional random field (SD-CRF), a post-processing step to refine the CNN prediction into the lesion segmentation based on a specific imaging component of mp-MRI. We trained and validated the proposed CNN in 5-fold cross-validation using 397 preoperative mp-MRI exams with whole-mount histopathology-confirmed lesion annotations. In the free-response receiver operating characteristics (FROC) analysis, the proposed CNN achieved 75.1% lesion detection sensitivity at the cost of 1 false positive per patient. In the evaluation for lesion segmentation, the proposed CNN improved the Dice coefficient by 20.6% from the baseline CNN.
@inproceedings{cao2019prostate, title = {Prostate Cancer Detection and Segmentation in Multi-parametric MRI via CNN and Conditional Random Field}, author = {Cao, Ruiming and Zhong, Xinran and Shakeri, Sepideh and Bajgiran, Amirhossein Mohammadian and Mirak, Sohrab Afshari and Enzmann, Dieter and Raman, Steven S and Sung, Kyunghyun}, booktitle = {2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)}, pages = {1900--1904}, year = {2019}, organization = {IEEE}, }
Medical Imaging
Prostate cancer inference via weakly-supervised learning using a large collection of negative MRI

Ruiming Cao, Xinran Zhong, Fabien Scalzo, and 2 more authors

In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, May 2019

Abs Bib PDF

Recent advances in medical imaging techniques have led to significant improvements in the management of prostate cancer (PCa). In particular, multi-parametric MRI (mp-MRI) continues to gain clinical acceptance as the preferred imaging technique for non-invasive detection and grading of PCa. However, the machine learning-based diagnosis systems for PCa are often constrained by the limited access to accurate lesion ground truth annotations for training. The performance of the machine learning system is highly dependable on both quality and quantity of lesion annotations associated with histopathologic findings, resulting in limited scalability and clinical validation. Here, we propose the baseline MRI model to alternatively learn the appearance of mp-MRI using radiology-confirmed negative MRI cases via weakly supervised learning. Since PCa lesions are case-specific and highly heterogeneous, it is assumed to be challenging to synthesize PCa lesions using the baseline MRI model, while it would be relatively easier to synthesize the normal appearance in mp-MRI. We then utilize the baseline MRI model to infer the pixel-wise suspiciousness of PCa by comparing the original and synthesized MRI with two distance functions. We trained and validated the baseline MRI model using 1,145 negative prostate mp-MRI scans. For evaluation, we used separated 232 mp-MRI scans, consisting of both positive and negative MRI cases. The 116 positive MRI scans were annotated by radiologists, confirmed with post-surgical whole-gland specimens. The suspiciousness map was evaluated by receiver operating characteristic (ROC) analysis for PCa lesions versus non-PCa regions classification and free-response receiver operating characteristic (FROC) analysis for PCa localization. Our proposed method achieved 0.84 area under the ROC curve and 77.0% sensitivity at one false positive per patient in FROC analysis.
@inproceedings{cao2019prostate_iccvw, title = {Prostate cancer inference via weakly-supervised learning using a large collection of negative MRI}, author = {Cao, Ruiming and Zhong, Xinran and Scalzo, Fabien and Raman, Steven and Sung, Kyunghyun}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops}, pages = {0--0}, year = {2019}, }
Medical Imaging
Joint prostate cancer detection and gleason score prediction in mp-MRI via FocalNet

Ruiming Cao, Amirhossein Mohammadian Bajgiran, Sohrab Afshari Mirak, and 5 more authors

IEEE transactions on medical imaging, May 2019

Abs Bib PDF

Multi-parametric MRI (mp-MRI) is considered the best non-invasive imaging modality for diagnosing prostate cancer (PCa). However, mp-MRI for PCa diagnosis is currently limited by the qualitative or semi-quantitative interpretation criteria, leading to inter-reader variability and a suboptimal ability to assess lesion aggressiveness. Convolutional neural networks (CNNs) are a powerful method to automatically learn the discriminative features for various tasks, including cancer detection. We propose a novel multi-class CNN, FocalNet, to jointly detect PCa lesions and predict their aggressiveness using Gleason score (GS). FocalNet characterizes lesion aggressiveness and fully utilizes distinctive knowledge from mp-MRI. We collected a prostate mp-MRI dataset from 417 patients who underwent 3T mp-MRI exams prior to robotic-assisted laparoscopic prostatectomy. FocalNet was trained and evaluated in this large study cohort with fivefold cross validation. In the free-response receiver operating characteristics (FROC) analysis for lesion detection, FocalNet achieved 89.7% and 87.9% sensitivity for index lesions and clinically significant lesions at one false positive per patient, respectively. For the GS classification, evaluated by the receiver operating characteristics (ROC) analysis, FocalNet received the area under the curve of 0.81 and 0.79 for the classifications of clinically significant PCa (GS ≥ 3 + 4) and PCa with GS ≥ 4 + 3, respectively. With the comparison to the prospective performance of radiologists using the current diagnostic guideline, FocalNet demonstrated comparable detection sensitivity for index lesions and clinically significant lesions, only 3.4% and 1.5% lower than highly experienced radiologists without statistical significance.
@article{cao2019joint, title = {Joint prostate cancer detection and gleason score prediction in mp-MRI via FocalNet}, author = {Cao, Ruiming and Bajgiran, Amirhossein Mohammadian and Mirak, Sohrab Afshari and Shakeri, Sepideh and Zhong, Xinran and Enzmann, Dieter and Raman, Steven and Sung, Kyunghyun}, journal = {IEEE transactions on medical imaging}, volume = {38}, number = {11}, pages = {2496--2506}, year = {2019}, publisher = {IEEE}, doi = {10.1109/TMI.2019.2901928}, }

Medical Imaging

Prostate Cancer Diagnosis from Multi-parametric Magnetic Resonance Imaging via Deep Learning

Ruiming Cao

University of California, Los Angeles. Master’s thesis , May 2019

Bib PDF

@phdthesis{cao2019msthesis,
  title = {Prostate Cancer Diagnosis from Multi-parametric Magnetic Resonance Imaging via Deep Learning},
  author = {Cao, Ruiming},
  year = {2019},
  school = {University of California, Los Angeles},
}

2018

Interpretability

Interpreting cnn knowledge via an explanatory graph

Quanshi Zhang, Ruiming Cao, Feng Shi, and 2 more authors

In Thirty-Second AAAI Conference on Artificial Intelligence, May 2018

Bib PDF Code

@inproceedings{zhang2018interpreting,
  title = {Interpreting cnn knowledge via an explanatory graph},
  author = {Zhang, Quanshi and Cao, Ruiming and Shi, Feng and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {Thirty-Second AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11819},
}

2017

Interpretability

Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning.

Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and 1 more author

In AAAI, May 2017

Bib PDF Code

@inproceedings{zhang2017growing,
  title = {Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning.},
  author = {Zhang, Quanshi and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {AAAI},
  pages = {2898--2906},
  year = {2017},
}

Interpretability

Mining Object Parts from CNNs via Active Question-Answering

Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and 1 more author

In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, May 2017

Bib PDF

@inproceedings{zhang2017mining,
  title = {Mining Object Parts from CNNs via Active Question-Answering},
  author = {Zhang, Quanshi and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),},
  year = {2017},
}

Interpretability

Interactively Transferring CNN Patterns for Part Localization

Quanshi Zhang, Ruiming Cao, Shengming Zhang, and 3 more authors

arXiv preprint arXiv:1708.01783, May 2017

Bib

@article{zhang2017interactively,
  title = {Interactively Transferring CNN Patterns for Part Localization},
  author = {Zhang, Quanshi and Cao, Ruiming and Zhang, Shengming and Redmonds, Mark and Wu, Ying Nian and Zhu, Song-Chun},
  journal = {arXiv preprint arXiv:1708.01783},
  year = {2017},
}