publications | Ruiming Cao

2024

Dynamics
Neural space-time model for dynamic scene recovery in multi-shot computational imaging systems

Ruiming Cao, Nikita S Divekar, James Nuñez, and 2 more authors

bioRxiv preprint: bioRxiv 2024.01.16.575950, 2024

Abs Bib PDF Code

Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space-time model (NSTM) that jointly estimates the scene and its motion dynamics. Hence, we can both remove motion artifacts and resolve sample dynamics. We demonstrate NSTM in three computational imaging systems: differential phase contrast microscopy, 3D structured illumination microscopy, and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.
@article{cao2024neural, title = {Neural space-time model for dynamic scene recovery in multi-shot computational imaging systems}, author = {Cao, Ruiming and Divekar, Nikita S and Nu{\~n}ez, James and Upadhyayula, Srigokul and Waller, Laura}, journal = {bioRxiv preprint: bioRxiv 2024.01.16.575950}, pages = {2024--01}, year = {2024}, publisher = {Cold Spring Harbor Laboratory}, }
Event Camera
Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras

Ruiming Cao^*, Dekel Galor^*, Amit Kohli, and 2 more authors

arXiv preprint arXiv:2404.01298, 2024

Abs Bib PDF

Event cameras capture changes of intensity over time as a stream of ’events’ and generally cannot measure intensity itself; hence, they are only used for imaging dynamic scenes. However, fluctuations due to random photon arrival inevitably trigger noise events, even for static scenes. While previous efforts have been focused on filtering out these undesirable noise events to improve signal quality, we find that, in the photon-noise regime, these noise events are correlated with the static scene intensity. We analyze the noise event generation and model its relationship to illuminance. Based on this understanding, we propose a method, called Noise2Image, to leverage the illuminance-dependent noise characteristics to recover the static parts of a scene, which are otherwise invisible to event cameras. We experimentally collect a dataset of noise events on static scenes to train and validate Noise2Image. Our results show that Noise2Image can robustly recover intensity images solely from noise events, providing a novel approach for capturing static scenes in event cameras, without additional hardware.
@article{cao2024noise2image, title = {Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras}, author = {Cao, Ruiming and Galor, Dekel and Kohli, Amit and Yates, Jacob L and Waller, Laura}, journal = {arXiv preprint arXiv:2404.01298}, year = {2024}, }
Label-free
Fast quantitative phase imaging with an LED array microscope using a perturbative approach

Kuan-Chen Shen, Ruiming Cao, Michael Unser, and 2 more authors

In Unconventional Optical Imaging IV, 2024

Abs Bib Video

Quantitative Phase Imaging is the method of choice to observe unlabelled biological cells, enabling the exploration of their intricate structures and dynamics. Differential Phase Contrast (DPC) is a fast and flexible approach but is limited to brightfield measurements, which limits the achievable resolution. In this study, we revisit DPC as a perturbative approach for high-resolution phase imaging with a minimal number of images. Our methodology combines both a perturbative phase retrieval algorithm and ring-shaped darkfield illumination patterns. The proposed approach leverages the advantages of DPC, such as its minimal image requirement, while addressing its inability to extend to darkfield imaging. By reducing the number of acquired images, our technique offers a rapid solution for examining dynamic systems.
@inproceedings{dong2024fast, title = {Fast quantitative phase imaging with an LED array microscope using a perturbative approach}, author = {Shen, Kuan-Chen and Cao, Ruiming and Unser, Michael and Waller, Laura and Dong, Jonathan}, booktitle = {Unconventional Optical Imaging IV}, pages = {PC129960D}, year = {2024}, organization = {SPIE}, }

2023

Dynamics
Speckle flow structured illumination microscopy for dynamic super-resolution imaging

Ruiming Cao, Fanglin Linda Liu, Li-Hao Yeh, and 2 more authors

In High-Speed Biomedical Imaging and Spectroscopy VIII, 2023

Best Presentation Award Abs Bib Video

Hitachi High-Tech Best Presentation Award, SPIE Photonics West High-Speed Biomedical Imaging and Spectroscopy conference

We propose a new method, Speckle Flow SIM, to super-resolve every frame of a dynamic scene from a sequence of diffraction-limited images at different acquisition timepoints. Speckle Flow SIM uses a fixed speckle-structured illumination to encode the super-resolved information into measurements and a neural space-time model to exploit the temporal redundancy for the dynamic scene reconstruction. We built a simple, inexpensive setup for Speckle Flow SIM and experimentally demonstrated 1.88× super-resolution for a dynamic scene.
@inproceedings{cao2023speckle_pw, title = {Speckle flow structured illumination microscopy for dynamic super-resolution imaging}, author = {Cao, Ruiming and Liu, Fanglin Linda and Yeh, Li-Hao and Meng, Guanghan and Waller, Laura}, booktitle = {High-Speed Biomedical Imaging and Spectroscopy VIII}, volume = {12390}, pages = {1239002}, year = {2023}, organization = {SPIE}, }
Dynamics
Speckle Structured Illumination of Dynamic Samples with a Neural Space-time Model

Ruiming Cao, Guanghan Meng, and Laura Waller

In Computational Optical Sensing and Imaging, 2023

Best Student Paper Award Abs Bib PDF Video

Best Student Paper Award at Computational Optical Sensing and Imaging conference

We propose a structured illumination microscopy (SIM) method, Speckle Flow SIM, that uses static speckle illumination with a dynamic sample and reconstructs super-resolved images via a neural space-time model, without sacrificing temporal resolution.
@inproceedings{cao2023speckle_cosi, title = {Speckle Structured Illumination of Dynamic Samples with a Neural Space-time Model}, author = {Cao, Ruiming and Meng, Guanghan and Waller, Laura}, booktitle = {Computational Optical Sensing and Imaging}, pages = {CW4D--2}, year = {2023}, organization = {Optica Publishing Group}, }

2022

Medical Imaging

Systems, methods and media for automatically segmenting and diagnosing prostate lesions using multi-parametric magnetic resonance imaging data

Kyung Hyun Sung, and Ruiming Cao

May 2022

US Patent 11,340,324

Bib

@misc{sung2022systems,
  title = {Systems, methods and media for automatically segmenting and diagnosing prostate lesions using multi-parametric magnetic resonance imaging data},
  author = {Sung, Kyung Hyun and Cao, Ruiming},
  year = {2022},
  month = may,
  note = {US Patent 11,340,324},
  url = {https://patents.google.com/patent/US11340324B2/en},
}

Label-free
Self-calibrated 3D differential phase contrast microscopy with optimized illumination

Ruiming Cao, Michael Kellman, David Ren, and 2 more authors

Biomedical Optics Express, May 2022

Abs Bib PDF

3D phase imaging recovers an object’s volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope’s focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.
@article{cao2022self, title = {Self-calibrated 3D differential phase contrast microscopy with optimized illumination}, author = {Cao, Ruiming and Kellman, Michael and Ren, David and Eckert, Regina and Waller, Laura}, journal = {Biomedical Optics Express}, volume = {13}, number = {3}, pages = {1671--1684}, year = {2022}, publisher = {Optica Publishing Group}, }
Dynamics
Dynamic Structured Illumination Microscopy with a Neural Space-time Model

Ruiming Cao, Fanglin Linda Liu, Li-Hao Yeh, and 1 more author

In IEEE International Conference on Computational Photography (ICCP), May 2022

Abs Bib PDF Video

Structured illumination microscopy (SIM) reconstructs a super-resolved image from multiple raw images captured with different illumination patterns; hence, acquisition speed is limited, making it unsuitable for dynamic scenes. We propose a new method, Speckle Flow SIM, that uses static patterned illumination with moving samples and models the sample motion during data capture in order to reconstruct the dynamic scene with super-resolution. Speckle Flow SIM relies on sample motion to capture a sequence of raw images. The spatio-temporal relationship of the dynamic scene is modeled using a neural space-time model with coordinate-based multi-layer perceptrons (MLPs), and the motion dynamics and the super-resolved scene are jointly recovered. We validate Speckle Flow SIM for coherent imaging in simulation and build a simple, inexpensive experimental setup with off-the-shelf components. We demonstrate that Speckle Flow SIM can reconstruct a dynamic scene with deformable motion and 1.88x the diffraction-limited resolution in experiment.
@inproceedings{cao2022dynamic, title = {Dynamic Structured Illumination Microscopy with a Neural Space-time Model}, author = {Cao, Ruiming and Liu, Fanglin Linda and Yeh, Li-Hao and Waller, Laura}, booktitle = {IEEE International Conference on Computational Photography (ICCP)}, year = {2022}, organization = {IEEE}, }
Dynamics
Space-Time DiffuserCam Video Reconstruction using Implicit Neural Representations

Tiffany Chien, Ruiming Cao, Fanglin Linda Liu, and 1 more author

In 3D Image Acquisition and Display: Technology, Perception and Applications, May 2022

Abs Bib PDF

We train space-time neural representations in an unsupervised, physics-based pipeline to reconstruct videos from DiffuserCam lensless imagers. Our method takes advantage of redundant information between frames, demonstrating improved quality when compared to frame-by-frame approaches.
@inproceedings{chien2022space, title = {Space-Time DiffuserCam Video Reconstruction using Implicit Neural Representations}, author = {Chien, Tiffany and Cao, Ruiming and Liu, Fanglin Linda and Waller, Laura}, booktitle = {3D Image Acquisition and Display: Technology, Perception and Applications}, pages = {JW5B--1}, year = {2022}, organization = {Optica Publishing Group}, }

2021

Medical Imaging

Performance of Deep Learning and Genitourinary Radiologists in Detection of Prostate Cancer Using 3-T Multiparametric Magnetic Resonance Imaging

Ruiming Cao, Xinran Zhong, Sohrab Afshari, and 7 more authors

Journal of Magnetic Resonance Imaging, May 2021

Bib PDF

@article{cao2021performance,
  title = {Performance of Deep Learning and Genitourinary Radiologists in Detection of Prostate Cancer Using 3-T Multiparametric Magnetic Resonance Imaging},
  author = {Cao, Ruiming and Zhong, Xinran and Afshari, Sohrab and Felker, Ely and Suvannarerg, Voraparee and Tubtawee, Teeravut and Vangala, Sitaram and Scalzo, Fabien and Raman, Steven and Sung, Kyunghyun},
  journal = {Journal of Magnetic Resonance Imaging},
  volume = {54},
  number = {2},
  pages = {474--483},
  year = {2021},
  publisher = {John Wiley \& Sons, Inc. Hoboken, USA},
}

Label-free

Algorithmic self-calibration for optimized 3D quantitative differential phase contrast microscopy

Ruiming Cao, Michael Kellman, David Yonghuan Ren, and 2 more authors

In Quantitative Phase Imaging VII, May 2021

Bib Video

@inproceedings{cao2021algorithmic,
  title = {Algorithmic self-calibration for optimized 3D quantitative differential phase contrast microscopy},
  author = {Cao, Ruiming and Kellman, Michael and Ren, David Yonghuan and Eckert, Regina and Waller, Laura},
  booktitle = {Quantitative Phase Imaging VII},
  volume = {11653},
  pages = {116530R},
  year = {2021},
  organization = {International Society for Optics and Photonics},
}

2020

Computer Vision

Mining interpretable AOG representations from convolutional networks via active question answering

Quanshi Zhang, Jie Ren, Ge Huang, and 3 more authors

IEEE transactions on pattern analysis and machine intelligence, May 2020

Bib

@article{zhang2020mining,
  title = {Mining interpretable AOG representations from convolutional networks via active question answering},
  author = {Zhang, Quanshi and Ren, Jie and Huang, Ge and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  journal = {IEEE transactions on pattern analysis and machine intelligence},
  volume = {43},
  number = {11},
  pages = {3949--3963},
  year = {2020},
  publisher = {IEEE},
}

Computer Vision

Extraction of an Explanatory Graph to Interpret a CNN

Quanshi Zhang, Xin Wang, Ruiming Cao, and 3 more authors

IEEE transactions on pattern analysis and machine intelligence, May 2020

Bib

@article{zhang2020extraction,
  title = {Extraction of an Explanatory Graph to Interpret a CNN},
  author = {Zhang, Quanshi and Wang, Xin and Cao, Ruiming and Wu, Ying Nian and Shi, Feng and Zhu, Song-Chun},
  journal = {IEEE transactions on pattern analysis and machine intelligence},
  volume = {43},
  number = {11},
  pages = {3863--3877},
  year = {2020},
  publisher = {IEEE},
}

Label-free

3D Differential Phase Contrast Microscopy with Axial Motion Deblurring

Ruiming Cao, Michael Kellman, David Ren, and 1 more author

In Computational Optical Sensing and Imaging, May 2020

Bib

@inproceedings{cao20203d,
  title = {3D Differential Phase Contrast Microscopy with Axial Motion Deblurring},
  author = {Cao, Ruiming and Kellman, Michael and Ren, David and Waller, Laura},
  booktitle = {Computational Optical Sensing and Imaging},
  pages = {CF4C--2},
  year = {2020},
  organization = {Optical Society of America},
}

2019

Medical Imaging

Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI

Xinran Zhong, Ruiming Cao, Sepideh Shakeri, and 6 more authors

Abdominal Radiology, May 2019

Bib

@article{zhong2019deep,
  title = {Deep transfer learning-based prostate cancer classification using 3 Tesla multi-parametric MRI},
  author = {Zhong, Xinran and Cao, Ruiming and Shakeri, Sepideh and Scalzo, Fabien and Lee, Yeejin and Enzmann, Dieter R and Wu, Holden H and Raman, Steven S and Sung, Kyunghyun},
  journal = {Abdominal Radiology},
  volume = {44},
  number = {6},
  pages = {2030--2039},
  year = {2019},
  publisher = {Springer US},
}

Medical Imaging
Prostate Cancer Detection and Segmentation in Multi-parametric MRI via CNN and Conditional Random Field

Ruiming Cao, Xinran Zhong, Sepideh Shakeri, and 5 more authors

In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), May 2019

Best Paper Award Abs Bib PDF

Best Paper 2nd place, IEEE International Symposium on Biomedical Imaging

Multi-parametric MRI (mp-MRI) is a powerful diagnostic tool for prostate cancer (PCa). However, interpreting prostate mp-MRI requires high-level expertise, causing significant inter-reader variations. Convolutional neural networks (CNNs) have recently shown great promise for various tasks. In this study, we propose an improved CNN to jointly detect PCa lesions and segment for accurate lesions contours. Specifically, we adapt focal loss to overcome the imbalance between cancerous and non-cancerous areas for improved lesion detection and design selective dense conditional random field (SD-CRF), a post-processing step to refine the CNN prediction into the lesion segmentation based on a specific imaging component of mp-MRI. We trained and validated the proposed CNN in 5-fold cross-validation using 397 preoperative mp-MRI exams with whole-mount histopathology-confirmed lesion annotations. In the free-response receiver operating characteristics (FROC) analysis, the proposed CNN achieved 75.1% lesion detection sensitivity at the cost of 1 false positive per patient. In the evaluation for lesion segmentation, the proposed CNN improved the Dice coefficient by 20.6% from the baseline CNN.
@inproceedings{cao2019prostate, title = {Prostate Cancer Detection and Segmentation in Multi-parametric MRI via CNN and Conditional Random Field}, author = {Cao, Ruiming and Zhong, Xinran and Shakeri, Sepideh and Bajgiran, Amirhossein Mohammadian and Mirak, Sohrab Afshari and Enzmann, Dieter and Raman, Steven S and Sung, Kyunghyun}, booktitle = {2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)}, pages = {1900--1904}, year = {2019}, organization = {IEEE}, }
Medical Imaging
Prostate cancer inference via weakly-supervised learning using a large collection of negative MRI

Ruiming Cao, Xinran Zhong, Fabien Scalzo, and 2 more authors

In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, May 2019

Abs Bib PDF

Recent advances in medical imaging techniques have led to significant improvements in the management of prostate cancer (PCa). In particular, multi-parametric MRI (mp-MRI) continues to gain clinical acceptance as the preferred imaging technique for non-invasive detection and grading of PCa. However, the machine learning-based diagnosis systems for PCa are often constrained by the limited access to accurate lesion ground truth annotations for training. The performance of the machine learning system is highly dependable on both quality and quantity of lesion annotations associated with histopathologic findings, resulting in limited scalability and clinical validation. Here, we propose the baseline MRI model to alternatively learn the appearance of mp-MRI using radiology-confirmed negative MRI cases via weakly supervised learning. Since PCa lesions are case-specific and highly heterogeneous, it is assumed to be challenging to synthesize PCa lesions using the baseline MRI model, while it would be relatively easier to synthesize the normal appearance in mp-MRI. We then utilize the baseline MRI model to infer the pixel-wise suspiciousness of PCa by comparing the original and synthesized MRI with two distance functions. We trained and validated the baseline MRI model using 1,145 negative prostate mp-MRI scans. For evaluation, we used separated 232 mp-MRI scans, consisting of both positive and negative MRI cases. The 116 positive MRI scans were annotated by radiologists, confirmed with post-surgical whole-gland specimens. The suspiciousness map was evaluated by receiver operating characteristic (ROC) analysis for PCa lesions versus non-PCa regions classification and free-response receiver operating characteristic (FROC) analysis for PCa localization. Our proposed method achieved 0.84 area under the ROC curve and 77.0% sensitivity at one false positive per patient in FROC analysis.
@inproceedings{cao2019prostate_iccvw, title = {Prostate cancer inference via weakly-supervised learning using a large collection of negative MRI}, author = {Cao, Ruiming and Zhong, Xinran and Scalzo, Fabien and Raman, Steven and Sung, Kyunghyun}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops}, pages = {0--0}, year = {2019}, }

Medical Imaging

Prostate Cancer Diagnosis from Multi-parametric Magnetic Resonance Imaging via Deep Learning

Ruiming Cao

University of California, Los Angeles. Master’s thesis , May 2019

Bib PDF

@phdthesis{cao2019msthesis,
  title = {Prostate Cancer Diagnosis from Multi-parametric Magnetic Resonance Imaging via Deep Learning},
  author = {Cao, Ruiming},
  year = {2019},
  school = {University of California, Los Angeles},
}

Medical Imaging
Joint prostate cancer detection and gleason score prediction in mp-MRI via FocalNet

Ruiming Cao, Amirhossein Mohammadian Bajgiran, Sohrab Afshari Mirak, and 5 more authors

IEEE transactions on medical imaging, May 2019

Abs Bib PDF

Multi-parametric MRI (mp-MRI) is considered the best non-invasive imaging modality for diagnosing prostate cancer (PCa). However, mp-MRI for PCa diagnosis is currently limited by the qualitative or semi-quantitative interpretation criteria, leading to inter-reader variability and a suboptimal ability to assess lesion aggressiveness. Convolutional neural networks (CNNs) are a powerful method to automatically learn the discriminative features for various tasks, including cancer detection. We propose a novel multi-class CNN, FocalNet, to jointly detect PCa lesions and predict their aggressiveness using Gleason score (GS). FocalNet characterizes lesion aggressiveness and fully utilizes distinctive knowledge from mp-MRI. We collected a prostate mp-MRI dataset from 417 patients who underwent 3T mp-MRI exams prior to robotic-assisted laparoscopic prostatectomy. FocalNet was trained and evaluated in this large study cohort with fivefold cross validation. In the free-response receiver operating characteristics (FROC) analysis for lesion detection, FocalNet achieved 89.7% and 87.9% sensitivity for index lesions and clinically significant lesions at one false positive per patient, respectively. For the GS classification, evaluated by the receiver operating characteristics (ROC) analysis, FocalNet received the area under the curve of 0.81 and 0.79 for the classifications of clinically significant PCa (GS ≥ 3 + 4) and PCa with GS ≥ 4 + 3, respectively. With the comparison to the prospective performance of radiologists using the current diagnostic guideline, FocalNet demonstrated comparable detection sensitivity for index lesions and clinically significant lesions, only 3.4% and 1.5% lower than highly experienced radiologists without statistical significance.
@article{cao2019joint, title = {Joint prostate cancer detection and gleason score prediction in mp-MRI via FocalNet}, author = {Cao, Ruiming and Bajgiran, Amirhossein Mohammadian and Mirak, Sohrab Afshari and Shakeri, Sepideh and Zhong, Xinran and Enzmann, Dieter and Raman, Steven and Sung, Kyunghyun}, journal = {IEEE transactions on medical imaging}, volume = {38}, number = {11}, pages = {2496--2506}, year = {2019}, publisher = {IEEE}, }

2018

Computer Vision

Interpreting cnn knowledge via an explanatory graph

Quanshi Zhang, Ruiming Cao, Feng Shi, and 2 more authors

In Thirty-Second AAAI Conference on Artificial Intelligence, May 2018

Bib

@inproceedings{zhang2018interpreting,
  title = {Interpreting cnn knowledge via an explanatory graph},
  author = {Zhang, Quanshi and Cao, Ruiming and Shi, Feng and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {Thirty-Second AAAI Conference on Artificial Intelligence},
  year = {2018},
}

2017

Computer Vision

Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning.

Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and 1 more author

In AAAI, May 2017

Bib

@inproceedings{zhang2017growing,
  title = {Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning.},
  author = {Zhang, Quanshi and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {AAAI},
  pages = {2898--2906},
  year = {2017},
}

Computer Vision

Mining Object Parts from CNNs via Active Question-Answering

Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and 1 more author

In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, May 2017

Bib

@inproceedings{zhang2017mining,
  title = {Mining Object Parts from CNNs via Active Question-Answering},
  author = {Zhang, Quanshi and Cao, Ruiming and Wu, Ying Nian and Zhu, Song-Chun},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),},
  year = {2017},
}

Computer Vision

Interactively Transferring CNN Patterns for Part Localization

Quanshi Zhang, Ruiming Cao, Shengming Zhang, and 3 more authors

arXiv preprint arXiv:1708.01783, May 2017

Bib

@article{zhang2017interactively,
  title = {Interactively Transferring CNN Patterns for Part Localization},
  author = {Zhang, Quanshi and Cao, Ruiming and Zhang, Shengming and Redmonds, Mark and Wu, Ying Nian and Zhu, Song-Chun},
  journal = {arXiv preprint arXiv:1708.01783},
  year = {2017},
}