Rapid patient-specific neural networks for X-ray to volume registration

Unberath, M. et al. The impact of machine learning on 2D/3D registration for image-guided interventions: a systematic review and perspective. Front. Robot. AI 8, 716007 (2021).

Yip, M. et al. Artificial intelligence meets medical robotics. Science 381, 141–146 (2023).

Penney, G. P. et al. A comparison of similarity measures for use in 2D/3D medical image registration. IEEE Trans. Med. Imaging 17, 586–595 (1998).

Knaan, D. & Joskowicz, L. Effective intensity-based 2D/3D rigid registration between fluoroscopic X-ray and CT. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Ellis, R. E. & Peters, T. M.) 351–358 (Springer, 2003).

Grupp, R. B. et al. Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. Int. J. Comput. Assist. Radiol. Surg. 15, 759–769 (2020).

Grimm, M., Esteban, J., Unberath, M. & Navab, N. Pose-dependent weights and domain randomization for fully automatic X-ray to CT registration. IEEE Trans. Med. Imaging 40, 2221–2232 (2021).

Mahesh, M., Ansari, A. J. & Mettler, F. A. Jr Patient exposure from radiologic and nuclear medicine procedures in the United States and worldwide: 2009–2018. Radiology 307, e221263 (2022).

Cornelis, F. H., Dzaye, O., Schoellnast, H. & Solomon, S. B. Imaging of interventional therapies in oncology: image guidance, robotics, and fusion systems. In Interventional Oncology (eds Fong, Y. et al.) 1–17 https://doi.org/10.1007/978-3-030-51192-0_19-1 (Springer, 2023).

Jhawar, B. S., Mitsis, D. & Duggal, N. Wrong-sided and wrong-level neurosurgery: a national survey. J. Neurosurg. Spine 7, 467–472 (2007).

Tonetti, J., Boudissa, M., Kerschbaumer, G. & Seurat, O. Role of 3D intraoperative imaging in orthopedic and trauma surgery. Orthop. Traumatol. Surg. Res. 106, S19–S25 (2020).

Abumoussa, A. et al. Machine learning for automated and real-time two-dimensional to three-dimensional registration of the spine using a single radiograph. Neurosurg. Focus 54, E16 (2023).

Naik, R. R., Hoblidar, A., Bhat, S. N., Ampar, N. & Kundangar, R. A hybrid 3D-2D image registration framework for pedicle screw trajectory registration between intraoperative X-ray image and preoperative CT image. J. Imaging 8, 185 (2022).

Metz, C. T. et al. Patient specific 4D coronary models from ECG-gated CTA data for intra-operative dynamic alignment of CTA with X-ray images. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Guang-Zhong, Y. et al.) 369–376 (Springer, 2009).

Wagner, M., Schafer, S., Strother, C. & Mistretta, C. 4D interventional device reconstruction from biplane fluoroscopy. Med. Phys. 43, 1324–1334 (2016).

Huynh, E. et al. Artificial intelligence in radiation oncology. Nat. Rev. Clin. Oncol. 17, 771–781 (2020).

Kim, Y. et al. Telerobotic neurovascular interventions with magnetic manipulation. Sci. Robot. 7, eabg9907 (2022).

Gu, W., Gao, C., Grupp, R., Fotouhi, J. & Unberath, M. Extended capture range of rigid 2D/3D registration by estimating Riemannian pose gradients. In Proc. International Workshop on Machine Learning in Medical Imaging (eds Liu, M. et al.) 281–291 (Springer, 2020).

Gopalakrishnan, V. & Golland, P. Fast auto-differentiable digitally reconstructed radiographs for solving inverse problems in intraoperative imaging. In Proc. Workshop on Clinical Image-Based Procedures (eds Chen, Y. et al.) 1–11 (Springer, 2022).

Gao, C. et al. A fully differentiable framework for 2D/3D registration and the projective spatial transformers. IEEE Trans. Med. Imaging 43, 275–285 (2023).

Grupp, R. B. et al. Pose estimation of periacetabular osteotomy fragments with intraoperative X-ray navigation. IEEE Trans. Biomed. Eng. 67, 441–452 (2019).

Bier, B. et al. Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views. Int. J. Comput. Assist. Radiol. Surg. 14, 1463–1473 (2019).

Shrestha, P., Xie, C., Yoshii, Y. & Kitahara, I. Rayemb: arbitrary landmark detection in X-ray images using ray embedding subspace. In Proc. Asian Conference on Computer Vision (eds Cho, M. et al.) 665–681 (Springer, 2024).

Miao, S., Wang, Z. J. & Liao, R. A CNN regression approach for real-time 2D/3D registration. IEEE Trans. Med. Imaging 35, 1352–1363 (2016).

Bui, M., Albarqouni, S., Schrapp, M., Navab, N. & Ilic, S. X-ray posenet: 6 DoF pose estimation for mobile X-ray devices. In Proc. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) 1036–1044 (IEEE, 2017).

Zhang, B. et al. A patient-specific self-supervised model for automatic X-ray/CT registration. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Greenspan, H. et al.) 515–524 (Springer, 2023).

Gopalakrishnan, V., Dey, N. & Golland, P. Intraoperative 2D/3D image registration via differentiable X-ray rendering. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 11662–11672 (IEEE, 2024).

Wasserthal, J. et al. TotalSegmentator: robust segmentation of 104 anatomic structures in CT images. Radiol. Artif. Intell. 5, e230024 (2023).

Jaus, A. et al. Towards unifying anatomy segmentation: automated generation of a full-body CT dataset. In Proc. 2024 IEEE International Conference on Image Processing (ICIP) 41–47 (IEEE, 2024).

Magnetic Resonance Angiography Atlas Dataset. NeuroImaging Tools & Resources Collaboratory https://www.nitrc.org/projects/icbmmra (2017).

Liu, P. et al. Deep learning to segment pelvic bones: large-scale CT datasets and baseline models. Int. J. Comput. Assist. Radiol. Surg. 16, 749–756 (2021).

Yushkevich, P. A. et al. Fast automatic segmentation of hippocampal subfields and medial temporal lobe subregions in 3 Tesla and 7 Tesla T2-weighted MRI. Alzheimers Dement. 12, P126–P127 (2016).

Zhou, Y., Barnes, C., Lu, J., Yang, J. & Li, H. On the continuity of rotation representations in neural networks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 5745–5753 (IEEE, 2019).

Geist, A. R., Frey, J., Zhobro, M., Levina, A. & Martius, G. Learning with 3D rotations, a hitchhiker’s guide to SO(3). In Proc. 41st International Conference on Machine Learning Vol. 235 (eds Salakhutdinov, R. et al.) 15331–15350 (PMLR, 2024).

Lin, C., Hanson, A. J. & Hanson, S. M. Algebraically rigorous quaternion framework for the neural network pose estimation problem. In Proc. IEEE/CVF International Conference on Computer Vision 14097–14106 (IEEE, 2023).

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

Flepp, R. et al. Automatic multi-view X-ray/CT registration using bone substructure contours. Int. J. Comput. Assist. Radiol. Surg. 20, 1401–1408 (2025).

Mitrović, U., Špiclin, Ž, Likar, B. & Pernuš, F. 3D-2D registration of cerebral angiograms: a method and evaluation on clinical images. IEEE Trans. Med. Imaging 32, 1550–1563 (2013).

Xu, M. et al. VesselBoost: a Python toolbox for small blood vessel segmentation in human magnetic resonance angiography data. Aperture Neuro https://doi.org/10.52294/001c.123217 (2024).

Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Navab, N. et al.) 234–241 (Springer, 2015).

Terzakis, G. & Lourakis, M. A consistently fast and globally optimal solution to the perspective-n-point problem. In Proc. European Conference on Computer Vision (eds Vedaldi, A. et al.) 478–494 (Springer, 2020).

Potente, M. & Mäkinen, T. Vascular heterogeneity and specialization in development and disease. Nat. Rev. Mol. Cell Biol. 18, 477–494 (2017).

Nuñez, F. B. & Dohna-Schwake, C. Epidemiology, diagnostics, and management of vein of Galen malformation. Pediatr. Neurol. 119, 50–55 (2021).

Varghese, C., Harrison, E. M., O’Grady, G. & Topol, E. J. Artificial intelligence in surgery. Nat. Med. 30, 1257–1268 (2024).

Jena, R., Sethi, D., Chaudhari, P. & Gee, J. Deep learning in medical image registration: magic or mirage? In Proc. 38th Annual Conference on Neural Information Processing Systems (eds Globerson, A. et al.) 108331–108353 (Curran Associates Inc., 2024).

Dey, N. et al. Learning general-purpose biomedical volume representations using randomized synthesis. In Proc. 13th International Conference on Learning Representations (eds Yue, Y. et al.) (ICLR, 2025).

Gagoski, B. et al. Automated detection and reacquisition of motion-degraded images in fetal HASTE imaging at 3T. Magn. Reson. Med. 87, 1914–1922 (2022).

Arsigny, V., Commowick, O., Ayache, N. & Pennec, X. A fast and log-euclidean polyaffine framework for locally linear registration. J. Math. Imaging Vis. 33, 222–238 (2009).

Gopalakrishnan, V., Dey, N. & Golland, P. PolyPose: deformable 2D/3D registration via polyrigid transformations. In Advances in Neural Information Processing Systems 60633–60659 (NeurIPS, 2025).

Hartley, R. & Zisserman, A. Multiple View Geometry in Computer Vision (Cambridge Univ. Press, 2003).

Swinehart, D. F. The Beer-Lambert law. J. Chem. Educ. 39, 333 (1962).

Siddon, R. L. Fast calculation of the exact radiological path for a three-dimensional CT array. Med. Phys. 12, 252–255 (1985).

Unberath, M. et al. DeepDRR—a catalyst for machine learning in fluoroscopy-guided procedures. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Frangi, A. F. et al.) 98–106 (Springer, 2018).

Milletari, F., Navab, N. & Ahmadi, S.-A. V-net: fully convolutional neural networks for volumetric medical image segmentation. In Proc. 2016 Fourth International Conference on 3D Vision (3DV) 565–571 (IEEE, 2016).

Grupp, R. B., Armand, M. & Taylor, R. H. Patch-based image similarity for intraoperative 2D/3D pelvis registration during periacetabular osteotomy. In Proc. International Workshop on Computer-Assisted and Robotic Endoscopy 153–163 (Springer, 2018).

Ferrara, D. et al. Sharing a whole-/total-body [18F] FDG-PET/CT dataset with CT-derived segmentations: an ENHANCE.PET initiative. Sci. Data 13, 869 (2026).

Sundar, L. K. S. et al. Fully automated, semantic segmentation of whole-body 18F-FDG PET/CT images based on data-centric artificial intelligence. J. Nucl. Med. 63, 1941–1948 (2022).

Grupp, R. et al. Data and code associated with the publication: Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. Johns Hopkins Research Data Repository https://doi.org/10.7281/T1/IFSXNV (2020).

Flepp, R. et al. Automatic multi-View X-Ray/CT registration using bone substructure contours. Zenodo https://zenodo.org/records/15753063 (2025).

Mitrović, U. & Špiclin, Ž. Gold standard for 3D-2D registration of cerebral angiograms. Laboratory of Imaging Technologies https://lit.fe.uni-lj.si/en/research/resources/3D-2D-GS-CA (2013).

Pengbo, L. et al. CTPelvic1K Dataset. Zenodo https://doi.org/10.5281/zenodo.4588402 (2020).

Wasserthal, J. Dataset with segmentations of 117 important anatomical structures in 1228 CT images. Zenodo https://doi.org/10.5281/zenodo.6802613 (2023).

Ferrara, D. et al. ENHANCE.PET 1.6k: a whole-/total-body [18F]FDG-PET/CT dataset with CT-derived segmentations. Science Data Bank https://doi.org/10.57760/sciencedb.34150 (2026).