Detection and classification of objects in three-dimensional images using deep learning methods

Cover Page

Cite item

Full Text

Open Access Open Access
Restricted Access Access granted
Restricted Access Subscription Access

Abstract

The issue of automatic object detection and categorization in three-dimensional, single-channel, raster images is considered. Objects may have low contrast and substantial shape variability, making it challenging to explicitly construct a model. The proposed solution employs machine learning techniques based on a labeled database of use scenarios. A two-step algorithm is presented, with the first stage being the detection of objects within the image and the second being the reduction of false positives and object categorization. Deep learning approach is applied with a single input and trained for the simultaneous solution of multiple tasks. The practical goal of developing a clinically viable automatic decision support system to detect and classify rib fractures based on CT scans is being solved. Computational experiments were conducted on the publicly available RibFrac dataset. The proposed system was shown to achieve a detection sensitivity of 0.935, with an average number of false positive predictions per image of 4.7. The resulting algorithm was compared with existing methods using quantitative measures.

About the authors

I. A. Matveev

Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences

Author for correspondence.
Email: matveev@frccsc.ru
Russian Federation, Moscow

A. A. Yurchenko

Moscow Institute of Physics and Technology (National Research University); Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences

Email: yurchenko.aa@phystech.edu
Russian Federation, Moscow; Moscow

References

  1. Topker M., Ringl H., Lazar M. The Ribs Unfolded – a CT Visualization Algorithm for Fast Detection of Rib Fractures: Effect on Sensitivity and Specificity in Trauma Patients // Eur Radiol. 2015. V. 25. P. 1865–1874.
  2. Urbaneja A., De Verbizier J., Formery A. S. et al. Automatic Rib Cage Unfolding with CT Cylindrical Projection Reformat in Polytraumatized Patients for Rib Fracture Detection and Characterization: Feasibility and Clinical Application // European J. Radiology. 2019. V. 110. P. 121–127.
  3. Jang Y.C., Lee C.W., Huang C.C. Diagnostic Accuracy for Acute Rib Fractures: A Cross-sectional Study Utilizing Automatic Rib Unfolding and 3D Volume-Rendered Reformation // Acad. Radiol. 2024. V. 31. № 4. P. 1538–1547.
  4. Erdemir A., Onur M.R., Idilman I. et al. Pros and Cons of Rib Unfolding Software: a Reliability and Reproducibility Study on Trauma Patients // Turkish J. Trauma and Emergency Surgery. 2023. V. 29. № 6. P. 717–723.
  5. Ronneberger O., Fissher P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation // arXiv preprint. 2015. arXiv:1505.04597.
  6. Chen L.C., Zhu Y. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation // Computer Vision – ECCV. 2018. V. 11211. P. 833–851.
  7. Kaiming H. Deep Residual Learning for Image Recognition // arXiv preprint. 2015. arXiv:1512.03385.
  8. Jin L., Yang J., Kuang K. et al. Deep-Learning-Assisted Detection and Segmentation of Rib Fractures from CT Scans: Development and Validation of FracNet // EBioMedicine. 2020. V. 62. № 12. P. 103106.
  9. Yang J., Shi R., Jin. L. et al. Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge // arXiv preprint. 2024. arXiv:2402.09372.
  10. Charles R.Q. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space // arXiv preprint. 2017. arXiv:1706.02413.
  11. Wang Y., Sun. Y., Liu Z. et al. Dynamic Graph CNN for Learning on Point Clouds // ACM Transactions on Graphics. 2019. V. 38. № 5. P. 146.
  12. Cao Z., Xu L., Chen D. et al. A Robust Shape-Aware Rib Fracture Detection and Segmentation Framework With Contrastive Learning // IEEE Transactions on Multimedia. 2023. V. 1. P. 1–8.
  13. Wu M., Chai Z., Qian G. et al. Development and Evaluation of a Deep Learning Algorithm for Rib Segmentation and Fracture Detection from Multicenter Chest CT Images // Radiology: Artificial Intelligence. 2021. V. 3. № 9. P.e200248.
  14. Isensee F., Wald T., Ulrich C. et al. nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation // arXiv preprint. 2024. arXiv:2404.09556.
  15. Cardoso M.J., Li W., Brown R. et al. MONAI: An Open-source Framework for Deep Learning in Healthcare // arXiv preprint. 2022. arXiv:2211.02701.
  16. Hatamizadeh A., Nath V., Tang Y. et al. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images // arXiv preprint. 2022. arXiv:2201.01266.
  17. Myronenko A. 3D MRI Brain Tumor Segmentation Using Autoencoder Regularization // arXiv preprint. 2018. arXiv:1810.11654.
  18. Huang Z., Wang H., Deng Z. et al. STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training // arXiv preprint. 2023. arXiv:2304.06716.
  19. D’Antonoli T.A., Berger L.K., Indrakanti A.K. et al. TotalSegmentator MRI: Sequence-Independent Segmentation of 59 Anatomical Structures in MR Images // arXiv preprint. 2024. arXiv:2405.19492.
  20. Jin L., Yang J., Kuang K. et al. Deep-Learning-Assisted Detection and Segmentation of Rib Fractures from CT Scans: Development and Validation of FracNet // EBioMedicine. 2020. V. 62. № 12. P. 103106.
  21. Yang J., Shi R., Jin L. et al. Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge // arXiv preprint. 2024. arXiv:2402.09372.
  22. Jin L., Gu S., Wei D. et al. RibSeg v2: A Large-scale Benchmark for Rib Labeling and Anatomical Centerline Extraction // IEEE Transactions on Medical Imaging (TMI). 2024. V. 43. № 1. P. 570–581.
  23. Yang J., Gu S., Wei D. et al. RibSeg Dataset and Strong Point Cloud Baselines for Rib Segmentation from CT Scans // Intern. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Strasbourg, 2021. P. 611–621.
  24. Li C., Zia M.Z., Tran Q.H. et al. Deep Supervision with Intermediate Concepts // arXiv preprint. 2018. arXiv:1801.03399.
  25. He Z., Nath V., Yang D. et al. SwinUNETR-V2: Stronger Swin Transformers withВ Stagewise Convolutions forВ 3D Medical Image Segmentation // Medical Image Computing and Computer Assisted Intervention (MICCAI). Vancouver, 2023. P. 416–426.
  26. Drozdzal M., Vorontsov E., Chartrand G. et al. The Importance of Skip Connections in Biomedical Image Segmentation // arXiv preprint. 2016. arXiv:1608.04117.
  27. Redmon J., Divvala S., Girshick R. et al. You Only Look Once: Unified, Real-Time Object Detection // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016. P. 779–788.
  28. Tian Z., Shen C., Chen H. et al. FCOS: Fully Convolutional One-Stage Object Detection // IEEE/CVF Intern. Conf. on Computer Vision (ICCV). Seoul, 2019. P. 9626–9635.
  29. Zhang S., Chi C., Yao Y. et al. Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Seattle, 2020. P. 9756–9765.
  30. Lin T.Y., Goyal P., Girshick R. et al. Focal Loss for Dense Object Detection // IEEE Intern. Conf. on Computer Vision (ICCV). Venice, 2017. P. 2999–3007.
  31. Tan M., Pang R., Quoc V.L. EfficientDet: Scalable and Efficient Object Detection // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Seattle, 2020. P. 10778–10787.
  32. Hu J., Shen L., Albanie S. et al. Squeeze-and-Excitation Networks // IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City, 2018. P. 7132–7141.
  33. Xie S., Girshick R., Dollar P. et al. Aggregated Residual Transformations for Deep Neural Networks // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Honolulu, 2017. P. 5987–5995.
  34. Wang J., Sun K., Cheng T. et al. Deep High-Resolution Representation Learning for Visual Recognition // arXiv preprint. 2019. arXiv: 1908.07919.
  35. Tan M., Le Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks // arXiv preprint. 2019. arXiv: 1905.11946.
  36. Lin T.Y., Dollar P., Girshick R.B. et al. Feature Pyramid Networks for Object Detection // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Honolulu, 2017. P. 936–944.
  37. Liu S., Johns E., Davison A.J. End-To-End Multi-Task Learning With Attention // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Long Beach, 2019. P. 1871–1880.
  38. Zhou Z., Siddiquee M., Tajbakhsh N. et al. UNet++: A Nested U-Net Architecture for Medical Image Segmentation // Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. 2018. V. 11045. P. 3–11.
  39. Oktay O., Schlemper J., Folgoc L.L. et al. Attention U-Net: Learning Where to Look for the Pancreas // arXiv preprint. 2018. arXiv: 1804.03999.
  40. Ioffe S., Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift // arXiv preprint. 2015. arXiv: 1502.03167.
  41. Fu C.Y., Shvets M., Berg A.C. RetinaMask: Learning to Predict Masks Improves State-of-the-art Single-Shot Detection for Free // arXiv preprint. 2019. arXiv: 1901.03353.
  42. Sudre C.H., Li W., Vercauteren T. et al. Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations // Lecture Notes in Computer Science. 2017. V. 2017. P. 240–248.
  43. Wang C., MacGillivray T., Macnaught G. et al. A Two-stage 3D Unet Framework for Multi-class Segmentation on Full Resolution Image // arXiv preprint. 2018. arXiv: 1804.04314.
  44. Jeon U., Kim H., Hong H., Wang J.H. Two-stage Meniscus Segmentation Framework Integrating Multiclass Localization Network and Adversarial Learning-based Segmentation Network in Knee MR Images // Medical Imaging 2021: Computer-Aided Diagnosis. 2021. V. 11597. P. 1159714.
  45. Jin D., Ma X., Zhang C. et al. Towards Overcoming False Positives in Visual Relationship Detection // arXiv preprint. 2020. arXiv:2012.12510.
  46. Shi W., Chen J., Feng F. et al. On the Theories Behind Hard Negative Sampling for Recommendation // Proc. ACM Web Conf. Austin, 2023. P. 812–822.
  47. Baumgartner M., Jaeger P.F., Isensee F. et al. NnDetection: A Self-configuring Method for Medical Object Detection // Lecture Notes in Computer Science. 2021. V. 12905. P. 530–539.

Supplementary files

Supplementary Files
Action
1. JATS XML
Download

Copyright (c) 2024 Russian Academy of Sciences