Digital DiagnosticsDigital DiagnosticsDigital DiagnosticsDigital Diagnostics2712-84902712-8962Eco-Vector32197110.17816/DD321971Original Study ArticlesОригинальные исследования原创性科研成果Research ArticleMethodology for testing and monitoring artificial intelligence-based software for medical diagnosticsМетодология тестирования и мониторинга программного обеспечения на основе технологий искусственного интеллекта для медицинской диагностики基于人工智能技术的医疗诊断软件测试和监测方法学https://orcid.org/0000-0002-0208-52184458-5608VasilievYuri A.ВасильевЮрий АлександровичVasilievYuri A.
Russian Federation

MD, Cand. Sci. (Med.)

канд. мед. наук

MD, Cand. Sci. (Med.)

VasilevYA1@zdrav.mos.ruhttps://orcid.org/0000-0002-2990-77363602-7120VlazimirskyAnton V.ВладзимирскийАнтон ВячеславовичVlazimirskyAnton V.
Russian Federation

MD, Dr. Sci. (Med.)

д-р мед. наук

MD, Dr. Sci. (Med.)

VladzimirskijAV@zdrav.mos.ruhttps://orcid.org/0000-0002-0245-44318948-6152OmelyanskayaOlga V.ОмелянскаяОльга ВасильевнаOmelyanskayaOlga V.
Russian Federation
OmelyanskayaOV@zdrav.mos.ruhttps://orcid.org/0000-0001-7786-03493160-8062ArzamasovKirill M.АрзамасовКирилл МихайловичArzamasovKirill M.
Russian Federation

MD, Cand. Sci. (Med.)

канд. мед. наук

MD, Cand. Sci. (Med.)

ArzamasovKM@zdrav.mos.ruhttps://orcid.org/0000-0002-3097-88813815-8870ChetverikovSergey F.ЧетвериковСергей ФедоровичChetverikovSergey F.
Russian Federation

Cand. Sci. (Engin.)

канд. тех. наук

Cand. Sci. (Engin.)

ChetverikovSF@zdrav.mos.ruhttps://orcid.org/0000-0001-7670-73858734-2085RumyantsevDenis A.РумянцевДенис АндреевичRumyantsevDenis A.
Russian Federation
x.radiology@mail.ruhttps://orcid.org/0000-0001-7458-53963823-6872ZelenovaMaria A.ЗеленоваМария АлександровнаZelenovaMaria A.
Russian Federation
ZelenovaMA@zdrav.mos.ruMoscow Center for Diagnostics and TelemedicineНаучно-практический клинический центр диагностики и телемедицинских технологийMoscow Center for Diagnostics and Telemedicine3008202326092023432522670604202315062023Copyright ©; 2023, Eco-VectorCopyright ©; 2023, Эко-векторCopyright ©; 2023, Eco-Vector2023Eco-VectorЭко-векторEco-Vectorhttps://creativecommons.org/licenses/by-nc-nd/4.0https://jdigitaldiagnostics.com/DD/article/view/321971

BACKGROUND: The global amount of investment in companies developing artificial intelligence (AI)-based software technologies for medical diagnostics reached $80 million in 2016, rose to $152 million in 2017, and is expected to continue growing. While software manufacturing companies should comply with existing clinical, bioethical, legal, and methodological frameworks and standards, there is a lack of uniform national and international standards and protocols for testing and monitoring AI-based software.

AIM: This objective of this study is to develop a universal methodology for testing and monitoring AI-based software for medical diagnostics, with the aim of improving its quality and implementing its integration into practical healthcare.

MATERIALS AND METHODS: The research process involved an analytical phase in which a literature review was conducted on the PubMed and eLibrary databases. The practical stage included the approbation of the developed methodology within the framework of an experiment focused on the use of innovative technologies in the field of computer vision to analyze medical images and further application in the health care system of the city of Moscow.

RESULTS: A methodology for testing and monitoring AI-based software for medical diagnostics has been developed, aimed at improving its quality and introducing it into practical healthcare. The methodology consists of seven stages: self-testing, functional testing, calibration testing, technological monitoring, clinical monitoring, feedback, and refinement.

CONCLUSION: Distinctive features of the methodology include its cyclical stages of monitoring and software development, leading to continuous improvement of its quality, the presence of detailed requirements for the results of the software work, and the participation of doctors in software evaluation. The methodology will allow software developers to achieve significant outcomes and demonstrate achievements across various areas. It also empowers users to make informed and confident choices among software options that have passed an independent and comprehensive quality check.

Обоснование. Мировая сумма инвестиций в компании по разработке программного обеспечения на основе технологий искусственного интеллекта для медицинской диагностики составила 80 млн долларов в 2016 году, 152 млн долларов ― в 2017 и, ожидаемо, продолжает расти. Активная деятельность компаний-производителей программного обеспечения должна соответствовать существующим клиническим, биоэтическим, правовым и методологическим основам и стандартам. Как на национальном, так и на международном уровне не существует единых стандартов и протоколов проведения испытаний и мониторинга программного обеспечения на основе технологий искусственного интеллекта для медицинской диагностики.

Цель ― разработать универсальную методологию тестирования и мониторинга программного обеспечения на основе технологий искусственного интеллекта для медицинской диагностики, направленную на повышение его качества и внедрение в практическое здравоохранение.

Материалы и методы. В ходе аналитического этапа был проведён обзор литературы по базам данных PubMed и eLIBRARY. Практический этап включал апробацию разработанной методологии в рамках Эксперимента по использованию инновационных технологий в области компьютерного зрения для анализа медицинских изображений и дальнейшего применения в системе здравоохранения города Москвы.

Результаты. Разработана методология тестирования и мониторинга программного обеспечения на основе технологий искусственного интеллекта для медицинской диагностики, направленная на повышение качества данного программного обеспечения и его внедрение в практическое здравоохранение. Методология состоит из 7 этапов: самотестирование, функциональное тестирование, калибровочное тестирование, технологический мониторинг, клинический мониторинг, обратная связь и доработка.

Заключение. Отличительными особенностями методологии являются цикличность этапов тестирования, мониторинга и доработки программного обеспечения, приводящие к постоянному повышению его качества, наличие подробных требований к результатам его работы, участие врачей в его оценке. Методология позволит разработчикам программного обеспечения достичь высоких результатов и продемонстрировать достижения в различных направлениях, а пользователям ― сделать осознанный и уверенный выбор среди программ, прошедших независимую и всестороннюю проверку качества.

论证。2016年，全球对基于人工智能技术开发医疗诊断软件的公司的投资额为8000万美元，2017年为1.52亿美元，并预料还将继续增长。软件公司的积极活动必须符合现有的临床、生物伦理、法律和方法学原理和标准。在国家和国际范围，基于人工智能技术的软件还没有统一的测试和监测标准和协议。

该研究的目的是开发一种通用方法，用于测试和监测基于人工智能技术的医疗诊断软件，以提高其质量和在实际医疗中的应用。

材料和方法。在分析阶段，对PubMed和eLIBRARY数据库进行了文献综述。实用阶段包括 在《使用创新计算机视觉技术进行医学图像分析并进一步应用于莫斯科市医疗系统的实验》框架内批准所开发的方法学，并将其进一步应用于莫斯科的医疗保健系统。

结果。我们开发了一套基于人工智能技术的医疗诊断软件测试和监测方法学，旨在提高该软件的质量，并将其应用于实际医疗保健中。该方法学包括7个阶段：自我测试、功能测试、校准测试、技术监测、临床监测、反馈和改进。

结论。该方法学的显著特点是对软件进行周期性的监测和改进，从而不断提高其质量；对软件性能结果并医生参与软件评估提出详细要求。该方法学可使软件开发人员在各个领域取得优异成绩并展示成就，也可使用户在通过独立、全面质量控制的程序中做出明智、自信的选择。

softwareartificial intelligenceradiologydiagnostic imagingmethodologyquality controlпрограммное обеспечениеискусственный интеллектрентгенологиядиагностическая визуализацияметодологияконтроль качества软件人工智能放射学诊断成像方法学质量控制This article was prepared by a group of authors as a part of the research and development effort titled “Development of a platform for improving the quality of AI services for clinical diagnostics”, No. 123031400006-0 in accordance with the Order No. 1196 dated December 21, 2022 “On approval of state assignments funded by means of allocations from the budget of the city of Moscow to the state budgetary (autonomous) institutions subordinate to the Moscow Health Care Department, for 2023 and the planned period of 2024 and 2025” issued by the Moscow Health Care DepartmentДанная статья подготовлена авторским коллективом в рамках работы № ЕГИСУ: «Разработка платформы повышения качества ИИ-Сервисов для медицинской диагностики», № 123031400006-0, в соответствии с Приказом Департамента здравоохранения города Москвы от 21.12.2022 № 1196 «Об утверждении государственных заданий, финансовое обеспечение которых осуществляется за счёт средств бюджета города Москвы государственным бюджетным (автономным) учреждениям, подведомственным Департаменту здравоохранения города Москвы, на 2023 год и плановый период 2024 и 2025 годов»This article was prepared by a group of authors as a part of the research and development effort titled “Development of a platform for improving the quality of AI services for clinical diagnostics”, No. 123031400006-0 in accordance with the Order No. 1196 dated December 21, 2022 “On approval of state assignments funded by means of allocations from the budget of the city of Moscow to the state budgetary (autonomous) institutions subordinate to the Moscow Health Care Department, for 2023 and the planned period of 2024 and 2025” issued by the Moscow Health Care DepartmentOakden-Rayner L, Palme LJ. Artificial intelligence in medicine: Validation and study design. In: Ranschart E, Morozov S, Algra P, eds. Artificial intelligence in medical imaging. Cham: Springer; 2019. Р. 83–104.Oakden-Rayner L., Palme L.J. Artificial intelligence in medicine: Validation and study design. In: Ranschart E., Morozov S., Algra P., eds. Artificial intelligence in medical imaging. Cham: Springer, 2019. Р. 83–104.Oakden-Rayner L, Palme LJ. Artificial intelligence in medicine: Validation and study design. In: Ranschart E, Morozov S, Algra P, eds. Artificial intelligence in medical imaging. Cham: Springer; 2019. Р. 83–104.Morozov SP, Zinchenko VV, Khoruzhaya AN, et al. Standardization of artificial intelligence in healthcare: Russia is becoming a leader. Doctor Inform Technol. 2021;(2):12–19. (In Russ). doi: 10.25881/18110193_2021_2_12Морозов С.П., Зинченко В.В., Хоружая А.Н., и др. Стандартизация искусственного интеллекта в здравоохранении: Россия выходит в лидеры // Врач и информационные технологии. 2021. № 2. С. 12–19. doi: 10.25881/18110193_2021_2_12Morozov SP, Zinchenko VV, Khoruzhaya AN, et al. Standardization of artificial intelligence in healthcare: Russia is becoming a leader. Doctor Inform Technol. 2021;(2):12–19. (In Russ). doi: 10.25881/18110193_2021_2_12Mello AA, Utkin LV, Trofimova TN. Artificial intelligence in medicine: The current state and main directions of development of intellectual diagnostics. Radiation Diagnost Therapy. 2020;(1):9–17. (In Russ). doi: 10.22328/2079-5343-2020-11-1-9-17Мелдо А.А., Уткин Л.В., Трофимова Т.Н. Искусственный интеллект в медицине: современное состояние и основные направления развития интеллектуальной диагностики // Лучевая диагностика и терапия. 2020. № 1. С. 9–17. doi: 10.22328/2079-5343-2020-11-1-9-17Mello AA, Utkin LV, Trofimova TN. Artificial intelligence in medicine: The current state and main directions of development of intellectual diagnostics. Radiation Diagnost Therapy. 2020;(1):9–17. (In Russ). doi: 10.22328/2079-5343-2020-11-1-9-17Zinchenko VV, Arzamasov KM, Chetverikov SF, et al. Methodology of post-registration clinical monitoring for software using artificial intelligence technologies. Modern Technol Med. 2022;14(5):15–25. (In Russ). doi: 10.17691/stm2022.14.5.02Зинченко В.В., Арзамасов К.М., Четвериков С.Ф., и др. Методология проведения пострегистрационного клинического мониторинга для программного обеспечения с применением технологий искусственного интеллекта // Современные технологии в медицине. 2022. Т. 14, № 5. С. 15–25. doi: 10.17691/stm2022.14.5.02Zinchenko VV, Arzamasov KM, Chetverikov SF, et al. Methodology of post-registration clinical monitoring for software using artificial intelligence technologies. Modern Technol Med. 2022;14(5):15–25. (In Russ). doi: 10.17691/stm2022.14.5.02Tanguay W, Acar P, Fine B, et al. Assessment of radiology artificial intelligence software: A validation and evaluation framework. Can Assoc Radiol J. 2023;74(2):326–333. doi: 10.1177/08465371221135760Tanguay W., Acar P., Fine B., et al. Assessment of radiology artificial intelligence software: A validation and evaluation framework // Can Assoc Radiol J. 2023. Vol. 74, N 2. Р. 326–333. doi: 10.1177/08465371221135760Tanguay W, Acar P, Fine B, et al. Assessment of radiology artificial intelligence software: A validation and evaluation framework. Can Assoc Radiol J. 2023;74(2):326–333. doi: 10.1177/08465371221135760Kohli A, Jha S. Why CAD failed in mammography. J Am Coll Radiol. 2018;15(3 Pt B):535–537. doi: 10.1016/j.jacr.2017.12.029Kohli A., Jha S. Why CAD failed in mammography // J Am Coll Radiol. 2018. Vol. 15, N 3, Pt. B. Р. 535–537. doi: 10.1016/j.jacr.2017.12.029Kohli A, Jha S. Why CAD failed in mammography. J Am Coll Radiol. 2018;15(3 Pt B):535–537. doi: 10.1016/j.jacr.2017.12.029Recht MP, Dewey M, Dreyer K, et al. Integrating artificial intelligence into the clinical practice of radiology: Challenges and recommendations. Eur Radiol. 2020;30(6):3576–3584. doi: 10.1007/s00330-020-06672-5Recht M.P., Dewey M., Dreyer K., et al. Integrating artificial intelligence into the clinical practice of radiology: Challenges and recommendations // Eur Radiol. 2020. Vol. 30, N 6. Р. 3576–3584. doi: 10.1007/s00330-020-06672-5Recht MP, Dewey M, Dreyer K, et al. Integrating artificial intelligence into the clinical practice of radiology: Challenges and recommendations. Eur Radiol. 2020;30(6):3576–3584. doi: 10.1007/s00330-020-06672-5Higgins DC, Johner C. Validation of artificial intelligence containing products across the regulated healthcare industries. Ther Innov Regul Sci. 2023;57(4):797–809. doi: 10.1007/s43441-023-00530-4Higgins D.C., Johner C. Validation of artificial intelligence containing products across the regulated healthcare industries // Ther Innov Regul Sci. 2023. Vol. 57, N 4. Р. 797–809. doi: 10.1007/s43441-023-00530-4Higgins DC, Johner C. Validation of artificial intelligence containing products across the regulated healthcare industries. Ther Innov Regul Sci. 2023;57(4):797–809. doi: 10.1007/s43441-023-00530-4Rudolph J, Schachtner B, Fink N, et al. Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis. Sci Rep. 2022;12(1):12764. doi: 10.1038/s41598-022-16514-7Rudolph J., Schachtner B., Fink N., et al. Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis // Sci Rep. 2022. Vol. 12, N 1. Р. 12764. doi: 10.1038/s41598-022-16514-7Rudolph J, Schachtner B, Fink N, et al. Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis. Sci Rep. 2022;12(1):12764. doi: 10.1038/s41598-022-16514-7Allen B, Dreyer K, Stibolt R, et al. Evaluation and real-world performance monitoring of artificial intelligence models in clinical practice: Try it, buy it, check it. J Am Coll Radiol. 2021;18(11):1489–1496. doi: 10.1016/j.jacr.2021.08.022Allen B., Dreyer K., Stibolt R., et al. Evaluation and real-world performance monitoring of artificial intelligence models in clinical practice: Try it, buy it, check it // J Am Coll Radiol. 2021. Vol. 18, N 11. Р. 1489–1496. doi: 10.1016/j.jacr.2021.08.022Allen B, Dreyer K, Stibolt R, et al. Evaluation and real-world performance monitoring of artificial intelligence models in clinical practice: Try it, buy it, check it. J Am Coll Radiol. 2021;18(11):1489–1496. doi: 10.1016/j.jacr.2021.08.022Strohm L, Hehakaya C, Ranschaert ER, et al. Implementation of artificial intelligence (AI) applications in radiology: Hindering and facilitating factors. Eur Radiol. 2020;30(10):5525–5532. doi: 10.1007/s00330-020-06946-yStrohm L., Hehakaya C., Ranschaert E.R., et al. Implementation of artificial intelligence (AI) applications in radiology: Hindering and facilitating factors // Eur Radiol. 2020. Vol. 30, N 10. Р. 5525–5532. doi: 10.1007/s00330-020-06946-yStrohm L, Hehakaya C, Ranschaert ER, et al. Implementation of artificial intelligence (AI) applications in radiology: Hindering and facilitating factors. Eur Radiol. 2020;30(10):5525–5532. doi: 10.1007/s00330-020-06946-ySohn JH, Chillakuru YR, Lee S, et al. An open-source, vender agnostic hardware and software pipeline for integration of artificial intelligence in radiology workflow. J Digit Imaging. 2020;33(4):1041–1046. doi: 10.1007/s10278-020-00348-8Sohn J.H., Chillakuru Y.R., Lee S., et al. An open-source, vender agnostic hardware and software pipeline for integration of artificial intelligence in radiology workflow // J Digit Imaging. 2020. Vol. 33, N 4. Р. 1041–1046. doi: 10.1007/s10278-020-00348-8Sohn JH, Chillakuru YR, Lee S, et al. An open-source, vender agnostic hardware and software pipeline for integration of artificial intelligence in radiology workflow. J Digit Imaging. 2020;33(4):1041–1046. doi: 10.1007/s10278-020-00348-8Wichmann JL, Willemink MJ, De Cecco CN. Artificial intelligence and machine learning in radiology: Current state and considerations for routine clinical implementation. Invest Radiol. 2020;55(9):619–627. doi: 10.1097/RLI.0000000000000673Wichmann J.L., Willemink M.J., De Cecco C.N. Artificial intelligence and machine learning in radiology: Current state and considerations for routine clinical implementation // Invest Radiol. 2020. Vol. 55, N 9. Р. 619–627. doi: 10.1097/RLI.0000000000000673Wichmann JL, Willemink MJ, De Cecco CN. Artificial intelligence and machine learning in radiology: Current state and considerations for routine clinical implementation. Invest Radiol. 2020;55(9):619–627. doi: 10.1097/RLI.0000000000000673Larson DB, Harvey H, Rubin DL, et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: Summary and recommendations. J Am Coll Radiol. 2021;18(3 Pt A):413–424. doi: 10.1016/j.jacr.2020.09.060Larson D.B., Harvey H., Rubin D.L., et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: Summary and recommendations // J Am Coll Radiol. 2021. Vol. 18, N 3, Pt. A. Р. 413–424. doi: 10.1016/j.jacr.2020.09.060Larson DB, Harvey H, Rubin DL, et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: Summary and recommendations. J Am Coll Radiol. 2021;18(3 Pt A):413–424. doi: 10.1016/j.jacr.2020.09.060Milam ME, Koo CW. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States. Clin Radiol. 2023;78(2):115–122. doi: 10.1016/j.crad.2022.08.135Milam M.E., Koo C.W. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States // Clin Radiol. 2023. Vol. 78, N 2. Р. 115–122. doi: 10.1016/j.crad.2022.08.135Milam ME, Koo CW. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States. Clin Radiol. 2023;78(2):115–122. doi: 10.1016/j.crad.2022.08.135De Silva D, Alahakoon D. An artificial intelligence life cycle: From conception to production. Patterns (NY). 2022;3(6):100489. doi: 10.1016/j.patter.2022.100489De Silva D., Alahakoon D. An artificial intelligence life cycle: From conception to production // Patterns (NY). 2022. Vol. 3, N 6. Р. 100489. doi: 10.1016/j.patter.2022.100489De Silva D, Alahakoon D. An artificial intelligence life cycle: From conception to production. Patterns (NY). 2022;3(6):100489. doi: 10.1016/j.patter.2022.100489Cerdá-Alberich L, Solana J, Mallol P, et al. MAIC-10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging. 2023;14(1):11. doi: 10.1186/s13244-022-01355-9Cerdá-Alberich L., Solana J., Mallol P., et al. MAIC-10 brief quality checklist for publications using artificial intelligence and medical images // Insights Imaging. 2023. Vol. 14, N 1. Р. 11. doi: 10.1186/s13244-022-01355-9Cerdá-Alberich L, Solana J, Mallol P, et al. MAIC-10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging. 2023;14(1):11. doi: 10.1186/s13244-022-01355-9Vasey B, Novak A, Ather S, et al. DECIDE-AI: A new reporting guideline and its relevance to artificial intelligence studies in radiology. Clin Radiol. 2023;78(2):130–136. doi: 10.1016/j.crad.2022.09.131Vasey B., Novak A., Ather S., et al. DECIDE-AI: A new reporting guideline and its relevance to artificial intelligence studies in radiology // Clin Radiol. 2023. Vol. 78, N 2. Р. 130–136. doi: 10.1016/j.crad.2022.09.131Vasey B, Novak A, Ather S, et al. DECIDE-AI: A new reporting guideline and its relevance to artificial intelligence studies in radiology. Clin Radiol. 2023;78(2):130–136. doi: 10.1016/j.crad.2022.09.131Regulations for the preparation of data sets with a description of approaches to the formation of a representative sample of data. Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2022. 40 p. (Best practices in radiological and instrumental diagnostics; Part 1). (In Russ).Регламент подготовки наборов данных с описанием подходов к формированию репрезентативной выборки данных. Москва: Научно-практический клинический центр диагностики и телемедицинских технологий Департамента здравоохранения города Москвы, 2022. 40 с. (Лучшие практики лучевой и инструментальной диагностики; Часть 1).Regulations for the preparation of data sets with a description of approaches to the formation of a representative sample of data. Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2022. 40 p. (Best practices in radiological and instrumental diagnostics; Part 1). (In Russ).Chetverikov S, Arzamasov KM, Andreichenko AE, et al. Approaches to sampling for quality control of artificial intelligence systems in biomedical research. Modern Technol Med. 2023;15(2):19–27. (In Russ). doi: 10.17691/stm2023.15.2.02Четвериков С.Ф., Арзамасов К.М., Андрейченко А.Е., и др. Подходы к формированию выборки для контроля качества работы систем искусственного интеллекта в медико-биологических исследованиях // Современные технологии в медицине. 2023. Т. 15, № 2. С. 19–27. doi: 10.17691/stm2023.15.2.02Chetverikov S, Arzamasov KM, Andreichenko AE, et al. Approaches to sampling for quality control of artificial intelligence systems in biomedical research. Modern Technol Med. 2023;15(2):19–27. (In Russ). doi: 10.17691/stm2023.15.2.02Morozov SP, Vladzimirsky AV, Klyashtorny VG, et al. Clinical trials of software based on intelligent technologies (radiation diagnostics). Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2019. 33 р. (In Russ).Морозов С.П., Владзимирский А.В., Кляшторный В.Г., и др. Клинические испытания программного обеспечения на основе интеллектуальных технологий (лучевая диагностика). Москва: Научно-практический клинический центр диагностики и телемедицинских технологий Департамента здравоохранения города Москвы, 2019. 33 с.Morozov SP, Vladzimirsky AV, Klyashtorny VG, et al. Clinical trials of software based on intelligent technologies (radiation diagnostics). Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2019. 33 р. (In Russ).Kim DW, Jang HY, Kim KW, et al. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: Results from recently published papers. Korean J Radiol. 2019;20(3):405–410. doi: 10.3348/kjr.2019.0025Kim D.W., Jang H.Y., Kim K.W., et al. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: Results from recently published papers // Korean J Radiol. 2019. Vol. 20, N 3. Р. 405–410. doi: 10.3348/kjr.2019.0025Kim DW, Jang HY, Kim KW, et al. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: Results from recently published papers. Korean J Radiol. 2019;20(3):405–410. doi: 10.3348/kjr.2019.0025