Analysis of the diagnostic and economic impact of the combined artificial intelligence algorithm for analysis of 10 pathological findings on chest computed tomography

Full Text

List of abbreviations

DATA SET: a data set, a collection of logical records

CI: confidence interval

AI: artificial intelligence

CT: computed tomography

ICD-10: International Statistical Classification of Diseases and Related Health Problems, Tenth Edition

MRI: magnetic resonance imaging

CNMS: cost of non-provided medical services

BACKGROUND

According to the World Health Organization, most deaths are associated with cardiovascular diseases (CVDs), infections, lung diseases, and cancers.¹ Based on large randomized trials of lung cancer screening, the use of chest low-dose computed tomography in asymptomatic patients at risk resulted in a 6.7% decrease in mortality from lung cancer and from all causes in the National Lung Screening Trial (USA) and in a 39% decrease in mortality during follow-up in the Year 5–10 of a Multicentric Italian Lung Detection study (Italy) due to incidental detection of clinically significant findings and treatment and prevention of relevant diseases [1, 2].

Lung cancer screening programs are cost effective in high-risk patients. This effect varies across healthcare systems in different countries [3]. Moreover, these programs have significant differences in lung cancer mortality and total mortality. In one of the lung cancer screening studies, 77.1% of patients died not from the disease itself but from other causes such as CVDs, lung diseases, other tumors, and infectious diseases [1]. By focusing on lung cancer searching, a radiologist may miss abnormal findings associated with other diseases. For example, during lung cancer screening, 58% of clinically significant findings are not reflected in the radiologist protocol texts [4].

During the coronavirus disease-2019 (COVID-19) pandemic, lung cancer screening programs were suspended because computed tomography (CT) scanners are required to perform mass chest CT to diagnose COVID-19. In half of the patients who underwent chest CT, incidental findings were detected, and 1/4 had clinically significant findings [5]. Chest CT data allow diagnosis of diseases in the lungs, other organs, and systems [6-8]. Medical personnel shortage, burnout, pandemic effects, and increased workload can lead to missing clinically significant findings.

Artificial intelligence (AI) is the most promising technology for solving this problem; thus, assessing economic benefits of such innovations is necessary. Numerous AI healthcare products were developed for diagnostic radiology, and the number of such solutions exceeds the number of other medical AI services in several times.² In Russia, the largest AI imaging project is related to an experiment on the use of innovative computer vision technologies for evaluating medical images and further use of the Moscow healthcare system (hereinafter referred to as the Moscow experiment). For this project, more than 7.5 million imaging examinations were performed, including X-ray imaging, mammography, and CT [9].³

Briefly, the use of AI single-pathology algorithms is of limited practical value for controlling diseases that are among the leading causes of death worldwide. Given the need for simultaneous detection of several pathologies using AI, the first software products have been developed for a comprehensive analysis of chest CT. They have passed all testing stages and have been approved for prospective use in 153 Moscow healthcare organizations.⁴ One of these products is a multi-IRA combined AI service by IRA Labs, which can simultaneously search for 10 abnormal signs of various diseases during CT [10-13], including the following:

1) Infiltrative lung lesions typical for viral pneumonia (COVID-19 pandemic; U07 according to ICD-10), with calculation of the percentage of lung damage.

2) Lung nodules with assessing their size, volume, and density to detect lung malignancies (ICD-10 code C34).

3) Free pleural fluid (effusion) (ICD-10 code J94).

4) Pulmonary emphysema as a sign of chronic obstructive pulmonary disease (ICD-10 code J44).

5) Measuring thoracic aorta diameter to detect aorta dilatation and aneurysms (ICD-10 codes I70 and I71).

6) Measuring the diameter of the pulmonary artery trunk to detect potential causes of pulmonary hypertension (ICD-10 code I27).

7) Evaluating the extent of coronary calcification using the Agatston score to assess the severity of coronary atherosclerosis and the risk of coronary heart disease (ICD-10 codes I20–I25).

8) Measuring the adrenal gland size to detect lesions and hyperplasia (ICD-10 code C74).

9) Measuring vertebral body heights for diagnosing compression fractures (ICD-10 codes M80–M85).

10) Analyzing the density of the vertebral bodies to detect signs of osteoporosis (ICD-10 codes M80–M85).

This study aimed to evaluate the frequency of significant abnormal findings and the economic potential of using combined AI technologies in the analysis of chest CT, validated by radiologists, compared with analysis performed radiologists without AI access in a private medical center.

MATERIALS AND METHODS

Study design

In this observational single-center retrospective study, patient informed consent was not required. The study was prepared in accordance with the CHEERS 2022 checklist for the economic evaluation of medical studies [14].⁵ An economic analysis plan was developed for a private medical center. It included the assessment of the potential additional costs of non-provided medical services (CNMS) that should be provided to patients with certain abnormal findings according to clinical guidelines and best practices of evidence-based medicine. The use of combined AI services added to CNMS resulted from radiologist activities, through further diagnostic actions to clarify the nature and severity of CT findings.

Treatment cost was not assessed in this study. The study design is presented in Fig. 1.

 

Fig. 1. Study design.

 

Eligibility criteria

Inclusion and exclusion criteria were used to form the study group.

Inclusion criteria: Chest CT in men and women who got medical assistance in a primary and specialized care organization for adult population of Moscow; chest CT performed and interpreted by radiologists between June 01, 2022, and July 31, 2022; chest CT performed without intravenous contrast enhancement; patients aged >18 years; availability of CT scans in DICOM format and protocol texts prepared by radiologists in the clinic; and the patient’s first visit to a clinic.

Exclusion criteria: age >85 years; previous chest CT within 1 year; AI could not process scans because of reasons out of its control (e.g., inappropriate modality, scan region other than the chest, insufficient number of slices, i.e., <30); AI could not process scans due to reasons related to algorithm features (e.g., incorrect work due to the presence of significant metal artifacts at the scan level).

Conditions of the study

CT was performed at the Clinical Hospital on Yauza LLC, which is a multidisciplinary private healthcare organization providing primary and specialized care to the adult population of Moscow.

Study duration

The study was conducted using CT performed between June 1, 2022, and July 31, 2022. Results were retrospectively analyzed using the AI algorithm and verified by experts between October 01, 2022, and November 30, 2022.

Description of the study

Chest CT was performed without intravenous contrast enhancement using a Philips Ingenuity CT scanner. This tomograph performed 128 slices per tube rotation. The chest CT protocol was performed in accordance with standard equipment manufacturer recommendations and national guidelines. CT findings were provided to experts and AI in two series reconstructed with a soft tissue kernel (60 HU for the center of the window, 360 HU for the width of the window) and a pulmonary kernel (−500 HU for the center of the window and 1,500 HU for the width of the window). The slice thickness was 1.0 mm. Iterative model reconstruction algorithms were used to improve image quality (reduce noise) and reduce the radiation dose to the patient.

All the included CT examinations were processed using multi-IRA combined AI software (IRA Labs) integrated into the clinic’s picture archiving and communication system. The AI algorithms used in this study were previously tested on specially prepared calibration datasets as part of the Moscow AI experiment.⁶

The criterion for the possibility of AI use was algorithm accuracy not lower than the area under the ROC curve (ROC AUC) of 0.81 for each pathological finding, according to the guidelines for clinical trials of software based on intelligent technologies [15]. The diagnostic accuracy metrics for AI algorithms based on developer-independent closed datasets of the Moscow experiment are presented in Table 1 [9, 10].

 

Table 1. Diagnostic accuracy metrics for a combined AI solution for chest CT, based on the datasets of the Moscow experiment

Name of the multi-IRA AI algorithm for certain pathologies	ROC AUC	Sensitivity	Specificity	Accuracy
COVID-IRA (detection of lung infiltrative lesions)	0.98	0.95	0.94	0.94
LungNodule-IRA (detection of lung nodules)	0.932	0.86	0.9	0.88
PleuralEffusion-IRA (detection of pleural effusion)	0.999	0.98	1	0.99
Aorta-IRA (chest) (evaluation of the thoracic aorta diameter)	0.997	0.96	1	0.98
Aorta-IRA (chest + abdomen) (evaluation of the thoracic and abdominal aorta diameters)	1	0.98	1	0.99
PulmTrunk-IRA (evaluation of the pulmonary trunk diameter)	1	1	0.98	0.99
Agatston-IRA (evaluation of the Agatston coronary calcification score)	0.986	0.96	0.96	0.96
Genant-IRA (evaluation of the height of vertebral bodies)	0.995	1	0.98	0.99
Emphysema-IRA (detection of emphysema)	0.989	0.94	0.98	0.96
Adrenal-IRA (assessment of the adrenal size for masses and hyperplasia)	0.96	1	0.96	0.98

 

Primary study outcome

For all the findings detected and missed by physicians in the clinic, “second stages” were determined (consultations with specialists and various types of additional clinical, instrumental, and laboratory examinations), i.e., further routing of the patient in accordance with current clinical guidelines for each pathological finding.

Then, for each patient, the CNMS was calculated according to the price list of the clinic, which was determined based on non-provided medical services, required according to clinical guidelines for missed pathological findings. The CNMS was also calculated for the missed significant pathological findings, as shown in Table 2 [16-31]. For radiologists, abnormal findings were considered missed if it was not reported in the final CT descriptions in the electronic medical information system of the clinic provided that such information was available according to true-positive results (ground truth) after reviewing CT data by an expert and AI analysis. Findings were true positive if they were detected by the AI algorithm and confirmed by two medical experts analyzing CT.

Additional study outcomes

The number of protocols with significant and insignificant missed pathological findings and the percentage of erroneous protocols were calculated for each clinic radiologist.

Subgroup analysis

To eliminate erroneous program triggering, two experts (radiologists with 10 and 13 years of experience, respectively, who are not employees of the clinic that provided data for the study) reviewed CT scans with results of AI processing. In the case of controversial opinions, a single option was selected after a collective discussion. As a result of this analysis, the true-positive results of the AI analysis, confirmed by two experts, were selected. Then, AI findings were compared with the final chest CT descriptions prepared by radiologists in the clinic without using AI, and some cases of missed pathological findings were identified. All pathological findings missed by radiologists in the clinic were divided into significant and insignificant. The significance criteria for pathological findings were evaluated in accordance with basic diagnostic requirements for AI analysis in the Moscow experiment, agreed by the Commission for Scientific Problems of the Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Healthcare of Moscow (Protocol No. 9/2021 dated December 10, 2021, No. 1/2022 dated February 28, 2022, No. 7/2022 dated December 06, 2022, No. 1/2023 dated January 13, 2023). These requirements are based on clinical guidelines and evidence-based best practices.

In this study, protocol texts were not evaluated for the presence or absence of the “epicardial fat” term because the radiologists in the clinic did not have tools to measure the volume of adipose tissue. The criteria for pathological findings and distribution by the missed pathological finding significance are presented in Table 2.

 

Table 2. Criteria for abnormal findings and omissions

Health condition	Criteria for pathological findings	Criteria for skipping significant pathological findings	Criteria for skipping non-significant pathological findings
Health condition	Criteria for pathological findings	Criteria for skipping significant pathological findings	Criteria for skipping non-significant pathological findings
Lung nodules	At least one solid or subsolid (only the solid component is measured) nodule with an average size of ≥6 mm (volume ≥100 mm3) [16]	All findings for this condition that met the described criteria were considered significant, excluding perifissural lung nodules. Nodules along the pleura are benign and do not require further observations.
Infiltrative lung changes typical for viral pneumonia (COVID-19) in a pandemic	1. Bilateral frosted glass infiltration of the lung parenchyma with peripheral predominance, with, or without consolidation infiltration of the lung parenchyma with a positive sign of an air bronchogram. 2. Bilateral cobblestone infiltration of the lung parenchyma (thickening of the interlobular interstitium with frosted glass), with peripheral predominance, with, or without consolidation infiltration of the lung parenchyma with a positive sign of an air bronchogram [17, 18]	Lung involvement >50%	Lung involvement <50%
Lung emphysema	In total ≥6% (by volume) of areas in both lungs (excluding bronchial lumen) with CT density ≤-950 HU [19, 20]	All findings for this condition that met the described criteria were considered significant.
Free pleural fluid (effusion)	A crescentic liquid accumulation (effusion) in the pleural cavity with a density of 0–30 HU in the gravity-dependent chest regions [21]	Maximum layer thickness >10 mm	Maximum layer thickness <10mm
Aneurysm / aorta dilatation	1. Dilatation of the ascending thoracic aorta: on native scans, the largest axial diameter of the ascending thoracic aorta is 40 to ≤49 mm. 2. Aneurysm of the ascending thoracic aorta: on native scans, the largest axial diameter of the ascending thoracic aorta is ≥50 mm. 3. Dilatation of the descending thoracic aorta: on native scans, the largest axial diameter of the descending thoracic aorta ranges from 31 to 39 mm. 4. Aneurysm of the descending thoracic aorta: on native scans, the largest axial diameter of the descending thoracic aorta is ≥40 mm.# [22] 5. Dilatation of the abdominal aorta: the largest diameter is 25–≤29 mm. 6. Aneurysm of abdominal aorta: the largest diameter ≥30 mm [23, 24]	Aneurysm of the ascending thoracic aorta (diameter ≥50 mm) and descending thoracic aorta (diameter ≥40 mm). Aneurysm of the abdominal aorta (diameter ≥30 mm).	Dilatation of the ascending thoracic aorta (diameter of 40–49 mm) and descending thoracic aorta (diameter of 31–39 mm). Dilatation of the abdominal aorta (diameter ≥30 mm).
Dilatation of the pulmonary trunk	Diameter of the pulmonary trunk ≥29 mm## [25]	>29 mm	29 mm
Coronary calcification by Agatston score	On native scans, the Agatston score (the sum of areas in the projection of coronary vessels, multiplied by individual density factors*) is ≥1, or CAC-DRS class A1–A3. [26, 27] * Factor 1: 130–199 HU Factor 2: 200–299 HU Factor 3: 300–399 HU Factor 4: ≥400 HU	Agatston score >10	Agatston score 1–10
Measurement of the adrenal size for masses and hyperplasia	Nodular formations of the adrenal body or pedicles with a short-axial diameter measuring≥10 mm [28]	All findings for this condition that met the described criteria are considered significant.
Evaluation of the density of the spongy substance of vertebral bodies for the detection of osteoporosis/ osteopenia	Decrease in bone mineral density in Th11–L3 (optimally L1–L2) vertebral bodies according to ACR 2018 criteria, ISCD 2019 statement [29]	Density <+100 HU	Density +100 to +150 HU
Vertebral compression fractures related to osteoporosis	Vertebrae with compressive body deformity ≥25% according to the Genant semi-quantitative scale, grades II–III [30, 31]. The deformation grade (DG) is calculated by the following formula: DG = (ratio of the maximum vertebral body size - minimum/maximum size) × 100%	All findings for this condition that met the described criteria were considered significant.

 

Ethical review

The Independent Ethical Committee of the Moscow Regional Branch of the Russian Society of Radiologists was notified about this this retrospective study (minutes dated March 01, 2023).

Statistical analysis

Results were presented using descriptive statistics methods, and the absolute number (n) and percentage (%) of observations were indicated for each category. Frequencies of pathological finding detection by different methods were compared using the Z-test for proportions. The p-values obtained for each of the nine pathological findings were adjusted for multiple testing (for general hypothesis of the absence of a statistically significant difference in diagnostic results) by the Bonferroni correction. Financial parameters were analyzed using a paired t-test. The level of statistical significance for p was 0.05. The statistical analysis was performed using the R program, version 4.1.3.

RESULTS

Study subjects

A total of 185 examinations were selected (male-to-female ratio: 47%:53%; age, 19–83 years; mean age, 49.5 years) in accordance with established criteria. Of them, 25 examinations were not processed by AI for the following reasons:

1) AI-independent (23 examinations).

• Inappropriate modality (9 examinations: 7 MRIs; 2 mammographies).
• Scan region other than the chest (9 examinations).
• Insufficient number of slices (<30), including localizers (5 examinations).

2) AI-dependent (2 examinations)

• Incorrect operation due to the presence of significant metal artifacts at the scan level (2 examinations).

The final group for analysis consisted of 160 chest CT scans with descriptions prepared by radiologists. Additional information on the presence of oncological, cardiovascular, and other chronic diseases in patients was not collected because patients visited this clinic only under compulsory or voluntary medical insurance policies and were on a paid basis to consult specialists. The medical information system about established diagnoses highly likely contained incomplete information.

Primary study outcomes

Automatic processes were set up to anonymize and transfer CT scans from the clinic to the developer of combined AI for chest CT and return the analysis results to the clinic and experts for AI validation. For quality control, experts provided a list of all discrepancies between the verified AI results and the text protocols to the clinic staff (Fig 2). No claims were obtained from the clinic. The largest number of clinically significant omissions in the protocol texts was found for osteoporosis and adrenal masses (n = 14 each). The largest number of insignificant omissions was recorded in aortic dilatation (n = 36) and osteopenia (n = 40). Detailed data on number of findings are presented in Fig. 3.

 

Fig. 2. Study result by the number of findings detected with and without AI.

 

Fig. 3. Number of findings (ranked by the number of significant missed pathological findings).

 

With AI, 90 (56%) examinations with pathological findings were identified, and at least one pathological finding was missed in 81 (51%) radiologist protocols. In 70 studies, the AI algorithm did not detect any pathological findings. In this study, examinations presented could have other abnormal findings that were not included into the AI validation program. Analysis results are summarized in Table 3.

 

Table 3. Summary data on the number of patients with detected and missed pathological findings

Parameter	Number	Total cases, %	Pathological findings assessment, %
Total patients	160	100	-
Without pathological findings	70	44	-
With pathological findings	90	56	100
A radiologist in the clinic has detected at least one pathological finding of those detected by AI.	35	22	39
A radiologist in the clinic has missed at least one pathological finding of those detected by AI.	81	51	90
Computed tomography examinations in which abnormal changes were detected only by an AI algorithm.	55	34	61

 

A CT could reveal several pathological findings; some were detected by a radiologist in the clinic (and they were described in the protocol), and some were detected only by AI (potential benefit of AI).

A detailed CNMS analysis for missed pathological findings in every 90 patients is presented in Fig. 4. When comparing two diagnostic approaches (algorithm + expert verification vs. expert without AI), some statistically significant differences were obtained for the following abnormal findings: aortic aneurysms/dilatations, increased diameter of the pulmonary trunk, Agatston score, vertebral compression fractures, decreased density of the vertebral bodies, and adrenal gland thickening (Table 4).

 

Fig. 4. Analysis of the cost of medical services not provided because of missed pathological findings, and all CT scans performed.

 

Table 4. Comparison of two diagnostic approaches to detect pathological findings

Parameters	Number of cases detected	Percentage [95% CI]	95% CI for proportion difference	р Z-test H0: proportions are equal (Bonferonni adjustment)
Parameters	Number of cases detected	Percentage [95% CI]	95% CI for proportion difference	р Z-test H0: proportions are equal (Bonferonni adjustment)
Signs of COVID-19 according to CT
Algorithm + Expert	3	0,016 [0, 003; 0, 047]	[-0,008; 0,04]	0,246 (1)
Protocol	0	0 [0; 0, 02]	-	-
Lung nodules
Algorithm + Expert	15	0,082 [0, 047; 0, 132]	[-0,036; 0,08]	0,542 (1)
Protocol	11	0,06 [0, 031; 0, 106]	-	-
Effusion
Algorithm + Expert	6	0,033 [0, 012; 0, 07]	[-0,028; 0,05]	0,749 (1)
Protocol	4	0,022 [0, 006; 0, 055]	-	-
Aorta
Algorithm + Expert	40	0,22 [0, 162; 0, 287]	[0, 141; 0, 276]	<0,001 (<0,001)
Protocol	2	0,011 [0, 001; 0, 039]	-	-
Pulmonary trunk
Algorithm + Expert	16	0,088 [0, 051; 0, 139]	[0, 021; 0, 122]	0,005 (0,042)
Protocol	3	0,016 [0, 003; 0, 047]	-	-
Coronary calcification
Algorithm + Expert	29	0,159 [0, 109; 0, 221]	[0, 043; 0, 177]	0,001 (0,01)
Protocol	9	0,049 [0, 023; 0, 092]	-	-
Vertebral compression fractures
Algorithm + Expert	18	0,1 [0, 06; 0, 152]	[0, 03; 0, 135]	0,002 (0,015)
Protocol	3	0,016 [0, 003; 0, 047]	-	-
Assessment of the mineral density of the vertebral bodies
Algorithm + Expert	74	0,407 [0, 335; 0, 482]	[0, 207; 0, 387]	<0,001 (<0,001)
Protocol	20	0,11 [0, 068; 0, 165]	-	-
Thickening of adrenal glands
Algorithm + Expert	26	0,143 [0, 095; 0, 202]	[0, 06; 0, 181]	<0,001 (0,001)
Protocol	4	0,022 [0, 006; 0, 055]	-	-

 

The total estimated “second step” CNMS for all missed pathological findings in 81 patients was RUB 2,847,760 ($ 37,250.99 or CNY 256,217.95) with RUB 17,799 ($232.83 or CNY 1601.41) per patient. The cost of the total “second step” CNMS for pathological findings missed by radiologists in the clinic, but detected by AI, and confirmed by experts, was RUB 2,065,360 ($ 27,016.57 or CNY 185,824.05) with RUB 12,909 ($168.86 or CNY 1,161.45) per patient. The results of CNMS calculations for all findings are presented in Table 5.

 

Table 5. Analysis of the estimated cost of non-provided medical services from all missed CT findings

Parameter	Calculation for all patients of the pilot project (rubles/US dollars/renminbi)				Calculation per one patient (rubles/US dollars/renminbi)
	Cost
	Total (all)	Consultations (all)	Additional examinations (all)	Follow-up (all)	Total (all)	Consultations (all)	Additional examinations (all)	Follow-up (all)
Cost of the “second step” of diagnostics for all pathological findings	2 847 760/ 37 250,99/ 256 217,95	463 300/ 6060,34/ 41 683,91	2 049 760/ 26 812,51/ 184 420,49	334 700/ 4378,15/ 30 113,54	17 799/ 232,83/ 1601,41	2896/ 37,88/ 260,56	12 811/ 167,58/ 1152,63	2092/ 27,37/ 188,22
Cost of the second step only for those pathological findings that were missed by a radiologist and detected by the AI algorithm	2 065 360/ 27 016,57/ 185 824,05	326 800/ 4274,81/ 29 402,77	1 519 460/ 19 875,76/ 136 708,47	219 100/ 2866/ 19 712,81	12 909/ 168,86/ 1161,45	2043/ 26,72/ 183,81	9497/ 124,23/ 854,46	1369/ 17,91/ 123,17
Cost of the second step for pathological findings detected by radiologists	782 400/ 10 234,42/ 70 393,9	136 500/ 1785,53/ 12 281,14	530 300/ 6936,75/ 47 712,02	115 600/ 1512,14/ 10 400,73	4890/ 63,97/ 439,96	853/ 11,16/ 76,75	3314/ 43,35/ 298,17	723/ 9,46/ 65,05

 

The total estimated CNMS for significant missed pathological findings was RUB 770,855 ($ 10,083.4 or CNY 69,355.17) with RUB 4,818 ($63.02 or CNY 433.48) per patient. Results are presented in Table 6.

 

Table 6. Cons of medical services not provided due to missed significant pathological findings

Parameter	Calculations for all patients of the pilot project (RUB / $ / CNY)				Calculations per one patient (RUB/$/CNY)
	Cost
	Total (all)	Consultations (all)	Additional examinations (all)	Follow-up (all)	Total (all)	Consultations (all)	Additional examinations (all)	Follow-up (all)
Cost of the second step for significant pathological findings missed by radiologists	770 855/ 10 083,4/ 69 355,17	113 100/ 1479,44/ 10 175,8	584 255/ 7642,53/ 52 566,44	73 500/ 961,44/ 6612,92	4818/ 63,02/ 433,48	707/ 9,25/ 63,61	3652/ 47,77/ 328,58	459/ 6/ 41,3
Cost of the second step for pathological findings detected by radiologists	782 400/ 10 234,42/ 70 393,9	136 500/ 1785,53/ 12 281,14	530 300/ 6936,75/ 47 712,02	115 600/ 1512,14/ 10 400,73	4890/ 63,97/ 439,96	853/ 11,16/ 76,75	3314/ 43,35/ 298,17	723/ 9,46/ 65,05

 

Costs were compared using a paired t-test with calculated average difference per patient and using 95% confidence interval (CI). Therefore, 160×12908.5 [160×9833.5; 160×15,983.5] correspond to the total CNMS for analyzing the population with own CIs. Results are presented in Table 7.

 

Table 7. Cost-effectiveness

Cohort	Statistics	Cost			р (paired t-test)
		Total (all)	Total (findings by radiologists)	Mean difference with 95% CI
Full cohort	n	160	160	12 908,5	<0,001
	Mean	17 798,5	4890	[9833, 5; 15 983, 5]
	SD	23 304,74	11 945	-
	Minimum	0	0	-
	First quartile	0	0	-
	Median	12 635	0	-
	Third quartile	27 900	0	-
	Maximum	91 095	52 420	-
	Percentage of zero values	46,88	79,38	-
Cohort with all detected pathological findings	n	81	81	24 298,4	<0,001
	Mean	33 503,06	9204,71	[19 701,9; 28 894,8]
	SD	22 262,46	15 162,13	-
	Minimum	3900	0	-
	First quartile	14 395	0	-
	Median	27 900	0	-
	Third quartile	51 995	14 395	-
	Maximum	91 095	52 420	-
	Percentage of zero values	-	61,18	-
Cohort with significant pathological findings	n	32	32	17 104,7	<0,001
	Mean	24 089,22	6984,53	[8726, 8; 25 482, 6]
	SD	15 568,37	13 799,84	-
	Minimum	7600	0	-
	First quartile	14 395	0	-
	Median	19 700	0	-
	Third quartile	27 900	4825	-
	Maximum	69 500	52 420	-
	Percentage of zero values	-	71,88	-

 

The final cost-effectiveness for the AI used in healthcare organizations is shown in Fig. 5.

 

Fig. 5. Range of costs of medical services not provided due to the use of the combined AI service for chest CT scans in a clinic. AI, artificial intelligence; CT, computed tomography; CNMS, cost of non-provided medical services.

 

An example of calculating the CNMS

A radiologist in the clinic correctly described the pulmonary trunk dilation up to 34 mm, an increased Agatston score up to 350, and decreased density of the vertebrae to a maximum of +90 HU. The AI algorithm also detected these pathological findings. The AI algorithm also detected pathological findings that were not described in the radiologist’s protocol, such as a lung nodule up to 10 × 9 mm, thoracic aorta dilatation up to 45 mm, and adrenal gland thickening up to 14 mm. An example of CNMS calculation for this case is presented in Table 8.

 

Table 8. An example of calculating the cost of non-provided medical services based on data of one chest CT

Recommendations and cost for all pathological findings detected in patients
Consultations (all)	Additional examinations (all)	Over time monitoring findings (all)	Стоимость
			Total (all)	Consultations (all)	Additional examinations (all)	Over time monitoring findings (all)
Consultation by cardiologists orendocrinologists	Stress ECG, AP activity, CBC, calcium, abdomen ultrasonography, coronary CT angiography, stress echocardiography, hormonal activity of an adrenal tumor, echocardiography, blood biochemistry, CT with intravenous contrast enhancement	CT over time, consultations of the endocrinologist	RUB 91,095 $1,191.6 CNY 8,195.98	RUB 7,800 $102.03 CNY 701.78	RUB 71,795 $939.14 CNY 6,459.52	RUB 11,500 $150.43 CNY 1,034.68
Recommendations and costs for pathological findings missed by radiologists and detected by AI
Consultations	Additional examinations	Follow-up monitoring	Cost
			Total	Consultations	Additional examinations	Follow-up monitoring
-	Stress ECG, coronary CT angiography, stress echocardiography, hormonal activity of the adrenal tumor, echocardiography, and CT with intravenous contrast enhancement	CT over time	RUB 65,300 $854.18 CNY 5,875.16	0	RUB 57,700 $754.76 CNY 5,191.37	RUB 7,600 $99.41 CNY 683.79
Recommendations and cost for pathological findings detected by radiologists
Consultation by cardiologist, endocrinologist	AP activity, CBC, calcium, abdomen ultrasound, and blood biochemistry	Consultation of an endocrinologist	RUB 25,795 $337.42 CNY 2,320.82	RUB 7,800 $102.03 CNY 701.78	RUB 14,095 $184.37 CNY 1,268.15	RUB 3,900 $51.02 CNY 350.89

Note. AP, alkaline phosphatase; CBC, complete blood count; CT, computed tomography; EchoCG, echocardiography.

 

Additional data

Final data on the number of protocols with significant and insignificant missed pathological findings and the percentage of erroneous protocols are presented in Table 9. For the protocol, both significant, and insignificant missed pathological findings could be found. Of the 160 analyzed protocols, significant, and insignificant pathological findings were missed in 81 protocols, i.e., 50.6% of the total number of CT scans. The average percentage of protocols was 28.1% with significant missed pathological findings (max, 56.9; min, 5) and 27.2% with insignificant missed pathological findings (max, 74.1; min, 5).

 

Table 9. Results by the number of protocols with significant and non-significant missed pathological findings

No. of the radiologist	Total number of protocols prepared	Number of protocols with insignificant missed pathological findings (%)	Number of protocols with significant missed pathological findings (%)	Total number of protocols with significant and insignificant missed pathological findings (%)
Physician 2	58	33 (56,9)	28 (48,3)	47 (81)
Physician 5	23	7 (30,4)	9 (39,1)	11 (39,1)
Physician 3	23	7 (30,4)	7 (30,4)	13 (56,5)
Physician 1	16	2 (12,5)	3 (18,7)	5 (31)
Physician 6	20	1 (5)	1 (5)	2 (10)
Physician 7	18	1 (5,5)	1 (5,5)	2 (11,1)
Physician 4	2	1 (50)	1 (50)	1 (50)
Total	160	52 (32,5)	50 (31,2)	81 (50,6)

 

The total number of protocols was associated with statistically significant increase in the number of errors. The total work experience in radiology (excluding residency) and thoracic radiology (including residency) was associated with a decrease in the number of errors. However, these data are not representative because of the small sample of radiologists and the presence of a dominant case. Detailed data on the experience of specialists are presented in supplementary materials (Tables 10, 11). Examples of AI algorithm use are presented in Fig. 6, 7.

 

Fig. 6. An example of AI use. Patient B, 76 years old. A radiologist correctly identified bilateral hydrothorax and emphysematous changes but did not describe the lung nodule in the right lung. An AI algorithm revealed all three pathological findings: hydrothorax is highlighted with a yellow line, emphysematous changes are highlighted in orange, and the lung nodule is indicated by a red square.

 

Fig. 7. An example of AI use. Patient B., 79 years old. Chest CT scans: a) axial section: a radiologist and an algorithm correctly identified a lung nodule in the left lung (indicated by a red square) and coronary calcification (outlined by an orange line). In addition, the algorithm indicated an increase in the volume of epicardial fat (filled in yellow; this pathological finding was not considered in the study); b) sagittal section: a radiologist and an algorithm correctly identified compression fractures of Th6 and Th9 vertebral bodies, Genant 3 (three columns are marked with red lines); however, the radiologist did not indicate deformities of Th5 and Th12 vertebral bodies, Genant 2 (three columns are marked with yellow lines) in the protocol.

 

Adverse events

In this study, no adverse events were noted.

DISCUSSION

Summary of the primary study outcome

For the first time, this study showed the expected economic effect of using a combined AI-based software product for the analysis of chest CT data. The estimated economic effect is based on CNMS assessment.

Medical services should be provided to patients in accordance with current clinical guidelines. The total CNMS of the “second step” of the necessary diagnostics for pathological findings missed by radiologists and detected by AI was just over 2 million rubles, or 3.6 times more than the cost of medical services that could be provided by the clinic based on CT findings detected by radiologists in 160 patients. According to the calculation model used, only the CNMS, as significant missed pathological findings were slightly more than 770 thousand rubles, according to the price list of the clinic, or 98% more than the CNMS that the clinic could provide based on incidental findings.

In addition, the possibility of using combined AI-based software was proven for auditing CT description protocols.

Discussion of primary study outcome

An analysis of socioeconomic burden of the COVID-19 pandemic can show an example of significant social and economic consequences of a mass disease for the Russian healthcare system and society, and this require focusing attention not only on the clinical aspects but also on the economic importance of investing in disease control strategies [32]. According to experts, the social, and economic burden of COVID-19 in the Russian Federation in 2020 was approximately 5.4 trillion rubles (5% of the nominal GDP in 2020), which corresponds to 2486.30 years of life lost due to premature mortality (YLL) in men and 1378.22 YLL in women [32]. In 2020, the economic burden of non-communicable diseases in the Russian Federation was four trillion rubles, whereas the damage from chronic diseases is comparable to the entire healthcare budget of the Russian Federation, and funds saved through effective prevention could become a huge additional resource for the development of the country.⁷

In the available literature, no studies have evaluated the effect of combined AI analysis of chest CT on the economic aspects of clinic activities.

Pickhardt et al. [33] built a model of the cost and clinical effectiveness of combined AI screening with abdomen CT. Based on the expected disease prevalence, probability of transition between health conditions, associated healthcare costs, and treatment effectiveness in three diseases (CVD, osteoporosis, and sarcopenia), three mutually exclusive screening models were evaluated: (1) ignoring outcomes (“do not treat”; no intervention regardless of CT findings), (2) statin therapy for all (“treat all” for the prevention of CVD without considering CT findings), and (3) opportunistic screening for CVD, osteoporosis, and sarcopenia with AI-based abdomen CT (targeted treatment for at-risk individuals). For baseline scenarios in simulated in 10-year groups of 55-year-old men and women, AI-based opportunistic CT screening proved to be a cost effective and more clinically effective strategy the than “ignoring” and “treating all” approaches. Thus, AI-based opportunistic CT screening appears to be a highly cost effective and clinically effective strategy with a wide range of input assumptions and cost savings in most scenarios. However, compared with our study, a really working combined AI service was not presented. In addition, our study used the combined AI service for 10 target pathological findings, whereas the mentioned paper reviewed three diseases. In this regard, there are reasons to expect an increase in the potential cost-effectiveness of using AI, combined with its positive effect on diagnostics.

Few publications have examined the economic effect of such programs using an integrated approach for several pathological findings without using AI systems. For example, in the Netherlands, a comprehensive low-dose CT screening of three diseases, such as lung cancer, chronic obstructive pulmonary disease, and CVD in people aged 50–75, could be cost effective if its cost is <971 euros per patient [34]. In a 5-year study of nearly 7,500 low-dose CT scans, extrapulmonary malignant tumors incidentally detected during lung cancer screening were on early stages and had favorable outcomes, and additional examinations required much less cost than in cases with later stages [35]. Thus, the analysis of the costs of additional diagnostic and therapeutic measures associated with extrapulmonary changes detected during low-dose CT of lung cancer is one of the main steps in proving the cost-effectiveness of such measures. This approach (despite the limited lung cancer screening using chest CT) allows proposing the use of combined AI services to improve the diagnostic effectiveness and cost-effectiveness of examinations.

This study also used the combined AI service to detect 10 pathological findings. Such reserve analysis techniques are useful in making informed decisions about further research [36-38]. These analysis techniques are recommended for diagnostic intervention and evidence development to optimize data collection and more accurately estimate long-term economic health effect as a large amount of clinical data become available.

Before the COVID-19 pandemic, AI algorithms were used to detect radiologic symptoms for disease detection, classification, image optimization, radiation dose reduction, and workflow improvement [39]. Medical studies on the effectiveness of AI make such programs more understandable, more efficient, safer, and more integrated into medical staff workflows [40]. Currently, studies are ongoing in the IMALife project, which evaluates the decrease in mortality not only from lung cancer but also from the consequences of emphysema (a biomarker of chronic obstructive pulmonary disease) and coronary calcification (a biomarker of atherosclerosis) [41].

At present, AI effectiveness was evaluated for the detected target pathological finding. Ziegelmayer et al. [42] showed a negative incremental cost-effectiveness ratio (ICER) in the baseline CT+AI scenario compared with CT, demonstrating lower costs, and higher efficiency. To support the use of AI, threshold analysis showed that ICER remained negative until the $68 threshold. Therefore, the use of a monosystem for the analysis of low-dose CT data using AI for lung cancer screening is a reasonable diagnostic strategy in terms of cost-effectiveness.

The constantly growing volume of radiological examinations provides additional burden on radiologists [43]. Excessive workload can increase the likelihood of errors and deteriorate the quality of care [44]. An audit system with a retrospective double review of studies is widely used in radiology. A RADPEER system of the American Society of Radiology is the best-known system [45]. However, according to Lauritzen et al. [46], double reading of the 1/3 of examinations performed at their clinic takes 20%–25% of the working time of HCPs. AI algorithms can significantly reduce the time for examination review and increase audit volume and quality. However, the algorithm used should have the minimum number of false-positive errors. The use of AI had some effects on the quality of work of radiologists such as a change in the lung lesion severities in the case of suspected COVID-19 toward a decrease in the proportion of severe and critical lung lesions [47].

Our study also demonstrated the feasibility of using combined AI services for study description audit and identified >28% of protocols with significant missed pathological findings and 27% of protocols with non-significant missed pathological findings. For CT, radiologists in the clinic provide information on the main diagnostic tasks for which patients were referred to scan. In the clinic where our study was conducted, radiologists did not have tools at all workstations for quick measurement of the Agatston score, and the measurement of the density of the vertebral bodies was not included in the standard description of examinations performed in this radiology department. In addition, the average error rate was comparable between radiologists; thus, by dismissing 1–2 radiologists with the worst results according to audits, the clinic will not solve the problem of missing pathological findings.

In this study, the economic effect was calculated without considering AI costs and cost of AI validation by experts. These expenses are variable and depend on the number of algorithms, level of experts involved, and other factors. Any rate for the AI-based service will be cost effective, provided that the total expenses are lower than the profit obtained by a clinic using AI (Fig. 8). However, such an analysis is beyond the scope of this study.

 

Fig. 8. Potential cost of medical services not provided because a combined AI service was not used for chest CT scans in a clinic, considering the cost of using AI.

 

Study Limitations

This pilot study has several limitations. The study has a retrospective design and evaluates the maximum potential CNMS of a private healthcare organization in accordance with recommendations for abnormal findings. In practice, not all patients tend to comply with doctor’s recommendations, particularly when additional paid examinations and consultations are required. In addition, each clinic has its conversion that is not evaluated in this study.

This study aimed to estimate the cost of the “second step” without the cost of the “third” and subsequent steps, i.e., the cost of treatment and rehabilitation was not considered. However, for nearly all pathologies that the combined AI service can detect, the cost of treatment significantly exceeds the cost of the “second stage.” Every clinic has many factors that affect the quality of medical personnel work. The quality of work parameter (number of missed clinically significant radiographic findings) can vary depending on the experience of the radiologists, number of examinations per day, time of day, day of the week, and many other additional factors that can affect knowledge level, attentiveness, and readiness of the radiologist to include in the protocol all abnormal findings and reasonable recommendations for the “second step.”

This study did not evaluate the potential for false-negative AI findings because the combined AI service used was validated during independent testing by employing closed datasets of the Moscow experiment, and the selected AI settings were found to be acceptable and calibrated for subsequent use.

The aim of this study was not to assess the economic effect at the level of city and federal healthcare systems. However, each clinic in the Russian Federation has some opportunities to provide paid medical services supported by evidence-based medicine principles. This study did not evaluate compliance of patients according to invitations based on a retrospective analysis, whereas the findings are closely related to the time intervals between CT and the “second step.”

CONCLUSION

A combined AI service can be used as an additional tool for radiologists when analyzing chest CT data to better detect 10 common and significant types of abnormal findings. When using such an approach, CNMS is 3.6 times larger than the standard operation model of radiologists not using AI.

Opportunistic screening of multiple diseases requires a detailed investigation of comorbidities to determine the optimal target group for diagnostic intervention using combined AI services. Combined AI solutions for chest CT are highly likely to be cost effective because such an approach allows the detection of various significant abnormal changes that require additional healthcare services.

ADDITIONAL INFORMATION

Funding source. This article was not supported by any external sources of funding.

Competing interests. The authors declare that they have no competing interests.

Authors’ contribution. The authors confirm that their authorship complies with the international ICMJE criteria (all authors contributed significantly to developing the concept, research, and preparation of the article and read and approved the final version before publication).

Chernina V.Yu. — search for publications on the article’s topic, formation of a data set, writing the text of the manuscript; Belyaev M.G. — expert evaluation of the data, manuscript’s text editing; Silin A. Yu. — the concept of the study; Pyatnitskiy I.A. — manuscript’s text editing; English text translation; Avetisov I.O., Petrash E.A. — manuscript’s text editing; Basova M.V. — formation of a data set, processing of research results; Sinitsyn V.E. — expert evaluation of the data, manuscript’s text editing; Omelyanovsky V.V. — expert evaluation data; Gombolevsky V.A. — the concept of the study, expert evaluation of data, manuscript’s text writing, approval of the final version.

Acknowledgments. The authors’ team would like to gratefully acknowledge Maksim E. Pisov, Technical Director of IRA Labs, and the development team (Proskurov V.A., Samoilenko A.I., Borzov A.S., Bukharaev A.N., Goncharov M.Y., Shirokikh B.N., Kurmukova A.I., Nachinkin I.A., Telepov A.Y., Shimovolos S.A., Donskova M.A., Samokhin V.Y., Shevtsova A.E., Yaushev F., Tkachenko S.M, Zakharov A.A., Saparov T.N., Filipenko V.K. Tomin V.D., Tomin J.D., Samoilenko A.I., Malevannu V.M., Leonov A. Yu, Nogova D.S., Bazarova A.I., Marakhovsky K.V., Belkov A.S., Vasin A.A., Berezhnaya D.S., Musikhin M.M.); Maria N. Dugova, Medical Director, and the team of medical experts (Ilyicheva D.V., Sevrukov D.D., Shchipakhina Y.A, Lyubimaya Y.O., Aleshina O.O., Tsybulskaya Y.A.); Mikhail S. Lamzin, Chief Operating Officer; Regina R. Gareeva, Product Manager, and Elena V. Panina, Deputy Director for Business Development.

¹ who.int [Internet]. Top 10 death causes in the World [cited on December 08, 2020]. Available from: https://www.who.int/ru/news-room/fact-sheets/detail/the-top-10-causes-of-death.

² IQVIA [Internet]. FDA Publishes Approved List of AI/ML-enabled Medical Devices [cite 2021 Oct 29]. Michaela Miller, Technology & Analytics Practice Lead, U.S. MedTech, IQVIA. Available from: https://www.iqvia.com/locations/united-states/blogs/2021/10/fda-publishes-approved-list-of-ai-ml-enabled-medical-devices.

³ Center for Diagnostics and Telemedicine [Internet]. Experiment (https://mosmed.ai/ai/); data sets (https://mosmed.ai/datasets/). Accessed on March 17, 2023.

⁴ Center for Diagnostics and Telemedicine [Internet]. Chest-IRA. IRA Labs LLC. Available from: https://mosmed.ai/service_catalog/chestira/.

⁵ EQUATOR Network [Internet]. Enhancing the QUAlity and Transparency Of health Research. Available from: https://www.equator-network.org/reporting-guidelines/cheers/.

⁶ Center for Diagnostics and Telemedicine [Internet]. IRA Labs LLC: Chest-IRA (https://mosmed.ai/service_catalog/chestira/); Adrenal-IRA Abd (https://mosmed.ai/service_catalog/adrenalira/); Aorta-IRA Abd (https://mosmed.ai/service_catalog/aortaira/); Genant-IRA Abd (https://mosmed.ai/service_catalog/genant-ira/). Accessed on March 17, 2023.

^# The Task Force for the Diagnosis and Treatment of Aortic Diseases of the European Society of Cardiology (ESC). ESC Guidelines on the diagnosis and treatment of aortic diseases, 2014. Available from: https://scardio.ru/content/Guidelines/Recom%20po%20aorte%207_rkj_15.pdf.

^## The Task Force for the diagnosis and treatment of pulmonary hypertension of the European Society of Cardiology (ESC) and the European Respiratory Society (ERS). ESC/ERS Guidelines for the diagnosis and treatment of pulmonary hypertension, 2015. Available from: https://scardio.ru/content/Guidelines/ESC%20_L_hypert_2015.pdf.

⁷ Medvestnik [Internet]. The economic burden of non-communicable diseases in Russia has grown by a trillion rubles in two years [Ekonomicheskoye bremya neinfektsionnykh zabolevaniy v Rossii vyroslo za dva goda na trillion rubley] (cited: June 04, 2021). Available from: https://medvestnik.ru/content/news/Ekonomicheskoe-bremya-neinfekcionnyh-zabolevanii-v-Rossii-vyroslo-za-dva-goda-na-trillion-rublei.html.

 

SUPPLEMENTARY MATERIALS

 

Table 10. Recommendations for additional methods

Health condition	Clinical guidelines	Consultation/additional examinations/follow-up monitoring
Health condition	Clinical guidelines	Consultation/additional examinations/follow-up monitoring
Lung nodules	Gombolevsky V.A., Blokhin I.A., Laipan A.Sh., etc. Guidelines for lung cancer screening. [Metodicheskiye rekomendatsii po skriningu raka logkogo.] Series: “Best practices of radiological and instrumental diagnostics.” [Luchshiye praktiki luchevoy i instrumental'noy diagnostiki.] Issue 56. Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Healthcare of Moscow. 2020. 57 pp.	Consultations: oncologist. Additional examinations: biopsy, bronchoscopy under CT/ultrasound control, cytology, blood test (HIV, syphilis, and hepatitis), PFT, spirometry, CBC, ECG, PET/CT, and brain MRI with an intravenous contrast-enhancement Follow-up monitoring: depending on the number and size of lung nodules. Chest CT is recommended after 3–6 months.
Infiltrative lung changes typical for viral pneumonia (COVID-19 in a pandemic)	Interim Guidelines. Prevention, Diagnosis and Treatment of Novel Coronavirus Infection (COVID-19). The Ministry of Health of the Russian Federation. Version 17 (December 14, 2022). Available from: https://static-0.minzdrav.gov.ru/system/attachments/attaches/000/061/254/original/%D0%92%D0%9C%D0%A0_COVID-19_V17.pdf?1671088207	Consultations: primary care doctor. Additional examinations: pulse oximetry; ECG; SARS-CoV-2 RNA by nucleic acid amplification techniques; SARS-CoV-2 antigens by immunochromatographic techniques; CBC with erythrocytes, hemoglobin, hematocrit, leukocytes, platelets, and leukogram; blood biochemistry (urea, creatinine, electrolytes, glucose, ALT, AST, bilirubin, albumin, lactate, and lactate dehydrogenase, as well as troponin as a marker of myocardial damage and ferritin as a protein of acute inflammation phase to assess the disease severity and prognosis); CRP; hormonal testing (procalcitonin and NT-proBNP); coagulogram (activated partial thromboplastin time, prothrombin time, prothrombin ratio and/or quick prothrombin %, fibrinogen, and D-dimer by the quantitative method). Follow-up monitoring: 8 weeks after discharge, a visit to a doctor and imaging studies (if indicated) are recommended: chest X-ray*, spirography*, blood oxygen saturation at rest and during exercise (it is possible to conduct a 6-min walk test and determining saturation before and after the test)*, echocardiography, and other techniques (diffusion test, arterial blood gases, etc.) * If abnormalities are detected, lung CT is indicated.
Lung emphysema	Lung emphysema. Clinical guidelines (approved by the Ministry of Health of the Russian Federation, 2021). Available from: https://www.garant.ru/products/ipo/prime/doc/402775957/	Consultations: pulmonologist/primary care doctor. Additional examinations: blood gas test. If a hereditary A1AT deficiency of A1AT is suspected, determine the blood activity of A1AT, spirometry with a bronchodilator test, body plethysmography, and lung diffusion test. For all patients with clinical signs of A1AT deficiency and/or decreased serum A1AT, phenotype, and genotype testing is recommended. Follow-up monitoring: for all patients with A1AT deficiency, annual blood biochemistry for albumin, total bilirubin, AST, ALT, GGT, and platelets is recommended. For patients with pulmonary A1AT deficiency, spirometry is initially recommended every 6–12 months to rule out rapid disease progression, followed up with less frequent examinations. For patients with A1AT deficiency, an annual abdomen ultrasound is recommended to exclude liver manifestations of the disease.
Free pleural fluid (effusion)	Maskell N., Butland R. British Thoracic Society (BTS) guidelines for the investigation of adult patients with unilateral pleural effusion [Rekomendatsii Britanskogo torakal'nogo obshchestva (BTS) po obsledovaniyu vzroslykh bol'nykh s odnostoronnim plevral'nym vypotom]. Pulmonology [Pul'monologiya]. 2006. No. 2. P. 13–26. doi: 10.18093/0869-0189-2006-2-13-26	Consultations: pulmonologist. If tuberculosis is detected, a TB specialist should be consulted. If a malignant lesion is detected, an oncologist should be consulted. Additional examinations: consider pleural aspiration (ASAP, “as soon as possible”), cytology, protein, LDH, pH, Gram staining, culture, and sensitivity determination, staining for acid-resistant rods, and culture for Mycobacterium tuberculosis. Later, if necessary, chest CT with intravenous contrast enhancement and pleural biopsy. If no underlying cause is found after all investigations, consider thoracoscopy.
Aneurysm/dilatation of the aorta	Clinical guidelines. Recommendations for the diagnosis and treatment of aortic diseases [Rekomendatsii po diagnostike i lecheniyu zabolevaniy aorty] (2017) // Cardiology and Cardiovascular Surgery [Kardiologiya i serdechno-sosudistaya khirurgiya]. 2018. No. 1. P. 7–67	Consultations: cardiologist/vascular surgeon. Additional examinations: CT aortography with intravenous contrast enhancement. Follow-up monitoring: transthoracic echocardiography and abdominal ultrasound.
Dilatation of pulmonary trunk	Clinical guidelines. Pulmonary hypertension, including chronic thromboembolic pulmonary hypertension, 2020 (June 03, 2021). Approved by the Ministry of Health of the Russian Federation. Available from: http://disuria.ru/_ld/10/1026_kr20I27MZ.pdf	Consultations: cardiologist. Additional examinations: echocardiography; ECG; blood gas tests; perfusion scintigraphy with ventilation scintigraphy; CBC with hemoglobin and hematocrit, erythrocyte, leukocyte, platelets, and ESR; blood biochemistry (creatinine, sodium, potassium, glucose, total protein, CRP, AST, ALT, total bilirubin, and uric acid); blood antibodies to cardiolipin, phospholipids, beta-2-glycoprotein in the case of suspected chronic thromboembolic pulmonary hypertension to identify risk factors; content of antibodies to cell nucleus and DNA antigens to exclude association with systemic diseases of the connective tissue; NT-proBNP; IgM, IgG antibodies to HIV-1, and HIV-2 in blood; HBV antigen, anti-HCV, anti-treponema pallidum; clinical urinalysis; thyroid function test (free triiodothyronine, free thyroxine, and thyroid-stimulating hormone); abdominal ultrasound (complex) to exclude liver disease and/or portal hypertension.
Coronary calcification	Clinical guidelines. Stable coronary heart disease, 2020 (Approved by the Scientific and Practical Council of the Ministry of Health of the Russian Federation). Available from: https://cr.minzdrav.gov.ru/schema/155_1	Consultations: cardiologist. Additional examinations: CBC (clinical) with hemoglobin, erythrocytes, and leukocytes; blood creatinine and estimated GFR or creatinine clearance for kidney function assessment; blood biochemistry, including total blood cholesterol, low-density lipoprotein cholesterol, and triglycerides; blood NT-proBNP; ECG at rest; ambulatory ECG monitoring; echocardiography; heart MRI when receiving non-conclusive echocardiography results (including with contrast enhancement); duplex scanning of the extracranial carotid arteries to detect atherosclerotic plaques; coronary CT angiography; stress ECG, stress echocardiography, stress MRI, and SPECT. Follow-up: in the case of repeated examinations, in all patients with a diagnosed stable coronary heart disease, annual monitoring of the clinical CBC, blood biochemistry, blood test for lipid metabolism abnormalities, blood creatinine, fasting blood glucose for timely therapy adjustment, if necessary, is recommended.
Measurement of adrenal size for masses and hyperplasia	Clinical guidelines. Adrenocortical cancer, 2020 (Approved by the Scientific and Practical Council of the Ministry of Health of the Russian Federation). Available from: https://cr.minzdrav.gov.ru/schema/341_1	Consultations: endocrinologist and oncologist. Additional examinations: test for the hormonal activity of the adrenal tumor (cortisol, ACTH, free plasma metanephrines, or 24-h urinary fractionated metanephrines, aldosterone, plasma renin, and serum potassium), abdominal CT with intravenous contrast enhancement (in the case of contraindications for CT with contrast enhancement, perform an abdominal MRI with the involved retroperitoneal space). In the case of adrenal gland lesion with an indeterminate CT phenotype and without hormonal hypersecretion, the situation with a multidisciplinary team is recommended. Three options are possible: (1) additional imaging studies (PET/CT with fluorodeoxyglucose), (2) watchful waiting with repeat non-contrast CT (or MRI) 3–6 months later, and (3) surgical treatment. Follow-up: in the case of lesions with an indeterminate CT phenotype, to assess changes in the tumor size over time (if observation was chosen based on the primary diagnosis), a repeat CT/MRI is recommended after 3–6 months. Surgical treatment in the case of a 20% increase in the lesion size (or an increase in the maximum diameter >5 mm) during a short follow-up period.
Evaluation of the density of the spongy substance of vertebral bodies for detection of osteoporosis/ osteopenia	Clinical guidelines. Osteoporosis, 2022 (Approved by the Scientific and Practical Council of the Ministry of Health of the Russian Federation). Available from: https://cr.minzdrav.gov.ru/recomend/87_4	Consultations: endocrinologist. Additional examinations: CBC (clinical); blood biochemistry (total calcium, creatinine with GFR, inorganic phosphorus, AP activity, and glucose); blood test for C-terminal telopeptide and/or N-terminal propeptide of type 1 procollagen (P1NP, a resorption marker for prescribing antiresorptive therapy and a bone formation marker for prescribing anabolic therapy) in patients receiving self-therapy for osteoporosis, initially, and after 3 months from the start of therapy for early assessment of therapy effectiveness and adherence; dual-energy X-ray densitometry (DXA) of the lumbar spine and proximal femur; 3D measurement of maximum oxygen consumption by quantitative CT (optional).
Vertebral compression fractures related to osteoporosis	Clinical guidelines. Pathologic fractures in osteoporosis, 2022 (Approved by the Scientific and Practical Council of the Ministry of Health of the Russian Federation). Available from: https://cr.minzdrav.gov.ru/recomend/614_2	Consultations: endocrinologist. Additional examinations: CBC (clinical); urinalysis; blood biochemistry with total and/or ionized calcium, inorganic phosphorus, total AP, and blood creatinine with GFR; blood level of parathyroid hormones, calcium, and phosphorus in 24-h urine; dual-energy X-ray densitometry. In the case of a low-energy fracture of the vertebral body in people aged >50 years, blood, and urine tests for paraproteins and M-gradient are required to exclude multiple myeloma.

Note. A1AT, alpha-1 antitrypsin; ACTH, adrenocorticotropic hormone; ALT, alanine aminotransferase; AST, aspartate aminotransferase; CBC, complete blood count; CRP, C-reactive protein; CT, computed tomography; ECG, electrocardiogram; EchoCG, echocardiography; ESR, erythrocyte sedimentation rate; GFR, glomerular filtration rate; GGT, gamma-glutamyl transferase; HBV, hepatitis B virus; HCV, hepatitis C virus; HIV, human immunodeficiency virus; LDH, lactate dehydrogenase; MRI, magnetic resonance imaging; NT-proBNP, aminoterminal pro B-type natriuretic peptide; AP, alkaline phosphatase; PET/CT, positron emission tomography with computed tomography; PFT, pulmonary function test; SPECT, single-photon emission computed tomography

 

Table 11. Employment history of physicians

Physician	Work experience			Employment	Academic degree
	in radiology (including residency)	in thoracic radiology (excluding residency)	in CT
Physician 1	15	13	13	Part-time	Yes
Physician 2	17	5	5	Part-time	No
Physician 3	7	5	5	Full-time	No
Physician 4	16	14	14	Part-time	No
Physician 5	7	5	5	Part-time	No
Physician 6	5	3	3	Part-time	Yes
Physician 7	14	10	10	Part-time	No

 

Generalized linear model

The generalized linear model assumes that each finding Y_i has linear relationship with variables X_ip, p=1,2,...m:

Y_i=b_i0+b₁X_i1+b₂X_i2+...+b_mX_im+ε_i,

where X variables can be either categorical (considering groups, classes, and categories) or continuous. In the generalized linear model, coefficients {bj, j=0,1,2,...,m} are estimated for the model of Y parameter dependence on factors {Xj, j=1,2,...,m}. For these coefficients, their statistical significance is determined (calculation of p-values for testing hypotheses H_j0: b_j=0, j=0,1,2,...,m), which shows the significance of the corresponding factors’ effect on the target parameter.

In our case, some parameters have a lognormal distribution; thus, a logarithmic transformation was used for them. The final model is as follows:

Y = log (total number of protocols with critical and non-critical errors)

X1 = all protocols

X2 = log (work experience in thoracic radiology [excluding residency])

X3 = log (length of experience in CT)

Categorical parameter X4 = academic degree.

Results of estimating coefficients for the generalized linear model and their statistical significance are presented in the table:

Thus, the total number of protocols statistically significantly increases the number of errors. The total work experience in radiology and thoracic radiology decreases the number of errors. However, these data are not representative because of the small sample of HCPs and presence of a dominant case.

Partially similar results (for statistical significance and effects of one experience types) are also considered in the correlation analysis:

• ρ (log (total number of protocols with critical and non-critical errors), all protocols) = 0.88 with 95% CI [0.39; 0.98] and р-value of 0.008 (statistically significant correlation),
• ρ (log (total number of protocols with critical and non-critical errors), log (work experience in thoracic radiology [excluding residency]) = -0.45 with 95% CI [-0.9; 0.46] and р-value of 0.31 (statistically insignificant correlation),
• ρ (log (total number of protocols with critical and non-critical errors), log (work experience in thoracic radiology (excluding residency)) = -0.27 with 95% CI [-0.85; 0.61] and р-value of 0.56 (statistically insignificant correlation).
• However, the absence of statistical significate correlations does not allow establishing any trends.

	Estimate	Std. Error	t.value	Pr...t..
(Intercept)	0,132922	0,352989	0,37656	0,742697
bd_lr$"All protocols"	0,037246	0,006111	6,094863	0,025879
log(bd_lr$"Work experience in thoracic radiology/r/n(excluding residency)/r/n")	-7,96969	1,009013	-7,8985	0,015654
factor(bd_lr$"Academic degree")no	-0,92975	0,229399	-4,05297	0,055828

 

About the authors

Valeria Yu. Chernina

IRA Labs

Email: v.chernina@ira-labs.com
ORCID iD: 0000-0002-0302-293X
SPIN-code: 8896-8051
Scopus Author ID: 57210638679
ResearcherId: AAF-1215-2020
Russian Federation, Moscow

Mikhail G. Belyaev

IRA Labs

Email: belyaevmichel@gmail.com
ORCID iD: 0000-0001-9906-6453
SPIN-code: 2406-1772

Cand. Sci. (Phys.-Math.), Professor

Russian Federation, Moscow

Anton Yu. Silin

Clinical Hospital on Yauza

Email: silin@yamed.ru
ORCID iD: 0000-0003-4952-2347
SPIN-code: 4411-8745
Russian Federation, Moscow

Ivan O. Avetisov

Clinical Hospital on Yauza

Email: avetisov@yamed.ru
ORCID iD: 0009-0007-3550-7556
Russian Federation, Moscow

Ilya A. Pyatnitskiy

IRA Labs; The University of Texas at Austin

Email: i.pyatnitskiy@ira-labs.com
ORCID iD: 0000-0002-2827-1473
SPIN-code: 6150-4961
Russian Federation, Moscow; Austin, Texas, USA

Ekaterina A. Petrash

IRA Labs; N.N. Blokhin National Medical Research Center of Oncology

Email: e.a.petrash@gmail.com
ORCID iD: 0000-0001-6572-5369
SPIN-code: 6910-8890

MD, Cand. Sci. (Med.)

Russian Federation, Moscow; Moscow

Maria V. Basova

IRA Labs

Email: m.basova@ira-labs.com
ORCID iD: 0009-0000-3325-8452
Russian Federation, Moscow

Valentin E. Sinitsyn

Lomonosov Moscow State University; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: vsini@mail.ru
ORCID iD: 0000-0002-5649-2193
SPIN-code: 8449-6590

MD, Dr. Sci. (Med.), Professor

Russian Federation, Moscow; Moscow

Vitaly V. Omelyanovskiy

The Center for Healthcare Quality Assessment and Control; Russian Medical Academy of Continuous Professional Education; Scientific and research financial institute

Email: vvo@rosmedex.ru
ORCID iD: 0000-0003-1581-0703
SPIN-code: 1776-4270

MD, Dr. Sci. (Med.), Professor

Russian Federation, Moscow; Moscow; Moscow

Victor A. Gombolevskiy

IRA Labs; Artificial Intelligence Research Institute

Author for correspondence.
Email: gombolevskii@gmail.com
ORCID iD: 0000-0003-1816-1315
SPIN-code: 6810-3279

MD, Cand. Sci. (Med.)

Russian Federation, Moscow; Moscow

References

Supplementary files