How does artificial intelligence effect on the assessment of lung damage in COVID-19 on chest CT scan?

Full Text

Acronyms and abbreviations

data set: a set of data, a collection of logical records

URIS: Unified Radiological Information System

AI service: artificial intelligence software

CT: computed tomography

CT (computed tomography) 0–4 represents an empirical scale for visual assessment on the severity dependence of general condition of the nature and pronouncement of radiological changes in the lungs with Coronavirus disease 2019 (COVID-19) according to CT, where CT-0 implies the absence of signs of viral pneumonia; CT-1 indicates mild pneumonia with ground glass opacity areas, with <25% severity of pathological changes; CT-2 is moderate pneumonia, with affection of 25%–50% of the lungs; CT-3 implies moderate pneumonia with 50%–75% of the lungs affected; and CT-4 is severe pneumonia, affecting >75% of the lungs.

Digital Imaging and Communications in Medicine: Structured Reporting is a standardized format of the medical industry standard for creating, storing, transferring, and visualizing digital medical images and documents of patients examined (structured reporting).

BACKGROUND

In 2020, the Coronavirus disease 2019 (COVID-19) pandemic challenged healthcare systems worldwide, which prompted global governments to seek new solutions under resource-constrained conditions. On August 27, 2020, the government commission on digital development of the Russian Federation approved the certificate of the federal project “Artificial Intelligence” within the national program “Digital Economy”. Regardless of this, back in 2019, a Decree of the Moscow Government was drafted, followed by an Order of the Moscow Healthcare Department at the beginning of 2020 on conducting an experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further application in the health care system of Moscow (Experiment) [1].

In the pandemic, computed tomography (CT) is used as a key tool for assessing changes in the lungs, associated with infection [2]. In the first months of the COVID-19 pandemic, semi-quantitative scales were mainly used to assess the severity of changes, with an insignificant frequency in routine clinical work [3–7]. Work with scales, which were based on severe acute respiratory syndrome viral pneumonia studies results, involves a separate calculation of the volume and type of lesions for lobes and segments of the lungs with subsequent summation of results [8]. A visual assessment was proposed based on the approximate volume determination of indurated tissue in both lungs without separate calculations for segments and lobes [9].

Radiologists in Moscow used the adapted CT 0–4 scale to assess visually the severity dependence of the general condition on the nature and pronouncement of radiological changes in the lungs with COVID-19 according to CT data. The percentage of damage was assessed separately for each lung, and the degree of change was assessed for the lung with the greatest lesion (regardless of postoperative changes). Every 25% of the volume of lung lesions increases the scale by one category [10, 11]. The proposed method of visual assessment CT 0–4 was validated by predicting lethal outcomes in patients with COVID-19 [12]. The risk of lethal outcome increases by an average of 38% (95% confidence interval 17.1–62.6) with the transition from one CT 0–4 category to the next [12].

In case of a large stream of studies, the doctor may omit findings and make mistakes in assessing lung damage volume [13].

A task has been added to the experiment to process chest CT data for COVID-19 diagnostics using artificial intelligence software (AI services). The AI services added CT series with lung lesion segmentation, lesion volume information for each lung, and a CT 0–4 category.

This study aimed to compare the distribution of CT 0–4 categories in reports generated by radiographers with and without AI services.

METHODS

Study design

This is a retrospective study based on a study registered in Clinical Trials (NCT04489992). Data analyzed in the course of the work was provided by experts of the Moscow Information Technology Department.

Compliance criteria

Inclusion and exclusion criteria were used to form the CT study group.

Inclusion criteria:

• CT scans of the chest organs of male and female patient who sought medical help with suspected COVID-19;
• age of patients over 18 years old;
• CT examinations of the chest organs were performed and interpreted by radiologists in the period from April 8, 2020 to December 1, 2020 in outpatient healthcare organizations;
• availability of information on the assessment category by CT 0–4 in protocols of medical reports of radiologists;
• CT description protocols were formed in the Unified Radiological Information System (URIS).

Exclusion criteria:

• CT studies with conclusion of other changes not associated with viral pneumonia.

Implementation conditions

CT examinations were conducted in all medical organizations subordinate to the Moscow Healthcare Department, providing outpatient care for the adult population. During the pandemic, outpatient medical organizations were transformed into outpatient CT centers that provided a special round-the-clock operation. Taking into account the epidemiological situation, it can be assumed that the gender and age distribution of patients who underwent CT scan corresponds to the same distribution in Moscow.

Study duration

The study was conducted in the period from April 8, 2020 to December 1, 2020. Additionally, an assessment was performed in November 2020 (from November 1, 2020 to December 1, 2020) (Fig. 1).

 

Fig. 1. Chronology of the use of AI services for COVID-19 diagnosis according to computed tomography of the thoracic organs (CT TO).

 

Description of the medical intervention

Over the entire period under consideration and separately for November 2020, the test and control comparison groups were formed (Fig. 2). The test group included CTs processed by AI services, and the control group included CTs without AI processing.

 

Fig. 2. Examples of original (control group) and additional CT series from various AI services (test group with subgroups) with demonstration of automatic image processing for segmentation of lung lesions in COVID-19, as well as summary information on lung damage and DICOM SR information.

 

Each algorithm was tested on a specially prepared calibration data set before including the AI service in the experiment. The calibration data set included CT scans of patients with laboratory-verified COVID-19 and an assessment by expert doctors. The criterion for inclusion of an AI service in the URIS was the algorithm accuracy not less than the area under the ROC curve (ROC AUC) of 0.81, according to the guidelines for clinical trials of software based on intelligent technologies [14].

Each AI service added a new series of AI-processed CT scans and information in Digital Imaging and Communications in Medicine: Structured Reporting (DICOM SR) format of the study. The additional CT series is based on the original CT series of the current study, with original image supplemented with the segmentation of lung lesions that are caused by COVID-19 according to the AI assumption. The AI developers participating in the experiment were advised to supplement the CT series sent by the AI service with summary information on lung damage and a CT 0–4 score. The DICOM SR data, available to each radiologist in the test group during the formation of the conclusion, contained information about a specific AI service, instructions for using the processing results, and automatically generated report including the severity assessment according to the CT 0–4 scale (Fig. 2).

CT examinations were performed on 48 computed tomographs (Toshiba Aquilion 64, Canon Medical Systems, Japan; HiSpeed GE, USA; Optima CT 660, GE, USA; Somatom Emotion 16, Siemens, Germany; Somatom Sensation 40, Siemens, Germany) according to standard chest scanning protocols recommended by manufacturers.

The comparison was conducted between categories on the CT 0–4 scale from the conclusions of radiologists who had access to AI service results and those who did not have such access.

Main study outcome

AI services were included in the experiment after passing qualitative and quantitative tests on databases prepared by experts of the Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Healthcare Department. Research has been distributed randomly based on the computational capabilities of developers since the inclusion of five AI services. One study could be processed by several AI services. AI results were used only for research purposes, and the radiologist made the final decision on the category on the CT 0–4 scale.

Subgroup analysis

The test group included subgroups since the experiment involved various AI services to diagnose changes in the lungs with COVID-19 according to CT data.

The experiment involved 7 different AI services for diagnosing COVID-19, namely RADlogics COVID-19 (RADLogics, USA), COVID-IRA (IRA labs, Russia), Care Mentor AI, COVID (CareMertor AI, Russia), Third Opinion CT-COVID-19 (Third Opinion, Russia), Multivox COVID19 (Gammamed, Russia), IRYM (Russia), and CVL (CVisionLab, Russia); however, the last two were not included in the test subgroups due to the small number of processed studies over the entire study period. The remaining 5 AI services were randomly coded: AI-A, AI-B, AI-C, AI-D, and AI-E (Fig. 2).

Ethical considerations

Approval of the Independent Ethics Committee of the Moscow Regional Branch of the Russian Society of Roentgenologists and Radiologists was obtained (Protocol No. 2 [1-II-2020] dated February 20, 2020).

Statistical analysis

Principles for calculating the sample size: all those having valid data were included in the statistical analysis. Methods for restoration of missing data were not applied.

Methods for statistical data analysis: descriptive statistics methods were used to present results, indicating the absolute number (n) and proportion (%) of cases in each category. Intergroup comparison of the frequency distribution in different categories between the control group and the test subgroups within each of the 2 periods was performed using the Pearson’s chi-squared test (χ2). The level of statistical significance was considered as a value of 0.05. Statistical analysis was performed using the Stata 14 software.

RESULTS

Participants of the study

During the entire period, results of primary CT studies of 260,594 patients were analyzed (male/female ratio was 44%/56%, aged from 18 to 100 years, [average age 49.5 years]), performed and interpreted in the period from April 8, 2020 to December 1, 2020 in outpatient medical organizations in Moscow, repurposed for the pandemic period into outpatient CT centers.

Main results of the study

The test group included 115,618 CT studies (44.4% of the total sample), and the control group included 144,976 CT studies (55.6%). The ratio of studies in the control and test groups was similar. Distribution by subgroups in the test group was as follows: 98,953 studies (37.9% of the total sample) were for AI-A, 44,194 (17%) for AI-B, 24,067 (9.2%) for AI-C, 22,679 (8.7%) for AI-D, and 10,645 (4.1%) for AI-E.

For different subgroups with AI services, 2.3%–18.5% fewer CT-0 categories were found (no COVID-19 lesions were detected) compared to the control group, and 4.7%–27.6% fewer CT 3–4 categories was found in different subgroups with AI services than in the control group. In addition, 40%–60% fewer CT-4 categories were found in different subgroups with AI services than without AI (Fig. 3; p <0.0001).

 

Fig. 3. Results of comparison of primary chest CT scans performed in outpatient CT centers in terms of the severity of CT 0–4 categories between the control group and test subgroups for the entire period (April 8, 2020–December 1, 2020). n=260 594; p <0,0001.

 

In November 2020, results of primary CT scans of 41,386 patients were analyzed (male/female ratio was 44%/56%, aged from 18 to 100 years, with the average age of 53.2 years), performed and interpreted in the period from November 1, 2020 to December 1, 2020 in outpatient medical organizations in Moscow, repurposed for the period of the pandemic into outpatient CT centers.

The test group included 28,881 CT scans (69.8% of the total sample in November), and the control group included 12,505 CT scans (30.2%). Distribution by subgroups in the test group was as follows: 12,266 studies (29.6% of the total sample in November) were for AI-A, 11,922 (28.8%) for AI-B, 9,785 (23.6%) for AI-C, 9,942 (24%) for AI-D, and 8,602 (20.8%) for AI-E.

In different subgroups with AI services, 1%–2.6% fewer CT-0 (no COVID-19 lesions were detected), 0.2%–15.7% more CT 3–4, and 25% fewer CT-4 categories were found compared to the control group (Fig. 4; p = 0.0010).

 

Fig. 4. Results of comparison of primary chest CT scans performed in outpatient CT centers, according to the severity of the CT 0–4 categories between the control group and test subgroups for November 2020. n=41 386; p=0,0010.

 

For the period of November 2020, statistically significant differences were revealed only for CT-0, whereas no differences were found for the rest of the categories. However, a minimal statistically significant difference was found in CT-0 (18.6% vs. 17.0%). By the general monitoring period, on the contrary, all categories were statistically significantly different between data “without AI” and “total for all AIs”. The critical χ2 value was 4. For the total period, all CT 0–4 categories made a significant contribution to the differences. The minimum χ2 value was 26.2 for CT-3 (p <0.0001).

DISCUSSION

Summary of the main result of the study

Study results revealed a change between the CT 0–4 scale categories generated by radiologists in the presence of results from processing by AI services and categories formed without the use of AI services.

Discussion of the main result of the study

In general, prior to the COVID-19 pandemic, AI services were used to reveal radiological symptoms to detect diseases, classify and improve images, reduce radiation exposure, and improve the workflow [15].

Medical research makes AI applications more understandable, safer, more efficient, and more integrated into medical workflows [16]. It should be understood that validation of the AI algorithm should include not only a retrospective analysis of accuracy compared to the golden standard, but also a randomized clinical trial to assess the influence of AI on the decision making by healthcare professionals [17]. For example, in a randomized clinical study HYPE, the effect of the machine learning-based early diagnostics system of intraoperative hypotension was demonstrated, so in the intervention group, the median time of hypotension was 8.0 min versus 32.7 min in the control group (p <0.001), and the number of lethal outcomes in the intervention group was 0 versus 2 in the control group [18]. In another major study using AI to analyze chest X-rays in COVID-19, 20% of surveyed doctors reported that the algorithm influenced clinical decision making [19].

Available literature provides no similar studies assessing the effect of AI results on the final decision making by radiologists when diagnosing lung changes with suspected COVID-19 based on chest CT data.

AI for COVID-19 has two tasks, namely detection and classification. The detection task is determined by the difference between CT-0 and all other categories. The classification task consists of identifying differences between different CT categories (CT1–CT4), i.e., different volumes of damage to the pulmonary parenchyma.

The finding 1 was the difference in the proportion of the CT-0 category between all subgroups. In the case of AI service A, the radiologist was shown only a part of the slices with changes characteristic of COVID-19, while all other AI services had equal number of slices as in the original CT series. Opinions of doctors without AI-A were close to those of the control group.

The finding 2 was the difference in the proportion of CT 3–4 categories between all subgroups. Probably, with a large amount of lung lesions, the empirical visual estimation of the radiologist may exaggerate the extent of the lesion. This overdiagnosis is compensated for when the radiologist monitors the segmentation of lung injuries, performed with the use of AI. This increases the degree of confidence in the automatic calculation of volume and category according to the CT 0–4 scale. Since a lung injury threshold of >50% (CT 3–4 categories) was used as justification for hospitalization, this, combined with clinical and laboratory findings, may reduce the number of hospitalizations.

Based on publications by S.P. Morozov et al. on prediction of lethal outcomes in COVID-19 according to chest CT, when transfer from one CT category to the next, the risk increased by an average of 38% (95% confidence interval 17.1–62.6), and in an additional study among patients with laboratory-verified COVID-19, the risk of lethal outcome with the CT-4 category was 3 times higher than with CT-0 [12, 20]. Our study has demonstrated a fewer CT-4 categories in the test subgroup than in the control one. Previous studies have revealed that CT-4 assessment of the degree of lung damage is associated with lethal outcomes in COVID-19 more than all other categories (CT 0–3) [12, 20]. Thus, the change in the number of patients with CT-4 categories is essential for the formation of calculators of mortality risks for patients with COVID-19.

Due to the pandemic, the simultaneous launch of all AI services was recognized to be limiting the potential benefits of using AI, since all developers would have to be expected to participate in the experiment. Therefore, throughout 2020, unevenness of different AI services in joining the experiment was reported. An analysis was performed for November obtained additional results when the number of CT studies in the control group and each test subgroup was comparable to each other.

According to the authors, the difference revealed between results of periods 1 and 2 is associated with several factors as follows:

1) different number of AI services;

2) technical factor, as until 2020, the AI services participating in the experiment did not have the opportunity to train their algorithms for assessing and diagnosing lung damage to diagnose COVID-19, therefore, during the experiment, the possibility of changing the AI based software version in order to improve the quality of algorithms and potentially more benefits was recognized justified;

3) the human factor, as until 2020, doctors did not use the assessment of the chest CT according to the CT 0–4 scale, which presents a certain difficulty in terms of assessing the volume of multiple lesions in the lungs. It should be noted that doctors could independently improve their skills in assessing the volume of lung lesions in COVID-19, since they evaluated a significant number of CT studies during the pandemic. In addition, doctors could gain experience by checking the markup performed by the AI service, which could lead to an improvement in the skill in more correct assessment of the lesion amount.

In URIS, the radiologist has the opportunity to leave feedback on the work of the AI service in a special feedback field. The study prospect is the comparison of distribution of categories on the CT 0–4 scale among radiologists who have not encountered AI services during the pandemic and use the results of AI services based on their feedback.

Research limitations

Our research has a number of limitations. It did not include patients with positive results of the polymerase chain reaction test for COVID-19 verification, since results of these studies were after the CT scan. The study was not randomized. The extent of agreement of radiologists with the results of AI services was not assessed. In the test group, some of the CT scans were analyzed by several AI services. AI services were not registered as medical devices. Over the course of the pandemic, AI services were changed as the quality of CT processing improved, and this fact was not further evaluated in this study. The adaptation of radiologists to the use of the CT 0–4 scale was not taken into account.

URIS, where doctors formed medical reports, provides a special field for feedback on the work of AI services. However, at the time of the publication formation, results of feedback from doctors were being processed; therefore, it cannot be presented in the current study.

CONCLUSION

Results reveal that the use of AI services for primary chest CT scans in outpatient settings leads to a decrease in the number of CT-0 and CT 3–4 categories, which can influence the management of patients with COVID-19.

Additional research is required to assess whether reducing the choice of the above categories is appropriate for patient management, and how change in routing further affects recovery and mortality rates.

ADDITIONAL INFORMATION

Funding source. The study had no sponsorship.

Competing interests. The authors declare that they have no competing interests.

Author contribution. The authors confirm that they meet the ICMJE international criteria for authorship (have read and approved the final version before publication). The largest contributions are distributed as follows: S.P. Morozov — research concept; V.Yu. Chernina — search for publications on the topic of the article, writing the text of the manuscript; A.E. Andreychenko — data set formation, editing the manuscript text; A.V. Vladzymyrskyy — editing the manuscript text; O.A. Mokienko — expert information evaluation, editing the manuscript text; V.A. Gombolevskiy — research concept, expert information evaluation, writing the manuscript text, approval of the final manuscript version. Acknowledgements: The authors would like to express their gratitude to the teams of radiology departments of medical organizations of the Department of Healthcare of Moscow. The authors would like to thank the teams of the Department of Information Technology of the City of Moscow and Laval LLC for their concerted efforts in implementing artificial intelligence in practical healthcare at the level of a large metropolitan city. In addition, the authors thank separately the developers of Binomics ray, RADLogics, IRA labs, CareMentor AI, Third Opinion and Gammamed. Each of the contributors made important research efforts during a difficult time of the epidemic. The authors thank separately O.V. Omelyanskaya, E.G. Bakhteeva, I.A. Vinogradova, S.O. Ermolaev, L.G. Rodionova, K.V. Khripunova, K.M. Arzamasov, P.A. Nikolaev, S.F. Chetverikov, I.A. Blokhin, for administrative, pedagogical, and test work in preparing and implementing the Experiment; special thanks to V.G. Klyashtorny for statistical analysis.

About the authors

Sergey P. Morozov

Moscow Center for Diagnostics and Telemedicine

Email: morozov@npcmr.ru
ORCID iD: 0000-0001-6545-6170
SPIN-code: 8542-1720
Scopus Author ID: 57200964938
ResearcherId: T-9163-2017

Dr. Sci. (Med.), Professor

Russian Federation, Moscow

Valeria Y. Chernina

Moscow Center for Diagnostics and Telemedicine

Email: v.chernina@npcmr.ru
ORCID iD: 0000-0002-0302-293X
SPIN-code: 8896-8051
Scopus Author ID: 57210638679
ResearcherId: AAF-1215-2020

MD

Russian Federation, Moscow

Anna E. Andreychenko

Moscow Center for Diagnostics and Telemedicine

Email: a.andreychenko@npcmr.ru
ORCID iD: 0000-0001-6359-0763
SPIN-code: 6625-4186
Scopus Author ID: 42960997200
ResearcherId: E-4930-2017

Cand. Sci. (Phys.-Math.)

Russian Federation, Moscow

Anton V. Vladzymyrskyy

Moscow Center for Diagnostics and Telemedicine

Email: a.vladzimirsky@npcmr.ru
ORCID iD: 0000-0002-2990-7736
SPIN-code: 3602-7120
Scopus Author ID: 8944262100
ResearcherId: D-1447-2017

Dr. Sci. (Med.)

Russian Federation, Moscow

Olesya А. Mokienko

Moscow Center for Diagnostics and Telemedicine

Email: Lesya.md@yandex.ru
ORCID iD: 0000-0002-7826-5135
SPIN-code: 8088-9921
Scopus Author ID: 55155448000
ResearcherId: J-3210-2016

Cand. Sci. (Med.)

Russian Federation, Moscow

Victor A. Gombolevskiy

Moscow Center for Diagnostics and Telemedicine

Author for correspondence.
Email: v.gombolevskiy@npcmr.ru
ORCID iD: 0000-0003-1816-1315
SPIN-code: 6810-3279
Scopus Author ID: 57196441765
ResearcherId: J-3389-2017
https://www.scopus.com/authid/detail.uri?authorId=57204359134

Cand. Sci. (Med.), Head of Medical Research Department

Russian Federation, Moscow

References

Supplementary files