Diagnostic accuracy of computed tomography for identifying hospitalizations for patients with COVID-19

Full Text

BACKGROUND

By January 14, 2021, the COVID-19 epidemic had caused approximately 92 million registered cases of infection worldwide, as well as approximately 2,000,000 lethal outcomes [1]. The SARS-CoV-2 virus disease can result in several scenarios. Symptoms may be completely absent or similar to flu-like symptoms (80%). In severe and critical cases, oxygen support or the use of an artificial lung ventilation apparatus (15% and 5%, respectively) is required [2]. Since a common manifestation of COVID-19 is viral damage to the lungs, methods of X-ray diagnostics represent one of the main tools for assessing disease severity and deciding whether hospitalization is required.

In accordance with the Temporary Methodological Recommendations of the Ministry of Health of the Russian Federation for the prevention, diagnosis, and treatment of new coronavirus infection COVID-19, for the assessment of changes in the lungs and differential diagnosis with other diseases, a visual semi-quantitative “empirical” scale of the degree of lung damage (computed tomography (CT) 0–4) should be used [3]. To this end, in their work, Morozov et al. [4] demonstrated not only the value of the scale as a predictor of lethal outcomes in COVID-19 patients but also its practical significance for routine patient management. However, despite the widespread use of the tool, as of this writing, an assessment of the numerical values of the indicators of its diagnostic accuracy has not been performed.

The study aimed to determine the diagnostic accuracy of the scale by four sequential CT studies to track the dynamics of the disease and make a decision on hospitalization of COVID-19 patients.

MATERIALS AND METHODS

Study design

Within this retrospective study, we used the database of the Unified Medical Information Analysis System (UMIAS) in Moscow to analyze a sequential sample of patients who were treated in medical organizations in Moscow from March 1 to August 1, 2020.

Compliance criteria

Inclusion criteria were patients with a confirmed diagnosis of new coronavirus infection, aged 18 years or older, who underwent 4 consecutive CT examinations of the thoracic organs (TO). The diagnosis of COVID-19 in the enrolled patients was confirmed by detection of SARS-CoV-2 viral RNA in throat swabs by reverse transcription polymerase chain reaction (RT-PCR).

Exclusion criteria were age under 18 years, duplicate records, and records with incomplete data.

Outcome registration methods

Studies of the chest organs were conducted using the recommended scanning parameters for patients with average anthropometric parameters (height 170 cm, weight 70 kg), voltage 120 kV, automatic adjustment of the current strength, scanning area 350 mm, and slice thickness 1.5 mm or thinner. The presentation was interpreted by roentgenologists who received special training in chest CT for COVID-19. Each medical description was sent to expert approbation at the Moscow Reference Center for Radiation Diagnostics.

For assessment of the results of CT studies, a semi-quantitative CT scale of 0–4 was used [5] in accordance with the Temporary Methodological Recommendations of the Ministry of Health of the Russian Federation for the prevention, diagnostics, and treatment of the new coronavirus infection COVID-19 [3]. According to the CT 0–4 scale, there are five degrees of lung tissue damage in COVID-19, namely normal (CT0), mild (CT1), moderate (CT2), severe (CT3), and critical (CT4). For patients with a mild to moderate course of the disease, medical care is allowed at home. Patients in severe and critical condition are subject to immediate hospitalization in a structural unit of a medical organization for the treatment of COVID-19.

If the radiologist, when interpreting medical descriptions, indicated the course of the disease as moderate (CT1–CT2) or severe (CT3–CT4), the patient was assigned the more severe of the two categories. If the doctor did not indicate the category on the CT scale of 0–4, but noted a positive or negative tendency, then in the presence of data on the previous CT study, the category was lowered or raised by one level, respectively.

To assess the duration of the convalescence period from COVID-19 pneumonia, the difference between the date of laboratory confirmation of the diagnosis and the date of the first of four CT examinations was determined, and according to the results, the patient was assigned to the CT0 category, provided that the category did not change in subsequent studies. Patients who had not recovered by August 1, 2020 were not included in the convalescence time analysis. Missing values were processed by excluding a specific indication from the corresponding analysis

Statistical analysis

When assessing the sensitivity and specificity of the CT scale 0–4 to identify the need for hospitalization, the disease dynamics were studied according to the results of repeated CT studies in a sequential sample of patients.

In diagnostic tests, sensitivity is defined as the conditional probability that the test results will be positive in the presence of the condition under study. Specificity is the conditional probability that test results will be negative if the condition under study is not noted. In the present study, a positive test result corresponded to worsening of the patient’s condition, and based on the results of a repeated CT scan, the patient was assigned to category CT3 or CT4 (the result is “worse”). The test result was recognized as negative if, based on the results of the repeated study, the patient was assigned to categories CT 0–2, in other words, if their condition did not worsen and they were not subject to hospitalization (the result is “better”).

In the model presented, the need for hospitalization of the patient due to belonging to the CT3 or CT4 category (the condition “hospital”) was taken as the condition under study. The absence of the condition under study included all cases when treatment at home was prescribed for patients of CT categories 0–2 (condition “home”).

Thus, in the model presented, the sensitivity (Se) was estimated as the conditional probability P_{worse/hospital:}

$P_{worse/hospital}=\frac{Number of \text{'}hospital\text{'} patients after second CT study}{Total number of \text{'}hospital\text{'} patients}$ (1).

The specificity (Sp) of the model corresponded to the conditional probability P_better/home:

$P_{better/home}=\frac{Number of \text{'}home\text{'} patients after second CT study}{Total number of \text{'}home\text{'} patients}$ (2).

The Youden index J was calculated from the values of sensitivity and specificity:

J = Se + Sp – 1 (3).

The positive predictive value (PPV) and negative predictive value (NPV) of the test depend on the disease prevalence. The Exponential Smoothing (ETS [6]) and Auto-Regressive Integrated Moving Average (ARIMA [7]) models were used to predict the incidence of COVID-19 in Moscow. Daily information on all cases of COVID-19 infection in the period from March 6 to November 28, 2020 was obtained from the Russian Agency for Health and Consumer Rights website [8]. Time series analysis was performed using R 3.6.3 [9] with the use of the forecast [10] and ggplot2 [11] packages. The development of the disease prevalence was assessed for a period of 120 days. For accuracy of assessment, the model was trained on morbidity data from March 6 to November 15, 2020, after which the predicted and actual values for the period from November 15 to November 28, 2020 were compared using the metrics of mean absolute percentage error (MAPE) and mean absolute scaled error (MASE).

Using the prevalence value, PPV was calculated as follows:

$PPV = \frac{Sensitivity\times prevalence}{(sensitivity\times prevalence)+(1-specificity)\times (1-prevalence)}$ (4).

The test NPV was calculated in the same way:

$NPV=\frac{Specificity\times (1-prevalence)}{Specificity\times (1-prevalence)+(1-sensitivity)\times prevalence}$ (5).

RESULTS

Objects (participants) of the study

Records of 139,592 patients of medical organizations in Moscow for the period from March 1 to August 1, 2020 were assessed for compliance with the criteria for inclusion in the study. After exclusion of 139,017 participants for various reasons, for statistical analysis, data from 575 patients with confirmed COVID-19 was used, each of whom underwent four consecutive TCT (thoracic computed tomography) scans (Fig. 1).

 

Fig. 1. Scheme of the examination of the study participants.

 

Main research findings

The mean age of patients in the final sample was 57.2±13.9 (range 2292) years; the sample included 314 women (54.6%). During the study, 49 lethal outcomes were registered. CT scanning 1 revealed no signs of viral pneumonia in 70 patients; 223 participants had a mild degree of lung tissue changes (CT1), a moderate degree (CT2) was noted in 163 patients, severe degree (CT3) was registered in 84 cases, and critical (CT4) was revealed in 16 patients. For 19 patients, the severity level was not indicated due to other pulmonological diseases (n = 13), or the study was conducted earlier than March 1, 2020 (n = 6). The average time period between TCT 1 and 2 was 9.4±8.3, that between TCT 2 and 3 was 10.2±8.1, and the period between TCT 3 and 4 was 22.6±17.5 days.

The dynamics of the distribution of disease cases in the sample according to the degree of severity followed a clear pattern. In the interval between TCT studies 1 and 2, the number of patients in categories CT0 and CT1 decreased with an increase in the number of patients of categories CT2, CT3, and CT4. In the interval between studies 2 and 3, the number of patients in categories CT0, CT3, and CT4 stabilized, and relative stability in the number of patients with mild and moderate changes was noted in the presence of an increase in the number of CT1 cases and a decrease in the number of CT2 cases. Finally, at stage 3, the tendencies of stage 1 were reversed, as there was a significant increase in the number of patients in the categories CT0 and CT1, with an equally noticeable decrease in the number of patients in the categories CT2, CT3, and CT4 (Fig. 2).

 

Fig. 2. Dynamics of the distribution of the number of patients according to the degree of changes in the lung tissue.

 

Convalescence time

To assess the convalescence time, the sample (n = 124) was divided into three cohorts:

1) patients of the categories CT1–CT4 according to the results of CT scan 1 of the TO, who had resolution of COVID-19 pneumonia after study 2 (CT0) without further deterioration of the clinical condition. This cohort included four patients with an average convalescence time of 23.5±4.9 days, all patients of the CT1 category according to the results of CT scan 1. Note that two patients from this cohort had positive RT-PCR tests for COVID-19 5 and 21 days later, respectively, after elimination of the characteristic manifestations of the disease;

2) patients of the categories CT1–CT4 according to the results of TCT study 2, who changed the category to CT0 according to the results of study 3 without further deterioration of the clinical condition. The cohort consisted of 12 patients, including 11 patients with CT1 based on study 2, and one patient had CT2. The mean convalescence time in the cohort was 36.3±21.3 days. Five patients also had positive test results 11.0±13.1 days after being assigned to category CT0;

3) patients of categories CT1–CT4 according to TCT study 3, who changed the category to CT0 according to the results of study 4. The cohort included 108 patients, one of whom died due to pathological changes not associated with COVID-19. According to the results of study 3, 81 patients from the cohort had category CT1, 16 patients had category CT2, 9 patients had category CT3, and two patients had category CT4. The average convalescence time was 36.0±24.3 days; four patients tested positive for COVID-19 16.0±17.1 days after being assigned to category CT0.

Assessment of diagnostic accuracy

When assessing the diagnostic accuracy of TCT and the semi-quantitative CT scale 0–4 to determine the need for hospitalization of COVID-19 patients, three stages of the study were considered separately (Fig. 2). According to the results of TCT study 2, the greatest specific changes in the clinical condition occurred among patients of the CT0 category, 53% of which changed category to CT1, 19% changed to CT2, and 6% had a severe disease course (Table 1).

 

Table 1. Categorization of participants between thoracic computed tomography scanning 1 and 2

Indicator	CT	Total*	Category according to the study 2 results
			CT0	CT1	CT2	CT3	CT4
Category according to the study 1 results	CT0	70	16	37	13	3	1
	CT1	223	7	122	71	18	4
	CT2	163	1	29	85	40	5
	CT3	84	0	1	15	56	12
	CT4	16	0	0	1	4	11

Note. *Here and in Tables 3 and 5, the discrepancy in the total number of patients according to the results of the adjacent TCT scan is associated with the missing results of the numerical study for some participants. Such cases were not excluded, since such data were available for other studies of these patients.

 

To calculate the values of specificity and sensitivity at stage 1 of the study from the data presented in Table 1, a 2 × 2 cross-classification table can be drawn up (Table 2).

 

Table 2. 2 × 2 table for stage 1 of the study

Indicator	Improvement	Deterioration	Total
Home	381	71	452
Hospital	17	83	100
Total	398	154	552

 

According to Table 2, from equations (1) and (2), the specificity and sensitivity of TCT when using the CT scale 0–4 to determine the need for hospitalization of COVID-19 patients were 84.3% and 83%, respectively.

When comparing the results of TCT studies 2 and 3, the greatest specific changes occurred in patients of the CT4 category, the clinical condition of 64% of whom improved (Table 3).

 

Table 3. Categorization of participants between thoracic computed tomography scanning 2 and 3

Indicator	CT	Total*	Category according to the study 3 results
			CT0	CT1	CT2	CT3	CT4
Category according to the study 2 results	CT0	24	11	5	7	1	0
	CT1	197	13	158	20	6	0
	CT2	192	1	59	110	21	1
	CT3	122	0	4	39	75	3
	CT4	33	0	1	3	17	12

 

The specificity and sensitivity of TCT and the CT scale 0–4 for the stage 2 of the study were 92.9 and 69.5%, respectively [equations (1) and (2); Table 4].

 

Table 4. 2 × 2 table for stage 2 of the study

Indicator	Improvement	Deterioration	Total
Home	384	29	413
Hospital	47	107	154
Total	431	136	567

 

Finally, at the stage 3, the greatest specific changes in the clinical condition of patients were registered in the categories CT2 (the condition improved in 54% of patients and worsened in 3% of cases) and CT3 (the condition improved in 49% of patients and worsened in 3% of cases); Table 5.

 

Table 5. Categorization of participants between thoracic computed tomography scanning 3 and 4

Indicator	CT	Total*	Category according to the study 4 results
			CT0	CT1	CT2	CT3	CT4
Category according to the study 3 results	CT0	26	22	4	0	0	0
	CT1	229	81	144	3	1	0
	CT2	181	16	81	77	4	1
	CT3	121	9	17	33	57	4
	CT4	16	2	1	1	5	7

 

At stage 3 of the study, the specificity of TCT and the CT scale 0–4 to determine the need for hospitalization of COVID-19 patients was 98.8%, and sensitivity was 53.7% [equations (1) and (2); Table 6].

 

Table 6. 2 × 2 table for stage 3 of the study

Indicator	Improvement	Deterioration	Total
Home	428	5	433
Hospital	63	73	136
Total	491	78	569

 

If we consider all stages of the study as a single block, the approach sensitivity was 91.8% (95% CI 83.7–100), and the specificity was 68.7% (95% CI 52.1–85.3).

Assessment of NPV and PPV values

Prediction of the prevalence of COVID-19 in Moscow

According to the Russian Agency for Health and Consumer Rights website [8], the infection curve demonstrated exponential growth until July 1, 2020. After that, the number of daily new cases reached a constant level of 658±42 (Fig. 3).

 

Fig. 3. Prediction of the prevalence of COVID-19 in Moscow: actual data (black curve); ETS MMM model (yellow curve); ARIMA model (0,2,1) (red curve). The prognosis of the ETS ZZZ model is not displayed as it coincides with ARIMA (0,2,1). For each of the models, the 95% confidence intervals are shown in the corresponding dimmed color.

 

Phase II of exponential growth started between September 15 and 23, 2020 (Fig. 3). To select a predictive model, the data on the incidence of COVID-19 were divided into groups for training and testing, followed by training for various models of EST and ARIMA. According to the MAPE and MASE values, the ARIMA (0,2,1) and ETS ZZZ (autoselectable parameters) models were the best predictors of the test data (Table 7).

 

Table 7. Accuracy statistics for different predictive models

Model	MAPE	MASE
ARIMA (0,2,1)	0.233	0.634
ETS ZZZ	0.233	0.634
ETS MMM	1.436	4.063

 

The ARIMA (0,2,1) and ETS ZZZ models predicted an almost linear increase in new COVID-19 cases after the end of phase II of exponential growth (Figure 3). The most optimistic scenario offered by the ETS MMM model did not prove true already on the testing sample (Fig. 3, Table 7). As a result, the incidence curve is not likely to flatten over the estimated time period. The curve corresponding to the predictions of the ARIMA (0,2,1) and ETS ZZZ models is not asymptotic, which complicates the estimation of the limit value for the total number of COVID-19 cases in Moscow (Fig. 3).

Since the infection curve has not flattened, only the annual number of cases of COVID-19 infection can be estimated. According to the ARIMA (0,2,1) and ETS ZZZ models, this figure will reach 1,220,500 cases per year, provided the current tendency continues. It should be noted that this prognostic estimate is approximate, and there may be effects of seasonal fluctuations and expected mass vaccination, which cannot be predicted due to the available data. According to Rosstat data as of January 1, 2020, the population of Moscow was 12,678,079 people¹. Based on this information, the Moscow instantaneous prevalence of COVID-19, characterized as the percentage of cases in the urban population at risk of the disease, will amount to 9.63% by March 6, 2021.

Predictive value

Considering the value of the instantaneous prevalence of COVID-19 in Moscow, as well as the values of sensitivity and specificity presented in Tables 2, 4, and 6, the PPV and NPV can be calculated using equations (3) and (4) (Table 8).

 

Table 8. PPV and NPV values for different stages of the study

Stage	1	2	3	General value (95% CI)
PPV, %	35.9	50.9	82.6	56.5 (29.5–83.4)
NPV, %	97.9	96.6	95.2	96.6 (95.0–98.1)

 

Thus, the results of a retrospective follow-up of 575 patients with laboratory confirmed COVID-19 demonstrated 69% specificity, 92% sensitivity, 56% PPV, and 97% NPV of CT for making a decision on hospitalization of the patients.

DISCUSSION

This work aimed to evaluate the numerical indicators of the diagnostic accuracy of TCT and the CT scale 0–4 for making a decision on hospitalization of COVID-19 patients. According to the results obtained, the scale showed average values of specificity and PPV with high sensitivity and NPV.

The role and importance of CT for the detection of COVID-19 pneumonia, their complications, and differential diagnostics with other lung diseases have caused an onslaught of discussion in the medical community [12, 13]. TCT scan has low rates of underdiagnosis [14]. In addition, a positive correlation of the CT scale 0–4 with mortality rates among patients with coronavirus pneumonia COVID-19 [4] has been demonstrated. However, the severity of the patient’s condition, as well as its dynamics, do not always correlate with the quantitative assessment of the volume of indurated lung tissue [15].

In the present work, we used the results of four consecutive TCT studies in patients with confirmed COVID-19. This enabled us to divide the study into three stages with pronounced trends in the change in the clinical condition of patients in the sample. At stage 1, corresponding to the period between CT examinations 1 and 2, deterioration in the clinical condition of the majority of patients was registered (Fig. 2). This stage was characterized by maximum sensitivity of the CT scale 0–4 (83.0%), the maximum Youden’s index (0.673), and the lowest PPV value (35.9%).

At stage 2, no significant changes in the number of categories of patients of different severity were registered (Fig. 2). This situation was accompanied by a decrease in sensitivity (−13.5%) and an increase in specificity (+8.6%) and PPV (+15%) of the CT scale 0–4, and Youden’s index was 0.624.

Finally, at the stage 3, corresponding to the period between CT studies 3 and 4, the majority of patients showed an improvement in their clinical condition (Fig. 2). At the same time, a further decrease in the method’s sensitivity (−29.3%) and an increase in specificity (+14.5%) and PPV (+46.7%) were noted. Youden’s index at this stage reached a minimum value of 0.525. All changes are given relative to the values of stage 1.

Study limitations

This study has limitations. The resulting convalescence times are higher than previously published values for cohorts 2 and 3 of participants (about 36 days). According to Bi et al. [16], the median convalescence time from COVID-19 is estimated at 20.8 days, while for patients aged 50–70 years and older, the period is increased to 22.6 days, and up to 28.3 days for patients with severe symptoms. This may be due to the methodology for assessing the indicator used in this work. The moment of convalescence was identified with the date of the CT scan, according to which the patient was transferred to category CT0, which is not always the proper approach [15].

Another study limitation was that, when the diagnostic value indicators were calculated, all patients with mild and moderate lung tissue changes were included in the “home” group, whereas, according to the Temporary Methodological Recommendations of the Ministry of Health of the Russian Federation for the prevention, diagnostics and treatment of new coronavirus infection COVID- 19, such patients should be hospitalized, and treatment at home is allowed only under certain conditions.

CONCLUSION

The CT scale 0–4 demonstrated the maximum diagnostic value under conditions of a high probability of deterioration in the condition of study participants, which confirms its practical significance for triage of patients in an unfavorable epidemic situation. The scale enables the development of pathological changes in patients of the CT0–CT1 categories to be ruled out with high confidence, thereby optimizing the workload on in-patient hospitals.

ADDITIONAL INFORMATION

Funding source. This study was not supported by any external sources of funding.

Competing interests. The authors declare that they have no competing interests.

Author contribution. S.P. Morozov ― conceptualization and funding of the study, editing of the text of the article; R.V. Reshetnikov ― collection and analysis of literary sources, statistical analysis, preparation and writing of the text of the article; V.A. Gombolevsky ― conceptualization of the study, data preparation; N.V. Ledikhova ― collection and preparation of data; I.A. Blokhin ― writing and editing the text of the article; O.A. Mokienko ― conceptualization of the study, editing of the text of the article. All authors made a substantial contribution to the conception of the work, acquisition, analysis, interpretation of data for the work, drafting and revising the work, final approval of the version to be published and agree to be accountable for all aspects of the work».

About the authors

Sergey P. Morozov

Moscow Center for Diagnostics and Telemedicine

Email: morozov@npcmr.ru
ORCID iD: 0000-0001-6545-6170
SPIN-code: 8542-1720

MD, Dr.Sci. (Med), Professor

Russian Federation, Moscow

Roman V. Reshetnikov

Moscow Center for Diagnostics and Telemedicine; I.M. Sechenov First Moscow State Medical University (Sechenov University)

Author for correspondence.
Email: reshetnikov@fbb.msu.ru
ORCID iD: 0000-0002-9661-0254
SPIN-code: 8592-0558

Cand.Sci. (Phys-Math)

Russian Federation, Moscow

Victor A. Gombolevskiy

Moscow Center for Diagnostics and Telemedicine

Email: gombolevskiy@npcmr.ru
ORCID iD: 0000-0003-1816-1315
SPIN-code: 6810-3279

MD, Cand.Sci. (Med)

Russian Federation, Moscow

Natalya V. Ledikhova

Moscow Center for Diagnostics and Telemedicine

Email: n.ledikhova@npcmr.ru
SPIN-code: 6907-5936
Russian Federation, Moscow

Ivan A. Blokhin

Moscow Center for Diagnostics and Telemedicine

Email: i.blokhin@npcmr.ru
ORCID iD: 0000-0002-2681-9378
SPIN-code: 3306-1387
Russian Federation, Moscow

Olesya A. Mokienko

Moscow Center for Diagnostics and Telemedicine

Email: o.mokienko@npcmr.ru
ORCID iD: 0000-0002-7826-5135
SPIN-code: 8088-9921

MD, Cand.Sci. (Med)

Russian Federation, Moscow

References

Supplementary files