A new artificial intelligence program for the automatic evaluation of scoliosis on frontal spinal radiographs: Accuracy, advantages and limitations

Full Text

Introduction

Scoliosis is a common spinal pathology that can affect any age group. It is usually defined as a lateral spinal curvature in the coronal plane with an angle of ≥10° and torsion of the vertebral bodies and their posterior elements. Scoliosis is mainly diagnosed based on frontal radiographs. Cobb’s method is considered the gold standard method to measure the scoliosis angle [1-8]. In recent years, many researchers have actively explored the use of artificial intelligence (AI) for evaluating scoliosis on different imaging modalities to objectify scoliosis evaluation and decrease intra- and interobserver measurement variability of Cobb’s method [9–14].

In this study, a new program based on machine learning was developed. In the first stage of system development, an artificial neural network was trained so that it can identify vertebral bodies automatically. Accordingly, 1,000 images were selected from the database (XrScl) [15]. Vertebral bodies in each image were marked by identifying their boreder susing four reference points, and all thoracic and lumbar vertebrae found on each image were numbered. After the training, the network could independently define the vertebral bodies and their numbers. In the second stage, Cobb’s angle was measured automatically, and scoliosis grade was determined automatically by the algorithm. Three methods can be used to measure Cobb’s angle using the automatic system: (1) the classical Cobb’s method, where the scoliosis angle is measured between the upper and lower endplates of the upper and lower end vertebrae (EV), respectively; (2) the method which is used in practice by radiologists (at Mariinsky City Hospital), where the scoliosis angle is constructed between the maximally inclined upper or lower endplates of the EV, and (3) the method of middle lines, in which the scoliosis angle is constructed between the lines drawn along the midpart of the vertebral bodies of the EV. The accuracy of the new automatic system was tested in measuring Cobb’s angle and in defining the scoliosis grade.

STUDY AIM

To analyze the accuracy, advantages, and disadvantages of a newly developed AI program for automatic diagnosis of scoliosis and measurement of Cobb’s angle on frontal spinal radiographs.

MATERIALS AND METHOD

Study Design

This is a diagnostic accuracy study.

Study Description and Statistical Analysis

To assess the reliability of the program in measuring Cobb’s angle, 114 digital spinal radiographs and chest X-ray (CXR) images, were selected from the database XrScl (test set 1) [15]. This database includes >2,500 radiographs; thus, we could use different sets of images for the training and evaluation of the system. The agreement of Cobb’s angles measured by the new automatic program and the radiologist was tested using the Blant–Altman method on Microsoft Excel. The mean difference between the two methods (“bias”) and 95% limits of agreement (2 standard deviations [SD]) was calculated. Only the angles that were obtained by both the radiologist and the AI system were compared.

To test the possibility of using the new system in medical practice, a limited clinical test was conducted in Mariinsky City Hospital. For this study, 120 radiographs were collected from the archive of Mariinsky City Hospital and G.A. Albrecht Federal State Budgetary Research Center (test set 2). The reliability of the system in defining the scoliosis grade was tested by calculating sensitivity as TP/(TP + FN), specificity as TN/(TN + FP), accuracy as (TP + TN)/(TP + TN + FP + FN), and area under the receiver operating characteristic curve (ROC AUC), where TP, TN, FP, and FN indicated true positive, true negative, false positive, and false negative, respectively.

The radiographs in data sets 1 and 2 were evaluated by two radiologists, one with more than 25 years of experience and the other of more than 9 year-experience.

RESULTS

Study Subjects

In test set 1, the radiographs were distributed into four severity groups (according to Russian Federation Law No. 565 [16]) as follows: grade I (5°–10°, 16%); grade II (11°–25°, 15%); grade III (26°–50°, 16%), and grade IV (>50°, 16%). A normal group (grade 0) was added to this classification (<5°, 37%). Moreover, 179 angles which ranged from 5.1° to 91.3°were found by both the radiologist and the system.

For the limited clinical test of the system, test set 2 was divided into normal or pathologic (with scoliosis) in equal groups (60 radiographs for each). The images with scoliosis were evenly distributed into four grades from 1 to 4 (15 studies for each grade).

Primary Findings

The agreement of Cobb’s angle measurements between the system and the radiologist was found mostly in grade 1 and 2 scoliosis, with an average measurement difference of −0.10 and 0.46 and SDs of 1.29 and 1.73, respectively. Angles measured by the two methods differed by <4.5° in 95% of cases in all scoliosis grades, except for the group with grade 3 scoliosis, where the 95% of the variability (limits of agreement) ranged from −6.60° to 7.85°. The largest SD (3.69°) was found also in this group. The results of statistical data analysis for each scoliosis grade are presented in Fig. 1 and 2 (Bland–Altman plots) and Table 1. Only 2.8% of the results were clinically unsatisfying with an angle variability of >5°.

Fig. 1. Results of the Bland–Altman method. Agreement between measurements of the two methods in radiographs with grade 1 and 2 scoliosis.

Fig. 2. Results of the Bland–Altman method. Agreement between measurements of the two methods in radiographs with grade 4 scoliosis.

Table 1. Statistical parameters for evaluating the agreement between the two methods

The diagnostic accuracy metrics of the limited clinical test in Mariinsky City Hospital using (test set 2) also proved the reliability of the system in defining the scoliosis grade, with a sensitivity of 0.97, specificity of 0.88, accuracy (general validity) of 0.93, and ROC AUC of 0.93 (Fig. 3). For each study, the average analysis time of the image by the newly proposed system was 5 s. These results confirm the effectiveness of the system in determining the scoliosis grade.

Fig. 3. ROC curve confirms the accuracy of the new AI program in defining the scoliosis grade.

DISCUSSION

Summary of Primary Findings

The bulk of this work was performed in a hospital where adolescents are screened for scoliosis, as part of multiple examinations for medical commissions, to determine whether adolescents match the criteria for army enlistment. Thus, radiographs in test set 1 and test set 2 were divided according to the severity classification included in Federal Law No. 565, where any small curves with angles <10° are considered grade 1 scoliosis [16]. The new automatic system for evaluating scoliosis on digital radiographs can help radiologists to define the scoliosis grade and measure the curve’s angle, particularly in situations such as screening for scoliosis and at times of heavy workload in outpatient clinics. In these situations, radiologists can use the program as an objective tool, increasing the accuracy of scoliosis evaluation on frontal radiographs because the interobserver variability can be avoided if radiologists use the same instrument. Moreover, it can significantly save the radiologist’s time. In addition, Cobb’s angles measured by the system were acceptable, with no significant clinical variability in most of the evaluated curves. Fig. 4 shows an example of how the system works.

Fig. 4. X-ray image showing grade 2 scoliosis. Analysis by the system (left) and by the radiologist (right). The radiologist did not measure the thoracolumbar curve as the lower EV is not shown in the image.

Discussion of Primary Findings

The proposed program provides multiple options for the radiologist to measure Cobb’s angle in three different methods. These options can help radiologists, particularly if he/she does not use the standard Cobb’s angle measurement method (such as that used in Mariinsky City Hospital when screening for scoliosis). The step of choosing the most tilted endplate is time consuming and can cause interobserver measurement variability. Therefore, the ability to objectively and automatically define the most tilted endplates, which form the largest angle in one curve, would be beneficial for the radiologist.

In addition, when using the program, the radiologist can change vertebral markings made by the system automatically. This is very essential to overcome any errors in vertebral markings that can cause a false final result. Further analysis of the results showed that the inaccurate marking of vertebral bodies and their borders predominantly led to less accurate results of angle measurement and scoliosis grade definition. This was mainly seen when evaluating scoliosis in poor-quality images and CXR images. The borders of the midthoracic vertebrae in CXR images usually are not seen posterior to the mediastinum. Multiple normal CXR images (grade 0, defined by the radiologist) were recognized by the system as grade 1 scoliosis because of detecting an FP curve (proximal or midthoracic curve) (Fig. 5). Other common error was also seen in marking the borders of L5 vertebral body (Fig. 6). The adjacent bony structures (adjacent sacrum and iliac bones) limit the definition of L5 borders, particularly its lower endplate.

Fig. 5. Errors in vertebral markings on radiographs with grade 0 (normal). In CXR images, poor definition of the vertebral borders may lead to false measurements.

Fig. 6. Errors in vertebral markings on radiographs with grade 0 (normal). Errors in defining the L5 vertebral body may lead to false positive curve detection (left).

Regarding errors in evaluating images with group 0 scoliosis, most of the measurements by the AI system resulted in angles with >5° but very close to it. However, such minimal variability in measurements between the system and the radiologist led to a change in the grade from 0 to 1 (Fig. 7). The majority of such angles (70%) were found in the range of 5°–6°, as shown in the pie chart (Fig. 8).

Fig. 7. Errors of the system in evaluating images with grade 0 scoliosis. Cobb’s angles measured by the radiologist (left) and the AI system (right). Measurement variability is not significant (1.4°); however, the scoliosis grade is 0 by the doctor and 1 by the system.

Fig. 8. Distribution of angles measured by the AI system on normal X-ray images; 70% of the 5°–6° range.

In addition, in most cases, the system accurately evaluated radiographs with severe scoliosis (grades III and IV), although errors in vertebral detection and numbering or errors in measuring Cobb’s angle sometimes were noted. Grade IV scoliosis is characterized by maximum vertebral rotation, with displacement of the pedicle beyond the midline and deformation of the vertebral body. Thus, the usual shape of the vertebra changes, and the edges of its body become less defined (Fig. 9).

Fig. 9. Specifications of vertebral marking in radiographs with grade 3 and 4 scoliosis. Errors in detection and numbering of vertebrae caused by the unusual shape of the vertebrae (yellow arrow).

In some cases, the accuracy of the system in diagnosing grade III or IV scoliosis was not affected by significant measurement variability of Cobb’s angle between the system and the radiologist. Usually, these errors had been mainly found when assessing a nonprimary (secondary) curvature (Fig. 10).

Fig. 10. Grade 3 scoliosis diagnosed by the radiologist (left) and the AI system (right). Significant variability in measuring the lumbar curve (7.8°) did not affect the overall scoliosis grade.

As another limitation, the proposed AI program can only be used to evaluate frontal radiographs. This system cannot evaluate scoliosis on sagittal radiographs and other modalities such as CT.

Most of the abovementioned errors of the system can be fixed by the radiologists when they are using the system because they are provided with an access to change vertebral markings to get more accurate results. The time needed to obtain the results even if the radiologist must make changes in the automatic vertebral markings done by the system was still shorter than getting the results through the usual Cobb’s method. In addition, in the future, we believe that vertebral marking will be more accurate when providing the network with a much larger dataset.

CONCLUSION

The automatic system can be used as a reliable objective tool to define the scoliosis grade and measure Cobb’s angle on frontal spinal radiographs, significantly saving the radiologist’s time. Major factors that can affect the results of the program are the quality of the radiographs and accuracy of the vertebral markings. Those factors can be overcome in practice when using the program, as the radiologist can correct the vertebral markings and obtain the best results.

ADDITIONAL INFORMATION

Funding source. This study was not supported by any external sources of funding.

Competing interests. The authors declare that they have no competing interests.

Authors’ contribution. All authors made a substantial contribution to the conception of the work, acquisition, analysis, interpretation of data for the work, drafting and revising the work, final approval of the version to be published and agree to be accountable for all aspects of the work. D.Kh.I. Kassab — literature review, collection and analysis of literary sources, writing the text and editing the article, statistical analysis; I.G. Kamyshanskaya — research concepts, data collection, analysis and review of the results, organizing communications between co-authors, writing the text, editing the article; S.V. Trukhan — program creation, research concepts and statistical analysis.

About the authors

Dima Kh. I. Kassab

Author for correspondence.
Email: DimaKK87@gmail.com
ORCID iD: 0000-0001-5085-6614
SPIN-code: 4907-7850

MD

Irina G. Kamyshanskaya

Email: irinaka@mail.ru
ORCID iD: 0000-0002-8351-9216
SPIN-code: 2422-5191

MD, Dr. Sci. (Medicine), Assistant Professor

Stanislau V. Trukhan

Email: stas.truhan@gmail.com
ORCID iD: 0000-0003-0688-0988
Russian Federation, Tver

References

Supplementary files

Supplementary Files

Action

1. JATS XML

Download

2. Fig. 1. Results of the Bland–Altman method. Agreement between measurements of the two methods in radiographs with grade 1 and 2 scoliosis.

Download (206KB)

Indexing metadata

3. Fig. 2. Results of the Bland–Altman method. Agreement between measurements of the two methods in radiographs with grade 4 scoliosis.

Download (227KB)

Indexing metadata

4. Fig. 3. ROC curve confirms the accuracy of the new AI program in defining the scoliosis grade.

Download (71KB)

Indexing metadata

5. Fig. 4. X-ray image showing grade 2 scoliosis. Analysis by the system (left) and by the radiologist (right). The radiologist did not measure the thoracolumbar curve as the lower EV is not shown in the image.

Download (267KB)

Indexing metadata

6. Fig. 5. Errors in vertebral markings on radiographs with grade 0 (normal). In CXR images, poor definition of the vertebral borders may lead to false measurements.

Download (181KB)

Indexing metadata

7. Fig. 6. Errors in vertebral markings on radiographs with grade 0 (normal). Errors in defining the L5 vertebral body may lead to false positive curve detection (left).

Download (166KB)

Indexing metadata

8. Fig. 7. Errors of the system in evaluating images with grade 0 scoliosis. Cobb’s angles measured by the radiologist (left) and the AI system (right). Measurement variability is not significant (1.4°); however, the scoliosis grade is 0 by the doctor and 1 by the system.

Download (270KB)

Indexing metadata

9. Fig. 8. Distribution of angles measured by the AI system on normal X-ray images; 70% of the 5°–6° range.

Download (58KB)

Indexing metadata

10. Fig. 9. Specifications of vertebral marking in radiographs with grade 3 and 4 scoliosis. Errors in detection and numbering of vertebrae caused by the unusual shape of the vertebrae (yellow arrow).

Download (204KB)

Indexing metadata

11. Fig. 10. Grade 3 scoliosis diagnosed by the radiologist (left) and the AI system (right). Significant variability in measuring the lumbar curve (7.8°) did not affect the overall scoliosis grade.

Download (244KB)

Indexing metadata