A new AI program for the automatic evaluation of scoliosis on frontal spinal radiographs: Accuracy, pros and cons.

Full Text

Introduction

Scoliosis is a common spinal pathology worldwide that can affect any age group. It is defined as a lateral curvature of the spine in the coronal plane by 10 degrees or more, with torsion of the vertebral bodies and their posterior elements during growth. The diagnosis of scoliosis is mainly done on frontal radiographs. The standard method which is used to measure the angle of scoliosis is Cobb’s method [‎1, ‎2]. In recent years, using artificial intelligence (AI) for evaluating scoliosis on different imaging modalities has been being studied actively by different researchers, in order to objectify scoliosis evaluation and decrease intra and inter observer measurement variability of the usual Cobb’s method [‎3, ‎4, ‎5]. We developed a new program based on machine learning (ML). The first stage of creating the system was training the artificial neural network (ANN) to identify vertebral bodies. For this purpose, 1000 images were selected from the database (XrScl) [‎6]. We performed markings for each image by identifying the vertebral bodies using four reference points and also performed numbering of all thoracic and lumbar vertebrae which were found on each image. After training, the network was successfully able to independently define the vertebral bodies and their numbers. The second stage of our method is the automatic measurement of Cobb’s angle and determining the grade of scoliosis, which is done automatically by the algorithm. Three different methods can be used to measure Cobb’s angle using the automatic system: (1) the classical Cobb’s method, where the angle of scoliosis is measured between upper and lower endplates of the upper and lower end vertebrae (EV); (2) the method which is used in practice by radiologists (at the Mariinsky city hospital), where the angle of scoliosis is constructed between the maximally inclined upper or lower endplates of the EV and (3) the method of middle lines, in which the angle of scoliosis is constructed between lines drawn along the midpart of vertebral bodies of the EV.  We tested the accuracy of the new automatic system in measuring Cobb’s angle and in defining the grade of scoliosis.

AIM

To analyze the accuracy, advantages and disadvantages of a newly developed AI program for automatic diagnosis of scoliosis and measurement of Cobb’s angle on frontal spinal radiographs.

MATERIALS AND METHOD

Design of the study

Diagnostic accuracy study

Study description and statistical analysis

To assess the reliability of the program in measuring curve’s angle, 112 digital spinal radiographs, as well as chest X-rays (CXR), were selected from the database XrScl (test set 1) [‎6]. We tested the agreement of Cobb’s angle measurements between the new automatic method and the radiologist using Blant-Altman method on Microsoft Excel. We calculated the mean difference (MD) between the two methods (the 'bias'), and 95% limits of agreement (2 standard deviations SD). Only the angles that were found by both the doctor and the AI system were compared

To test the possibility of using the new system in medical practice, a limited clinical test was conducted in Mariinsky city Hospital. For this study, 120 radiographs were collected from the archive of Mariinsky city Hospital and G.A. Albrecht federal state budgetary research center (test set 2). We tested the reliability of the system in defining the grade of scoliosis by calculating sensitivity TP/(TP+FN); specificity TN/(TN+FP); accuracy (TP+TN)/(TP+TN+FP+FN); and area under the ROC curve (ROC AUC).

RESULTS

Objects of the study

In (test set 1), the radiographs were distributed into 4 severity groups according to decree No. 565 of the government of the Russian Federation [‎‎7], grade I (5°-10°―17%); grade II (11°-25°―16%); grade III (26°-50°―15%) and grade IV (>50°―13%). A normal group /grade 0 was added to this classification (<5°― 39%). 179 angles with a range from 5,1° to 88,1° were found by both the radiologist and the system. 

For the limited clinical test of the system, (test set 2) was divided by two radiologists into two equal groups (60 radiographs for each), as either normal or pathologic (with scoliosis). The images with scoliosis were evenly distributed into 4 grades from 1 to 4 (15 studies for each grade).

The main results of the study

The agreement of Cobb’s angle measurement between the system and the radiologist was found mostly in scoliosis with grades 1 and 2 with an average measurement difference of -0.10 and 0.46 respectively and SD of 1.29 and 1.73 respectively. Angles measured by the two methods differed by less than 4.5° in 95% of cases in all grades of scoliosis, except for the group with grade 3 scoliosis, where 95% of variability (limits of agreement) ranged from -6.60° to 7.85°. The largest standard deviation (3.69°) was found also in this group. The results of statistical data analysis for each grade of scoliosis are presented in figures 1 and 2 (Bland Altman plots) and in table 1. Only 2.8% of the results were found to be clinically unsatisfying with an angle variability of more than 5°.

Table 1 - statistical parameters for evaluating the agreement between the two methods

Diagnostic accuracy metrics of the limited clinical test in Mariinsky city hospital using (test set 2) had also proved the reliability of the system, with sensitivity = 0.97; specificity = 0.88; accuracy (general validity) = 0.93; area under the ROC curve (ROC AUC) = 0.93 (Figure 3). The average analysis time of the image by the newly proposed system was 5 seconds for each study. These results confirm the effectivity of the system in determining the grade of scoliosis.

DISCUSSION

Summary of the main research result

The new automatic system for evaluating scoliosis on digital radiographs can help the radiologist to define the grade of scoliosis and measure curve’s angle, especially in situations such as screening adolescents for scoliosis to recognize if they match the criteria for army enlistment and also in situations with heavy workload in outpatients’ clinics. In these situations, the program can be used by the radiologist as an objective tool, significantly saving his time and increasing the accuracy of scoliosis evaluation on frontal radiographs. Moreover, the results of measuring Cobb’s angle by the system were shown to be acceptable with no significant clinical variability in most of the evaluated curves. So, applying this method into clinical practice can help in decreasing inter and intra- observer variability which is a common disadvantage of the usual Cobb’s method. Figure 4 shows an example of how the system works.

Discussion of the main results of the study

A significant advantage of our program is that it provides multiple options for the radiologist for measuring Cobb’s angle in three different methods. These options can be hugely helpful for the radiologist especially if he does not use the standard Cobb’s angle measurement method (such as in Mariinsky city hospital when screening for scoliosis). The step of choosing which endplate is the most tilted is time consuming and can be an added cause for inter observer measurement variability. Therefore, the ability to objectively and automatically define the most tilted endplates which form the largest angle in one curve would offer a great benefit for the radiologist.

 Also, when using the program, the radiologist has an access to change vertebral markings which are done by the system automatically. This is very essential to overcome any errors in vertebral markings that can cause a false final result by the system.

Further analysis of the results had shown that the main factor that had led to less accurate results of angle measurement and scoliosis grade definition is the inaccurate marking of vertebral bodies and their borders. This was mainly seen when evaluating scoliosis in images with poor quality and also in CXRs. The borders of mid thoracic vertebrae in CXRs usually are not seen posterior to the mediastinum. Multiple normal CXRs (grade 0, defined by the radiologist) were recognized by the system as grade 1 scoliosis because of detecting a false positive curve (proximal or mid thoracic curve) (Figure 5). Other common error was also seen in marking the borders of L5 vertebral body (Figure 6). The adjacent bony structures (adjacent sacrum and iliac bones) limit the definition of L5 borders, especially its lower endplate.

In addition, regarding errors in evaluating images with group 0 scoliosis, it was noted that most of the measurements by the AI system resulted in angles more than 5 degrees but very close to it. However, such minimal variability in measurement between the system and the radiologist had led to change the grade from 0 to 1 (Figure 7). The majority of such angles (70%) were found in the range of 5°- 6°, as shown in the pie chart (Figure 8).

Also, analyzing results of system in evaluating images with sever scoliosis (grade 3 or 4) has shown that in most cases, the system accurately evaluated radiographs with sever scoliosis though errors in vertebral detection and numbering or errors in measuring Cobb’s angle sometimes were noted. Grade IV scoliosis is characterized by maximum vertebral rotation, with displacement of the pedicle beyond midline and deformation of vertebral body. So that the usual shape of the vertebra changes, and the edges of its body become less defined (Figure 6).

It was interesting to find that sometimes errors that led to significant variability in measuring Cobb’s angle between the radiologist and the AI system on radiographs with grade 3 and 4 scoliosis did not affect the accuracy of the system in diagnosing grade III or IV scoliosis. As usually these errors had been mainly found when assessing a non-primary (secondary) curvature (Figure 7).

Other limitation of the current AI program is that it can only use to evaluate frontal radiographs. Evaluating of scoliosis on sagittal radiographs and on other modalities such as CT cannot be applied. 

Most of the above-mentioned errors of the system can be fixed be the radiologists by using the provided access to change vertebral markings in order to get the right results. The time needed to get the results even if the radiologist has to make changes in the automatic vertebral markings done by the system is still shorter than getting the results through the usual Cobb’s method. Also, we believe that in future, marking of the vertebrae will be more accurate when providing the network by much larger dataset. 

CONCLUSION

The automatic system can be used as a reliable objective tool to define the grade of scoliosis and measure Cobb’s angle on frontal spinal radiographs, significantly saving the time of the radiologist. The major factors that can affect the results of the program are the quality of radiographs and the accuracy of vertebral markings. Those factors can be overcome in practice when using the program, as the radiologist has an access to correct the vertebral markings and get the best results.

ADDITIONAL INFORMATIONS

Funding source. This study was not supported by any external sources of funding.

Competing interests. The authors declare that they have no competing interests.

Author contribution. All authors confirm that their authorship meets the international ICMJE criteria (all authors have made a significant contribution to the development of the concept, research and preparation of the article, read and approved the final version before publication). D. KH. I. Kassab —literature review, collection and analysis of literary sources, writing the text and editing the article, statistical analysis; I. G. Kamyshanskaya —; S.V. Trukhan — program creation, research concepts and statistical analysis.

About the authors

Дима Халед Ибрагим Кассаб

Author for correspondence.
Email: dimakk87@gmail.com
ORCID iD: 0000-0001-5085-6614

Рентгенолог, Аспирант

References

Supplementary files