Application of artificial intelligence technologies in detecting adrenal neoplasms on computed tomography scans

Full Text

BACKGROUND

Adrenal neoplasms are one of the most common findings on computed tomography (CT) scans. According to population data, the detection rate of adrenal neoplasms using CT scans is 4% [1, 2]. However, this rate increases to 7%–10% in patients over 70 years of age [3–5]. Pooled autopsy data revealed that the prevalence of adrenal incidentalomas ranged from 1% to 32%, with an average of 6% [6, 7]. Various authors report that adrenocortical cancer accounts for 4%–12% of all adrenal neoplasms [8, 9].

Adrenal neoplasms often have distinctive features that can be identified through non-invasive diagnostic imaging. CT is the primary imaging technique that can identify the type of neoplasm in most cases [10–12].

Artificial intelligence (AI)-based solutions are being actively developed and implemented in clinical settings for the detection of adrenal neoplasms on CT scans. In addition to lesion detection, these systems perform segmentation on a separate Digital Imaging and Communications in Medicine (DICOM) series and provide graphical visualization of the data, including lesion size and attenuation in Hounsfield units [13, 14].

Moreover, AI-based technologies are used to interpret chest and abdominal CT scans and identify various abnormalities, including impaired lung aeration and lung neoplasms¹ [15–18]. Notably, AI-based solutions that detect adrenal neoplasms can also interpret chest CT images because the adrenal glands are usually within the scan field. This study assesses their diagnostic performance in identifying adrenal neoplasms using chest and abdominal CT scans.

AIM

To evaluate the performance of AI-based solutions in detecting adrenal neoplasms on CT images.

METHODS

Study Design

This was a retrospective, sample-based, single-center study.

Study Setting

The study used patient data extracted from the Unified Radiological Information Service of the Unified Medical Information Analysis System.

Two sets of CT data were used:²

• Abdominal CT (dataset 1);
• Chest CT (dataset 2).

Each dataset consisted of 100 previously anonymized CT scans with (n = 50) and without (n = 50) evidence of adrenal neoplasms [19, 20]. Two datasets were prepared using CT scans from patients over 18 years of age. The dataset was created using principles developed at the Center for Diagnostics and Telemedicine [21, 22].

Two radiologists with more than three years of experience reviewed the CT scans to verify the data. The scans were considered abnormal based on inter-rater agreement. In case of disagreement, an expert with more than 5 years of experience was consulted. The expert decided which scans to include in the dataset. Abnormal cases were identified by the presence of an adrenal lesion (body or limbs) measuring ≥10 mm in the short axis on an unenhanced CT image. Otherwise, cases were classified as normal.

Eligibility Criteria

Inclusion criteria:

Dataset 1 included:

• Abdominal and pelvic CT images with and without intravenous contrast enhancement;
• Abdominal CT images with and without contrast enhancement;
• CT images of the kidney and urinary tract with and without contrast enhancement;
• Adrenal CT images.

Dataset 2 included chest CT scans with and without intravenous contrast enhancement.

Non-inclusion criteria:

• Status post surgery;
• Technical defects;
• Images acquired with scanning protocol violations;
• Lack of expert validation.

Artificial Intelligence-Based Solutions

Two companies, Intel Diagnostik LLC and AIRA Labs LLC (which are anonymized and depersonalized throughout this text), have presented their solutions in the Experiment on the Use of Innovative Computer Vision Technologies for Medical Image Analysis and Further Use in the Moscow Healthcare System for the Detection and Assessment of Adrenal Neoplasms. Each company's solution is presented in two modifications:

• Asingle-target service designed only foradrenal neoplasm detection;
• Amulti-target AI service for theassessment ofabdominal andchest CT images.

The multi-target AI service was designed to detect up to 11 abnormalities in chest CT scans and up to six in abdominal CT scans, including adrenal neoplasms.

AI-based models interpret unenhanced and intravenous contrast-enhanced abdominal and chest CT scan series presented in the DICOM format.

The AI-based results are presented in two formats. First, a text report (DICOM Structured Report [SR]) providing details on the presence/absence, number, size, and attenuation (in Hounsfield units) of adrenal neoplasms. Second, a supplementary DICOM series with graphic delineation of regions of interest, along with tumor size and attenuation data. Each developer chose their own methodology for detecting abnormal findings, but this work did not consider that.

Table 1 shows the claimed metrics of the diagnostic performance of AI-based solutions for adrenal neoplasm detection using chest and abdominal CT scans.

 

Table 1. Claimed diagnostic accuracy parameters of AI-based solutions for detecting adrenal neoplasms in chest and abdominal computed tomography scans
Artificial intelligence services	Area under the curve	Sensitivity	Specificity	Accuracy
MS 1 (abdomen)	0.82	0.81	0.83	0.81
MTS 1 (chest)	0.84	0.84	0.84	0.81
STS 1 (chest)	0.84	0.84	0.84	0.81
MTS 2 (abdomen)	0.93	0.84	0.91	0.88
STS 2 (chest)	0.93	0.84	0.91	0.88
STS 2 (abdomen)	0.93	0.89	0.92	0.9
Note. MTS, multi-target AI service; STS, single-target AI service.

 

Fig. 1 shows examples of using AI-based services to detect adrenal neoplasms.

 

Fig. 1. Examples of using artificial intelligence (AI) services in adrenal neoplasm detection on chest and abdominal computed tomography (CT) scans: a, b, single-target AI service 1: blue outlines indicate adrenal neoplasms on abdominal CT images; c, d, single-target AI service 2: yellow outlines indicate adrenal neoplasms on abdominal CT images; e, f, multi-target AI service 2: yellow outlines indicate a left adrenal neoplasm on an abdominal CT image (e) and a right adrenal neoplasm on a chest CT image (f).

 

Main Study Outcome

Evaluation of the diagnostic performance of AI-based solutions in detecting adrenal neoplasms using chest and abdominal CT scans.

Outcomes Registration

Accuracy of AI-based solutions was assessed by calculating the following metrics:³

• area under the curve (AUC),
• accuracy,
• sensitivity,
• specificity.

Accuracy (Ac) was defined as the ratio of correctly classified cases to the total number of observations:

$Ac=\frac{TP+TN}{TP+FN+TN+FP}$, (1)

where: TR is the number of true positives; TN is the number of true negatives; FN is the number of false negatives; and FP is the number of false positives.

Sensitivity (Se) was defined as the percentage of true positives:

$Se=\frac{TP}{TP+FN}$, (2)

where: TR is the number of true positives; FN is the number of false negatives.

Specificity (Sp) was defined as the percentage of true negatives:

$Sp=\frac{TN}{FP+TN}$, (3)

where: TN is the number of true negatives; FP is the number of false positives.

Ethics Approval

This study was based on the Experiment on the Use of Innovative Computer Vision Technologies for Medical Image Analysis and Further Use in the Moscow Healthcare System approved by the Independent Ethics Committee of the Moscow Regional Branch of the Russian Society of Radiographers and Radiologists (Minutes No. 2 of February 20, 2020), also registered at ClinicalTrials.gov (NCT04489992).

Statistical Analysis

The overall AI service metrics are reported with 95% confidence intervals. The roc.test function (method=“delong”) in the R programming language was used to calculate and compare p-values. The null hypothesis, which states that there are no significant differences across AI-based solutions, was tested. The level of significance was set as p = 0.05 (two-sided).

RESULTS

Sample Characteristics

Table 2 shows the demographic characteristics of patients whose CT scans were included in the datasets.

 

Table 2. Demographic characteristics of the patient sample with CT scans included in the datasets
Dataset	Age, years			Sex, n
	Min	Max	Mean	Male	Female
№ 1	19	90	59	29	71
№ 2	50	94	68	40	60

 

Primary Results

A receiver operating characteristic (ROC) curve was plotted for each AI-based solution. The ROC curves are smoothed; the dots indicate the changes in sensitivity and specificity at the discrete thresholds corresponding to the incremental increases in probability of abnormal findings (see Figs. 2, 3).

 

Fig. 2. ROC curves of artificial intelligence (AI) services for adrenal neoplasm detection on chest and abdominal computed tomography (CT) scans: a, multi-target AI service 1 (chest); b, single-target AI service 1 (abdomen); c, single-target AI service 1 (chest); d, multi-target AI service 2 (abdomen); e, multi-target AI service 2 (chest); f, single-target AI service 2 (abdomen).

 

Fig. 3. ROC analysis results for artificial intelligence (AI) services for adrenal neoplasm detection on chest and abdominal computed tomography (CT) scans: yellow, multi-target AI service 1 (chest); red, single-target AI service 1 (abdomen); orange, single-target AI service 1 (chest); light blue, multi-target AI service 2 (abdomen); green, multi-target AI service 2 (chest); blue, single-target AI service 2 (abdomen).

 

Table 3 shows the results of testing AI-based solutions for adrenal neoplasm detection using chest and abdominal CT images.

 

Table 3. Diagnostic accuracy metrics of AI-based solutions for detecting adrenal neoplasms in chest and abdominal computed tomography scans using a test dataset
Artificial intelligence services	Area under the curve (95% CI)	Sensitivity (95% CI)*	Specificity (95% CI)*	Accuracy (95% CI)*
MTS 1 (abdomen)	0.978 (0.949–1.000)	0.940 (0.874–1.000)	0.980 (0.941–1.000)	0.960 (0.922–0.998)
MTS 1 (chest)	0.888 (0.823–0.952)	0.837 (0.733–0.940)	0.920 (0.845–0.995)	0.879 (0.814–0.943)
STS 1 (chest)	0.858 (0.793–0.926)	0.739 (0.612–0.866)	0.938 (0.869–1.000)	0.840 (0.766–0.914)
MTS 2 (abdomen)	0.995 (0.985–1.000)	1.000 (1.000–1.000)	0.958 (0.902–1.000)	0.979 (0.950–1.000)
STS 2 (chest)	0.964 (0.927–1.000)	0.878 (0.786–0.969)	1.000 (1.000–1.000)	0.938 (0.889–0.986)
STS 2 (abdomen)	0.993 (0.982–1.000)	1.000 (1.000–1.000)	0.940 (0.874–1.000)	0.969 (0.935–1.000)
Note. * The metrics were calculated based on the optimal threshold determined by the maximum Youden index. CI, confidence interval; MTS, multi-target AI service; STS, single-target AI service.

 

Table 4 compares the areas under the curve (AUCs) of AI services.

 

Table 4. Comparison of areas under the curve for the detection of adrenal neoplasms using artificial intelligence services and chest and abdominal computed tomography scans
i	j
	MTS 1 (abdomen)	MTS 1 (chest)	STS 1 (chest)	MTS 2 (abdomen)	STS 2 (chest)	STS 2 (abdomen)
MTS 1 (abdomen)	0.978	0.090	0.120	−0.017	0.014	−0.015
MTS 1 (chest)	−0.090	0.888	0.030	−0.107	−0.076	−0.105
STS 1 (chest)	−0.120	−0.030	0.858	−0.137	−0.106	−0.135
MTS 2 (abdomen)	0.017	0.107	0.137	0.995	0.031	0.002
STS 2 (chest)	−0.014	0.076	0.106	−0.031	0.964	−0.029
STS 2 (abdomen)	0.015	0.105	0.135	−0.002	0.029	0.993
Note. The cells show the difference in areas under the curve between the artificial intelligence services in row (i) and column (j). The main diagonal shows the area under the curve for each row and column of the artificial intelligence service. Differences were considered significant at p <  0.05. Significant differences are highlighted in green and in bold. MTS, multi-target AI service; STS, single-target AI service.

 

The presented data demonstrate the high diagnostic accuracy of AI-based solutions; the AUC, sensitivity, specificity, and accuracy exceeded 86%, 74%, 92%, and 84%, respectively. The maximum specificity and sensitivity were reported for AI Services 2. Accuracy can be used for a comprehensive assessment of the trade-off between these metrics. Multi-target AI service 2 (for interpretation of abdominal CT scans) showed the best results in absolute terms.

DISCUSSION

The clinical implementation of new techniques and solutions requires an evaluation of their impact on current diagnostic processes. This is especially true for AI-based solutions used in diagnostic radiology, where defining the optimal scope of use is essential [23].

Testing AI services for adrenal neoplasm detection using test datasets revealed high diagnostic accuracy, with some parameters showing the highest levels.

The relatively low specificity of AI services in adrenal neoplasm detection is attributed to false positives, where anatomical structures (such as vessels, diaphragmatic crura, or lymph nodes) were misinterpreted as abnormal findings, or normal adrenal glands were erroneously delineated. The lower sensitivity is attributed to false negative cases involving overlooked adrenal neoplasms or incomplete segmentation. This may be related to the use of CT scans reconstructed with bone filters, which do not clearly visualize soft tissue structures.

Figure 4 shows examples of AI errors in adrenal neoplasm detection.

 

Fig. 4. Examples of AI errors in adrenal neoplasm detection: a, overlooked adrenal neoplasm (yellow arrow); b, failure to detect a left adrenal neoplasm (false negative); c, abdominal CT images; d, pancreatic tail delineation as an adrenal neoplasm (false positive); e, undetected adrenal neoplasms (red arrows); f, failure to detect adrenal neoplasms on chest CT in the lung window (false negative).

 

Some international authors also reported using AI-based solutions to detect adrenal tumors. Robinson-Weiss et al. [24] developed a machine learning model that can segment adrenal glands in intravenous contrast-enhanced CTs and classify them as either normal or neoplastic. Two retrospective datasets were used to evaluate an AI-based model. Dataset 1 included 274 CT scans (mean age: 61 years; 133 women). Dataset 2 included 991 CT scans (mean age: 62 years; 578 women). Sensitivity and specificity were 83% and 89% for dataset 1, and 69% and 91% for dataset 2, respectively.

We used AI-based solutions to detect adrenal neoplasms in unenhanced CT scans. In practice, this reduced the risk of overlooking abnormal findings and alerted radiologists. However, intravenous contrast enhancement is required to classify neoplasms. Alimu et al. [25] used deep learning-based AI services to quantify and segment functional adrenal tumors in contrast-enhanced CT images. This model achieved an AUC of 0.915.

Chai et al. [26] employed a similar approach, presenting an automated method for interpreting contrast-enhanced CT images. This method included algorithms for segmenting and identifying radiomic features and classifying adrenal tumors. The proposed tool was integrated into the graphical user interface (GUI) of MATLAB® (MathWorks, Inc., USA) and tested on 236 CT scans. The results showed tumor classification accuracy rates of up to 90%.

In our study, all the AI-based solutions for adrenal neoplasm detection demonstrated high diagnostic accuracy, providing sensitivity and specificity of 74%–100% and 92%–100%, respectively.

Study Limitations

This study did not evaluate morphometric data, such as the size and attenuation of adrenal neoplasms. Further research is needed to understand the impact of data dispersion on the performance of AI services.

CONCLUSION

The findings suggest that AI-based solutions can be effectively used in clinical practice for adrenal neoplasm detection, demonstrating sensitivity, specificity, and accuracy of 74%–100%, 92%–100%, and 84%–98%, respectively However, there is a potential for false positives and false negatives, which may be associated with syntopy (for example, in patients who have undergone abdominal surgery or have a low body mass index). The quality of AI-based solutions can be improved by training them on more diverse, representative, and high-quality data. This reduces the likelihood of misinterpreting anatomical structures as abnormal changes and minimizes overlooked findings.

ADDITIONAL INFORMATION

Author contributions: D.U. Shikhmuradov: data curation, formal analysis, investigation, writing—original draft, writing—review & editing, visualization; T.M. Bobrovskaya, E.F. Savkina, R.A. Erizhokov, L.D. Pestrenin: data curation, formal analysis, investigation; K.M. Arzamasov: conceptualization, methodology, supervision, writing—review & editing. All the authors approved the final version of the manuscript for publication and agreed to be accountable for all aspects of the work, ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Ethics approval: This study is based on the results of the Experiment on the use of innovative computer vision technologies for analysis of medical images in the Moscow healthcare system, which was approved by the Ethics Committee of the Independent Ethics Council of the Moscow Regional Branch of the Russian Society of Radiologists (Minutes No. 2 dated February 20, 2020), registered at ClinicalTrials (NCT04489992).

Funding sources: This article was prepared by the author team as part of the research project “Scientific Methodologies for the Sustainable Development of Artificial Intelligence Technologies in Medical Diagnostics” (EGISU No. 123031500004-5), in accordance with Order No. 1196 dated December 21, 2022, On the Approval of State Assignments Funded from the Budget of the City of Moscow for State Budgetary (Autonomous) Institutions Subordinate to the Moscow City Health Department for 2023 and the Planned Period of 2024–2025, issued by the Moscow City Health Department.

Disclosure of interests: The authors have no relationships, activities, or interests for the last three years related to for-profit or not-for-profit third parties whose interests may be affected by the content of the article.

Statement of originality: No previously published material (text, images, or data) was used in this study or article.

Data availability statement: The editorial policy regarding data sharing does not apply to this work.

Generative AI: No generative artificial intelligence technologies were used to prepare this article.

Provenance and peer-review: This article was submitted unsolicited and reviewed following the standard procedure. The peer review process involved two external reviewers and a member of the Editorial Board, and the in-house science editor.

 

¹ Basic Diagnostic Requirements for AI Service Output [Internet]. In: Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies; 2024–2024. Available at: https://mosmed.ai/ai/docs/. Accessed on: April 13, 2024.

² Certificate of state registration of database No. 2023621091 of April 4, 2023. Bull. No. 4. Vasiliev Yu.A., Turavilova E.V., Shulkin I.M. et al. MosMedData: CT Scans with Signs of Adrenal Lesions. Available at: https://elibrary.ru/download/elibrary_52121913_33574960.PDF. Accessed on: April 13, 2024.

³ Certificate of state registration of computer software No. 2022617324 of April 19, 2022. Bull. No. 4. Morozov S.P., Andreychenko A.E., Kirpichev Yu.S. et al. A web-based tool for ROC analysis of diagnostic tests. Available at: https://elibrary.ru/download/elibrary_48373757_42748544.PDF. Accessed on: April 13, 2024.

About the authors

David U. Shikhmuradov

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Author for correspondence.
Email: ShikhmuradovDU@zdrav.mos.ru
ORCID iD: 0000-0003-1597-5786
SPIN-code: 9641-0913

MD

Russian Federation, Moscow

Kirill M. Arzamasov

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: ArzamasovKM@zdrav.mos.ru
ORCID iD: 0000-0001-7786-0349
SPIN-code: 3160-8062

MD, Dr. Sci. (Medicine)

Russian Federation, Moscow

Tatiana M. Bobrovskaya

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: BobrovskayaTM@zdrav.mos.ru
ORCID iD: 0000-0002-2746-7554
SPIN-code: 3400-8575
Russian Federation, Moscow

Ekaterina F. Savkina

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: SavkinaEF@zdrav.mos.ru
ORCID iD: 0000-0001-9165-0719
SPIN-code: 4986-5592
Russian Federation, Moscow

Rustam A. Erizhokov

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: ErizhokovRA@zdrav.mos.ru
ORCID iD: 0009-0007-3636-2889
SPIN-code: 2274-6428

MD

Russian Federation, Moscow

Lev D. Pestrenin

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: PestreninLD@zdrav.mos.ru
ORCID iD: 0000-0002-1786-4329
SPIN-code: 7193-7706
Russian Federation, Moscow

References

Supplementary files