Machine Analyzes Joint X-Rays About as Well as Humans

A machine-learning system for analyzing rheumatoid arthritis (RA) patients’ X-rays was able to produce Sharp/van der Heijde (SvdH) scores, the standard way to quantify joint space narrowing and bone erosions, with good accuracy when compared with human readers, researchers said.

Called autoscoRA, the system matched the human reader’s scores for joint space narrowing in more than 95% of hand and foot images, according to Thomas Deimel, MD, of the Medical University of Vienna in Austria, and colleagues.

Performance in scoring erosions was more variable, the group reported in Arthritis & Rheumatology, but the level of agreement was still considered good. Scoring differences greater than 1 point in the SvdH method were seen for only 6.3% of hand images and 11.0% of foot X-rays.

Another finding in favor of autoscoRA came from a test in which images were scored by the first human reader, autoscoRA, and a second human reader. AutoscoRA agreed with the first reader for summed scores with an intraclass correlation of 0.94, whereas the second human reader’s scores agreed with the first’s with a correlation of 0.86. When scoring individual joints, autoscoRA readouts for joint space narrowing differed from the first reader’s by more than 1 point in fewer than 3% of instances, whereas for the second reader, about 10% of images had these differences.

“For the erosion score, the performance of the automated system roughly matched that of the second human reader numerically, although visual inspection indicated potentially more consistent predictions by the former,” Deimel and colleagues added.

One of the problems with standard SvdH scoring is that inter-reader (and even intra-reader) reliability is only so-so. Consistency, therefore, is a desirable goal for any method of radiograph analysis. For one thing, errors are more likely to be systematic and therefore more easily recognized and rectified than if they occur randomly.

Another reason to favor an automated system is cost and efficiency. SvdH scoring requires considerable training and experienced readers are therefore scarce, especially outside major referral centers. The reading itself is also time-consuming and, with the requirement for specialized staff to do it, expensive. “An automated system such as autoscoRA directly addresses the feasibility gap, offering a scalable and reproducible solution that transforms imaging into reliable, structured outcome data,” Deimel and colleagues wrote.

AutoscoRA has been under development for some time; Deimel gave a preliminary presentation on it at a 2020 rheumatology conference. This new study included many more images and additional analyses to better define the system’s potential.

The researchers drew on a large archive of hand and foot X-rays from 769 RA patients seen at the Medical University of Vienna, who had a total of 3,437 clinic visits and more than 12,000 radiographs. Some 60% of images were used for training, 20% for validation, and 20% as a “test set.” The comparative testing with autoscoRA and human readers was performed on this latter set.

Besides scoring individual radiographs, the study also looked at serial images from 54 patients with a total of 237 visits over time. This allowed an examination of how autoscoRA could quantify disease progression. Agreement with the human reader averaged 70% over a range of progression definitions (i.e., the degree of change in erosion and joint space scores over time). “Overall performance appeared to be relatively stable across the range of cutoffs,” the researchers wrote.

Deimel and colleagues stressed that autoscoRA needs additional “external validation” with images from other institutions, as well as more focus on the system’s ability to assess progression over time, before it could be considered for routine clinical use. In the meantime, however, the researchers suggested that it could find near-term application in clinical trials and for analyzing large image collections such as those in registries and other observational patient cohorts.