Original article| Volume 9, P310-325, January 2021

Ok
• PDF [11 MB]PDF [11 MB]
• Top

# Meta-MUMS DTA: Implementation, validation, and application of diagnostic test accuracy software for meta-analysis in radiology

Open AccessPublished:October 12, 2020

## Abstract

### Purpose

Generally, guiding clinical practice concentrates on the statistical techniques implemented for performing the diagnostic meta-analysis and test accuracy studies in a specific field of research. This study aims to implement a comprehensive diagnostic meta-analysis tool, which is user-friendly, free, and simple, and can be useful for diagnostic and bivariate model analysis purposes.

### Methods

The Meta-MUMS DTA tool for meta-analysis developed in Matlab R2013a for the Microsoft Windows operating systems (32-bit and 64-bit). Meta-DiSc, Open-MetaAnalyst, Stata (Deeks' test) were the tools used for comparison purposes.

### Results

The features include determination of heterogeneity and computations of chi-square (Q, df, and p-value), I2, Γ2, and Spearman correlation tests, subgroup analysis, meta-regression techniques to explore the relationships of study characteristics and accuracy estimates and performing statistical pooling of sensitivities, specificities, likelihood ratio, and diagnostic odds ratios on fixed- and random-effects models as well as providing figures for forest plots with high quality. The Egger's regression test (along with its smooth version SVE and SVT), Deeks' regression test with funnel plots, and trim and fill were the tools for detecting publication bias. Bivariate model analysis of sensitivity and specificity accuracy is also available in this software. Publication bias and bivariate model analysis are super-advantageous of the proposed software. Moreover, a worked example to evaluate the diagnostic accuracy of mammography and magnetic resonance imaging in breast cancer is proposed.

### Conclusion

The Meta-MUMS DTA tool shows its advantages for upcoming diagnostic meta-analysis studies, especially in radiology science and hopefully may become a platform for teaching purposes.

## 1. Introduction

In clinical practice, diagnostic meta-analysis is used increasingly as a synthesis of shreds of evidence of studies. Using inaccurate tests can result in severe diagnostic errors.
• Whiting P.F.
• Rutjes A.W.
• Westwood M.E.
• Mallett S.
• Group Q.-S.
A systematic review classifies sources of bias and variation in diagnostic test accuracy studies.
Accuracy test studies can be determining the level of agreement between the results of evaluation tests. Foundations of meta-analysis solve many challenging problems that may persist in the studies.
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
By the estimation of sensitivity and specificity, likelihood ratios, odds ratios (ORs), predictive values, and meta-analysis,
• Leeflang M.
Systematic reviews and meta-analyses of diagnostic test accuracy.
researchers can measure diagnostic accuracy studies.
• Whiting P.F.
• Rutjes A.W.
• Westwood M.E.
• Mallett S.
• Group Q.-S.
A systematic review classifies sources of bias and variation in diagnostic test accuracy studies.
,
• Honest H.
• Khan K.S.
Reporting of measures of accuracy in systematic reviews of diagnostic literature.
,
• Reitsma J.B.
• Glas A.S.
• Rutjes A.W.
• Scholten R.J.
• Bossuyt P.M.
• Zwinderman A.H.
Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
Meta-analysis allows precise estimation of test accuracy, which provides a reliable comparison of the accuracy of different statistics of sensitivity and specificity tests in contrast to single studies.
• Dinnes J.
• Mallett S.
• Hopewell S.
• Roderick P.J.
• Deeks J.J.
The Moses–Littenberg meta-analytical method generates systematic differences in test accuracy compared to hierarchical meta-analytical models.
Approaches of diagnostic test accuracy meta-analysis include pooling of sensitivity and specificity estimates, linear regression model to estimate receiver operating characteristics (SROC), and curve development of Moses and Littenberg model.
• Leeflang M.
Systematic reviews and meta-analyses of diagnostic test accuracy.
,
• Dinnes J.
• Mallett S.
• Hopewell S.
• Roderick P.J.
• Deeks J.J.
The Moses–Littenberg meta-analytical method generates systematic differences in test accuracy compared to hierarchical meta-analytical models.
• Leeflang M.M.
• Deeks J.J.
• Gatsonis C.
• Bossuyt P.M.
Systematic reviews of diagnostic test accuracy.
• Littenberg B.
• Moses L.E.
Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method.
After analyzing the weighting of the inverse variance of the log diagnostic Odds Ratio (DOR), it was estimated.
• Dinnes J.
• Mallett S.
• Hopewell S.
• Roderick P.J.
• Deeks J.J.
The Moses–Littenberg meta-analytical method generates systematic differences in test accuracy compared to hierarchical meta-analytical models.
One of the earliest well-known packages which were not published is Meta-Test, which was implemented by Lau J in New England Medical Center in 1997,
• Lau J.
• Schmid C.H.
• Chalmers T.C.
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
and related to the test accuracy of data and pooling of sensitivities, specificities and (SROC) analysis. It was a DOS-based tool and challenging to use it. It could not be able to pool the likelihood ratios (LRs) or to test heterogeneity and meta-regression facilities and has not user-friendly feature. It can convert each pair of sensitivity and specificity into a single measure of accuracy and diagnostic odds ratio. So in this state, detecting sensitivity and specificity will not be distinguished.
The two existing diagnostic meta-analysis tools Meta-DiSc and Open-MetaAnalyst were available in statistical frameworks for studying comparative outcomes, which were used widely in radiology, medicine, epidemiology, psychology, education, management to mention a few. More advanced analysis features include fixed and random effects meta-regression and bivariate diagnostic tests.
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
,
• Zamora J.
• Abraira V.
• Muriel A.
• Khan K.
• Coomarasamy A.
Meta-DiSc: a software for meta-analysis of test accuracy data.
The Meta-DiSc is the most reliable diagnostic meta-analysis software with forest plots of sensitivity, specificity, LRs, DOR, subgroup capacities, Spearman correlation coefficient, and ROC plane curve.
• Zamora J.
• Abraira V.
• Muriel A.
• Khan K.
• Coomarasamy A.
Meta-DiSc: a software for meta-analysis of test accuracy data.
The Open-MetaAnalyst tool proposed in 2009 has accessible features, and a graphical user interface with the spreadsheet-based layout along with including the evidence-based practices.
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
,
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
Open-MetaAnalyst generated different graphical output suitable to the data at hand. Its diagnostic test data included sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (PPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), summary/curve ROC, bivariate model, in both fixed and random effects models.
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
The bivariate model preserved the two-dimensional nature of data that are present in the current software. The bivariate analysis model is the improved and extended version of the traditional SROC approach.
• Reitsma J.B.
• Glas A.S.
• Rutjes A.W.
• Scholten R.J.
• Bossuyt P.M.
• Zwinderman A.H.
Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
The free Meta-MUMS DTA stands for diagnostic test accuracy meta-analysis developed by Mashhad University of Medical Sciences, is designed to propose a user-friendly interface and produce high-resolution figures. Additional features include statistical pooling of sensitivities, specificities, likelihood ratios, diagnostic odds ratios, summary receiver operating characteristics (SROC), determination of heterogeneity, meta-regression for publication bias detection, SVE, SVT, trim and fill, and subgroup calculations.
The currently implemented software works in windows-based and Linux-based environments to carry out its analyses.

## 2. Methods

### 2.1 Implementation

The Meta-MUMS DTA is a comprehensive update for the original Meta-MUMS tool
• Sokouti M.
• et al.
Treating empyema thoracis using video-assisted thoracoscopic surgery and open decortication procedures: a systematic review and meta-analysis by meta-mums tool.
• Sokouti M.
• et al.
A systematic review and meta-analysis on the treatment of liver hydatid cyst using meta-MUMS tool: comparing PAIR and laparoscopic procedures.
• Sokouti M.
• et al.
Comparative global epidemiological investigation of SARS-CoV-2 and SARS-CoV diseases using meta-MUMS tool through incidence, mortality, and recovery rates.
which was for conducting the traditional meta-analysis approach on mostly randomized clinical trials, in other words, the Meta-MUMS DTA which stands for Meta-analysis tool developed in Mashhad University of Medical Sciences that perform Diagnostic Test Accuracy. In this study, Meta-MUMS DTA tool is presented along with a worked example to propose the useful features provided in the tool. The development and validation of Meta-MUMS DTA were to satisfy two aims, as discussed below. The programming environment for Meta-MUMS DTA software was the Matlab version R2013a. And, the executable files compiled in Matlab were compatible with Microsoft Windows XP and higher versions (32-bit and 64-bit), which is freely available upon request. The user can install the Meta-MUMS DTA tool directly by the Matlab compiler installer (i.e., mcrInstaller.exe) as its initial requirements. After installing, the user can run the.exe file in any folder or location of windows. The user interface of Meta-MUMS DTA consists of six menu bars, including File, Edit, View, Graphical outputs, Numerical outputs, as well as Analysis. The program benefits from different statistical methods with a user-friendly interface proposing comprehensible menus along with informative dialog boxes. In this tool, entering data can be performed using a keyboard or copied from the spreadsheets of Microsoft Excel. The variables used in the datasheet are study characteristics, dichotomous (true-positive, true-negative, false-positive, false-negative). The quantitative results for the available tools of Meta-MUMS DTA are diagnostic odds ratio (DOR), forest plots (sensitivity, specificity, likelihood ratio positive, likelihood ratio negative (LRs), DOR), meta-regression (SVE, SVT), and threshold effect (Spearman correlation coefficient and ROC plane plots and their confidence intervals (shown in Table 1).
Table 1Available tools in Meta-MUMS-DTA tool.
 General characteristic•Program size: 7 MB•Compatibility: All versions of Microsoft windows XP and higherInstallation: Matlab compiler (32, 64 bits) Input options•Maximum 100 studies•True positive, True negative, False positive, False negative Fixed effect analysis•Inverse varianceRandom effect analysis•Dersimonian-LairdIndividual study data•Outcome results•P-values•Z-values•Weights Data input option•Manual Heterogeneity•Q Cochrane, Chi-square,•I2•Γ2•ROC & SROC curve•R2 for subgroup& Meta-regression•Meta-regression Diagnostic Meta-analysis•Diagnostic Odds Ratio (DOR)•Likelihood Ratios (LRs)•Sensitivity•Specificity•Publication bias•Bivariate model Graphical output options•Forest plots•Point proportional study weights•Meta Regression Scatter plot•Funnel Plot and (Trim & Fill, SVE, SVT regression plots)•Standard error, P-value, Selective analysis•Subgroup•OverallThreshold analysis•Spearman correlation Coefficient•ROC & SROC curve plot Export options•Output to Excel•Graphs exported to all windows graphic formats (e.g., JPG, Tiff, Gif)
By getting the extracted data into the datasheet of the Meta-MUMS DTA tool, several statistical analyses such as pooling and meta-analyzing are present. Fixed- and random-effects models use the inverse of variance for weighted, un-weighted for pooling the results from the target studies. The weights of different studies can be balanced using the random-effects model since it estimates the mean distribution of effects. As a result, the standard error and confidence intervals of the summary effect will cover more comprehensive ranges using the random-effects model. Forest plots generally illustrate the results of a meta-analysis. A two-column image includes the forest plots; the left column lists the name of studies, while the right column shows the measure of effect, such as the DOR of each study incorporating confidence intervals represented by parallel horizontal lines. Sometimes using the diagnostic odds ratio, the natural logarithmic scale is suitable for graphing the plot (Fig. 1). The Meta- MUMS DTA automatically generates the forest plots' ranges. Also, the horizontal and vertical scroll bars are incorporated to fit the customized area of the forest plot for users' needs. The users can store the forest plots in almost all image formats (e.g., JPG, TIFF, PNG, PDF, BMP, GIF). A meta-analysis of diagnostic test accuracy studies provides summaries of the results of pooled included studies, estimates the average diagnostic accuracy of a test, the variability of study findings around the estimates, and the uncertainty of the average.

### 2.2 Exploring the heterogeneity

Heterogeneity refers to variation in the results of studies. The variability is often higher than would be expected from within-study sampling error and may be explained by the change in characteristics of patients, chance, test methods, and study design.
For the evaluation of statistical pooling of accuracy of estimates of different studies and possible influencing factors, it needs to explore the heterogeneity. Threshold effect and some other than threshold effect factors can result in an accuracy of estimate that can cause heterogeneity in the studies. In the presence of the threshold effect, there are negative or positive correlations between sensitivities and specificities, which resulted in a spearman plot in an SROC space. Meta-MUMS DTA tool can assess the threshold effect influence by determining the sensitivity and specificity accuracy estimates in forest plots. In the presence of the threshold effect, forest plots are useful in sketching increased sensitivities with decreased specificities. The same inverse relationship will be present in likelihood ratio positive (LR+) and likelihood ratio negative (LR-) to measure the heterogeneity of pooled studies and the presence of the threshold effect. The strong positive correlation of logit sensitivities and specificities could also suggest the threshold effect. The Meta-MUMS DTA tool can also determine heterogeneity by visual inspection of forest plots and accuracy of estimates when a significant rate deviation from the line of pooled accuracy of estimate indicates the presence of heterogeneity (with lower p-values). The proposed tool can calculate Cochran's Q, p-value, I2, and Γ2. Calculating the weighted sum of squared differences between individual study effects and the pooled effect among the studies where the weights are those used in pooling meta-analysis results in the value of Q.

### 2.3 Meta-regression

The meta-regression techniques are beneficial in determining the heterogeneity and assessing the relationship between study-level covariates. These are available in Meta-MUMS DTA tool by fixed, mixed, and unrestricted maximum likelihood (ML) models. Meta-regression analysis is a form of a linear regression which aims to relate the size of the effect of one or more characteristics of the involved studies. In this case, by calculating slope and p-value, it is possible to find a significant relationship, and R2 demonstrated as a percentage value, which can determine how much meta-regression model could explain the heterogeneity. The sample generated scatter plots using Meta-MUMS DTA are illustrated in Fig. 2 with all types of available image formats (e.g., JPG, TIFF, PNG, PDF, BMP, GIF). Usually, DOR is measured overall diagnostic accuracy by encompassing both sensitivity and specificity or LR positive and LR negative, but has limitation due to unusable in clinical practice and masking them.
Meta-MUMS DTA implements meta-regression using the Moses Littenberg Linear model by fixed and random effect models and adding weighted scheme. The outcome is ln (DOR), which is about the linear model of any number of study-level covariates. The output of meta-regression modeling of the Meta-MUMS DTA tool has a co-efficient model, such as the ratio of DOR with confidence intervals. A low p-value refers to as the co-variates level of diagnostic accuracy.
More advanced meta-regression such as SROC model and bivariate analysis of paired sensitivity and specificity are also available in Meta-MUMS DTA tool. Also, Youden's index was also implemented in the Meta-MUMS tool along with AUC parameter.

### 2.4 Publication bias

Study qualities, heterogeneity, and publication may bias diagnostic meta-analysis.
• Deeks J.J.
• Irwig L.
The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed.
For detecting publication bias and other sample size effects in systematic reviews, assessing diagnostic test accuracy tests should be essential.
The validity of meta-analysis can be identifiable in the presence of possible publication bias. The Eggers, Deeks, SVE, and SVT tests are useful tools for the determination of publication bias. Among them, Deeks' test is preferred and recommended.
• Jin Z.-C.
• Wu C.
• Zhou X.-H.
• He J.
A modified regression method to test publication bias in meta-analyses with binary outcomes.
• van Enst W.A.
• Ochodo E.
• Scholten R.J.
• Hooft L.
• Leeflang M.M.
Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study.
• Viechtbauer W.
Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments.
Due to having the ability to separate within-, from-, between-study variance of studies, random-effects models are usually preferable in the meta-analysis.
• Bürkner P.C.
• Doebler P.
Testing for publication bias in diagnostic meta‐analysis: a simulation study.
The graphical shape of funnel plots is generally useful for the detection of publication bias. Any asymmetry in funnel plots can represent publication bias [37].
Another method for detecting publication bias is the weighted linear regression approach.
• Deeks J.J.
• Irwig L.
The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed.
,
• Egger M.
• Smith G.D.
• Schneider M.
• Minder C.
Bias in meta-analysis detected by a simple, graphical test.
Trim and fill, developed by Duval and Tweedie, is a non-parametric method for detecting publication bias,
• Duval S.
• Tweedie R.
Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis.
,
• Peters J.L.
• Sutton A.J.
• Jones D.R.
• Abrams K.R.
• Rushton L.
Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity.
in which "K" studies and K0 missing studies of meta-analysis produced asymmetry in funnel plot and can estimate K0.
• Bürkner P.C.
• Doebler P.
Testing for publication bias in diagnostic meta‐analysis: a simulation study.
Trim and fill is recommended in the application of diagnostic meta-analysis due to having superiority to other combinations of tests when assessing for publication bias in the diagnostic meta-analysis.
• Bürkner P.C.
• Doebler P.
Testing for publication bias in diagnostic meta‐analysis: a simulation study.

### 2.5 Subgroup analysis

The subgroup analysis is of interest in explaining the variance between studies. The subgroup analysis feature implemented in this tool consists of two effects models, namely fixed and random effects models. The fixed-effects model within subgroups computes the mean effect and variance for each "subgroup" and then compares the mean effect across the subgroups.
For comparing the effect sizes across the subgroups, the tool uses three algorithms; Z-test for comparing two effect sizes; Q-test to determine partition of the variance as well as to assess the dispersion of summary effects of combined effects.
• Sokouti M.
• Sokouti B.
Most accurate non-linear approximation of standard normal distribution integral based on artificial neural networks.
R2 is a measure of explaining the variation between studies, which is another advantage of the Meta-MUMS DTA tool. Moreover, forest plots and enhanced graphical options are implemented in Meta-MUMS DTA for subgroup analysis, as shown in Fig. 3.

### 2.6 Bivariate modeling analysis

This model can perform a meta-analysis of sensitivity and specificity to produce informative summary measures in diagnostic reviews, with preservation of the two-dimensional nature of data. The SROC approach utilizes these two outcomes as a single indicator of diagnostic accuracy. Bivariate modeling can estimate the amount of between-study variance in their sensitivity, specificity, and correlation by the random-effects model. This model also produces summary estimates of sensitivity, specificity, and 95% confidence intervals. One statistical property of bivariate model analysis is the estimation of correlation that might exist between sensitivity and specificity estimates. This result can produce the validity of bivariate model analysis.
• Egger M.
• Smith G.D.
• Schneider M.
• Minder C.
Bias in meta-analysis detected by a simple, graphical test.
,
• Gatsonis P.
• Deeks C.
• Harbord R.M.
• Takwoingi Y.
Analysing and presenting results.
This model is a common and valid method for performing a diagnostic meta-analysis. In the presence of moderate correlation, the SROC approach is useful; however, with small associations, separate pooling of sensitivity and specificity is needed.
Moreover, despite the availability of advanced statistical analysis modeling, diagnostic meta-analysis will remain challenging due to possible threatened publication bias and lack of information on vital elements of design and conduction.
The mean value of logit sensitivity and specificity can determine the possible negative correlation between them. Studies with a more precise estimate of sensitivity and specificity proposed higher weights in the analysis of sensitivity. For obtaining the SROC curve, the parameters of bivariate distribution must be beneficial. Diagnostic OR and LRs can be sketchable from the calculation of sensitivity and specificity. Co-variables can also input to bivariate modeling, which leads to effects on sensitivity and specificity, and these are different between two diagnostic techniques.
The current proposed Meta-MUMS DTA tool has the capability of bivariate modeling analysis as used in the SROC approach.

## 3. Results and discussion

There are certainly a lot of advantages in using the Meta-MUMS DTA tool representatively in a diagnostic meta-analysis article about breast cancer written by Zhang et al.
• Zhang Y.
• Ren H.
Meta-analysis of diagnostic accuracy of magnetic resonance imaging and mammography for breast cancer.
Breast cancer is one of the most common malignancies and leading causes of death in women. Therefore it is essential to identify breast cancer tumors accurately at early stages for the initial treatments. Differentiating breast cancer from benign or normal lesions of the breast is the most crucial action. Today clinicians widely use mammography (MG) and magnetic resonance imaging (MRI) to diagnose breast cancer.
• Zuo T.T.
• Zheng R.S.
• Zeng H.M.
• Zhang S.W.
• Chen W.Q.
Female breast cancer incidence and mortality in C hina, 2013.
Diagnostic accuracy of two abovementioned methods is necessary for evaluation and interpretations of test results. The Meta-MUMS DTA tool meets the current and pressing needs of the community for teaching meta-analysis, which conducts high-quality syntheses of data. Sensitivity, specificity, LRs, and DOR are particular parts of diagnostic tests, while SROC reflects the characteristics of diagnostic tests, which are available in Appendix (Figs. 1 and 2, …, 10).
In the study of Zhang et al., the values for sensitivity and specificity were 0.75 and 0.71 for mammography for breast cancer, while for MRI were 0.92 and 0.70, respectively.
The combination of sensitivity and specificity is called LR, which reflects the accuracy of diagnostic tests; LR+> 10 has a positive value while LR- < 0.1 has a negative value for detecting breast cancers.
• Reitsma J.B.
• Glas A.S.
• Rutjes A.W.
• Scholten R.J.
• Bossuyt P.M.
• Zwinderman A.H.
Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
We have shown some of the Meta-MUMS DTA software extra capabilities by reworking a diagnostic meta-analysis to complete the Zhang's work by some other analyses such as SVE, SVT, and "trim and fill."
In their study, Zhang et al. used Meta-Disc software for the pooling of data. Without statistical analysis, no one could determine the superiority of the groups without a subgroup analysis carried out by the Meta-MUMS DTA. Our investigations include a subset of the original data of Zhang's research, which will present the Meta-MUMS DTA tool capabilities. Furthermore, to ease the model comparison of mammography and MRI imaging, all analysis modes were performed by the random-effects model in subgroup analysis.
The current software (Meta-MUMS DTA) is advantageous for having SVE (smooth-variance of Eggers), SVT (smooth-variance of Thomson), and trim & fill capabilities for detecting publication bias.
For MRI and MG group, the Eggers and SVE tests propose no significant and significant publication bias, respectively. In contrast, SVT and Deeks tests show substantial and no significant publication bias, respectively. Moreover, trim & fill analysis of MRI and MG group proposed zero and three missed imputation studies, respectively. This outcome indicated that despite adding these studies, heterogeneity increased, and so there are no significant differences in two imaging methods.
The subgroup analyses of sensitivity, specificity, DOR, LR+, LR-, SROC are performed in the Meta-MUMS DTA tool to find out and confirm the absolute superiority of MRI versus MG.
Forest plots of two diagnostic procedures are illustrated in (Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5). While the following information reveals that MRI is 21.80% better than MG in diagnosing breast cancer. Sensitivity MRI = 0.908, p < 0.001, lower limit = 0.843, upper limit = 0.948; sensitivity MG = 0.745, p < 0.001, lower limit = 0.629, upper limit = 0.834 (shown in Fig. 1) (i.e., p-value = 0.003 and R
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
= 21.80%).
And, specificity MRI = 0.740, p < 0.001, lower limit = 0.615, upper limit = 0.836
specificity MG = 0.710, p < 0.002, lower limit = 0.584, upper limit = 0.810 (Fig. 2).
Due to insignificant p-value = 0.706, there was no difference between the specificities of MRI versus MG. Notably, the specificity values greater than 0.5 show that the results of both methods are sound.
LR + MRI = 3.303, p < 0.001, Lower limit = 2.291, Upper limit = 4.763.
LR + MG = 2.557, p < 0.001, Lower limit = 1.812, Upper limit = 3.607 (Fig. 3).
p-value = 0.317, and there was no significant differences between MRI versus MG.
LR- MRI = 0.151, p < 0.001, Lower limit = 0.096, Upper limit = 0.236.
LR- MG = 0.385, p < 0.001, Lower limit = 0.266, Upper limit = 0.556 (Fig. 4).
p-value < 0.001, R2 = 8.124 and MG was better than MRI by 8.124%.
DOR MRI = 29.05, p < 0.001, Lower limit = 15.864, Upper limit = 53.2.
DOR MG = 6.72, p < 0.001, Lower limit = 4.017, Upper limit = 11.242 (Fig. 5).
p-value < 0.001, R2 = 41.191 and MRI was better than MG by 41.191%.
SROC MRI→ (AUC = 0.93318, SE = 0.02059), Y MRI = 0.7379.
SROC MG→ (AUC = 0.78971, SE = 0.02593), Y MG = 0.4539.
With p-value < 0.001, Both AUC and Youden's index of MRI show better performance than AUC and Youden's index of MG.
The following formula shows the required calculations for identifying superiority
• Borenstein M.
• Hedges L.
• Higgins J.
• Rothstein H.
Comprehensive Meta-Analysis Version 3.
:
$p=2×(1−φ(|ZDiff|)),ZDiff=DiffSEDiff,Diff=MB−MA,SEDiff=VMA+VMB,V=SE2$

In summary, according to the reworking results of Zhang's research, re-analyzed by the Meta-MUMS DTA tool. Forest plots of sensitivity, specificity, SROC, DOR, LR-, illustrated that MRI imaging has superiority over mammography; however, there were no significant differences in terms of specificity and LR+ (Fig. 4).
Areas under the curve (AUC) of MRI has significant superiority over that of MG. And the results for the SROC of MRI and MG are shown; MRI has better than MG results.
To assess the features of Open-MetaAnalyst, Meta-Disc, and Meta-MUMS DTA tools, 20 researchers worked on them and evaluated them by assigning scores to their functions. Table 2 shows the scoring results of the three abovementioned tools. According to the total scores of Table 2, the Meta-MUMS DTA tool has the highest usability while compared to two other competitors. Additionally, the three meta-analysis tools have been demonstrated in terms of their basic and advanced analytical characteristics in Table 3, Table 4, respectively.
Table 2Comparing Features of Diagnostic meta-analysis software based on scores.
FeatureOpen-Meta

Analyst
Meta-DiScMeta-Mums

DTA
Installation9.5(9–10)8.5(7–10)9.7(9–10)
Getting Started9(8–10)9(8–10)9.5(9–10)
Data insertion9.3(8–10)9.2(8–10)9.7(9–10)
Effective analysis9.9(9–10)8.4(7–10)9.9(9–10)
Quality of Plots8.5(7–10)8.3(7–10)9.8(9–10)
extensibility of Numerical Outputs4.2(3–5)7(6–8)9.2(8–10)
Extensibility of Plots8.5(7–10)8.4(6–10)9.4(8–10)
Meta-Analysis Features9.3(8–10)8.1(6–9)9.7(8–10)
Total8.5258.3639.613
Table 3Comparing Basic characteristics of Meta-Mums DTA with other Diagnostic Meta-analysis software.
Open-MetaAnalystMeta-DiScMeta-Mums DTA
Single use price (Standard)FreeFreeFree
Size267 Mb2.29 Mb7 Mb
CompatibilityMac, Windows 7$8 64 bit Windows 32 bitWindows XP and higher 64 &32 bit Last update201920182019 LicenseOpenOpenOpen Input options Manual+++ Copy-paste+++ File import (Excel …. )+-+ Single data input+++ Maximum number of studiesunlimitedunlimited100 Export options Copy out put+++ Export to office applicationtxt-+ Report creation Picture typePngJpg, Png, emf, wmf, rtfjpg, tif, png, pdf, bmp, gif 96 dpi96 dpi600 dpi The '+' indicates presence of a feature. Abbreviation: JPG ′Joint Picture Expert Group', Tiff 'Tagged Image File Format', PNG ′Portable Network Graphics', GIF ′Graphics Interchange Format', BMP 'Bitmap', PDF ′Portable Document Format', emf 'Enhanced Windows MetaFiles', wmf 'Windows Meta File', mac 'apple macinoth', rtf 'Rich Text Format'. Table 4Analytical feature comparison of Meta-Mums DTA with other software. Open-MetaAnalystMeta-DiScMeta-Mums DTA Computational setting options Constant continuity correction+++ إBootstrap confidence intervals++ Numerical output Association measures-riskRD,RR,OR, log OR, log RRRD, log RR, log ORRD,RR,OR, log OR, log RR Fixed effect models/weighingIV,MH, PetoIV,MHIV Random effect models/weighingDLDL,HE, SJ, ML, REML, EBDL HeterogeneityQ,I22Q,I22Q,I22 Small study effect/publication biasFSN,RC,EGG,TFFSNEGG, SVE, SVT, TF Meta-regressionFixed, Random (DL, unREML)Fixed, Random (DL,HE, SJ, ML, REML, EB, HS)Fixed, Random (DL, unREML) Graphical output Forest plot+++ Scroll bar quality+ Points proportional to weights+++ Annotations in row possible+++ Funnel plot1/se,seSeSe Exclusion sensitivity plot+++ Trim$ fill plot++
Graph formatting+++
Scatter plot++
SVE plot+
SVT plot+
R
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
++
The '+' indicate the presence of a feature. Abbreviations, P ′p-value', RD ′Risk Difference', OR ′Odds Ratio', RR ′Risk Ratio', HG 'Hedge's g, PETO 'PETO's weighing', DL ′Dersimonian & Laird weighing', Q 'Cochran's Q′, I2 'Higgin's inconsistency statistics' t2 'Between study variance indicator', FSN ′Fail safe Number test', RC ′Random correlation test', EGG 'Eggere's Regression test, TF ′Trim & Fill', HE 'Hedg'es', SJ ′Sidik-Jonkman', ML ′Maximum likelihood', RML ′Restricted maximum likelihood', EB ′Eprical Bayes', URML ′Unrestricted Restricted maximum likelihood', HS ′Hunter Schmidt', SVE ′Smoothed Variance based on Egger', SVT "Smoothed Variance based on Thomson.
The Meta-MUMS DTA tool is a software developed for researchers interested in diagnostic meta-analysis. This tool is programmed in Matlab version R2013a environment and perform the diagnostic meta-analysis procedure using the retrieved data from different studies on the same research subject. The Meta-MUMS DTA tool is an innovative diagnostic meta-analysis tool for calculating various statistical analyses.
These include likelihood ratios, sensitivity, specificity, DOR, Spearman coefficient, exploration of heterogeneity, meta-regression, bivariate model analysis for estimation of sensitivity and specificity for detecting I2, p-value, Q Cochrane, Γ2, using fixed- and random-effects models as well as overall or within subgroups. It produces high-quality images (600 dpi) for all plots, such as forest plots and meta-regression scatter plots. High-resolution figures with manually manageable scroll bars are also other advantages of this tool. The improved formula based on artificial neural network obtained p-value from Z-value, which is available in the Meta-MUMS DTA tool.
The comparison of Meta-MUMS DTA tool with other diagnostic tools illustrated in Table 2 showed the novelties of the software. For inserting data in the Open-MetaAnalyst,
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
identifying study names and data is required. At the same time, in Meta-MUMS DTA, the workspace will be available during the starting point and choosing the diagnostic meta-analysis option, which is another advantage of Meta-MUMS DTA tool. The Open-MetaAnalyst tool is a powerful, open-source program for performing meta-analyses of diagnostic test analysis using a variety of fixed- and random-effects models. The Open-MetaAnalyst tool also enables us to do cumulative, leave-one-out functionalities. It also has an ease-of-use graphical user interface (GUI), methods of performing Bayesian, bivariate meta-analysis subgroup analysis, and meta-regression.
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
It can use a bivariate model to estimate the sensitivity and specificities for diagnostic test data. It conducts a joint meta-analysis of the sensitivity and specificity of diagnostic test data in a standard receiver operating curve.
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
Meta-DiSc tool is diagnostic software for meta-analysis of test accuracy data, which explore heterogeneity and variety of statistics such as I2, chi-square, and spearman correlation test. Meta-regression of the Meta-Disc tool can explore the relationships between studies and the accuracy of estimation. It can carry out statistical pooling of sensitivity, specificity, LRs, and DOR in fixed- and random-effects models and meta-analysis of them.
• Zamora J.
• Abraira V.
• Muriel A.
• Khan K.
• Coomarasamy A.
Meta-DiSc: a software for meta-analysis of test accuracy data.
Meta-Disc does not have the capability of performing bivariate analysis and subgroup analysis. Meta-test is not available and does not have the ability of analytical tools such as pooling of LRs, tests for heterogeneity, and meta-regression facilities.
• Lau J.
• Schmid C.H.
• Chalmers T.C.
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
The validations of the outputs of the current tools were compared and assessed with the Meta-Disc and Open-MetaAnalyst tool outputs.

## 4. Conclusion

The Meta-MUMS DTA tool was developed and validated in a new programming environment for conducting meta-analysis in the user-friendly environment and is useful for upcoming diagnostic meta-analysis studies. The proposed software provides several additional features in comparison to other existing diagnostic software. They include enhancements in data entry and storage, computations, output results, Bivariate analysis modeling, exploring heterogeneity, meta-regression, subgroup meta-analysis, and high-resolution images. And hence, the validation, assessment, and verification of this tool can make it the first choice for diagnostic meta-analysis studies. We hope that Meta-MUMS DTA will become a platform for teaching meta-analysis, as well as an essential tool for improving the quality and scope of research synthesis.

## Acknowledgements

This study is part of PhD thesis (No. 931507) and was approved and supported by Research Council of Mashhad University of Medical Sciences, Mashhad, Iran and carried out in Nuclear Medicine Research Center of Mashhad University of Medical Sciences, Mashhad, Iran.

## References

• Whiting P.F.
• Rutjes A.W.
• Westwood M.E.
• Mallett S.
• Group Q.-S.
A systematic review classifies sources of bias and variation in diagnostic test accuracy studies.
J Clin Epidemiol. 2013; 66: 1093-1104
• Wallace B.C.
• Schmid C.H.
• Lau J.
• Trikalinos T.A.
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data.
BMC Med Res Methodol. 2009; 9: 80
• Leeflang M.
Systematic reviews and meta-analyses of diagnostic test accuracy.
Clin Microbiol Infect. 2014; 20: 105-113
• Honest H.
• Khan K.S.
Reporting of measures of accuracy in systematic reviews of diagnostic literature.
BMC Health Serv Res. 2002; 2: 4
• Reitsma J.B.
• Glas A.S.
• Rutjes A.W.
• Scholten R.J.
• Bossuyt P.M.
• Zwinderman A.H.
Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
J Clin Epidemiol. 2005; 58: 982-990
• Dinnes J.
• Mallett S.
• Hopewell S.
• Roderick P.J.
• Deeks J.J.
The Moses–Littenberg meta-analytical method generates systematic differences in test accuracy compared to hierarchical meta-analytical models.
J Clin Epidemiol. 2016; 80: 77-87
• Leeflang M.M.
• Deeks J.J.
• Gatsonis C.
• Bossuyt P.M.
Systematic reviews of diagnostic test accuracy.
Ann Intern Med. 2008; 149: 889-897
• Littenberg B.
• Moses L.E.
Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method.
Med Decis Making. 1993; 13: 313-321
• Lau J.
• Schmid C.H.
• Chalmers T.C.
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
J Clin Epidemiol. 1995; 48: 45-57
• Wallace B.C.
• Lajeunesse M.J.
• Dietz G.
• et al.
Open MEE: intuitive, open‐source software for meta‐analysis in ecology and evolutionary biology.
Methods in Ecology and Evolution. 2017; 8: 941-947
• Zamora J.
• Abraira V.
• Muriel A.
• Khan K.
• Coomarasamy A.
Meta-DiSc: a software for meta-analysis of test accuracy data.
BMC Med Res Methodol. 2006; 6: 31
• Sokouti M.
• et al.
Treating empyema thoracis using video-assisted thoracoscopic surgery and open decortication procedures: a systematic review and meta-analysis by meta-mums tool.
Arch Med Sci : AMS. 2019; 15: 912-935
• Sokouti M.
• et al.
A systematic review and meta-analysis on the treatment of liver hydatid cyst using meta-MUMS tool: comparing PAIR and laparoscopic procedures.
Arch Med Sci : AMS. 2019; 15: 284-308
• Sokouti M.
• et al.
Comparative global epidemiological investigation of SARS-CoV-2 and SARS-CoV diseases using meta-MUMS tool through incidence, mortality, and recovery rates.
Arch Med Res. 2020; 51: 458-463
• Deeks J.J.
• Irwig L.
The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed.
J Clin Epidemiol. 2005; 58: 882-893
• Jin Z.-C.
• Wu C.
• Zhou X.-H.
• He J.
A modified regression method to test publication bias in meta-analyses with binary outcomes.
BMC Med Res Methodol. 2014; 14: 132
• van Enst W.A.
• Ochodo E.
• Scholten R.J.
• Hooft L.
• Leeflang M.M.
Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study.
BMC Med Res Methodol. 2014; 14: 70
• Viechtbauer W.
Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments.
Springer, 2007
• Bürkner P.C.
• Doebler P.
Testing for publication bias in diagnostic meta‐analysis: a simulation study.
Stat Med. 2014; 33: 3061-3077
• Egger M.
• Smith G.D.
• Schneider M.
• Minder C.
Bias in meta-analysis detected by a simple, graphical test.
BMJ. 1997; 315: 629-634
• Duval S.
• Tweedie R.
Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis.
Biometrics. 2000; 56: 455-463
• Peters J.L.
• Sutton A.J.
• Jones D.R.
• Abrams K.R.
• Rushton L.
Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity.
Stat Med. 2007; 26: 4544-4562
• Sokouti M.
• Sokouti B.
Most accurate non-linear approximation of standard normal distribution integral based on artificial neural networks.
Suranaree journal of science and technology. 2017; 24: 263-280
• Gatsonis P.
• Deeks C.
• Harbord R.M.
• Takwoingi Y.
Analysing and presenting results.
in: Deeks J.J. Bossuyt P.M. Gatsonis C. The Cochrane Collaboration Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 10 Ch 10. 2010 (2010)
• Zhang Y.
• Ren H.
Meta-analysis of diagnostic accuracy of magnetic resonance imaging and mammography for breast cancer.
J Canc Res Therapeut. 2017; 13: 862
• Zuo T.T.
• Zheng R.S.
• Zeng H.M.
• Zhang S.W.
• Chen W.Q.
Female breast cancer incidence and mortality in C hina, 2013.
Thoracic cancer. 2017; 8: 214-218
• Borenstein M.
• Hedges L.
• Higgins J.
• Rothstein H.
Comprehensive Meta-Analysis Version 3.
Biostat, Englewood, NJ2013