If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Generally, guiding clinical practice concentrates on the statistical techniques implemented for performing the diagnostic meta-analysis and test accuracy studies in a specific field of research. This study aims to implement a comprehensive diagnostic meta-analysis tool, which is user-friendly, free, and simple, and can be useful for diagnostic and bivariate model analysis purposes.
Methods
The Meta-MUMS DTA tool for meta-analysis developed in Matlab R2013a for the Microsoft Windows operating systems (32-bit and 64-bit). Meta-DiSc, Open-MetaAnalyst, Stata (Deeks' test) were the tools used for comparison purposes.
Results
The features include determination of heterogeneity and computations of chi-square (Q, df, and p-value), I2, Γ2, and Spearman correlation tests, subgroup analysis, meta-regression techniques to explore the relationships of study characteristics and accuracy estimates and performing statistical pooling of sensitivities, specificities, likelihood ratio, and diagnostic odds ratios on fixed- and random-effects models as well as providing figures for forest plots with high quality. The Egger's regression test (along with its smooth version SVE and SVT), Deeks' regression test with funnel plots, and trim and fill were the tools for detecting publication bias. Bivariate model analysis of sensitivity and specificity accuracy is also available in this software. Publication bias and bivariate model analysis are super-advantageous of the proposed software. Moreover, a worked example to evaluate the diagnostic accuracy of mammography and magnetic resonance imaging in breast cancer is proposed.
Conclusion
The Meta-MUMS DTA tool shows its advantages for upcoming diagnostic meta-analysis studies, especially in radiology science and hopefully may become a platform for teaching purposes.
In clinical practice, diagnostic meta-analysis is used increasingly as a synthesis of shreds of evidence of studies. Using inaccurate tests can result in severe diagnostic errors.
Accuracy test studies can be determining the level of agreement between the results of evaluation tests. Foundations of meta-analysis solve many challenging problems that may persist in the studies.
Meta-analysis allows precise estimation of test accuracy, which provides a reliable comparison of the accuracy of different statistics of sensitivity and specificity tests in contrast to single studies.
Approaches of diagnostic test accuracy meta-analysis include pooling of sensitivity and specificity estimates, linear regression model to estimate receiver operating characteristics (SROC), and curve development of Moses and Littenberg model.
and related to the test accuracy of data and pooling of sensitivities, specificities and (SROC) analysis. It was a DOS-based tool and challenging to use it. It could not be able to pool the likelihood ratios (LRs) or to test heterogeneity and meta-regression facilities and has not user-friendly feature. It can convert each pair of sensitivity and specificity into a single measure of accuracy and diagnostic odds ratio. So in this state, detecting sensitivity and specificity will not be distinguished.
The two existing diagnostic meta-analysis tools Meta-DiSc and Open-MetaAnalyst were available in statistical frameworks for studying comparative outcomes, which were used widely in radiology, medicine, epidemiology, psychology, education, management to mention a few. More advanced analysis features include fixed and random effects meta-regression and bivariate diagnostic tests.
The Meta-DiSc is the most reliable diagnostic meta-analysis software with forest plots of sensitivity, specificity, LRs, DOR, subgroup capacities, Spearman correlation coefficient, and ROC plane curve.
The Open-MetaAnalyst tool proposed in 2009 has accessible features, and a graphical user interface with the spreadsheet-based layout along with including the evidence-based practices.
Open-MetaAnalyst generated different graphical output suitable to the data at hand. Its diagnostic test data included sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (PPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), summary/curve ROC, bivariate model, in both fixed and random effects models.
The bivariate model preserved the two-dimensional nature of data that are present in the current software. The bivariate analysis model is the improved and extended version of the traditional SROC approach.
The free Meta-MUMS DTA stands for diagnostic test accuracy meta-analysis developed by Mashhad University of Medical Sciences, is designed to propose a user-friendly interface and produce high-resolution figures. Additional features include statistical pooling of sensitivities, specificities, likelihood ratios, diagnostic odds ratios, summary receiver operating characteristics (SROC), determination of heterogeneity, meta-regression for publication bias detection, SVE, SVT, trim and fill, and subgroup calculations.
The currently implemented software works in windows-based and Linux-based environments to carry out its analyses.
2. Methods
2.1 Implementation
The Meta-MUMS DTA is a comprehensive update for the original Meta-MUMS tool
Treating empyema thoracis using video-assisted thoracoscopic surgery and open decortication procedures: a systematic review and meta-analysis by meta-mums tool.
Comparative global epidemiological investigation of SARS-CoV-2 and SARS-CoV diseases using meta-MUMS tool through incidence, mortality, and recovery rates.
which was for conducting the traditional meta-analysis approach on mostly randomized clinical trials, in other words, the Meta-MUMS DTA which stands for Meta-analysis tool developed in Mashhad University of Medical Sciences that perform Diagnostic Test Accuracy. In this study, Meta-MUMS DTA tool is presented along with a worked example to propose the useful features provided in the tool. The development and validation of Meta-MUMS DTA were to satisfy two aims, as discussed below. The programming environment for Meta-MUMS DTA software was the Matlab version R2013a. And, the executable files compiled in Matlab were compatible with Microsoft Windows XP and higher versions (32-bit and 64-bit), which is freely available upon request. The user can install the Meta-MUMS DTA tool directly by the Matlab compiler installer (i.e., mcrInstaller.exe) as its initial requirements. After installing, the user can run the.exe file in any folder or location of windows. The user interface of Meta-MUMS DTA consists of six menu bars, including File, Edit, View, Graphical outputs, Numerical outputs, as well as Analysis. The program benefits from different statistical methods with a user-friendly interface proposing comprehensible menus along with informative dialog boxes. In this tool, entering data can be performed using a keyboard or copied from the spreadsheets of Microsoft Excel. The variables used in the datasheet are study characteristics, dichotomous (true-positive, true-negative, false-positive, false-negative). The quantitative results for the available tools of Meta-MUMS DTA are diagnostic odds ratio (DOR), forest plots (sensitivity, specificity, likelihood ratio positive, likelihood ratio negative (LRs), DOR), meta-regression (SVE, SVT), and threshold effect (Spearman correlation coefficient and ROC plane plots and their confidence intervals (shown in Table 1).
Table 1Available tools in Meta-MUMS-DTA tool.
General characteristic
•
Program size: 7 MB
•
Compatibility: All versions of Microsoft windows XP and higher
By getting the extracted data into the datasheet of the Meta-MUMS DTA tool, several statistical analyses such as pooling and meta-analyzing are present. Fixed- and random-effects models use the inverse of variance for weighted, un-weighted for pooling the results from the target studies. The weights of different studies can be balanced using the random-effects model since it estimates the mean distribution of effects. As a result, the standard error and confidence intervals of the summary effect will cover more comprehensive ranges using the random-effects model. Forest plots generally illustrate the results of a meta-analysis. A two-column image includes the forest plots; the left column lists the name of studies, while the right column shows the measure of effect, such as the DOR of each study incorporating confidence intervals represented by parallel horizontal lines. Sometimes using the diagnostic odds ratio, the natural logarithmic scale is suitable for graphing the plot (Fig. 1). The Meta- MUMS DTA automatically generates the forest plots' ranges. Also, the horizontal and vertical scroll bars are incorporated to fit the customized area of the forest plot for users' needs. The users can store the forest plots in almost all image formats (e.g., JPG, TIFF, PNG, PDF, BMP, GIF). A meta-analysis of diagnostic test accuracy studies provides summaries of the results of pooled included studies, estimates the average diagnostic accuracy of a test, the variability of study findings around the estimates, and the uncertainty of the average.
Fig. 1Forest plot comparison of MRI versus MG sensitivities.
Heterogeneity refers to variation in the results of studies. The variability is often higher than would be expected from within-study sampling error and may be explained by the change in characteristics of patients, chance, test methods, and study design.
For the evaluation of statistical pooling of accuracy of estimates of different studies and possible influencing factors, it needs to explore the heterogeneity. Threshold effect and some other than threshold effect factors can result in an accuracy of estimate that can cause heterogeneity in the studies. In the presence of the threshold effect, there are negative or positive correlations between sensitivities and specificities, which resulted in a spearman plot in an SROC space. Meta-MUMS DTA tool can assess the threshold effect influence by determining the sensitivity and specificity accuracy estimates in forest plots. In the presence of the threshold effect, forest plots are useful in sketching increased sensitivities with decreased specificities. The same inverse relationship will be present in likelihood ratio positive (LR+) and likelihood ratio negative (LR-) to measure the heterogeneity of pooled studies and the presence of the threshold effect. The strong positive correlation of logit sensitivities and specificities could also suggest the threshold effect. The Meta-MUMS DTA tool can also determine heterogeneity by visual inspection of forest plots and accuracy of estimates when a significant rate deviation from the line of pooled accuracy of estimate indicates the presence of heterogeneity (with lower p-values). The proposed tool can calculate Cochran's Q, p-value, I2, and Γ2. Calculating the weighted sum of squared differences between individual study effects and the pooled effect among the studies where the weights are those used in pooling meta-analysis results in the value of Q.
2.3 Meta-regression
The meta-regression techniques are beneficial in determining the heterogeneity and assessing the relationship between study-level covariates. These are available in Meta-MUMS DTA tool by fixed, mixed, and unrestricted maximum likelihood (ML) models. Meta-regression analysis is a form of a linear regression which aims to relate the size of the effect of one or more characteristics of the involved studies. In this case, by calculating slope and p-value, it is possible to find a significant relationship, and R2 demonstrated as a percentage value, which can determine how much meta-regression model could explain the heterogeneity. The sample generated scatter plots using Meta-MUMS DTA are illustrated in Fig. 2 with all types of available image formats (e.g., JPG, TIFF, PNG, PDF, BMP, GIF). Usually, DOR is measured overall diagnostic accuracy by encompassing both sensitivity and specificity or LR positive and LR negative, but has limitation due to unusable in clinical practice and masking them.
Fig. 2Forest plot comparison of MRI versus MG specificities.
Meta-MUMS DTA implements meta-regression using the Moses Littenberg Linear model by fixed and random effect models and adding weighted scheme. The outcome is ln (DOR), which is about the linear model of any number of study-level covariates. The output of meta-regression modeling of the Meta-MUMS DTA tool has a co-efficient model, such as the ratio of DOR with confidence intervals. A low p-value refers to as the co-variates level of diagnostic accuracy.
More advanced meta-regression such as SROC model and bivariate analysis of paired sensitivity and specificity are also available in Meta-MUMS DTA tool. Also, Youden's index was also implemented in the Meta-MUMS tool along with AUC parameter.
2.4 Publication bias
Study qualities, heterogeneity, and publication may bias diagnostic meta-analysis.
For detecting publication bias and other sample size effects in systematic reviews, assessing diagnostic test accuracy tests should be essential.
The validity of meta-analysis can be identifiable in the presence of possible publication bias. The Eggers, Deeks, SVE, and SVT tests are useful tools for the determination of publication bias. Among them, Deeks' test is preferred and recommended.
Due to having the ability to separate within-, from-, between-study variance of studies, random-effects models are usually preferable in the meta-analysis.
The graphical shape of funnel plots is generally useful for the detection of publication bias. Any asymmetry in funnel plots can represent publication bias [37].
Another method for detecting publication bias is the weighted linear regression approach.
Trim and fill is recommended in the application of diagnostic meta-analysis due to having superiority to other combinations of tests when assessing for publication bias in the diagnostic meta-analysis.
The subgroup analysis is of interest in explaining the variance between studies. The subgroup analysis feature implemented in this tool consists of two effects models, namely fixed and random effects models. The fixed-effects model within subgroups computes the mean effect and variance for each "subgroup" and then compares the mean effect across the subgroups.
For comparing the effect sizes across the subgroups, the tool uses three algorithms; Z-test for comparing two effect sizes; Q-test to determine partition of the variance as well as to assess the dispersion of summary effects of combined effects.
R2 is a measure of explaining the variation between studies, which is another advantage of the Meta-MUMS DTA tool. Moreover, forest plots and enhanced graphical options are implemented in Meta-MUMS DTA for subgroup analysis, as shown in Fig. 3.
Fig. 3Forest plot comparison of MRI versus MG LR+.
This model can perform a meta-analysis of sensitivity and specificity to produce informative summary measures in diagnostic reviews, with preservation of the two-dimensional nature of data. The SROC approach utilizes these two outcomes as a single indicator of diagnostic accuracy. Bivariate modeling can estimate the amount of between-study variance in their sensitivity, specificity, and correlation by the random-effects model. This model also produces summary estimates of sensitivity, specificity, and 95% confidence intervals. One statistical property of bivariate model analysis is the estimation of correlation that might exist between sensitivity and specificity estimates. This result can produce the validity of bivariate model analysis.
in: Deeks J.J. Bossuyt P.M. Gatsonis C. The Cochrane Collaboration Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 10 Ch 10. 2010
This model is a common and valid method for performing a diagnostic meta-analysis. In the presence of moderate correlation, the SROC approach is useful; however, with small associations, separate pooling of sensitivity and specificity is needed.
Moreover, despite the availability of advanced statistical analysis modeling, diagnostic meta-analysis will remain challenging due to possible threatened publication bias and lack of information on vital elements of design and conduction.
The mean value of logit sensitivity and specificity can determine the possible negative correlation between them. Studies with a more precise estimate of sensitivity and specificity proposed higher weights in the analysis of sensitivity. For obtaining the SROC curve, the parameters of bivariate distribution must be beneficial. Diagnostic OR and LRs can be sketchable from the calculation of sensitivity and specificity. Co-variables can also input to bivariate modeling, which leads to effects on sensitivity and specificity, and these are different between two diagnostic techniques.
The current proposed Meta-MUMS DTA tool has the capability of bivariate modeling analysis as used in the SROC approach.
3. Results and discussion
There are certainly a lot of advantages in using the Meta-MUMS DTA tool representatively in a diagnostic meta-analysis article about breast cancer written by Zhang et al.
Breast cancer is one of the most common malignancies and leading causes of death in women. Therefore it is essential to identify breast cancer tumors accurately at early stages for the initial treatments. Differentiating breast cancer from benign or normal lesions of the breast is the most crucial action. Today clinicians widely use mammography (MG) and magnetic resonance imaging (MRI) to diagnose breast cancer.
Diagnostic accuracy of two abovementioned methods is necessary for evaluation and interpretations of test results. The Meta-MUMS DTA tool meets the current and pressing needs of the community for teaching meta-analysis, which conducts high-quality syntheses of data. Sensitivity, specificity, LRs, and DOR are particular parts of diagnostic tests, while SROC reflects the characteristics of diagnostic tests, which are available in Appendix (Figs. 1 and 2, …, 10).
In the study of Zhang et al., the values for sensitivity and specificity were 0.75 and 0.71 for mammography for breast cancer, while for MRI were 0.92 and 0.70, respectively.
The combination of sensitivity and specificity is called LR, which reflects the accuracy of diagnostic tests; LR+> 10 has a positive value while LR- < 0.1 has a negative value for detecting breast cancers.
We have shown some of the Meta-MUMS DTA software extra capabilities by reworking a diagnostic meta-analysis to complete the Zhang's work by some other analyses such as SVE, SVT, and "trim and fill."
In their study, Zhang et al. used Meta-Disc software for the pooling of data. Without statistical analysis, no one could determine the superiority of the groups without a subgroup analysis carried out by the Meta-MUMS DTA. Our investigations include a subset of the original data of Zhang's research, which will present the Meta-MUMS DTA tool capabilities. Furthermore, to ease the model comparison of mammography and MRI imaging, all analysis modes were performed by the random-effects model in subgroup analysis.
The current software (Meta-MUMS DTA) is advantageous for having SVE (smooth-variance of Eggers), SVT (smooth-variance of Thomson), and trim & fill capabilities for detecting publication bias.
For MRI and MG group, the Eggers and SVE tests propose no significant and significant publication bias, respectively. In contrast, SVT and Deeks tests show substantial and no significant publication bias, respectively. Moreover, trim & fill analysis of MRI and MG group proposed zero and three missed imputation studies, respectively. This outcome indicated that despite adding these studies, heterogeneity increased, and so there are no significant differences in two imaging methods.
The subgroup analyses of sensitivity, specificity, DOR, LR+, LR-, SROC are performed in the Meta-MUMS DTA tool to find out and confirm the absolute superiority of MRI versus MG.
Forest plots of two diagnostic procedures are illustrated in (Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5). While the following information reveals that MRI is 21.80% better than MG in diagnosing breast cancer. Sensitivity MRI = 0.908, p < 0.001, lower limit = 0.843, upper limit = 0.948; sensitivity MG = 0.745, p < 0.001, lower limit = 0.629, upper limit = 0.834 (shown in Fig. 1) (i.e., p-value = 0.003 and R
Due to insignificant p-value = 0.706, there was no difference between the specificities of MRI versus MG. Notably, the specificity values greater than 0.5 show that the results of both methods are sound.
LR + MRI = 3.303, p < 0.001, Lower limit = 2.291, Upper limit = 4.763.
LR + MG = 2.557, p < 0.001, Lower limit = 1.812, Upper limit = 3.607 (Fig. 3).
p-value = 0.317, and there was no significant differences between MRI versus MG.
In summary, according to the reworking results of Zhang's research, re-analyzed by the Meta-MUMS DTA tool. Forest plots of sensitivity, specificity, SROC, DOR, LR-, illustrated that MRI imaging has superiority over mammography; however, there were no significant differences in terms of specificity and LR+ (Fig. 4).
Areas under the curve (AUC) of MRI has significant superiority over that of MG. And the results for the SROC of MRI and MG are shown; MRI has better than MG results.
To assess the features of Open-MetaAnalyst, Meta-Disc, and Meta-MUMS DTA tools, 20 researchers worked on them and evaluated them by assigning scores to their functions. Table 2 shows the scoring results of the three abovementioned tools. According to the total scores of Table 2, the Meta-MUMS DTA tool has the highest usability while compared to two other competitors. Additionally, the three meta-analysis tools have been demonstrated in terms of their basic and advanced analytical characteristics in Table 3, Table 4, respectively.
Table 2Comparing Features of Diagnostic meta-analysis software based on scores.
The Meta-MUMS DTA tool is a software developed for researchers interested in diagnostic meta-analysis. This tool is programmed in Matlab version R2013a environment and perform the diagnostic meta-analysis procedure using the retrieved data from different studies on the same research subject. The Meta-MUMS DTA tool is an innovative diagnostic meta-analysis tool for calculating various statistical analyses.
These include likelihood ratios, sensitivity, specificity, DOR, Spearman coefficient, exploration of heterogeneity, meta-regression, bivariate model analysis for estimation of sensitivity and specificity for detecting I2, p-value, Q Cochrane, Γ2, using fixed- and random-effects models as well as overall or within subgroups. It produces high-quality images (600 dpi) for all plots, such as forest plots and meta-regression scatter plots. High-resolution figures with manually manageable scroll bars are also other advantages of this tool. The improved formula based on artificial neural network obtained p-value from Z-value, which is available in the Meta-MUMS DTA tool.
The comparison of Meta-MUMS DTA tool with other diagnostic tools illustrated in Table 2 showed the novelties of the software. For inserting data in the Open-MetaAnalyst,
identifying study names and data is required. At the same time, in Meta-MUMS DTA, the workspace will be available during the starting point and choosing the diagnostic meta-analysis option, which is another advantage of Meta-MUMS DTA tool. The Open-MetaAnalyst tool is a powerful, open-source program for performing meta-analyses of diagnostic test analysis using a variety of fixed- and random-effects models. The Open-MetaAnalyst tool also enables us to do cumulative, leave-one-out functionalities. It also has an ease-of-use graphical user interface (GUI), methods of performing Bayesian, bivariate meta-analysis subgroup analysis, and meta-regression.
It can use a bivariate model to estimate the sensitivity and specificities for diagnostic test data. It conducts a joint meta-analysis of the sensitivity and specificity of diagnostic test data in a standard receiver operating curve.
Meta-DiSc tool is diagnostic software for meta-analysis of test accuracy data, which explore heterogeneity and variety of statistics such as I2, chi-square, and spearman correlation test. Meta-regression of the Meta-Disc tool can explore the relationships between studies and the accuracy of estimation. It can carry out statistical pooling of sensitivity, specificity, LRs, and DOR in fixed- and random-effects models and meta-analysis of them.
Meta-Disc does not have the capability of performing bivariate analysis and subgroup analysis. Meta-test is not available and does not have the ability of analytical tools such as pooling of LRs, tests for heterogeneity, and meta-regression facilities.
The validations of the outputs of the current tools were compared and assessed with the Meta-Disc and Open-MetaAnalyst tool outputs.
4. Conclusion
The Meta-MUMS DTA tool was developed and validated in a new programming environment for conducting meta-analysis in the user-friendly environment and is useful for upcoming diagnostic meta-analysis studies. The proposed software provides several additional features in comparison to other existing diagnostic software. They include enhancements in data entry and storage, computations, output results, Bivariate analysis modeling, exploring heterogeneity, meta-regression, subgroup meta-analysis, and high-resolution images. And hence, the validation, assessment, and verification of this tool can make it the first choice for diagnostic meta-analysis studies. We hope that Meta-MUMS DTA will become a platform for teaching meta-analysis, as well as an essential tool for improving the quality and scope of research synthesis.
Acknowledgements
This study is part of PhD thesis (No. 931507) and was approved and supported by Research Council of Mashhad University of Medical Sciences, Mashhad, Iran and carried out in Nuclear Medicine Research Center of Mashhad University of Medical Sciences, Mashhad, Iran.
Appendices. The generated plots using Meta-MUMS DTA
Fig. 1Sensitivity and Specificity plots of mammography.
Treating empyema thoracis using video-assisted thoracoscopic surgery and open decortication procedures: a systematic review and meta-analysis by meta-mums tool.
Comparative global epidemiological investigation of SARS-CoV-2 and SARS-CoV diseases using meta-MUMS tool through incidence, mortality, and recovery rates.
in: Deeks J.J. Bossuyt P.M. Gatsonis C. The Cochrane Collaboration Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 10 Ch 10. 2010 (2010)
Declaration of Competing Interest statements were not included in published version of the following articles that appeared in previous issues of Clinical Epidemiology and Global Health. Hence, the authors of the below articles were contacted after publication to request a Declaration of Interest statement:
30224-4&id=gr1.jpg){kind=link}
30224-4&id=gr2.jpg){kind=link}
30224-4&id=gr3.jpg){kind=link}
30224-4&id=gr4.jpg){kind=link}
30224-4&id=gr5.jpg){kind=link}
30224-4&id=fx1.jpg){kind=link}
30224-4&id=fx2.jpg){kind=link}
30224-4&id=fx3.jpg){kind=link}
30224-4&id=fx4.jpg){kind=link}
30224-4&id=fx5.jpg){kind=link}
30224-4&id=fx6.jpg){kind=link}
30224-4&id=fx7.jpg){kind=link}
30224-4&id=fx8.jpg){kind=link}
30224-4&id=fx9.jpg){kind=link}
30224-4&id=fx10.jpg){kind=link}