Research synthesis methods such as meta-analysis rely on either individual participant data or appropriate summary statistics for implementation. A commonly encountered problem arises when a measurement of variability is not explicitly included among the summary statistics in published studies, leading to potential publication bias and loss of statistical power from having to omit otherwise credible research from meta-analysis.
In the absence of individual participant data (a.k.a. raw data), standard meta-analytic approaches on continuous outcomes rely on reported summary metrics, such as the mean and a corresponding measure of variability (e.g., the sample standard deviation or standard error values for each treatment group). This is due to the fact that effect sizes in modern meta-analyses are weighted by study previsions, most commonly by the inverse of study variance. That is, high variance studies are given less weight whereas, low variance studies are given more weight in the analysis.
Variability metrics can also be calculated algebraically from alternative parametric summary statistics available in the primary report, inter alia, Fisher’s least significant difference (LSD), confidence intervals, and the p-value. These alternative summary statistics may be reported in tables or embedded in the text of the results section, although, in practice, many primary reports fail to include appropriate metrics of uncertainty for treatment means or their respective summary statistics.
Alternatively, post hoc tests (a.k.a. means separation or multiple comparison tests) are extensively used in the analysis of randomized trials in agricultural research. In post hoc procedures, the pairwise mean differences are compared against a test statistic, which is partially based on the MSE. Thus, with the test statistic at hand, we can estimate the MSE. Ngugi et al 2011 outlined this method for trials with missing estimates of variability by estimating the upper and lower values for non-significant and significant mean differences, respectively, and averaging them for a parameter called estimated LSD (ELSD), in the Fisher’s LSD procedure. The ELSD was then used in the respective post hoc test formula to obtain the MSE for each trial.
Building on this concept, MSE FindR was developed to help the estimation of MSE from ANOVA-type studies with missing variability metrics that, alternatively, report the treatment means, alpha significance level, number of replicates, and post hoc test results. The tool is equipped with a variety of post hoc tests (Fisher’s LSD, Tukey’s HSD, Bonferroni and Šidák correction for multiple comparisons, and Scheffé’s test obtained from agricolae, emmeans, and multcomp R packages) and experimental designs (Latin square, Completely randomized design [CRD], Randomized complete block design [RCBD], 2-way complete factorial and split-plot arranged either as CRD or RCBD) commonly used in agricultural research sciences. With this tool, researchers can conveniently obtain a metric of variability from published reports lacking summary statistics, enabling the inclusion of studies in the quantitative review of research data.
Examples files and a tutorial can be found below.
Vinicius Garnica
Research and Teaching Assistant
My research interests include agronomy and plant pathology matter.