forecasting//random-effects pooling
Random-effects pooling is a way of combining several estimates of one quantity that treats their disagreement beyond their own error bars as real variation between sources, and it is used in meta-analysis and in forecasting so that conflicting studies widen the result instead of being averaged into false precision.
Random-effects pooling is a way of combining several estimates of one quantity that treats their disagreement beyond their own error bars as real variation between sources, and it is used in meta-analysis and in forecasting so that conflicting studies widen the result instead of being averaged into false precision.
Plain inverse-variance weighting gives each estimate yiy_iyi the weight wi=1/σi2w_i = 1/\sigma_i^2wi=1/σi2 and assumes every source measures the same value. The random-effects version (DerSimonian and Laird) measures how much the estimates scatter around the weighted mean yˉ\bar yyˉ, with Q=∑iwi(yi−yˉ)2Q = \sum_i w_i (y_i - \bar y)^2Q=∑iwi(yi−yˉ)2, and turns the excess over what their own errors explain into a between-source variance:
τ2=max (0, Q−(k−1)∑iwi−∑iwi2/∑iwi)\tau^2 = \max\!\left(0,\; \frac{Q - (k-1)}{\sum_i w_i - \sum_i w_i^2 / \sum_i w_i}\right)τ2=max(0,∑iwi−∑iwi2/∑iwiQ−(k−1))
with kkk sources. Each estimate is then weighted by 1/(σi2+τ2)1/(\sigma_i^2 + \tau^2)1/(σi2+τ2). Take two estimates of the yearly gain from better algorithms in language models, about threefold and about tenfold, each with a fairly tight interval: inverse-variance weighting returns a confident middle, while random effects returns the middle with an interval wide enough to hold both, and the gap between the two results is the disagreement made visible.
The method still assumes the sources are independent. Several estimates from one research group, or built on one dataset, count as fewer sources than they look.
With two or three sources τ2\tau^2τ2 is itself poorly estimated, so the widened interval is a floor on the uncertainty, not a measurement of it.