Investigating publication bias

Funnel plots have been suggested as a means of investigating whether publication bias is likely to be a particular problem within individual meta-analyses. These plots are based on the fact that precision in estimating treatment effect increases with sample size. Thus if we plot estimated treatment effect against size or number of events in a group of similar trials, we would expect to see a wide scatter in the results for small trials with the spread narrowing as the trial size increases. If there is no bias then we would expect the shape of the plot to resemble a funnel (see Figs 11.13 and 11.14).

If there is publication bias, then we would expect the plot to be skewed, effectively with a gap where the small negative studies ought to be. Although formal tests for evaluating asymmetry have been proposed, observed asymmetry in a funnel plot could

0.33 0.6 1 Odds ratio

0.33 0.6 1 Odds ratio

0.33 0.6 1 Odds ratio

Fig. 11.13 Hypothetical funnel plots (a) symmetrical in the absence of bias (b) asymmetrical plot in the presence of publication bias smaller studies showing no statistically significant effects are missing (c) asymmetrical plot in the presence of bias due to low methodological quality of small studies (open circles show small studies of poor quality whose results are biased towards larger effects). Reproduced with permission from [56].

Fig. 11.14 Example funnel plot illustrating British MRC trials of interventions in solid tumours. Trials are included irrespective of publication status and so illustrate the type of pattern expected where there is unlikely to be publication bias. Reproduced with permission from [57].

Number of deaths

Fig. 11.14 Example funnel plot illustrating British MRC trials of interventions in solid tumours. Trials are included irrespective of publication status and so illustrate the type of pattern expected where there is unlikely to be publication bias. Reproduced with permission from [57].

be attributable to a number of factors other than publication bias [1]. For example, odds ratios overestimate risk reduction if event rates are high. If smaller trials were conducted in higher risk populations that might benefit differentially from treatment and larger trials in more general populations (as can happen if new cancer treatments are first explored in small trials of patients with advanced disease), then the estimated effect will be greater in smaller trials thereby leading to asymmetry. Similarly, hazard ratios tend to migrate towards unity over time (if the event rate remains constant). Thus larger trials that tend to follow up patients for longer will produce lower estimates of efficacy, again potentially leading to funnel plot asymmetry. Furthermore, it is likely to be particularly difficult to observe and measure asymmetry if, as is commonly the case, meta-analyses are based on only a moderate number of trials. In practice, interpretation is very difficult, and although funnel plots may serve as a useful exploratory tool, asymmetry in a funnel plot does not necessarily indicate publication bias. Conversely, symmetry cannot be taken as proof that publication bias (or indeed other forms of heterogeneity) do not exist. Ultimately there is no substitute for attempting to identify unpublished trials, and funnel plots should not be used as an excuse for avoiding doing so.

0 0

Post a comment