An Example of Publication Bias - Cognitive Advantage in Bilingualism

MindBlog has done several posts on experiments that reinforce what has become the conventional wisdom regarding bilingualism: that it enhances our executive control faculties. De Bruin et al., noting that they themselves had chosen to report positive results supporting this conclusion, but had not followed through on negative data (the file drawer effect), thought to do a systematic analysis of the publication fate of experiments with positive, mixed, or negative outcomes. They searched for conference abstracts on bilingualism and executive control in 169 conferences (31 different national and international meetings) organized between 1999 and 2012, and identified 128 abstracts (presented at 52 different conferences) that focused on bilingualism and executive control. Their result:

Sixty-eight percent of the studies that clearly found a bilingual advantage were published, compared with 50% of the studies that found mixed results supporting the bilingual-advantage theories, 39% of the studies that found mixed results partly challenging those theories, and 29% of the studies that found no differences between monolinguals and bilinguals or found a bilingual disadvantage. On the whole, 63% of the studies supporting the bilingual advantage were published, compared with only 36% of the studies that challenged it.

From the authors discussion:

This difference in publication percentage based on the outcomes of the study could be the result of a bias during several steps of the publication process: Authors, reviewers, and editors can decide to submit or accept only studies that showed positive results. In the first step of the publication process, the file-drawer problem could play an important role in the observed publication bias. Authors could decide not to publish studies with null or mixed results, or they could choose to submit their results only partially, for example, by leaving out tasks that did not show an effect of bilingualism. The article by Treccani et al. (2009)[Treccani is a co-author of the current paper] is an example of the file-drawer problem, as it excluded the experiments that did not show an effect of bilingualism.

On the next level, reviewers and editors might reject manuscripts reporting null, negative, or mixed results more often than manuscripts reporting positive effects. This rejection is often based on the argument that null effects are difficult to interpret, or the result of poor stimulus design... Mahoney (1977) asked journal reviewers to referee manuscripts reporting positive, negative, mixed, or null results with identical methodological procedures. Although the methodology was the same, reviewers scored the manuscripts reporting positive results as methodologically better than the manuscripts reporting negative or mixed results. For manuscripts with positive results, reviewers usually recommended acceptance with moderate revisions. For manuscripts with negative results, however, their usual recommendation was major revision or rejection. Manuscripts with mixed results were mostly rejected.