Biomedical Science and Research Journals | Assessing Citation Bias in Scientific Literature


Assessing Citation Bias in Scientific Literature


Our objective was to provide a further test of possible citation bias or confirmation bias in science based on differential outcomes of two articles published between 1979 and 2013. The two articles used here formed a natural experiment because the articles were from the same author and concerned similar topics except one article had more positive results for its participants. Based on Google Scholar citations, citation rates were compared using binomial tests, including normal approximation z scores, and one-sample chi-square tests. Between the two articles, the one that presented more favorable findings was cited far more often than the one with less positive results, providing empirical support for the existence of citation or confirmation bias, which may result in lower quality and less comprehensive literature reviews.

Keywords: Citation Rates; Citation Bias; Confirmation Bias; Research Methodology; Literature Reviews; Questionable Research Practices (QRP); Gay Fathers

Abbreviations: LGB: Lesbian Gay or Bisexual; LGBTQ: Lesbian Gay Bisexual Transgender Questioning; QRP, Questionable Research Practices

Short Communication

As we have discussed elsewhere [1], there are many types of research fraud, including authorship fraud, plagiarism, data falsification, publication fraud, grant fraud, and data fabrication [2], under a wider rubric of research misconduct or questionable research practices (QRPs), all of which can distort the scientific record. If the scientific record is distorted, then future research and related policy recommendations can be biased and ultimately ineffective. The QRP under investigation here is the tendency of researchers to cite information that supports their own viewpoints, often known as citation bias, a subset of confirmation bias, and to less often cite information that clashes with their own worldviews. Our view would agree with Thomas Jefferson (1820) who said “…. We are not afraid to follow truth wherever it may lead, nor to tolerate any error so long as reason is left free to combat it.” [3].

Duyx and colleagues [4] have noted that, “Citation bias is considered to be a questionable research practice (QRP). QRPs are suboptimal and undesirable behaviors of scientists that lie between responsible conduct of research and research misconduct or fraud (fabrication, falsification, and plagiarism). (p. 92)” and that QRPs “may have a strong negative impact on the development of knowledge (p. 92)” as well as concluding that “The scientific process stands or falls by a balanced representation of the available research. Citation bias distorts this balanced representation and may lead to false beliefs (p. 98).” Where citation bias exists, there is a risk that literature reviews will reflect that bias and become distorted, even misleading. The net result might be that scientific consensus could be incorrect, leading to problems with future research funding, clinical practice, legislation and policy making [5].

The objective of this study was to test for citation bias/ confirmation bias for a particular set of articles by the same author. Ideally, one would want not only the same author from the same institution but the ability to compare articles that were similar in year of publication, sample size, sample characteristics, and research topics. Elsewhere we have looked at two other cases of this problem [1].


It is not easy to test for confirmation/citation bias. In an ideal world, one might find two articles that were by the same authors, using the same data, published in the same journal, in the same year. Sometimes only less than ideal data are available. If the major difference between citation rates for two such articles is the positivity of the outcome and the positive article has significantly more citations than the article with the less favorable outcome, then we may have statistical evidence for citation or confirmation bias. If citation bias is not in operation, then the citation rates for two such articles should not be significantly different. Elsewhere we have considered two sets of articles that were nearly identical [1]. Here we will consider another set of articles that were not identical but may represent a conservative test because the more popular article has had far less time to have been cited, biasing the relative citations in a direction of greater equality.

The objective outcome measure used for comparing the papers was the Google citation count per article as of 01 June 2020.

Following [1], we used two analytic approaches. First, a one-sample chi-square test was used to test the null hypothesis that papers were cited with similar frequency regardless of the favorability of the findings (50/50 in the case for comparing two articles). Second, setting a binomial probability to 0.500 (comparing two articles), a binomial test was used to test the null hypothesis that the papers with favorable and unfavorable findings would be cited with similar frequencies. In both cases, the alternative hypothesis was that the papers with favorable findings would be cited more than those which were less favorable. Binomial tests were performed from a website calculator (www.socscistatistics. com) while we used SPSS to calculate one-sample chi-square tests.

The first study was based on a 1997 dissertation by Sirota [6] which compared 68 adult daughters of gay fathers with 68 adult daughters of heterosexual parents on numerous characteristics. Of the gay fathers’ group, 63 were gay, three were bisexual, and two were sometimes gay and at other times bisexual; all of their daughters knew of their father’s sexual orientation. She reported that the groups did not differ significantly in terms of age, education, occupation, income, degree of religiosity, attitudes towards homosexuals, attitudes towards their fathers, or current lifestyle/marital status. The two groups did not differ with respect to their fathers’ ages, incomes, or occupations. A significantly higher percentage of the daughters of gay fathers were white (87.7% vs. 69.7%, p < .01) or reported no religious affiliation (55.3% vs. 14.7%, p = .001). Daughters of gay fathers were more likely to have LGB relatives (37/66, 56% vs. 12/68, 18%, p = .001) or LGB friends (63/68, 93% vs. 41/67, 61%, p = .001).

Daughters of heterosexuals did report closer relationships with their mothers (65/68, 96%) than did daughters of gay fathers (49/66, 74%, p = .001). The parents of daughters with gay fathers were more likely to have divorced or been separated (39/68, 57%) than were those of daughters with heterosexual parents (17/68, 25%, p = .001). Roughly equal numbers of parents had died from each group (14/68 vs. 15/68), although none of the mothers of the daughters of heterosexual parents had died compared to five of the mothers of the daughters of gay fathers.

The former were more likely (23/67, 34.3%) than the latter (2/67, 3.0%) to identify as lesbian or bisexual (d = 0.87, p < .001) and, if currently heterosexual, more likely to have questioned their sexual orientation while growing up (30/43, 69.8% vs. 14/60, 23.3%; d = 1.32, p < .001). Sirota also found a significant (d = 0.68, p < .05) difference in having problems with alcohol or drug use (30/68, 44% vs. 10/68, 15%; table 13 on page 77 incorrectly reports 7.3%). The daughters of gay fathers were more likely to report one or both parents as having had problems using drugs or alcohol (40/68, 59%) than were the daughters of heterosexual fathers (21/67, 31%, p = .001). Later, Sirota [7] reported significant differences in insecure attachment (78% vs. 44%, d = 0.72, p < .001) and with respect to feeling uncomfortable with close relationships (44% vs. 12%, d = 0.75, p < .001). The differences could not be explained away by differentials in parental divorce rates [8], although later scholars [9] have implied or asserted, without evidence, that the differentials were divorce-related. The Google citation count for this first article was 28.

The comparison study was also by the same author and also concerned research on homosexuality in which problems with homophobic bias among nurses was investigated [10]. The sample was larger and the article had fewer years during which it could have been cited than the earlier comparison article [7]. The Google citation count for the second article was 52.


Comparing Sirota’s two articles’ citation counts of 28 and 52, by the binomial test, we obtained z = - 2.57 (p < .006). The onesample chi-square test (df = 1) obtained was 7.20 (p = .007). After eliminating citations by known conservative scholars from the two sets of citations, the one-sample chi-square test was 17.75 (p < .001) while z = - 4.09 (p < .001) for the binomial test.


In the case of the two articles compared, citation rates were significantly higher for the more positive article than for the more negative article. This type of pattern has the potential to bias literature reviews, lower their quality [11] and create a false scientific consensus [12]. Publication bias may exacerbate the problem, if positive manuscripts are more likely to be published as journal articles than negative manuscripts. It is possible that the dissertation discussed here was largely unpublished for such reasons.


It is our hope that by informing fellow scientists about the risks of citation bias and confirmation bias, those biases may be reduced in the future, even if little can be done about past issues. One advantage of our methodological approach is that the situation can be re- evaluated every few years to check if citation bias patterns are similar, decreasing, or increasing over time. Ultimately, we hope that literature reviews will more accurately reflect the “facts on the ground” and thereby draw more accurate conclusions about scientific and social phenomena, even if that means correcting the scholarly record [13,14] in some cases.

For More information:

Know More Information: Biomed Grid Please Click on :

For More information: Open Access Journals of Biomedical Science and ResearchJournal of Biomedical Science 

Know More Information: Biomed Grid Please Click on :


Popular posts from this blog

Biomedical Science and Research Journals | Advanced Applications of Biomaterials Based on Alginic Acid

Biomedical Science and Research Journals | The Need of Early Detection of Positive COVID-19 Patients in the Community by Viral Tests (e.g. RT-PCR Tests) and Antibody Tests (Serological Tests) to Stop the Spread

Biomedical Science and Research Journals | Substance Identification in Anti-Doping Control-Some Issues