What 80 percent of textbooks won't tell you: The rate of p-value fallacies and effect size inclusion across introductory psychology textbooks

In psychology, null hypothesis significance testing (NHST) is the most commonly-used method for assessing quantitative research results. Despite its pervasiveness NHST is not well-understood by either psychology professors or their students; previous research suggests that up to 89 percent of psychology professors (and 100 percent of psychology undergraduate students) hold at least one false belief about what NHST can tell a researcher about their data (Heller & Krauss, 2002). Given the ubiquity of these errors, there is reason to examine the ways in which NHST is taught at the undergraduate level. The present study used a content analysis approach to code the latest available editions of 23 different North American introductory psychology textbooks. Each textbook's definition of p-values was coded for the presence of five interpretation errors (drawn from Kline, 2009), as well as for the presence of an alternative statistical index (namely, effect size). The results indicated that 80 percent of textbooks included a p-value definition that contained at least one major interpretation error. Moreover, only 30 percent of textbooks included a definition of effect size. The implications of these findings are discussed with regard to addressing rising concerns around 'p-hacking' and the misuse of NHST in psychological research.