Department of Biology

Few students sitting in their introductory statistics class learn that they are being taught the product of a misguided effort to combine two methods of inference into one. Few students learn that many think the method they are being taught should be banned. In Understanding Null Hypothesis Tests, and Their Wise Use, I emphasize Ronald Fisher’s approach to null hypothesis testing. Fisher’s method is simple, intuitive, thoughtful, and pure. If we follow Fisher’s example, all the criticisms of null hypothesis tests melt away. Fisher on a good day. Do you collect data and then ask a friend how to analyze them? Once you have read this monograph, you will not have to do that anymore (at least not often). You will understand the concepts behind the mathematics. You will see why different types of data require different types of tests. Do you think that P is the probability of a type I error? It is not. If you fail to reject, do you accept the alternate or alternative hypothesis? You accomplish nothing by doing so. Do you equate statistical “significance” with importance? You should not. These and other misconceptions are explained and dispensed with in Understanding Null Hypothesis Tests, and Their Wise Use. After reading this monograph, you will understand why it is utter foolishness to say, We use confidence intervals instead. (Confidence intervals are wonderful, but they show the results of null hypothesis tests performed backwards.) More importantly, you will understand wise use. You will use P-values thoughtfully, not to make mindless, binary decisions. Most importantly, you will know the Big Secret that should not be a secret. A null hypothesis is infinitely precise, so many and maybe most null hypotheses cannot be correct—a fact we should know from the start. It was the legendary statistician John Tukey who explained why it is important to test such nulls anyway: to determine whether we can trust our data to tell us the direction of a difference. Getting the direction wrong is referred to as a type III or type S error. Once you have read about type S errors in Understanding Null Hypothesis Tests, and Their Wise Use, you will have a better understanding of null hypothesis tests than anyone on your block.

Understanding Null Hypothesis Tests, and Their Wise Use

Many interpret error bars to mean that if they do not overlap the difference is statistically significant. This overlap rule is really an overlap myth; the rule does not hold true for any conventional type of error bar. There are rules of thumb for estimating P values, but it would be better to show error bars for which the overlap rule holds true. Here I explain how to calculate what I call comparative confidence intervals which, when plotted as error bars, let us judge significance based on overlap or separation. Others have published on these intervals (the mathematical basis goes back to John Tukey) but here I advertise comparative confidence intervals in the hope that more people use them. Judging statistical significance by eye would be most useful when making multiple comparisons, so I show how comparative confidence intervals can be used to illustrate the results of Tukey tests or Dunnett’s test. I also explain how to use of comparative confidence intervals to explore the effects of multiple independent variables and explore the problems posed by heterogeneity of variance and repeated measures. When families of comparative confidence intervals are plotted around means, I show how box-and-whiskers plots make it easy to judge which intervals overlap with which. Comparative confidence intervals have the potential to be used in a wide variety of circumstances, so I describe an easy way to confirm the intervals’ validity. When sample means are being compared to each other, they should be plotted with error bars that indicate comparative confidence intervals, either along with or instead of conventional error bars. This paper, which can be downloaded below, is based on a submission that was rejected by Psychological Methods. The original submission along with the reviewer’s comments and my responses are also available below.

Department of Biology

Items in this collection

Understanding Null Hypothesis Tests, and Their Wise Use

Making the Error Bar Overlap Myth a Reality: Comparative Confidence Intervals

.repo-1h6ne0y{background:#ffffff;text-transform:uppercase;padding-right:1.5rem;}.repo-1h6ne0y:after{content:'';position:absolute;background:#adadad;transform:translateY(-50%);top:50%;left:0;width:100%;height:0.1rem;z-index:-1;}Items in this collection

Making the Error Bar Overlap Myth a Reality: Comparative Confidence Intervals

Items in this collection