Null hypothesis significance testing has been under attack in recent years, partly owing to the arbitrary nature of setting α (the decision-making threshold and probability of Type I error) at a constant value, usually 0.05. If the goal of null hypothesis testing is to present conclusions in which we have the highest possible confidence, then the only logical decision-making threshold is the value that minimizes the probability (or occasionally, cost) of making errors. Setting α to minimize the combination of Type I and Type II error at a critical effect size can easily be accomplished for traditional statistical tests by calculating the α associated with the minimum average of α and β at the critical effect size. This technique also has the flexibility to incorporate prior probabilities of null and alternate hypotheses and/or relative costs of Type I and Type II errors, if known. Using an optimal α results in stronger scientific inferences because it estimates and minimizes both Type I errors and relevant Type II errors for a test. It also results in greater transparency concerning assumptions about relevant effect size(s) and the relative costs of Type I and II errors. By contrast, the use of α = 0.05 results in arbitrary decisions about what effect sizes will likely be considered significant, if real, and results in arbitrary amounts of Type II error for meaningful potential effect sizes. We cannot identify a rationale for continuing to arbitrarily use α = 0.05 for null hypothesis significance tests in any field, when it is possible to determine an optimal α.

The optimum p value significance threshold after imposing the constraints (α < 0.05 and study power ≥ 0.8), C-PSTopt, for different combinations of sample size in each arm (n1 and n2) given that the acceptable effect size ≥ 0.5, probability of 0.5 (pr) that the alternative hypothesis (H1) is correct (odds = 1), and that the seriousness of type II error is one-fourth that of type I error (C = 0.25).

Note. In Null Hypothesis Significance Testing (NHST), the null hypothesis of no effect or no difference is either true (cells A and B) or false (cells C and D). When the null hypothesis is true (i.e., the left-hand column), it is possible for a researcher to make an incorrect decision by obtaining a significant result and rejecting the null hypothesis (cell B). The probability of this happening is equal to α and is set to 5%, by convention, to help minimize the false positive. When the null hypothesis is false (i.e., the right-hand column), the researcher can make a correct decision by obtaining a significant result (cell D). The probability of this happening is (1 – β), or the statistical power of the test. When the null hypothesis is false, one can make an inferential error by failing to obtain a significant result (cell C). This error rate is defined as beta (β) and is commonly referred to as Type II error.Correct and Incorrect Conclusions in NHST.

Overview of type I and type II errors (A), proportion of type I and type II errors between 2017–2019 (B) and proportion of type I and type II errors between 2020–2021 (C). For more details about type I and type II errors see [6].

The best estimate is in bold type.&optimal values reflect integer resolution for k and 0.25 resolution for r and a; *minimizes total error; $0.1 difference in sum due to rounding; **search resolution is a quarter of a percent apart.

ap-value used for significance testing is 0.0012 [11].Probabilities are calculated for a simple linear regression with N = 43, from [11].

Population, fatality, hospitalization, and prevalence statistics are often reported for Na age intervals [ak−1, ak) (k ∈ {1, …, Na}) with . Here, a0 is the smallest age value in the data set and Δaℓ is the width of the ℓ-th age window. We assume that the population size N(ak) is constant in the considered time window. The closed interval [0, 1] contains 0, 1, and all numbers in between, and denotes the set of non-negative integers.

Type I and type II errors as well as correct decisions in EMBO peer review.

Temporal and abiotic variables used to model the probability of insect flight.

ap-value used for significance testing is 0.02495 [12].Probabilities are calculated for a one-way ANOVA with N = 30, k = 3, and σp (within groups) = 3.4, and critical effect size = σp (within groups) from [12].

Probabilities of type II error "β" for D1LT1.

Probabilities of type II error "β" for D0LT0.

nPeds: number of pedigrees in stage II. Entries are the probability of meeting the specified validation criteria for a given sample size and model. Models are described in Table 1.

Probabilities of type II error "β" for D1LT0.

No sensitivity assumptions were made. The critical cutoff was ≥ 19 positive tests out of a sample size of 3000. The calculations were made for a true prevalence of 0%. At least 99.6% specificity is required to have a Type II error of < 10%.

No sensitivity assumptions were made. The critical cutoff was ≥ 3 positive tests in at least 1 village. The calculations were made for a true prevalence of 0%. At least 99.7% specificity is required to have a Type II error of < 10%.

Summary of type 2 errors by type of case (all cases, complex, simple), participant type (all, trainee, experienced), and p values.

A 2015 article in Science (Hu et al.) proposed a new way to reduce implicit racial and gender biases during sleep. The method built on an existing counter-stereotype training procedure, using targeted memory reactivation to strengthen counter-stereotype memory by playing cues associated with the training during a 90min nap. If effective, this procedure would have potential real-world usefulness in reducing implicit biases and their myriad effects. We replicated this procedure on a sample of n = 31 college students. Contrary to the results reported by Hu et al., we found no effect of cueing on implicit bias, either immediately following the nap or one week later. In fact, bias was non-significantly greater for cued than for uncued stimuli. Our failure to detect an effect of cueing on implicit bias could indicate either that the original report was a false positive, or that the current study is a false negative. However, several factors argue against Type II error in the current study. Critically, this replication was powered at 0.9 for detecting the originally reported cueing effect. Additionally, the 95% confidence interval for the cueing effect in the present study did not overlap with that of the originally reported effect; therefore, our observations are not easily explained as a noisy estimate of the same underlying effect. Ultimately, the outcome of this replication study reduces our confidence that cueing during sleep can reduce implicit bias.

Probability of making a Type I error (p < .05, two-tailed).

Proportions of type I and type II errors in the decisions of the EMBO peer review for the LTF and YI programmes.