Decision Mechanics

Insight. Applied.

Confidence intervals

By

Statistical confidence intervals are almost always misinterpreted. Consider the following statement.

"The prevalence of the disease P has a 95% confidence interval of 1% <= P <= 5%."

This is commonly taken to imply that there’s a 95% chance that the true prevalence is between 1% and 5%.

This isn’t the case.

Confidence intervals represent uncertainty about the interval, rather than the parameter of interest.

The correct interpretation of the confidence interval defined above is that if we collect many samples from the population and calculate confidence intervals from them, 95% of those confidence intervals will contain the true value of P.

In Bayesian statistics we generally calculate credible intervals which are compatible with the intuitive interpretation.

Absence of evidence is not evidence of absence

By

A statistical note published in the BMJ (formally the British Medical Journal) reminds us that

…trials that do not show a significant difference between the treatments being compared are often called "negative." This term wrongly implies that the study has shown that there is no difference, whereas usually all that has been shown is an absence of evidence of a difference. These are quite different statements.

Statisticians publish guidance on p-values

By

American Statistical Association building

The American Statistical Association has taken the unprecedented step of issuing guidance on the use of p-values. Its statement was prompted by concerns that misuse of p-values was driving bad science.

Basic and Applied Social Psychology has refused to accept papers containing p-values.

A p-value is “informally” defined in the statement as

…the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

Doesn’t really help much, but you have to applaud the attempt. After all, even scientists don’t really understand p-values.