Tidbit series: How to explain Data Science concepts to anyone.  

Yes, let us talk about good’ol P-Values.

P-values are tricky to understand and easy to misunderstand. Great properties to become the most common statistical measure if you ask me.

It does not have to be that way!

A quick first explanation I like is borrowed from Cassie Kozyrkov:

  • The lower the p-value is, the more likely someone was surprised by something.

The vagueness of that explanation is what is good about it. A p-value in isolation is not saying much. Only with the associated analysis it becomes valuable. One could also say:

  • The p value tells you how well a hypothesis and the data fit together.

Why the P-value makes sense

Let’s say we did an experiment and measured some kind of effect. This effect can have mainly 3 possible explanations:

  1. The effect is real and that is why we see the effect in the data. (What we usually hope for)
  2. The effect is not real and the reason is just coincidence.
  3. The effect is not real and the reason is some bias in the data / analysis / population.

With good study design (randomization or some other causal identification strategy) we can eliminate case 3:

  1. The effect is real and that is why we see the effect in the data. (What we usually hope for)
  2. The effect is not real and the reason is just coincidence.
  3. The effect is not real and the reason is some bias in the data / analysis / population.

Now onto the important bit:

  • The p value tells you how plausible the explanation “The effect is just coincidence” is.

This means a low p-value makes case 2 an unreasonable explanation as well:

  1. The effect is real and that is why we see the effect in the data. (What we usually hope for)
  2. The effect is not real and the reason is just coincidence.
  3. The effect is not real and the reason is some bias in the data / analysis / population.

…and Voila! With the help of study design and low p-values, case one is the only viable explanation left, so we conclude that an effect is real. This is how p-values actually work in “proving” an effect.

Why the P-value does not make sense

Instead of proving things directly, p-values do it indirectly, by showing that alternatives do not make sense. This is a bit unintuitive.

When testing a Hypothesis, p-values are not what we are looking for. They are only a tool to get to what we actually want: To know if something is true or false. Unfortunately, a p-value can not give us that, it assumes the premise of the question. (Nothing can prove 100% to be fair. But p-values cannot; even philosophically)

The next best thing besides proving would be having the likelihood that something is true. This can also not be provided by a p-value. Frequentist statistics (of which p-values are a part of) does not quanitfy uncertainty like that.

In the end, the straight forward alternative to this is Bayesian statistic. Bayesian Stats does not run into these troubles, even if there are other challenges with it. But that is a topic for another time.