learn-co-students / dsc-2-20-04-p-values-and-null-hypothesis-nyc-career-ds-062518

P-Values and the Null Hypothesis

Introduction

In this lesson, we'll learn about the relationship between P-values and the Null Hypothesis, and their role in designing an experiment.

Objectives

You will be able to:

• Understand and explain the null hypothesis, including its role in sound experiment design
• Understand, calculate, and interpret P-values

Understanding The Null Hypothesis

As we said previously, scientific experiments actually have 2 hypotheses:

Null Hypothesis: There is no relationship between A and B Example: "There is no relationship between this flu medication and a reduced recovery time from the flu".

The Null Hypothesis is usually denoted as $H_O$

Alternative Hypothesis: The hypothesis we traditionally think of when thikning of a hypothesis for an experiment Example: "This flu medication reduces recovery time for the flu."

The Alternative Hypothesis is usually denoted as $H_a$

P-Values and Alpha Values

No matter what you're experimenting on, good experiments come down down to one question: Is our p-value less than our alpha value? Let's dive into what each of these values represents, and why they're so important to experimental design.

P-value: The calculated probability of arriving at this data randomly.

If we calculate a p-value and it comes out to 0.03, we can interpret this as saying "There is a 3% chance that the results I'm seeing are actually due to randomness or pure luck".

$\alpha$ (alpha value): The marginal threshold at which we're okay with with rejecting the null hypothesis.

An alpha value can be any value we set between 0 and 1. However, the most common alpha value in science is 0.05 (although this is somewhat of a controversial topic in the scientific community, currently).

If we set an alpha value of $\alpha = 0.05$, we're essentially saying "I'm okay with accepting my alternative hypothesis as true if there is less than a 5% chance that the results that I'm seeing are actually due to randomness".

When we conduct an experiment, our goal is calculate a p-value and compare it to our alpha value. If $p < \alpha$, then we reject the null hypothesis and accept that there is not "no relationship" between our dependent variables. Note that any good scientist will admit that this doesn't prove that there is a direct relationship between our dependent and independent variables--just that we have enough evidence to the contrary to show that we can no longer believe that there is no relationship between them.

In simple terms:

$p < \alpha$: Reject the Null Hypothesis and accept the Alternative Hypothesis

$p >= \alpha$: Fail to reject the Null Hypothesis.

There are many different ways that we can structure a hypothesis statement, but they always come down to this comparison in the end. In normally distributed data, we calculate p-values from z-scores. This is done a bit differently with discrete data. We may also have One-Tail and Two-Tail tests.

A One-Tail Test is when we want to know if a parameter from our treatment group is greater than (or less than) a corresponding parameter from our control group.

Example One-Tail Hypothesis

"$H_a = \mu_1 < \mu_2 $ The treatment group given this weight loss drug will lost more weight on average than the control group that was given a competitor's weight loss drug

$ H_o = \mu1 >= \mu_2$ The treatment group given this weight loss drug will not lose more weight on average than the control group that was given a competitor's weight loss drug".

A Two-Tail Test is for when we want to test if a parameter falls between (or outside of) a range of two given values.

Example Two-Tail Hypothesis

$H_a = \mu_1 != \mu_2$ "People in the experimental group that are administered this drug will not lose the same amount of weight as the people in the control group. They will be heavier or lighter".

$H_o = \mu_1 = \mu_2$ "People in the experimental group that are administered this drug will lose the same amount of weight as the people in the control group."

Use the Charts!

The following charts are designed to help you remember how to phrase, set up, and evaluate hypotheses for any experiment. Note that before you can know which chart to use, you need to define if your experiment is with Continuous or Discrete data, and if your hypothesis test is a One-Tail or a Two-Tail test!

Charts for Continuous Data

Charts for Discrete Data

Important Note

Do Not make the mistake of thinking you need to memorize each of the charts above. Instead, focus on understanding the differences between them. You'll find each of the following charts stored as an image within the corresponding github repo for this lesson.

Download these charts and store them on your machine for when you need them!

What Does an Experiment Really Prove?

You may be wondering why we need Null Hypothesis at all. This is a good question. It has to do with being honest about what an experiment actually proves.

Scientists use the Null Hypothesis so that we can be very specific in our findings. This is because a successful experiment doesn't actually prove a relationship between our dependent and independent variable. Instead, it just proves that we do not have enough evidence to convincingly believe there is no relationship between the dependent and the independent variable. There can always be a lurking variable behind the scenes that is actually responsible for the relationship between our two variables--its almost impossible to cover every possible angle. However, a successful experiment where our p-value is less than our alpha value (typically, $p &lt; 0.05$) does give us enough information to confidently say that's statistically unlikely that there is no relationship between the two, which is what would have to be true in order for the null hypothesis to be correct!

The Null Hypothesis Loves You and Wants You To Be Happy

We've covered a lot about the null hypothesis and how it's used in experiments in this lesson, but there's a lot more to learn about it!

Read the following article, The Null Hypothesis Loves You and Wants You To Be Happy. This does an excellent job of explaining why the concept of the Null Hypothesis is crucial to good science.

Summary

In this lesson, you learned about the relationship between P-values and the Null Hypothesis. Now let's move on and see how group sizes affect our tests!