Correlation using Python

What is Correlation?

• Definition: Correlation is a statistical measure that describes the degree to which two or more variables move in relation to each other. It quantifies the strength and direction of a potential linear relationship.
• Correlation Coefficient (r): The most common measure of correlation is Pearson’s correlation coefficient. It ranges from -1 to +1.
  • -1: Perfect negative correlation (as one variable increases, the other decreases)
  • 0: No correlation (no predictable relationship)
  • +1: Perfect positive correlation (as one variable increases, the other increases)

Importance of Correlation

1. Feature Selection: Identify features in machine learning models that have a strong correlation with your target variable. Discard features with weak correlations to potentially improve model performance and reduce overfitting.
2. Exploratory Data Analysis (EDA): Understanding relationships between different variables in your dataset.
3. Experimental Design: Helps in designing experiments and interpreting results.

Correlation Methods in Python

1. NumPy:
   • np.corrcoef(x, y): Calculates the correlation matrix, including pairwise correlations between variables.
2. Pandas:
   • df.corr(): Calculates pairwise correlations between all columns in a DataFrame.
   • df['column1'].corr(df['column2']): Calculates correlation between specific columns.
3. SciPy:
   • scipy.stats.pearsonr(x, y): Calculates Pearson’s correlation coefficient and p-value

Example (Using Pandas)

import pandas as pd
import numpy as np

# Sample DataFrame
data = {'height': [170, 185, 165, 190, 174],
        'weight': [65, 80, 72, 92, 70],
        'age': [25, 30, 28, 35, 29]}
df = pd.DataFrame(data)

# Calculate correlation matrix
correlations = df.corr()
print(correlations)

Interpretation

The output of the correlation matrix provides insights into the relationships between the variables:

• A strong positive correlation between ‘height’ and ‘weight’.
• A moderate negative correlation between ‘height’ and ‘age’.

Visualization

Use libraries like Seaborn or Matplotlib for visualizing correlation matrices:

import seaborn as sns 

# Visualize as heatmap
sns.heatmap(correlations, annot=True)

Important Considerations

• Correlation does not imply causation: Finding correlation doesn’t necessarily mean one variable causes the other. There could be confounding factors or lurking variables.
• Types of Correlation: Pearson’s coefficient measures linear correlation. Other types exist (like Spearman’s for rank-based correlation). Choose the appropriate method based on your data characteristics.

Example: Height vs. Weight Correlation using NumPy and SciPy

import numpy as np
from scipy.stats import pearsonr

# Sample data (height in centimeters, weight in kilograms)
height = np.array([170, 185, 165, 190, 174])
weight = np.array([65, 80, 72, 92, 70])

# Correlation using NumPy (correlation matrix)
correlation_matrix = np.corrcoef(height, weight)
print("Correlation Matrix (NumPy):")
print(correlation_matrix)

# Correlation using SciPy (Pearson's coefficient and p-value)
correlation, p_value = pearsonr(height, weight)
print("\nCorrelation Coefficient (SciPy):", correlation)
print("p-value (SciPy):", p_value)

Explanation

1. Sample Data: We create sample arrays representing heights and weights of individuals.
2. NumPy’s corrcoef:
   • Calculates a correlation matrix, showing the correlation between each pair of variables. Since we have two variables, it will be a 2×2 matrix.
3. SciPy’s pearsonr:
   • Specifically calculates Pearson’s correlation coefficient, which measures linear correlation.
   • Provides the correlation coefficient (correlation) and the p-value, which helps assess the statistical significance of the correlation.

Interpretation

• The correlation matrix will show a strong positive correlation between height and weight.
• The p-value will likely be small, indicating that the observed correlation is statistically significant (not simply due to chance).

Key Points

• NumPy gives you the broader correlation matrix, helpful if you have multiple variables.
• SciPy’s pearsonr allows you to focus directly on Pearson’s correlation and its statistical significance.

Understanding Multivariable Correlation

When you have more than two variables, you need a way to visualize and understand the correlations between all of them at once. Here’s where the correlation matrix comes in

Steps

1. Arrange Your Data:
   • NumPy array: Create a 2D NumPy array where each row represents a data point and each column represents a variable. For example:

import numpy as np

data = np.array([
    [170, 65, 25, 70],  # Data point 1: height, weight, age, score 
    [185, 80, 30, 85],  # Data point 2
    # ... more data points
])

Calculate the Correlation Matrix (NumPy):

• Use np.corrcoef(data, rowvar=False) to calculate the correlation matrix. Note: rowvar=False is important to treat each column as a variable.

correlation_matrix = np.corrcoef(data, rowvar=False)
print(correlation_matrix)

1. Interpret the Correlation Matrix:
   • The correlation matrix is square (e.g., 4×4 if you have four variables).
   • Each cell (i, j) represents the correlation coefficient between variable i and variable j.
   • The diagonal values will always be 1 (a variable correlates perfectly with itself).

Individual Correlations (SciPy)

If you want to focus on the correlation between specific pairs of variables within the larger set:

• Use scipy.stats.pearsonr to calculate the Pearson correlation coefficient between two selected variables.

from scipy.stats import pearsonr 
 
# Correlation between variable 0 and variable 2 of your data
correlation, p_value = pearsonr(data[:, 0], data[:, 2]) 
print(correlation, p_value)

Example (Building on Previous Height/Weight Example)

import numpy as np
from scipy.stats import pearsonr

# Sample data (height, weight, age, test score)
data = np.array([
    [170, 65, 25, 70], 
    [185, 80, 30, 85],
    [165, 72, 28, 68],
    [190, 92, 35, 98],
    [174, 70, 29, 75] 
])

# Correlation matrix
correlation_matrix = np.corrcoef(data, rowvar=False)
print("Correlation Matrix:\n", correlation_matrix)

# Correlation between height and test score
corr_height_score, p_value = pearsonr(data[:, 0], data[:, 3])
print("Correlation between height and test score:", corr_height_score)

Key SciPy Function: scipy.stats.pearsonr

The pearsonr function from SciPy’s statistics module is your primary tool for calculating correlation between pairs of variables. Let’s see how this works for three or more variables:

Steps

1. Prepare Your Data:
   • Have your data in a format SciPy can work with, such as a NumPy array or Pandas DataFrame. Each column should represent a different variable.
2. Calculating Pairwise Correlations
   • Iterative approach: Use a loop or list comprehension to apply pearsonr to every combination of variables you want to examine.

from scipy.stats import pearsonr

# ... (Assuming you have your 'data' either as a NumPy array or DataFrame)

variables = data.columns  # Example if you're using a DataFrame

for i in range(len(variables)):
    for j in range(i + 1, len(variables)):
        var1 = variables[i]
        var2 = variables[j]
        corr, p_value = pearsonr(data[var1], data[var2])
        print(f"Correlation between {var1} and {var2}: {corr} (p-value: {p_value})")

Advantages of SciPy’s Approach

• Flexibility: You control exactly which pairs of variables get analyzed.
• Statistical Significance: The pearsonr function returns both the correlation coefficient and the p-value, helping you determine whether the correlation is likely to be a result of chance or not.

Example: Expanding on the Previous One

from scipy.stats import pearsonr
import pandas as pd

# Sample data (height, weight, age, test score)
data = {
    'height': [170, 185, 165, 190, 174], 
    'weight': [65, 80, 72, 92, 70],
    'age': [25, 30, 28, 35, 29], 
    'test_score': [70, 85, 68, 98, 75]
}
df = pd.DataFrame(data)

# Pairwise correlations
for i in range(len(df.columns)):
    for j in range(i + 1, len(df.columns)):
        var1 = df.columns[i]
        var2 = df.columns[j]
        corr, p_value = pearsonr(df[var1], df[var2])
        print(f"Correlation between {var1} and {var2}: {corr} (p-value: {p_value})")

Remember:

• SciPy’s pearsonr focuses on individual pairwise correlations. If you want a complete correlation matrix in one step, NumPy’s corrcoef is more convenient.
• SciPy is excellent when you need to analyze specific pairs or consider the statistical significance of the correlations.

Hypothesis Testing

What is a p-value?

• In the context of statistical hypothesis testing, a p-value is the probability of obtaining results at least as extreme as the observed results, assuming that the null hypothesis is true.
• Simplifying: It helps you decide whether your observed data is surprising if the null hypothesis were actually true.

Understanding p-values

1. Null Hypothesis (H0): This is the statement you start out assuming; usually, it’s a statement of “no effect” or “no difference.” For example: “There is no difference in average test scores between group A and group B.”
2. Alternative Hypothesis (H1): This is what you might suspect to be true instead. For example: “There is a difference in average test scores between group A and group B.”
3. Significance Level (α): This is a threshold you set before the experiment (often 0.05 or 0.01). It represents the risk you’re willing to take of rejecting the null hypothesis when its actually true (a false positive).
4. Calculating the p-value: You perform a statistical test that produces a test statistic. The p-value is calculated based on this test statistic and the assumption that the null hypothesis is true.

Interpreting p-values

• p-value ≤ α: You reject the null hypothesis. Your results are considered statistically significant, suggesting support for the alternative hypothesis.
• p-value > α: You fail to reject the null hypothesis. You don’t have enough evidence to conclude there’s a significant effect or difference.

Common Misconceptions

• It’s NOT the probability the null hypothesis is true.
• It’s NOT the probability of repeating the experiment and getting the same result.
• Doesn’t tell you the size or importance of an effect.

Importance of p-values

• Scientific research: Used extensively to determine whether observed results are likely due to chance or a real effect.
• Decision-making: Provides a measure of evidence against the null hypothesis, aiding informed conclusions.

Example

You’re testing a new weight-loss drug.

• Null hypothesis: The drug has no effect on weight loss compared to a placebo.
• Alternative hypothesis: The drug leads to greater weight loss than a placebo.
• Significance level: α = 0.05
• Test results: The p-value is 0.02.

Conclusion: Since the p-value is less than the significance level, you reject the null hypothesis. This suggests there’s evidence that the drug might have an effect on weight loss.

Here’s how to interpret p-values in the context of correlation:

Basics

• Null Hypothesis (H0): The population correlation coefficient (usually denoted by the Greek letter rho, ρ) is zero. This means there’s no real linear relationship between the two variables.
• Alternative Hypothesis (H1): The population correlation coefficient is not zero. This implies a linear relationship may exist (positive or negative).
• P-value: The p-value tells you the probability of observing a correlation coefficient as extreme or more extreme than your current calculated correlation, if the null hypothesis (no correlation) were true.

Interpretation

• Small p-value (e.g., p-value < 0.05): It’s unlikely that the observed correlation occurred by chance alone. You reject the null hypothesis and conclude there is likely a statistically significant correlation between the variables.
• Large p-value (e.g., p-value > 0.05): It’s more likely that the observed correlation could have occurred due to random sampling variability. You fail to reject the null hypothesis and conclude that you don’t have sufficient evidence for a statistically significant correlation.

Example

You calculate a correlation coefficient of 0.8 with a p-value of 0.003 between height and weight.

• Interpretation: The small p-value (0.003 < 0.05) suggests that a correlation this strong is unlikely to occur by chance alone if there was truly no relationship between height and weight in the population. You likely have a statistically significant positive correlation.

Important Notes:

• Statistical vs. Practical Significance: A statistically significant correlation doesn’t always mean a practically important one. Consider the size of the correlation coefficient too.
• Correlation is not Causation: Statistical significance doesn’t prove that one variable causes the other.
• Assumptions: Pearson’s correlation assumes a linear relationship and some other statistical assumptions. If the data violates these assumptions, the p-value might be misleading.

How p-values are Calculated

The calculation of the p-value for a correlation coefficient involves these key steps:

1. Calculate the t-statistic: This involves your sample correlation coefficient, sample size, and degrees of freedom.
2. Look up the t-distribution: Using the calculated t-statistic and the degrees of freedom, you find the corresponding p-value on a t-distribution table or using statistical software.