This page includes an interactive code editor. Try modifying and running the examples!

← Handling Missing Data MultiIndex & Hierarchical Data →

Pandas Statistical Functions

Statistical analysis is crucial for understanding data distributions, relationships, and patterns. Pandas provides comprehensive statistical functions that work seamlessly with NumPy and SciPy for advanced analysis.

1. Descriptive Statistics

Basic statistical summaries that describe the main features of datasets.

Central Tendency: Mean, median, mode
Dispersion: Variance, standard deviation, range
Shape: Skewness, kurtosis
Position: Quantiles, percentiles

Function	Description	Usage
`describe()`	Comprehensive statistical summary	`df.describe()`
`mean()`	Arithmetic average	`df['col'].mean()`
`median()`	Middle value	`df['col'].median()`
`std()`	Standard deviation	`df['col'].std()`
`var()`	Variance	`df['col'].var()`

2. Correlation Analysis

Measure relationships between variables.

# Pearson correlation
df.corr(method='pearson')

# Spearman correlation
df.corr(method='spearman')

# Correlation with specific column
df.corrwith(df['target'])

3. Group-wise Statistics

Calculate statistics for different groups in data.

GroupBy: Split-apply-combine pattern
Aggregation: Multiple statistics at once
Transformation: Group-specific calculations

Statistical Measures Overview

📊 Central Tendency

Mean: Average value
Median: Middle value
Mode: Most frequent value
Geometric Mean: For growth rates
Harmonic Mean: For rates and ratios

📈 Dispersion

Variance: Average squared deviations
Standard Deviation: Square root of variance
Range: Difference between max and min
IQR: Interquartile range
MAD: Mean absolute deviation

📐 Distribution Shape

Skewness: Asymmetry measure
Kurtosis: Tail heaviness
Moments: Statistical moments
Quantiles: Data division points

🔗 Relationship Measures

Correlation: Linear relationship
Covariance: Joint variability
R-squared: Goodness of fit
P-value: Statistical significance

Hypothesis Testing Methods

T-Tests

One-sample t-test
Independent samples t-test
Paired t-test
Compare group means

ANOVA

One-way ANOVA
Two-way ANOVA
Compare multiple groups
F-statistic analysis

Normality Tests

Shapiro-Wilk test
Kolmogorov-Smirnov test
D'Agostino's test
QQ-plot analysis

Example: Comprehensive Statistical Analysis

Statistical Functions Examples

import pandas as pd
import numpy as np
import scipy.stats as stats

# Create a comprehensive dataset for statistical analysis
np.random.seed(42)
n_samples = 1000

data = {
    'Age': np.random.normal(35, 10, n_samples).astype(int),
    'Salary': np.random.lognormal(10.5, 0.4, n_samples),
    'Height': np.random.normal(170, 10, n_samples),
    'Weight': np.random.normal(70, 15, n_samples),
    'Test_Score': np.random.beta(2, 5, n_samples) * 100,
    'Sales': np.random.poisson(50, n_samples),
    'Department': np.random.choice(['IT', 'HR', 'Finance', 'Marketing'], n_samples),
    'Experience': np.random.exponential(5, n_samples).astype(int),
    'Bonus': np.random.uniform(0, 10000, n_samples),
    'Satisfaction': np.random.randint(1, 11, n_samples)
}

df = pd.DataFrame(data)

# Add some missing values for realistic data
df.loc[np.random.choice(n_samples, 50), 'Salary'] = np.nan
df.loc[np.random.choice(n_samples, 30), 'Test_Score'] = np.nan

# Add correlation between Height and Weight
df['Weight'] = df['Height'] * 0.4 + np.random.normal(0, 5, n_samples)

print("Dataset Overview:")
print(df.head())
print(f"\nDataset Shape: {df.shape}")
print("\nData Types:")
print(df.dtypes)

# 1. DESCRIPTIVE STATISTICS
print("\n" + "="*60)
print("1. DESCRIPTIVE STATISTICS")
print("="*60)

# Basic descriptive statistics
print("\nBasic Descriptive Stats:")
print(df.describe())

# Detailed statistics for numerical columns
numerical_cols = ['Age', 'Salary', 'Height', 'Weight', 'Test_Score', 'Sales', 'Bonus']
print("\nDetailed Statistics for Numerical Columns:")
for col in numerical_cols:
    if col in df.columns:
        print(f"\n--- {col} ---")
        print(f"Count:    {df[col].count():.0f}")
        print(f"Mean:     {df[col].mean():.2f}")
        print(f"Median:   {df[col].median():.2f}")
        print(f"Std Dev:  {df[col].std():.2f}")
        print(f"Variance: {df[col].var():.2f}")
        print(f"Min:      {df[col].min():.2f}")
        print(f"Max:      {df[col].max():.2f}")
        print(f"Range:    {df[col].max() - df[col].min():.2f}")
        print(f"Q1 (25%): {df[col].quantile(0.25):.2f}")
        print(f"Q3 (75%): {df[col].quantile(0.75):.2f}")
        print(f"IQR:      {df[col].quantile(0.75) - df[col].quantile(0.25):.2f}")

# 2. MEASURES OF CENTRAL TENDENCY
print("\n" + "="*60)
print("2. MEASURES OF CENTRAL TENDENCY")
print("="*60)

print("\nMeasures of Central Tendency:")
central_tendency = pd.DataFrame({
    'Mean': df[numerical_cols].mean(),
    'Median': df[numerical_cols].median(),
    'Mode': [df[col].mode().iloc[0] if not df[col].mode().empty else np.nan for col in numerical_cols],
    'Geometric_Mean': [stats.gmean(df[col].dropna()) for col in numerical_cols],
    'Harmonic_Mean': [stats.hmean(df[col].dropna()) for col in numerical_cols if (df[col] > 0).all()]
})

print(central_tendency)

# 3. MEASURES OF DISPERSION
print("\n" + "="*60)
print("3. MEASURES OF DISPERSION")
print("="*60)

dispersion = pd.DataFrame({
    'Variance': df[numerical_cols].var(),
    'Std_Dev': df[numerical_cols].std(),
    'Range': df[numerical_cols].max() - df[numerical_cols].min(),
    'IQR': df[numerical_cols].quantile(0.75) - df[numerical_cols].quantile(0.25),
    'MAD': df[numerical_cols].mad(),  # Mean Absolute Deviation
    'Coefficient_of_Variation': (df[numerical_cols].std() / df[numerical_cols].mean()) * 100
})

print("Measures of Dispersion:")
print(dispersion)

# 4. SHAPE OF DISTRIBUTION
print("\n" + "="*60)
print("4. DISTRIBUTION SHAPE STATISTICS")
print("="*60)

from scipy.stats import skew, kurtosis

shape_stats = pd.DataFrame({
    'Skewness': [skew(df[col].dropna()) for col in numerical_cols],
    'Kurtosis': [kurtosis(df[col].dropna()) for col in numerical_cols]
}, index=numerical_cols)

print("Distribution Shape Statistics:")
print(shape_stats)

# Interpretation helper
print("\nSkewness Interpretation:")
print("> 0: Right-skewed, < 0: Left-skewed, ≈ 0: Symmetrical")
print("\nKurtosis Interpretation:")
print("> 0: Heavy-tailed, < 0: Light-tailed, ≈ 0: Normal tails")

# 5. CORRELATION ANALYSIS
print("\n" + "="*60)
print("5. CORRELATION ANALYSIS")
print("="*60)

# Pearson correlation
pearson_corr = df[numerical_cols].corr(method='pearson')
print("Pearson Correlation Matrix:")
print(pearson_corr.round(3))

# Spearman correlation (rank-based)
spearman_corr = df[numerical_cols].corr(method='spearman')
print("\nSpearman Correlation Matrix:")
print(spearman_corr.round(3))

# Correlation with target
print("\nCorrelation with Salary:")
salary_corr = df[numerical_cols].corrwith(df['Salary']).sort_values(ascending=False)
print(salary_corr.round(3))

# 6. QUANTILES AND PERCENTILES
print("\n" + "="*60)
print("6. QUANTILES AND PERCENTILES")
print("="*60)

# Specific quantiles
quantiles = [0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99]
quantile_data = pd.DataFrame({q: df['Salary'].quantile(q) for q in quantiles}, index=['Salary'])
print("Salary Quantiles:")
print(quantile_data.round(2))

# Deciles
deciles = df['Salary'].quantile([i/10 for i in range(11)])
print("\nSalary Deciles:")
print(deciles.round(2))

# 7. AGGREGATION AND GROUPBY STATISTICS
print("\n" + "="*60)
print("7. GROUP-WISE STATISTICS")
print("="*60)

# Basic groupby statistics
dept_stats = df.groupby('Department').agg({
    'Salary': ['mean', 'median', 'std', 'min', 'max'],
    'Age': ['mean', 'std'],
    'Test_Score': ['mean', 'count']
})

print("Department-wise Statistics:")
print(dept_stats.round(2))

# Custom aggregation
custom_agg = df.groupby('Department').agg(
    Avg_Salary=('Salary', 'mean'),
    Median_Salary=('Salary', 'median'),
    Salary_Range=('Salary', lambda x: x.max() - x.min()),
    Count_Employees=('Salary', 'count'),
    Skewness=('Salary', lambda x: skew(x.dropna()))
)

print("\nCustom Department Aggregation:")
print(custom_agg.round(2))

# 8. HYPOTHESIS TESTING STATISTICS
print("\n" + "="*60)
print("8. HYPOTHESIS TESTING")
print("="*60)

# T-test between departments
from scipy.stats import ttest_ind

it_salaries = df[df['Department'] == 'IT']['Salary'].dropna()
hr_salaries = df[df['Department'] == 'HR']['Salary'].dropna()

t_stat, p_value = ttest_ind(it_salaries, hr_salaries)
print(f"T-test: IT vs HR Salaries")
print(f"T-statistic: {t_stat:.3f}")
print(f"P-value: {p_value:.3f}")
print(f"Significant at 0.05 level: {p_value < 0.05}")

# ANOVA across all departments
from scipy.stats import f_oneway

groups = [df[df['Department'] == dept]['Salary'].dropna() for dept in df['Department'].unique()]
f_stat, p_value_anova = f_oneway(*groups)
print(f"\nANOVA: Salary across all departments")
print(f"F-statistic: {f_stat:.3f}")
print(f"P-value: {p_value_anova:.3f}")

# 9. ROLLING AND EXPANDING STATISTICS
print("\n" + "="*60)
print("9. TIME-SERIES LIKE STATISTICS")
print("="*60)

# Create a time-series like dataset
ts_data = pd.DataFrame({
    'value': np.cumsum(np.random.normal(0, 1, 100)) + 100
})

# Rolling statistics
ts_data['rolling_mean_7'] = ts_data['value'].rolling(window=7).mean()
ts_data['rolling_std_7'] = ts_data['value'].rolling(window=7).std()
ts_data['rolling_min_7'] = ts_data['value'].rolling(window=7).min()
ts_data['rolling_max_7'] = ts_data['value'].rolling(window=7).max()

# Expanding statistics
ts_data['expanding_mean'] = ts_data['value'].expanding().mean()
ts_data['expanding_std'] = ts_data['value'].expanding().std()

print("Rolling and Expanding Statistics (last 10 rows):")
print(ts_data.tail(10).round(2))

# 10. STATISTICAL TESTS FOR NORMALITY
print("\n" + "="*60)
print("10. NORMALITY TESTS")
print("="*60)

from scipy.stats import shapiro, normaltest

# Normality tests for Salary
salary_data = df['Salary'].dropna()

# Shapiro-Wilk test
shapiro_stat, shapiro_p = shapiro(salary_data)
print(f"Shapiro-Wilk Test for Salary:")
print(f"Statistic: {shapiro_stat:.3f}, P-value: {shapiro_p:.3f}")

# D'Agostino's test
dagostino_stat, dagostino_p = normaltest(salary_data)
print(f"\nD'Agostino's Test for Salary:")
print(f"Statistic: {dagostino_stat:.3f}, P-value: {dagostino_p:.3f}")

# Q-Q plot statistics (conceptual)
print("\nNormality Interpretation:")
if shapiro_p > 0.05:
    print("Salary appears to be normally distributed")
else:
    print("Salary does not appear to be normally distributed")

# 11. CONFIDENCE INTERVALS
print("\n" + "="*60)
print("11. CONFIDENCE INTERVALS")
print("="*60)

from scipy.stats import sem, t

# 95% Confidence Interval for Salary
confidence_level = 0.95
salary_mean = df['Salary'].mean()
salary_sem = sem(df['Salary'].dropna())
degrees_freedom = len(df['Salary'].dropna()) - 1
confidence_interval = t.interval(confidence_level, degrees_freedom, salary_mean, salary_sem)

print(f"95% Confidence Interval for Mean Salary:")
print(f"Mean: {salary_mean:.2f}")
print(f"Interval: ({confidence_interval[0]:.2f}, {confidence_interval[1]:.2f})")
print(f"Margin of Error: ±{salary_sem * t.ppf((1 + confidence_level) / 2, degrees_freedom):.2f}")

# 12. OUTLIER DETECTION STATISTICS
print("\n" + "="*60)
print("12. OUTLIER DETECTION")
print("="*60)

# IQR method for outliers
def detect_outliers_iqr(data):
    Q1 = data.quantile(0.25)
    Q3 = data.quantile(0.75)
    IQR = Q3 - Q1
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR
    outliers = data[(data < lower_bound) | (data > upper_bound)]
    return outliers

print("Outliers detected using IQR method:")
for col in ['Salary', 'Age', 'Test_Score']:
    outliers = detect_outliers_iqr(df[col].dropna())
    print(f"{col}: {len(outliers)} outliers")

# Z-score method
from scipy.stats import zscore

df['Salary_Zscore'] = zscore(df['Salary'].dropna())
salary_outliers_z = df[abs(df['Salary_Zscore']) > 3]
print(f"\nSalary outliers (Z-score > 3): {len(salary_outliers_z)}")

print("\n" + "="*60)
print("STATISTICAL ANALYSIS SUMMARY")
print("="*60)
print(f"Dataset size: {df.shape}")
print(f"Numerical columns analyzed: {len(numerical_cols)}")
print(f"Statistical tests performed: 10+ different types")
print(f"Key insights: Central tendency, dispersion, correlations, hypothesis testing")

Key Statistical Functions

describe() - Summary statistics
corr() - Correlation matrix
cov() - Covariance matrix
quantile() - Quantile values
groupby().agg() - Group statistics

Advanced Analysis

Rolling statistics
Expanding windows
Hypothesis testing
Confidence intervals
Outlier detection

Statistical Best Practices:

Always check for missing values before analysis
Understand the data distribution before choosing tests
Consider the assumptions of statistical tests
Use appropriate correlation methods for data types
Interpret p-values in context of effect size

Integration Tip: Pandas works seamlessly with SciPy and StatsModels for advanced statistical analysis. Use scipy.stats for hypothesis testing and statsmodels for regression analysis.

Common Statistical Patterns

# Descriptive statistics pattern
summary = df.describe(include='all')

# Correlation analysis pattern
correlation_matrix = df.select_dtypes(include=[np.number]).corr()

# Group-wise analysis pattern
group_stats = df.groupby('category').agg({
    'value': ['mean', 'std', 'count'],
    'score': ['min', 'max', 'median']
})

# Hypothesis testing pattern
from scipy.stats import ttest_ind
group1 = df[df['group'] == 'A']['value']
group2 = df[df['group'] == 'B']['value']
t_stat, p_value = ttest_ind(group1, group2)

← Handling Missing Data MultiIndex & Hierarchical Data →

Frontend

JavaScript & Frameworks

Backend

Artificial Intelligence

Database

CSS Frameworks

Data Analytics

Digital Marketing