Real-World Data Science Projects with Pandas

In algorithmic trading, raw market data arrives as unaligned ticks (Timestamp, Price, Volume). Traders must resample high-frequency streams into standard 5-minute OHLCV (Open, High, Low, Close, Volume)candlesticks and compute statistical volatility bands:

import pandas as pd
import numpy as np

# 1. Generate Synthetic 1-Second Trade Ticks
np.random.seed(42)
n_ticks = 100_000
timestamps = pd.date_range('2026-01-01 09:30:00', periods=n_ticks, freq='100ms')
prices = 150.0 + np.cumsum(np.random.normal(0, 0.05, size=n_ticks))
volumes = np.random.randint(10, 500, size=n_ticks)

ticks_df = pd.DataFrame({'price': prices, 'volume': volumes}, index=timestamps)

# 2. Resample into 5-Minute OHLCV Candlesticks
ohlcv = ticks_df.resample('5min').agg({
    'price': ['first', 'max', 'min', 'last'],
    'volume': 'sum'
})
ohlcv.columns = ['open', 'high', 'low', 'close', 'volume']
ohlcv.dropna(inplace=True)

# 3. Calculate 20-Period Rolling VWAP & Bollinger Bands
# Volume Weighted Average Price (VWAP) = Sum(Price * Volume) / Sum(Volume)
ohlcv['pv'] = ohlcv['close'] * ohlcv['volume']
ohlcv['rolling_vwap'] = ohlcv['pv'].rolling(20).sum() / ohlcv['volume'].rolling(20).sum()

# Bollinger Bands: 20-period Moving Average +/- 2 Standard Deviations
ohlcv['sma_20'] = ohlcv['close'].rolling(20).mean()
ohlcv['std_20'] = ohlcv['close'].rolling(20).std()
ohlcv['bollinger_upper'] = ohlcv['sma_20'] + (2 * ohlcv['std_20'])
ohlcv['bollinger_lower'] = ohlcv['sma_20'] - (2 * ohlcv['std_20'])

print("Processed 5-Minute Algorithmic Candlesticks:")
print(ohlcv[['open', 'high', 'low', 'close', 'rolling_vwap', 'bollinger_upper']].tail(5))

Measuring user retention across monthly signup cohorts is critical for evaluating product-market fit. Below is a vector-optimized pipeline that extracts cohort months, computes active period intervals, and produces a percentage retention matrix:

# 1. Sample Transaction Log
transactions = pd.DataFrame({
    'user_id': [101, 102, 101, 103, 102, 101, 104, 103, 102],
    'order_date': pd.to_datetime([
        '2025-01-15', '2025-01-20', '2025-02-10', '2025-02-14',
        '2025-02-18', '2025-03-05', '2025-03-12', '2025-04-01', '2025-04-10'
    ]),
    'amount': [49.99, 99.00, 25.00, 15.00, 80.00, 120.00, 45.00, 30.00, 110.00]
})

# 2. Extract Order Month and User First-Purchase Cohort Month
transactions['order_month'] = transactions['order_date'].dt.to_period('M')
transactions['cohort_month'] = transactions.groupby('user_id')['order_date'].transform('min').dt.to_period('M')

# 3. Calculate Cohort Index (Months Elapsed Since First Purchase)
transactions['cohort_index'] = (
    (transactions['order_month'].dt.year - transactions['cohort_month'].dt.year) * 12 +
    (transactions['order_month'].dt.month - transactions['cohort_month'].dt.month)
)

# 4. Construct Cohort Pivot Table
cohort_data = transactions.groupby(['cohort_month', 'cohort_index'])['user_id'].nunique().reset_index()
cohort_pivot = cohort_data.pivot(index='cohort_month', columns='cohort_index', values='user_id')

# 5. Convert to Percentage Retention Matrix
cohort_sizes = cohort_pivot.iloc[:, 0]
retention_matrix = cohort_pivot.divide(cohort_sizes, axis=0) * 100

print("Customer Retention Percentage Matrix (%):")
print(retention_matrix.round(1).fillna("-"))

In hardware engineering, different sensors (e.g. Engine Temperature vs Vibration Accelerometer) sample at different, unaligned clock frequencies. Traditional SQL joins fail because exact timestamp matches never happen. Pandas solves this with pd.merge_asof(), performing inexact nearest-neighbor time joins:

# Sensor A: Temperature (Sampled every 500ms with jitter)
sensor_temp = pd.DataFrame({
    'timestamp': pd.to_datetime(['2026-03-01 12:00:00.120', '2026-03-01 12:00:00.640', '2026-03-01 12:00:01.110']),
    'engine_temp_c': [78.4, 79.1, 80.2]
})

# Sensor B: Vibration (Sampled every 300ms)
sensor_vib = pd.DataFrame({
    'timestamp': pd.to_datetime(['2026-03-01 12:00:00.050', '2026-03-01 12:00:00.350', '2026-03-01 12:00:00.720', '2026-03-01 12:00:01.020']),
    'vibration_g': [0.12, 0.45, 0.88, 1.20]
})

# Inexact Asynchronous Join: Match temperature with the latest preceding vibration reading!
# tolerance='200ms' prevents joining stale measurements if a sensor dropped out.
aligned_telemetry = pd.merge_asof(
    sensor_temp.sort_values('timestamp'),
    sensor_vib.sort_values('timestamp'),
    on='timestamp',
    direction='backward',
    tolerance=pd.Timedelta('500ms')
)

print("Synchronized IoT Multi-Sensor Telemetry:")
print(aligned_telemetry)