Data Science & AI: NumPy Vectorization & Pandas

1. Why Standard Python Lists Fail at Data Science

A standard Python list is an array of pointers to scattered PyObject structs across RAM. Every element access incurs pointer-dereference overhead, type-checking, and cache misses.

A NumPy ndarray stores homogeneous primitive numbers in a single, contiguous block of memory. This allows CPU hardware to execute SIMD (Single Instruction, Multiple Data) vector instructions:

import numpy as np
import time

# Benchmark: Standard Python loop vs NumPy vectorization on 10 million floats
size = 10_000_000

# 1. Standard Python Loop
py_list = list(range(size))
start = time.perf_counter()
py_result = [x * 2 for x in py_list]
py_time = time.perf_counter() - start

# 2. NumPy Vectorized Operation
np_arr = np.arange(size)
start = time.perf_counter()
np_result = np_arr * 2  # Vectorized C-level computation!
np_time = time.perf_counter() - start

print(f"Python List Time: {py_time:.4f}s")
print(f"NumPy Vector Time: {np_time:.4f}s")
print(f"NumPy is {py_time / np_time:.1f}x FASTER!")

2. NumPy Broadcasting Rules

Broadcasting allows arithmetic operations between arrays of different shapes without copying data in memory:

# 2D Matrix (3 rows x 2 columns)
matrix = np.array([
    [10.0, 20.0],
    [30.0, 40.0],
    [50.0, 60.0]
])

# 1D Vector (1 row x 2 columns)
adjustment = np.array([1.5, 2.0])

# Broadcasting stretches adjustment automatically across all rows!
adjusted_matrix = matrix + adjustment
print(adjusted_matrix)

3. Tabular Analytics with Pandas DataFrames

Pandas builds on top of NumPy arrays to provide labeled rows and columns (DataFrames), handling missing values, datetime indexing, and SQL-style aggregations:

import pandas as pd

# Creating a DataFrame from raw records
data = {
    "user_id": [101, 102, 103, 104, 105],
    "department": ["Engineering", "Product", "Engineering", "Marketing", "Engineering"],
    "salary": [135000, 115000, 142000, 92000, 158000],
    "experience_years": [4, 3, 5, 2, 7]
}

df = pd.DataFrame(data)

# 1. Boolean Masking & Filtering
senior_engineers = df[(df["department"] == "Engineering") & (df["experience_years"] >= 5)]
print("Senior Engineers:\n", senior_engineers)

# 2. GroupBy Aggregations (like SQL GROUP BY)
dept_metrics = df.groupby("department")["salary"].agg(["mean", "count"]).reset_index()
print("\nAverage Salary by Department:\n", dept_metrics)

# 3. Label-based (.loc) vs Integer-based (.iloc) indexing
print("\nFirst row by integer index:\n", df.iloc[0])