Pandas Data Structures: Series, DataFrame & Memory
The efficiency of Pandas stems from its two core data abstractions: the Series (1D labeled vector) and the DataFrame (2D heterogeneous table). In this chapter, explore how data is arranged across contiguous memory buffers, how the Index accelerates lookups with $O(1)$ hash maps, and how to slash memory footprints by up to 90% via type downcasting.
1. The Pandas Series: Labeled 1D Homogeneous Array
A pd.Series consists of two synchronized components: an array of raw values (typically backed by a C-contiguous NumPy array) and an Index containing row labels:
['2026-01-01', '2026-01-02', '2026-01-03']
[142.50, 145.20, 143.80]
float64 (8 bytes/val)
| Property | NumPy 1D Array | Pandas Series |
|---|---|---|
| Access Mechanism | Integer position only (arr[0], arr[4]) | Position (.iloc[0]) AND semantic label (.loc['Mon']) |
| Alignment during Math | Fails if shapes differ; strictly position-based | Automatic alignment based on matching index labels |
| Missing Values | NaN supported only for floats; ints cannot be null | Native nullable types (Int64, boolean, pd.NA) |
2. The Pandas DataFrame: 2D Column-Oriented Matrix
A pd.DataFrame is structurally a dictionary of aligned Series objects where each column is guaranteed to share the exact same row index:
import pandas as pd
# 1. From Dictionary of Lists
df_dict = pd.DataFrame({
'ticker': ['AAPL', 'MSFT', 'GOOGL'],
'price': [225.40, 448.20, 178.60],
'pe_ratio': [32.4, 36.1, 24.8]
})
# 2. From List of Dictionaries (Typical API Payload)
records = [
{'order_id': 1001, 'status': 'completed', 'total': 49.99},
{'order_id': 1002, 'status': 'pending', 'total': 120.00}
]
df_records = pd.DataFrame(records)
# 3. Accessing a Column yields a Series:
print(type(df_dict['ticker'])) # <class 'pandas.core.series.Series'>3. Interactive Execution Sandbox
Modify and run the code below to inspect data structures, custom indexing, and memory allocation:
Data Structures & Memory Profiling
4. Memory Optimization: Slashing RAM Footprint by 90%
By default, Pandas assigns generic 64-bit types (int64, float64) and stores text as expensive Python object pointers. For large datasets, downcasting types saves vast amounts of RAM:
| Default Datatype | Memory Cost | Optimized Downcast | Memory Savings |
|---|---|---|---|
int64 (Whole numbers) | 8 bytes / cell | int8 / int16 / int32 | Up to 87.5% reduction |
float64 (Decimals) | 8 bytes / cell | float32 | 50% reduction |
object (Repeated text like country names) | Large Python heap pointers | category or string[pyarrow] | Up to 95% reduction! |
# Memory Downcasting Demonstration
large_df = pd.DataFrame({
'user_id': np.random.randint(1, 1000, size=500_000), # default int64
'country': np.random.choice(['US', 'UK', 'CA'], size=500_000), # default object
'score': np.random.uniform(0.0, 100.0, size=500_000) # default float64
})
print(f"Original RAM usage: {large_df.memory_usage(deep=True).sum() / 1e6:.2f} MB")
# Downcasting types:
large_df['user_id'] = large_df['user_id'].astype('int16')
large_df['country'] = large_df['country'].astype('category')
large_df['score'] = large_df['score'].astype('float32')
print(f"Optimized RAM usage: {large_df.memory_usage(deep=True).sum() / 1e6:.2f} MB")Explore Related Tools
Hand-picked utilities and calculators related to this tool.
