This page includes an interactive code editor. Try modifying and running the examples!

Pandas Data Manipulation

Data manipulation is at the core of data analysis with Pandas. This tutorial covers essential techniques for filtering, sorting, transforming, and aggregating data using Pandas DataFrames.

1. Filtering Data

Filtering allows you to select subsets of data based on conditions.

Boolean Indexing: Use boolean conditions to filter rows
Multiple Conditions: Combine conditions using & (and), | (or), ~ (not)
Query Method: Use df.query() for more complex filtering

Operation	Syntax	Description
Single Condition	`df[df['column'] > value]`	Filter rows where column values meet condition
Multiple Conditions	`df[(cond1) & (cond2)]`	Filter using AND logic
String Contains	`df[df['col'].str.contains('text')]`	Filter rows containing specific text

2. Sorting Data

Sorting helps in organizing data for better analysis and visualization.

Ascending/Descending: Control sort order with ascending parameter
Multiple Columns: Sort by multiple columns with priority order
In-place Sorting: Use inplace=True to modify original DataFrame

3. Adding and Modifying Columns

Create new columns or modify existing ones based on calculations or transformations.

Direct Assignment: df['new_col'] = values
Apply Function: Use apply() for element-wise operations
Vectorized Operations: Perform operations on entire columns

4. Grouping and Aggregation

Group data by categories and compute aggregate statistics.

GroupBy: df.groupby('column')
Aggregate Functions: mean(), sum(), count(), min(), max()
Multiple Aggregations: Use agg() for different functions per column

5. Handling Missing Data

Deal with NaN values in your dataset.

Detection: isna(), notna()
Filling: fillna() with mean, median, or specific value
Dropping: dropna() to remove rows/columns with missing values

6. String Operations

Perform text manipulation on string columns using vectorized string methods.

Case Conversion: str.upper(), str.lower()
Searching: str.contains(), str.startswith()
Extraction: str.extract() with regular expressions

Data Manipulation Workflow

Typical data manipulation follows this pattern:

Load Data → Clean Data → Filter/Sort → Transform → Aggregate → Analyze

Example: Comprehensive Data Manipulation

Data Manipulation Example

import pandas as pd
import numpy as np

# Creating a sample DataFrame
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'Age': [25, 30, 35, 28, 32],
    'City': ['New York', 'London', 'Paris', 'Tokyo', 'Sydney'],
    'Salary': [50000, 60000, 70000, 55000, 65000],
    'Department': ['HR', 'IT', 'Finance', 'Marketing', 'IT']
}
df = pd.DataFrame(data)
print("Original DataFrame:")
print(df)

# 1. FILTERING DATA
print("\n=== FILTERING ===")
# Filter employees older than 30
older_than_30 = df[df['Age'] > 30]
print("Employees older than 30:")
print(older_than_30)

# Filter IT department employees
it_employees = df[df['Department'] == 'IT']
print("\nIT Department Employees:")
print(it_employees)

# Multiple conditions
high_salary_it = df[(df['Department'] == 'IT') & (df['Salary'] > 60000)]
print("\nHigh Salary IT Employees:")
print(high_salary_it)

# 2. SORTING DATA
print("\n=== SORTING ===")
# Sort by Salary (descending)
sorted_by_salary = df.sort_values('Salary', ascending=False)
print("Sorted by Salary (descending):")
print(sorted_by_salary)

# Sort by multiple columns
sorted_multi = df.sort_values(['Department', 'Salary'], ascending=[True, False])
print("\nSorted by Department (asc) and Salary (desc):")
print(sorted_multi)

# 3. ADDING/MODIFYING COLUMNS
print("\n=== COLUMN OPERATIONS ===")
# Add a new column
df['Bonus'] = df['Salary'] * 0.1  # 10% bonus
print("DataFrame with Bonus column:")
print(df)

# Modify existing column
df['Age_Group'] = df['Age'].apply(lambda x: 'Young' if x < 30 else 'Senior')
print("\nDataFrame with Age Group:")
print(df)

# 4. GROUPING AND AGGREGATION
print("\n=== GROUPING AND AGGREGATION ===")
# Group by Department and calculate statistics
dept_stats = df.groupby('Department').agg({
    'Salary': ['mean', 'min', 'max', 'count'],
    'Age': 'mean'
})
print("Department Statistics:")
print(dept_stats)

# 5. HANDLING MISSING DATA
print("\n=== MISSING DATA HANDLING ===")
# Create DataFrame with missing values
df_with_na = df.copy()
df_with_na.loc[2, 'Salary'] = np.nan
df_with_na.loc[0, 'Age'] = np.nan
print("DataFrame with Missing Values:")
print(df_with_na)

# Fill missing values
df_filled = df_with_na.fillna({'Salary': df_with_na['Salary'].mean(), 'Age': df_with_na['Age'].median()})
print("\nDataFrame with Filled Missing Values:")
print(df_filled)

# Drop rows with missing values
df_dropped = df_with_na.dropna()
print("\nDataFrame after Dropping Rows with Missing Values:")
print(df_dropped)

# 6. STRING OPERATIONS
print("\n=== STRING OPERATIONS ===")
# String manipulation on Name column
df['Name_Upper'] = df['Name'].str.upper()
df['Name_Length'] = df['Name'].str.len()
print("DataFrame with String Operations:")
print(df[['Name', 'Name_Upper', 'Name_Length']])

# 7. PIVOT TABLES
print("\n=== PIVOT TABLES ===")
# Create a pivot table
pivot_table = df.pivot_table(values='Salary', index='Department', columns='Age_Group', aggfunc='mean')
print("Pivot Table - Average Salary by Department and Age Group:")
print(pivot_table)

Common Manipulation Methods

filter() - Filter columns
sort_values() - Sort data
groupby() - Group data
pivot_table() - Create pivot tables
merge() - Join DataFrames

Key Concepts

Boolean indexing
Vectorized operations
Method chaining
In-place vs. copy operations
Performance optimization

Tip: Always work with copies of your data when experimenting with manipulations to avoid modifying the original dataset unintentionally.

← Data Filtering GroupBy Operations →

Frontend

JavaScript & Frameworks

Backend

Artificial Intelligence

Database

CSS Frameworks

Data Analytics

Digital Marketing

Frontend

Backend

Artificial Intelligence

DevOps

Database

Cyber Security

Pandas Tutorial