This page includes an interactive code editor. Try modifying and running the examples!

Pandas Data Filtering

Data filtering is one of the most common and essential operations in data analysis with Pandas. It allows you to extract specific subsets of data based on conditions, making it easier to focus on relevant information.

1. Basic Boolean Filtering

The most fundamental way to filter data in Pandas is using boolean conditions.

Syntax: df[condition]
Mechanism: Create a boolean mask (True/False values) and apply it to the DataFrame
Example: df[df['Age'] > 30] returns rows where Age is greater than 30

Operator	Description	Example
>	Greater than	`df['Salary'] > 50000`
<	Less than	`df['Age'] < 40`
==	Equal to	`df['Department'] == 'IT'`
!=	Not equal to	`df['City'] != 'London'`
>=	Greater than or equal	`df['Rating'] >= 4.0`
<=	Less than or equal	`df['Stock'] <= 50`

2. Multiple Conditions Filtering

Combine multiple conditions using logical operators:

& (AND): Both conditions must be True
| (OR): At least one condition must be True
~ (NOT): Invert the condition
Important: Always use parentheses around each condition

Note: Use &, |, ~ instead of and, or, not for element-wise operations.

3. Special Filtering Methods

Pandas provides several convenient methods for common filtering scenarios:

Method	Description	Example
isin()	Check if value is in a list	`df['Dept'].isin(['HR', 'IT'])`
str.contains()	String pattern matching	`df['Name'].str.contains('John')`
between()	Check if value is between range	`df['Age'].between(25, 35)`
isna() / notna()	Filter missing values	`df[df['Salary'].notna()]`
query()	Filter using query string	`df.query('Salary > 50000 and Age < 40')`

Example: Basic Data Filtering

Data Filtering Examples

import pandas as pd
import numpy as np

# Create a sample DataFrame
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'Age': [25, 30, 35, 28, 32],
    'City': ['New York', 'London', 'Paris', 'Tokyo', 'Sydney'],
    'Salary': [50000, 60000, 70000, 55000, 65000],
    'Department': ['HR', 'IT', 'Finance', 'Marketing', 'IT']
}

df = pd.DataFrame(data)
print("Original DataFrame:")
print(df)

# Basic filtering: Select rows where Age > 30
print("\nEmployees older than 30:")
filtered_age = df[df['Age'] > 30]
print(filtered_age)

# Multiple conditions using & (and), | (or)
print("\nIT Department employees with Salary > 55000:")
filtered_multiple = df[(df['Department'] == 'IT') & (df['Salary'] > 55000)]
print(filtered_multiple)

# Using isin() for multiple values
print("\nEmployees in HR or Marketing departments:")
filtered_departments = df[df['Department'].isin(['HR', 'Marketing'])]
print(filtered_departments)

# String filtering with str.contains()
print("\nEmployees with 'a' in their name:")
filtered_names = df[df['Name'].str.contains('a', case=False)]
print(filtered_names)

# Using query() method
print("\nUsing query method - Age between 25 and 32:")
filtered_query = df.query('25 <= Age <= 32')
print(filtered_query)

Advanced Filtering Techniques

Advanced Filtering Examples

import pandas as pd

# Advanced filtering examples
data = {
    'Product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Headphones'],
    'Price': [1200, 25, 75, 300, 150],
    'Category': ['Electronics', 'Accessories', 'Accessories', 'Electronics', 'Audio'],
    'Stock': [15, 100, 50, 20, 30],
    'Rating': [4.5, 4.2, 4.0, 4.7, 4.3]
}

df = pd.DataFrame(data)
print("Products DataFrame:")
print(df)

# Filter with multiple conditions and select specific columns
print("\nHigh-rated Electronics products:")
high_rated_electronics = df[
    (df['Category'] == 'Electronics') & 
    (df['Rating'] > 4.3)
][['Product', 'Price', 'Rating']]
print(high_rated_electronics)

# Using loc for label-based filtering
print("\nUsing loc for complex filtering:")
filtered_loc = df.loc[
    (df['Price'] > 50) & 
    (df['Stock'] < 100) & 
    ~df['Category'].isin(['Audio']),  # ~ means NOT
    ['Product', 'Category', 'Price']
]
print(filtered_loc)

# Filtering with custom functions
print("\nProducts where price is more than 10% of average:")
avg_price = df['Price'].mean()
expensive_products = df[df['Price'] > avg_price * 1.1]
print(expensive_products)

# Handling missing values while filtering
df_with_na = df.copy()
df_with_na.loc[2, 'Rating'] = None  # Introduce a NaN value

print("\nFiltering with NaN handling:")
clean_ratings = df_with_na[df_with_na['Rating'].notna() & (df_with_na['Rating'] > 4.0)]
print(clean_ratings)

Filtering Best Practices

Use parentheses for multiple conditions
Chain filters for complex conditions
Use query() for readable complex filters
Consider performance for large datasets
Always test filters on small subsets first

Common Use Cases

Data cleaning (filtering outliers)
Segment analysis (by category)
Time-based filtering
Quality control (filtering valid records)
Performance analysis (top/bottom performers)

Pro Tip: When working with large datasets, consider using .loc[] for better performance and memory efficiency compared to chained boolean indexing.

← Data Selection Data Manipulation →

Frontend

JavaScript & Frameworks

Backend

Artificial Intelligence

Database

CSS Frameworks

Data Analytics

Digital Marketing

Frontend

Backend

Artificial Intelligence

DevOps

Database

Cyber Security

Pandas Tutorial