Pandas: From Data Understanding to Finding Insights

Pandas is one of the most important Python libraries used in Data Science and Data Analysis. It provides powerful and easy-to-use tools for working with structured data such as CSV files, Excel files, databases, and tables.

In real-world data analysis, raw data is rarely ready to use directly. We first need to understand the dataset, identify missing or incorrect values, clean the data, manipulate and transform it, and visualize important patterns. After that, we can create useful features and extract meaningful insights from the data

Pandas provides a wide range of functions and methods that make these tasks easier and more efficient.

In this article, we’ll explore Pandas through a complete data analysis workflow, starting from data understanding and moving through data cleaning, manipulation, transformation, visualization, feature engineering, and finding insights.

Contents

• What is Pandas?
• Why is Pandas Important in Data Science?
• Understanding Pandas Data Structures
• Data Understanding
• Data Cleaning
• Data Manipulation
• Data Transformation
• Data Visualization
• Feature Engineering
• Finding Insights from Data
• Conclusion

What is Pandas?

Pandas is an open-source Python library used for data manipulation and analysis.It provides convenient data structures and functions for working with structured and tabular data.

For example, a dataset containing customer information may look like this:

| Name | Age | City | Salary |

| — — — | — : | — — — — — | — — -: |

| Rahul | 25 | Hyderabad | 35000 |

| Priya | 28 | Chennai | 42000 |

| Anil | 24 | Bangalore | 30000 |

Pandas allows us to easily:

• Analyze relationships and patterns
• Explore the dataset
• Clean missing or incorrect values
• Filter and sort records
• Perform calculations
• Group and summarize data

Why is Pandas Important in Data Science?

Data Science involves working with data at different stages. Pandas is useful because it supports many of the common tasks required during data analysis.

For example, Pandas can help us:

• Load data from CSV, Excel, and other sources
• Understand the structure of a dataset
• Handle missing and duplicate values
• Filter and sort data
• Perform calculations and aggregations.

Therefore, learning Pandas is an important step for anyone starting a career in Data Science, Data Analytics, or Machine Learning.

Understanding Pandas Data Structures

Pandas mainly provides two important data structures:

• Series
• Pandas

Series

A Series is a one-dimensional labeled data structure.It can be compared to a single column in a table.

import pandas as pd
marks = pd.Series([70, 80, 90, 85])
print(marks)

Output:

0    70
1    80
2    90
3    85

Each value has an index, which helps us identify and access individual values.

DataFrame

A DataFrame is a two-dimensional labeled data structure consisting of rows and columns.It is similar to a table in Excel or a database.

data = {
    "Name": ["Rahul", "Priya", "Anil"],
    "Age": [25, 28, 24],
    "Salary": [35000, 42000, 30000]
}
df = pd.DataFrame(data)
print(df)

Output:

    Name  Age  Salary
0  Rahul   25   35000
1  Priya   28   42000
2   Anil   24   30000

A DataFrame is the most commonly used Pandas structure when working with datasets.

Data Understanding

Data understanding is usually the first step after loading a dataset.

Before cleaning or analyzing data, we need to understand:

• What columns are present?
• How many rows are there?
• What type of data does each column contain?
• Are there missing values?
• What are the basic statistics of numerical columns?

Let’s see how Pandas helps us to answer these questions.

Loading Data

Pandas can load data from different file formats.

For example, to read a CSV file:

df = pd.read_csv("customers.csv")
To read an Excel file:
df = pd.read_excel("customers.xlsx")

Once the dats is loaded, it is stored in a DataFrame.

Viewing Data

We can use head() to view the first few rows.

By default, it displays the first five rows.

df.head()

To view the last few rows:

df.tail()

We can also specify the number of rows:

df.head(10)

This helps us quickly understand what the dataset looks like.

Understanding Rows and Columns

The shape attribute tells us the number of rows and columns.

df.shape

Example output:

(1000, 5)

This means the dataset contains:

1000 rows and 5 columns.

We can also use:

df.columns

Data Types

The dtypes attribute shows the data type of each column.

df.dtypes

For example:

Name object
Age int64
Salary int64

Understanding data types is important because different operations are performed on different types of data.

Summary Statistics

The describe() function provides basic statistical information about numerical columns.

df.describe()

It commonly provides:

• Count
• Mean
• Standard Deviation
• Minimum
• Quartiles
• Maximum

This gives us a quick overview of the numerical data.

Data Cleaning

Raw datasets may contain missing values, duplicate records, incorrect values, or incorrect data types.Data cleaning is the process of identifying and fixing these issues.Clean data is important because incorrect or poor-quality data can lead to incorrect analysis and conclusions.

Handling Missing Values

Missing values are common in real-world datasets.

We can check missing values using:

df.isnull().sum()

This shows the number of missing values in each column.

We can remove rows containing missing values using:

df.dropna()

We can also fill missing values using:

df.fillna(0)

For example, missing salary values can be replaced with 0.

We can also fill missing numerical values using the mean:

df["Salary"] = df["Salary"].fillna(df["Salary"].mean())

The appropriate method depends on the dataset and the meaning of the missing values.

Removing Duplicates

Datasets may contain duplicate records.

We can identify duplicates using:

df.duplicated()

To remove duplicate rows:

df.drop_duplicated()

Removing unnecessary duplicates helps maintain data quality.

Handling Incorrect Data

Sometimes a dataset may contain incorrect or inconsistent values.

For example:

df["Gender"] = df["Gender"].str.lower()

Now the values can be represented consistently as:

male

female

Incorrect numerical values can also be identified and handled based on the context of the dataset.

Changing Data Types

Sometimes columns have incorrect data types.

For example, an Age column may be stored as text:

“21”

“25”

“30”

We can convert it into an integer:

df["Age"] = df["Age"].astype(int)

For dates, we can use:

df["Date"] = pd.to_datetime(df["Date"])

Correct data types make analysis and calculations easier.

Data Manipulation

Data manipulation means selecting, filtering, sorting, modifying, and combining data according to our requirements.

Pandas provides many functions for manipulating DataFrames.

Selecting Columns and Rows

To select a single column:

df["Name"]

To select multiple columns:

df[["Name", "Salary"]]

We can use loc to select rows and columns based on labels:

df.loc[0, "Name"]

We can use iloc to select data based on numerical positions:

df.iloc[0, 1]

Filtering Data

Filtering allows us to select only the rows that satisfy a specific condition.

Get D.Pavan Vamsi Krishna’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example, to find employees with salary greater than 40,000:

df[df["Salary"] > 40000]

We can also apply multiple conditions:

df[(df["Age"] > 25) & (df["Salary"] > 40000)]

Filtering is very useful when we want to focus on a specific part of a dataset.

Sorting Data

We can sort a DataFrame using sort_values().

For example, sorting employees by salary:

df.sort_values("Salary")

For descending order:

df.sort_values("Salary", ascending=False)

Sorting helps us identify the highest or lowest values easily.

Adding and Removing Columns

We can create a new column by assigning values to it.

df["Bonus"] = df["Salary"] * 0.10

This creates a new Bonus column.

To remove a column:

df.drop("Bonus", axis=1)

Adding and removing columns is useful when preparing data for analysis or Machine Learning.

GroupBy

groupby() is used to divide data into groups based on one or more columns.

For example, we can group employees by department:

df.groupby("Department")

We can then calculate statistics for each group.

df.groupby("Department")["Salary"].mean()

This gives the average salary for each department.

Aggregation

Aggregation means applying calculations such as:

• Sum
• Mean
• Count
• Minimum
• Maximum

For example:

df["Salary"].mean()

To calculate multiple statistics:

df["Salary"].agg(["mean", "min", "max"])

Aggregation is especially useful with groupby() for generating summaries.

Merging and Joining

Sometimes information is stored in multiple DataFrames.

For example, one DataFrame may contain customer information and another may contain order information.

We can combine them using merge():

result = pd.merge(customers, orders, on="Customer_ID")

Pandas also provides joining operations for combining related datasets.

Merging and joining are important when working with data coming from different tables or sources.

Data Transformation

Data transformation involves converting or modifying data into a more useful format.It helps prepare data for analysis and Machine Learning.

Applying Functions

The apply() function allows us to apply a function to values in a column.

For example:

df["Salary"] = df["Salary"].apply(lambda x: x * 1.10)

This increases each salary by 10%.

apply() is useful when we need to perform custom operations on data.

Mapping Values

map() can be used to replace or transform values based on a mapping.

For example:

df["Gender"] = df["Gender"].map({
"M": "Male",
"F": "Female"
})

This converts short codes into meaningful labels.

String Operations

Pandas provides useful string methods through the .str accessor.

For example:

df["Name"] = df["Name"].str.upper()

This converts names to uppercase.

To remove extra spaces:

df["Name"] = df["Name"].str.strip()

Other useful operations include:

• str.lower()
• str.upper()
• str.contains()
• str.replace()
• str.startswith()

These operations are especially useful when working with text data.

Date and Time Operations

Pandas provides powerful tools for working with dates and times.

First, we can convert a column into datetime format:

df["Date"] = pd.to_datetime(df["Date"])

Then we can extract information such as:

df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month
df["Day"] = df["Date"].dt.day

This allows us to perform time-based analysis easily.

Reshaping Data

Reshaping means changing the structure of a DataFrame to make it easier to analyze.

Common Pandas functions include:

• pivot()
• pivot_table()
• melt()

For example, pivot_table() can be used to summarize data based on different categories.

pd.pivot_table(
df,
values="Sales",
index="Category",
aggfunc="sum"
)

This creates a summarized table showing total sales for each category.

Data Visualization

Visualization helps us understand patterns, trends, distributions, and relationships in data.Pandas can create basic visualizations using its built-in plotting functionality.

We can use:

import matplotlib.pyplot as pltLine Plot

A line plot is useful for showing trends over time.

df.plot(x="Month", y="Sales", kind="line")
plt.show()

For example, it can show how sales change from month to month.

Bar Plot

A bar plot is useful for comparing values across different categories.

df.plot(x="Category", y="Sales", kind="bar")
plt.show()

For example, we can compare sales across different product categories.

Histogram

A histogram shows the distribution of numerical data.

df["Age"].plot(kind="hist")
plt.show()

It helps us understand how values are distributed across different ranges.

Box Plot

A box plot shows the distribution, spread, median, and potential outliers in numerical data.

df["Salary"].plot(kind="box")
plt.show()

Box plots are especially useful for identifying unusual or extreme values.

Feature Engineering

Feature engineering is the process of creating new useful features from existing data.

It is an important step in Data Science and Machine Learning because useful features can help models learn patterns more effectively.

Creating New Features

We can create a new feature using existing columns.

For example, if we have:

Price and Quantity

we can create:

df["Total_Amount"] = df["Price"] * df["Quantity"]

Now Total_Amount becomes a new feature that can provide additional information.

Extracting Information from Existing Features

We can extract useful information from existing columns.

For example, from a date:

df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month

Similarly, we can extract parts of text or numerical information depending on the dataset.

This helps convert existing information into features that are easier to analyze.

Binning

Binning means converting continuous numerical values into different groups or ranges.

For example, we can divide age into groups:

0–18 → Teenager

19–30 → Young Adult

31–50 → Adult

51+ → Senior

Using Pandas:

bins = [0, 18, 30, 50, 100]
labels = ["Teenager", "Young Adult", "Adult", "Senior"]
df["Age_Group"] = pd.cut(
df["Age"],
bins=bins,
labels=labels
)

Binning can make numerical data easier to analyze and interpret.

Finding Insights from Data

The ultimate goal of data analysis is not just to clean and manipulate data, but to find meaningful insights.

After understanding, cleaning, transforming, and analyzing the dataset, we can ask questions such as:

• Which product has the highest sales?
• Which Category generates the most revenue?
• Which location has the highest number of customers?
• What is the average purchase amount?
• Which month has the highest sales?
• Are there any unusual values or outliers?
• Which customer group contributes the most revenue?

For example:

df.groupby("Category")["Sales"].sum().sort_values(ascending=False)

Conclusion

Pandas provides a complete set of tools for working with structured data in Python.

In this article, we explored Pandas through a complete data analysis workflow:

Data Understanding → Data Cleaning → Data Manipulation → Data Transformation → Data Visualization → Feature Engineering → Finding Insights

We learned how to load and understand datasets, handle missing and duplicate values, filter and manipulate data, transform columns, combine datasets, create visualizations, engineer useful features, and finally extract meaningful insights.

Learning Pandas is an important step toward becoming a Data Analyst, Data Scientist, or Machine Learning Engineer, because it provides the practical foundation needed to work with real-world datasets.