Sitemap

Pandas Tutorial

8 min readMay 29, 2018

--

Press enter or click to view image in full size

1. Pandas Definition: a Python library used for data manipulation and analysis. It offers data structures and operations for manipulating numerical tables and time series.

2. Importing Pandas:

In [1]: import pandas
In [2]: import pandas as pd

In the first line, we directly call the library by referencing to it as “ pandas. ” For example, creating a series: pandas.Series(data) .

In the second line, we make reference to the library as “pd” instead of “pandas.” For example, creating a series: pd.Series(data). In this tutorial, we will be using this method.

3. Creating Data: using DataFrame and Series

a) DataFrame: is a table that contains an array (recall from the Numpy Tutorial: an array has individual entries each with a certain value located at [row, column]).

In [3]: pd.DataFrame({"Apples": [1], "Bananas": [4]})
Out[3]:

Note: DataFrame values can also be strings:

In [4]: pd.DataFrame({'Bob':['Good', 'Bad'], 'Sue':['Great', 'Ew']})
Out[4]:

Using “ pd.DataFrame ” does not allow us to change the columns labels (also known as Index) of our table. We can assign labels by using the index constructor:

In [5]: pd.DataFrame({'Bob':['Good', 'Bad'], 'Sue':['Great', 'Ew']},   
index = ['Product A', 'Product B'])
Out[5]:

b) Series: is a sequence of data values. Multiple Series ultimately make up a DataFrame. We can label columns in Series the same way we did in DataFrame, using an index constructor:

In [6]: pd.Series([1, 2, 3],index = ['Apples', 'Oranges', 'Grapes'],         
name = 'Fresh Produce Prices')
Out[6]: Apples 1
Oranges 2
Grapes 3
Name: Fresh Produce Prices

4. Reading Files:

a) CSV Files: table of values separated by commas.

“ pd.read_csv ” reads the data into a DataFrame:

In [7]: my_csv = pd.read_csv(" PATH TO FILE ")

The shape command returns how large the DataFrame is:

In [8]: my_csv.shape 
Out[8]: (rows, columns)

The head command returns the first five rows and all columns of our DataFrame:

In [9]: my_csv.head()

Sometimes, Pandas does not pick up on built in column labels within the CSV file. This can be fixed using the “ index_col ” command:

In [10]: my_csv = pd.read_csv(' PATH TO FILE ', index_col = 0)# Set index to return column values at column [0]

b) Excel Files: loading excel files requires an additional parameter: the name of the sheet

In [11]: my_excel = pd.read_xslx(' PATH TO FILE ', sheet_name = 'My Excel File')

5. Writing Files:

a) CSV Files: “ to_csv ” writes data into a CSV file

In [12]: my_csv.to_csv('my_csv')

b) Excel Files: “ to_excel” writes data into an Excel file

In [13]: my_excel.to_excel('My Excel File')

6. Selecting Data:

We can select data from a DataFrame by applying a retrieval filter. For example: we have a DataFrame called “produce” containing products and their price, we would like to see the available products

In [14]: produce.products
Out[14]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Cream

a) Index Selection: we can retrieve data by using the indexing command. We can also retrieve specific values by calling on their positions:

In [15]: produce['products']
Out[15]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Cream
In [16]: produce['products'][0]
Out[16]: 'Apples'

Pandas has its own accessor operators that follows the format “ .iloc [rows, columns] ’’ Note: in iloc, the first element of the range is included while the last one is excluded.

In [17]: produce.iloc[0]      # First row
Out[17]: Products Apples
Price 4.99
In [18]: produce.iloc[:, 0] # First column
Out[18]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Cream

In [19]: produce.iloc[:3, 0] # First three row and first column
Out[19]: 0 Apples
1 Celery
2 Bananas
In [20]: produce.iloc[[0, 1, 2], 0] # First three row and first column
Out[20]: 0 Apples
1 Celery
2 Bananas
In [21]: produce.iloc[-2:, 0] # Last two row and first column
Out[21]: 99 Lemons
100 Ice Cream

b) Label Selection: we can call specific values based on labels using “ .loc [rows, ‘column name’] ” Note: in loc, the first element of the range is included while the last one is also included.

In [22]: produce.loc[0, 'Products']   # First row first product
Out[22]: 'Apples'

The set_index tool allows us to rename the column / index component of our DataFrame. In the example below, we changed the index name from “products” to “items”:

In [23]: produce.set_index['items']

We can also use the rename function and indexes to rename a column or row:

In [24]: produce.rename(columns={'items': 'products'})In [25]: produce.rename(columns={0: 'products', 1: 'products'})
# Replacing index name at columns 0 and 1
In [26]: produce.rename_axis('grocery goods', axis='rows').rename_axis("products", axis='columns')
# Renaming entire rows and columns

c) Conditional Selection: this method selects data based on criteria

We first start off by creating a series of booleans to check which products fit the search:

In [27]: produce.products == 'Apples'
Out[27]: 0 True
1 False
...
99 False
100 False

We can then use this method inside of the “ .loc ” command to return the data we are looking for:

In [28]: produce.loc[produce.products == 'Apples']
Out[28]: 0 Apples 4.99

We can also use “ & ” to search for two criterion:

In [29]: produce.loc[(produce.products == 'Apples')&(produce.price <= 10)] 
Out[29]: 0 Apples 4.99

Another method we can use is “ .isin ” which is a built-in conditional selector that returns data value that is in the DataFrame:

In [30]: produce.loc[produce.products.isin (['Apples', 'Celery'])] 
Out[30]: 0 Apples 4.99
1 Celery 2.99

The last method is using “ notnul ” or “ isnull ” which are also built-in conditional selectors. In the example below, we use this method to filter products without a price tag:

In [31]: produce.loc[produce.price.notnull()] 
Out[31]: 19 Peaches NaN
31 Carrots NaN

7. Assigning Data: this section is about assigning data to a DataFrame

Get Lucy Ji’s stories in your inbox

Join Medium for free to get updates from this writer.

a) Constant Value:

In [32]: produce['price'] = 'expensive'
In [33]: produce['price']
Out[33]: 0 expensive
1 expensive
...
99 expensive
100 expensive

b) Iterable Value:

In [34]: produce['index_backwards'] = range(len(produce), 0, -1)
In [35]: produce['index_backwards']
Out[35]: 0 100
1 99
...
99 1
100 0

8. Summarizing Data:

a) describe: creates a high-level summary of attributes in a specified column. The output changes based on the dtype input, meaning it is type-aware:

In [36]: produce.price.describe()
Out[36]: count 100.000000
mean 4.32320000
...
max 13.9900000

b) unique: returns a list of unique values

In [37]: produce.products.unique()
Out[37]: array(['Apples', 'Celery', ... 'Lemons', 'Ice Cream'])

c) value_counts: returns unique values and how often they occur

In [38]: produce.products.value_counts()
Out[38]: Apples 1
Celery 1
...
Lemons 1
Ice Cream 1

9. Maps: this function creates new representations from existing data or transforming the format of data.

Let’s say we want to change all the values of the list to strings, map takes every value in the column and converts it into a new value using the function provided:

In [39]: my_list = [1, 2, 3]In [40]: for x in my_list:
In [41]: print type(my_list) # shows original value type
Out[41]: <type 'int'> # original value type
<type 'int'>
<type 'int'>
In [42]: my_list = map(str, my_list) # applying function on all valuesIn [43]: for x in my_list
In [44]: print type(my_list) # shows new value
Out[44]: <type 'str'> # new value type
<type 'str'>
<type 'str'>

We can also apply functions across columns, on the entire DataFrame. In the example below, we subtracted the mean price from all prices then applied these new values to our DataFrame:

In [45]: def new_price(y):
In [46]: y.price = y.price - produce.price.mean()
In [47]: return (y)
In [48]: produce.apply(new_price, axis='columns')
Out[48]: 0 Apples -1.99
1 Celery -2.99
...
1 Lemons -4.99
1 Ice Cream 1.99

10. Grouping Data:

a) groupby: groups specific data that was called upon. For example:

In [50]: your_location = pandas.DataFrame({'Name':['Bob', 'Sue',   
'Rob'], 'City':['Seattle', 'Portland', 'Portland']})
In [51]: your_location.groupby(['Name', 'City'])
Out[51]: Name City
Bob Seattle
Sue Portland
Rob Portland

b) agg: allows us to run different functions on our DataFrame simultaneously. For example:

In [52]: your_location.groupby(['Name']).City.agg([len])
Out[52]: Name len
Bob 7
Sue 8
Rob 8

c) Multi-Indexes:

In [53]: your_location.groupby(['Name', 'City']).City.agg([len])
Out[53]: Name City len
Bob Seattle 7
Sue Portland 8
Rob Portland 8

11. Sorting Data:

a) sort_values: returns data in the ascending order. For example:

In [54]: your_location.sort_values(by = ['len'])
# Can also sort by more than one column at a time
Out[54]: Name City len
Bob Seattle 7
Sue Portland 8
Rob Portland 8

We can also order data in descending order:

In [55]: your_location.sort_values(by='len', ascending = False)
Out[55]: Name City len
Rob Portland 8
Sue Portland 8
Bob Seattle 7

12. Data Types:

a) dtypes: grabs the property of a specific column

In [56]: produce.price.dtype
Out[56]: dtype('float64') #using 64-bit floating point number

It also returns the property of every column in the dataset:

In [57]: produce.dtype
Out[57]: products object
price float64

b) astype: converts a column type into another

In [58]: my_int = pandas.DataFrame({'Name':['Runner1', 'Runner2',   
'Runner3'], 'Speed':[1, 2, 3]})
In [59]: my_int.astype('float64')
Out[59]: 0 1.0
1 2.0
2 3.0

13. Missing Data:

a) fillna: can replace missing values. For example, we can replace each “NaN” with “Unknown”:

In [60]: produce.price.fillna('Unknown')
Out[60]: 67 Unknwon
82 Unknown

Conversly, we can also replace a non-null value using the replace operation:

In [61]: produce.price.replace('4.99', '8.99')
Out[61]: 0 8.99

14. Combining Data:

a) concat: allows us to combine data such as CSV files

In [62]: csv_1 = pd.read_csv("PATH TO FILE")
In [63]: csv_2 = pd.read_csv("PATH TO FILE")
In [64]: pd.concat([csv_1, csv_2])

b) join: combines different DataFrame objects with common indexes. Identifying the same trending video in both Canada and the US. lsuffix and rsuffix are necessary parameters the data would have the same column names in Canadian and UK datasets.

In [65]: left = can.set_index(['title', 'trending_date'])
In [66]: right = UK.set_index(['title', 'trending_date'])
In [67]: left.join(right, lsuffix='_CAN', rsuffix='_UK')

15. Method Chaining: performs operations on a DataFrame or Series that emphasizes continuity, essentially ‘chaining’ operations together:

In [68]: (ramen['Stars']
.replace('Unrated', None)
.dropna()
.astype('float64')
.head())
Out[68]: 2580 3.75
2579 1.00
2578 2.25
2577 2.75
2576 3.75
Name: Stars, dtype: float64

a) assign: allows us to either create new columns or modify old ones inside our DataFrame. Say we want to fill null ‘prices’ with the products it correlates to:

In [69]: produce.assign(price = produce.apply (lambda x: x.price if pd.notnull(x.price) else x.products , axis = 'columns'))
# lambda is a function that can be defined within the command
Out[69]: 67 Grapes
82 Lychees

b) pipe: lets us perform operations on the entire DataFrame at once and replaces current DataFrame with pipe output:

In [70]: def name_index(ni)
In [71]: ni.index.name = 'new_name'
In [72]: return ni
In [73]: produce.pipe(name_index)
# Entire index name becomes 'new_name'

Pandas Basics Cheat Sheet: https://assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf

Credits: this tutorial was created based on notes taken while completing lessons and exercises on https://www.kaggle.com/learn/pandas

--

--

Lucy Ji
Lucy Ji

Written by Lucy Ji

Here to promote data literacy