Pandas Tutorial
1. Pandas Definition: a Python library used for data manipulation and analysis. It offers data structures and operations for manipulating numerical tables and time series.
2. Importing Pandas:
In [1]: import pandas
In [2]: import pandas as pdIn the first line, we directly call the library by referencing to it as “ pandas. ” For example, creating a series: pandas.Series(data) .
In the second line, we make reference to the library as “pd” instead of “pandas.” For example, creating a series: pd.Series(data). In this tutorial, we will be using this method.
3. Creating Data: using DataFrame and Series
a) DataFrame: is a table that contains an array (recall from the Numpy Tutorial: an array has individual entries each with a certain value located at [row, column]).
In [3]: pd.DataFrame({"Apples": [1], "Bananas": [4]})
Out[3]:Note: DataFrame values can also be strings:
In [4]: pd.DataFrame({'Bob':['Good', 'Bad'], 'Sue':['Great', 'Ew']})
Out[4]:Using “ pd.DataFrame ” does not allow us to change the columns labels (also known as Index) of our table. We can assign labels by using the index constructor:
In [5]: pd.DataFrame({'Bob':['Good', 'Bad'], 'Sue':['Great', 'Ew']},
index = ['Product A', 'Product B'])
Out[5]:b) Series: is a sequence of data values. Multiple Series ultimately make up a DataFrame. We can label columns in Series the same way we did in DataFrame, using an index constructor:
In [6]: pd.Series([1, 2, 3],index = ['Apples', 'Oranges', 'Grapes'],
name = 'Fresh Produce Prices')
Out[6]: Apples 1
Oranges 2
Grapes 3
Name: Fresh Produce Prices4. Reading Files:
a) CSV Files: table of values separated by commas.
“ pd.read_csv ” reads the data into a DataFrame:
In [7]: my_csv = pd.read_csv(" PATH TO FILE ")The shape command returns how large the DataFrame is:
In [8]: my_csv.shape
Out[8]: (rows, columns)The head command returns the first five rows and all columns of our DataFrame:
In [9]: my_csv.head()Sometimes, Pandas does not pick up on built in column labels within the CSV file. This can be fixed using the “ index_col ” command:
In [10]: my_csv = pd.read_csv(' PATH TO FILE ', index_col = 0)# Set index to return column values at column [0]
b) Excel Files: loading excel files requires an additional parameter: the name of the sheet
In [11]: my_excel = pd.read_xslx(' PATH TO FILE ', sheet_name = 'My Excel File')5. Writing Files:
a) CSV Files: “ to_csv ” writes data into a CSV file
In [12]: my_csv.to_csv('my_csv')b) Excel Files: “ to_excel” writes data into an Excel file
In [13]: my_excel.to_excel('My Excel File')6. Selecting Data:
We can select data from a DataFrame by applying a retrieval filter. For example: we have a DataFrame called “produce” containing products and their price, we would like to see the available products
In [14]: produce.products
Out[14]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Creama) Index Selection: we can retrieve data by using the indexing command. We can also retrieve specific values by calling on their positions:
In [15]: produce['products']
Out[15]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Cream
In [16]: produce['products'][0]
Out[16]: 'Apples'
Pandas has its own accessor operators that follows the format “ .iloc [rows, columns] ’’ Note: in iloc, the first element of the range is included while the last one is excluded.
In [17]: produce.iloc[0] # First row
Out[17]: Products Apples
Price 4.99
In [18]: produce.iloc[:, 0] # First column
Out[18]: 0 Apples
1 Celery
...
99 Lemons
100 Ice Cream
In [19]: produce.iloc[:3, 0] # First three row and first column
Out[19]: 0 Apples
1 Celery
2 Bananas
In [20]: produce.iloc[[0, 1, 2], 0] # First three row and first column
Out[20]: 0 Apples
1 Celery
2 Bananas
In [21]: produce.iloc[-2:, 0] # Last two row and first column
Out[21]: 99 Lemons
100 Ice Cream
b) Label Selection: we can call specific values based on labels using “ .loc [rows, ‘column name’] ” Note: in loc, the first element of the range is included while the last one is also included.
In [22]: produce.loc[0, 'Products'] # First row first product
Out[22]: 'Apples'The set_index tool allows us to rename the column / index component of our DataFrame. In the example below, we changed the index name from “products” to “items”:
In [23]: produce.set_index['items']We can also use the rename function and indexes to rename a column or row:
In [24]: produce.rename(columns={'items': 'products'})In [25]: produce.rename(columns={0: 'products', 1: 'products'})
# Replacing index name at columns 0 and 1In [26]: produce.rename_axis('grocery goods', axis='rows').rename_axis("products", axis='columns')
# Renaming entire rows and columns
c) Conditional Selection: this method selects data based on criteria
We first start off by creating a series of booleans to check which products fit the search:
In [27]: produce.products == 'Apples'
Out[27]: 0 True
1 False
...
99 False
100 FalseWe can then use this method inside of the “ .loc ” command to return the data we are looking for:
In [28]: produce.loc[produce.products == 'Apples']
Out[28]: 0 Apples 4.99We can also use “ & ” to search for two criterion:
In [29]: produce.loc[(produce.products == 'Apples')&(produce.price <= 10)]
Out[29]: 0 Apples 4.99Another method we can use is “ .isin ” which is a built-in conditional selector that returns data value that is in the DataFrame:
In [30]: produce.loc[produce.products.isin (['Apples', 'Celery'])]
Out[30]: 0 Apples 4.99
1 Celery 2.99The last method is using “ notnul ” or “ isnull ” which are also built-in conditional selectors. In the example below, we use this method to filter products without a price tag:
In [31]: produce.loc[produce.price.notnull()]
Out[31]: 19 Peaches NaN
31 Carrots NaN7. Assigning Data: this section is about assigning data to a DataFrame
Get Lucy Ji’s stories in your inbox
Join Medium for free to get updates from this writer.
a) Constant Value:
In [32]: produce['price'] = 'expensive'
In [33]: produce['price']
Out[33]: 0 expensive
1 expensive
...
99 expensive
100 expensive b) Iterable Value:
In [34]: produce['index_backwards'] = range(len(produce), 0, -1)
In [35]: produce['index_backwards']
Out[35]: 0 100
1 99
...
99 1
100 08. Summarizing Data:
a) describe: creates a high-level summary of attributes in a specified column. The output changes based on the dtype input, meaning it is type-aware:
In [36]: produce.price.describe()
Out[36]: count 100.000000
mean 4.32320000
...
max 13.9900000b) unique: returns a list of unique values
In [37]: produce.products.unique()
Out[37]: array(['Apples', 'Celery', ... 'Lemons', 'Ice Cream'])c) value_counts: returns unique values and how often they occur
In [38]: produce.products.value_counts()
Out[38]: Apples 1
Celery 1
...
Lemons 1
Ice Cream 19. Maps: this function creates new representations from existing data or transforming the format of data.
Let’s say we want to change all the values of the list to strings, map takes every value in the column and converts it into a new value using the function provided:
In [39]: my_list = [1, 2, 3]In [40]: for x in my_list:
In [41]: print type(my_list) # shows original value type Out[41]: <type 'int'> # original value type
<type 'int'>
<type 'int'>In [42]: my_list = map(str, my_list) # applying function on all valuesIn [43]: for x in my_list
In [44]: print type(my_list) # shows new value Out[44]: <type 'str'> # new value type
<type 'str'>
<type 'str'>
We can also apply functions across columns, on the entire DataFrame. In the example below, we subtracted the mean price from all prices then applied these new values to our DataFrame:
In [45]: def new_price(y):
In [46]: y.price = y.price - produce.price.mean()
In [47]: return (y)
In [48]: produce.apply(new_price, axis='columns')
Out[48]: 0 Apples -1.99
1 Celery -2.99
...
1 Lemons -4.99
1 Ice Cream 1.9910. Grouping Data:
a) groupby: groups specific data that was called upon. For example:
In [50]: your_location = pandas.DataFrame({'Name':['Bob', 'Sue',
'Rob'], 'City':['Seattle', 'Portland', 'Portland']})In [51]: your_location.groupby(['Name', 'City'])
Out[51]: Name City
Bob Seattle
Sue Portland
Rob Portland
b) agg: allows us to run different functions on our DataFrame simultaneously. For example:
In [52]: your_location.groupby(['Name']).City.agg([len])
Out[52]: Name len
Bob 7
Sue 8
Rob 8c) Multi-Indexes:
In [53]: your_location.groupby(['Name', 'City']).City.agg([len])
Out[53]: Name City len
Bob Seattle 7
Sue Portland 8
Rob Portland 811. Sorting Data:
a) sort_values: returns data in the ascending order. For example:
In [54]: your_location.sort_values(by = ['len'])
# Can also sort by more than one column at a time
Out[54]: Name City len
Bob Seattle 7
Sue Portland 8
Rob Portland 8We can also order data in descending order:
In [55]: your_location.sort_values(by='len', ascending = False)
Out[55]: Name City len
Rob Portland 8
Sue Portland 8
Bob Seattle 712. Data Types:
a) dtypes: grabs the property of a specific column
In [56]: produce.price.dtype
Out[56]: dtype('float64') #using 64-bit floating point numberIt also returns the property of every column in the dataset:
In [57]: produce.dtype
Out[57]: products object
price float64b) astype: converts a column type into another
In [58]: my_int = pandas.DataFrame({'Name':['Runner1', 'Runner2',
'Runner3'], 'Speed':[1, 2, 3]})In [59]: my_int.astype('float64')
Out[59]: 0 1.0
1 2.0
2 3.0
13. Missing Data:
a) fillna: can replace missing values. For example, we can replace each “NaN” with “Unknown”:
In [60]: produce.price.fillna('Unknown')
Out[60]: 67 Unknwon
82 UnknownConversly, we can also replace a non-null value using the replace operation:
In [61]: produce.price.replace('4.99', '8.99')
Out[61]: 0 8.9914. Combining Data:
a) concat: allows us to combine data such as CSV files
In [62]: csv_1 = pd.read_csv("PATH TO FILE")
In [63]: csv_2 = pd.read_csv("PATH TO FILE")
In [64]: pd.concat([csv_1, csv_2])b) join: combines different DataFrame objects with common indexes. Identifying the same trending video in both Canada and the US. lsuffix and rsuffix are necessary parameters the data would have the same column names in Canadian and UK datasets.
In [65]: left = can.set_index(['title', 'trending_date'])
In [66]: right = UK.set_index(['title', 'trending_date'])
In [67]: left.join(right, lsuffix='_CAN', rsuffix='_UK')15. Method Chaining: performs operations on a DataFrame or Series that emphasizes continuity, essentially ‘chaining’ operations together:
In [68]: (ramen['Stars']
.replace('Unrated', None)
.dropna()
.astype('float64')
.head())
Out[68]: 2580 3.75
2579 1.00
2578 2.25
2577 2.75
2576 3.75
Name: Stars, dtype: float64 a) assign: allows us to either create new columns or modify old ones inside our DataFrame. Say we want to fill null ‘prices’ with the products it correlates to:
In [69]: produce.assign(price = produce.apply (lambda x: x.price if pd.notnull(x.price) else x.products , axis = 'columns'))
# lambda is a function that can be defined within the commandOut[69]: 67 Grapes
82 Lychees
b) pipe: lets us perform operations on the entire DataFrame at once and replaces current DataFrame with pipe output:
In [70]: def name_index(ni)
In [71]: ni.index.name = 'new_name'
In [72]: return ni
In [73]: produce.pipe(name_index)
# Entire index name becomes 'new_name' Pandas Basics Cheat Sheet: https://assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf
Credits: this tutorial was created based on notes taken while completing lessons and exercises on https://www.kaggle.com/learn/pandas





