Pandas is the most popular Python data analysis library available today and can read in data directly from a wide variety of sources, including CSVs, Excel Workbooks, JSON files, SQL databases, parquet files, and even from your clipboard. Currently, there is no direct method using pandas to read in data trapped within a PDF file. Thankfully, the tabula-py library is available to read in these tables within a PDF as pandas DataFrames.

The tabula-py library itself is a wrapper around tabula-java, a command line tool for extracting trapped data within a PDF. Get started by installing it with pip install tabula-py.

Sample PDF — Book Sales

For this tutorial, we will extract a single table within the royalty statement from Pandas Cookbook, a book I published in 2017. An image of the PDF is displayed below.

Automatic table detection

Tabula-py automatically detects all tables on the pages provided with its read_pdf function. All we have to do is provide it the file path of the PDF document and the pages to extract. The returned result is a list of DataFrames, stored here as the variable dfs.

import pandas as pd
import tabula
dfs = tabula.read_pdf('2018Q1.pdf', pages=1)
len(dfs)

1

Tabula has detected a single table. Let’s output the DataFrame to see how well the extraction worked.

df = dfs[0]
df

For this PDF, I wanted to extract the table beginning in the middle box with the four columns Format, Copies, Revenue, Royalties. Instead, Tabula extracted the two boxes above this and returned a three-column table combining Copies and Revenue. While the result isn’t bad, it’s not usable in the current form. We could use pandas at this point to clean up the data, but tabula-py provides us parameters to define the boundary of the entire table itself and the positioning of the columns.

The relative_area and relative_columns parameters are set to True which allows us to provide boundaries to both the area and columns parameters as a percentage of the page, as opposed to point coordinates. In my opinion, it is easier to define a boundary using relative page percentage.

The area column is passed a four-item list of top, left, bottom, and right. These represent the boundaries of the table relative to the page. The three-item list for columns represent the vertical lines separating the four columns. Let's re-read in the data and output the resulting DataFrame.

dfs = tabula.read_pdf('2018Q1.pdf', pages=1, 
                      relative_area=True,
                      relative_columns=True,
                      area=[42, 0, 60, 100], 
                      columns= [45, 65, 82])
df = dfs[0]
df

Convert to numeric

This extraction looks much better and almost exactly what we want. Let’s check the data types of each column to understand which ones need more processing.

df.dtypes

Format       object
Copies        int64
Revenue      object
Royalties    object
dtype: object

Conveniently, the Copies column was successfully converted to an integer. The Revenue and Royalties columns were read in as strings. We’ll now have to turn to pandas to convert these columns to floats. We do so by replacing any dollar signs, commas, and spaces with empty spaces using a regular expression.

df[['Revenue', 'Royalties']] = (df[['Revenue', 'Royalties']]
                                  .replace('[$, ]', '', regex=True)
                                  .astype('float64'))
df

Summary

In this tutorial, we passed custom inputs into tabula-py’s read_pdf function to extract a table trapped within a PDF as a pandas DataFrame. We then cleaned up the result with pandas ability to replace characters using a regular expression. Several more options are available to tabula-py to customize the extraction process.

Master Data Analysis with Python

If you enjoyed this lesson and are looking to become an expert with Pandas, then check out my book, Master Data Analysis with Python. It is the most comprehensive Pandas book available, and comes with video lessons, 500+ exercises with solutions, and certification exams.

Convert Trapped Tables within PDFs to Pandas DataFrames was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lots of courses promise to teach you X or turn you into Y within Z weeks. This learning as fast as possible approach may indeed cover lots of material with you producing interesting final products. You will feel a sense of accomplishment that you’ve quickly reached certain milestones towards X/Y.

In all likelihood, you’ve skipped over certain areas of development in order to reach some final stage. You may find it harder to progress from this point because you’ve taken a narrow path without a complete understanding of the fundamentals. When you enter the real-world, projects demand deep knowledge of a wide array of skills.

You many have to go back and cover these fundamentals that were originally ignored. This might lead you to frustration because you’ll be producing far less interesting results than what was covered in the original course.

This is why I highly recommend a slow approach where material is covered thoroughly without knowledge gaps. Skills are built incrementally, one small step at a time. It will be tempting to reach up high to that shiny project that others are sharing on social media, but you may find yourself falling without that firm understanding of the fundamentals.

All of the courses I teach use this slow, thorough approach without knowledge gaps, where skills are built incrementally. Yes, they may be boring at times, but if your goal is to have deep knowledge in a professional setting then I believe this path is effective.

And as a bonus, I believe this path to be faster in the long run as the knowledge you gain will be retained easier and for a longer period of time compared to skipping around to final results.

True for other learning pursuits

I believe this approach is also best for other learning pursuits — whether it is a new language, playing an instrument, a sport, etc… learning slowly, and thoroughly without knowledge gaps is best.

Comprehensive Python Data Science Courses

Immerse yourself in my comprehensive path for mastering data science with Python. Purchase the All Access Pass to get lifetime access to all current and future courses. Some of the courses it contains:

• Master the Fundamentals of Python — A comprehensive introduction to Python (300+ pages, 150+ exercises, 25 hours of video)
• Master Data Analysis with Python — The most comprehensive course available to learn pandas. (800+ pages and 500+ exercises, 10 hours of video)

Get the All Access Pass now!

In this post, you’ll learn how to use the fantastic str.get_dummies Pandas Series method to instantly restructure trapped data within a string. We begin by reading in a small sample dataset containing people's favorite fruits.

import pandas as pd
df = pd.read_csv('data/fruits.csv')
df

Notice that the fruit column has multiple fruit names in each cell separated by a pipe character. While this format compactly represents the data, it isn’t the most suitable for answering basic questions such as:

• What is the number of fruit per person?
• How many people enjoy banana?
• Which people enjoy both oranges and bananas?
• How any fruits are in-common with each person?

Attempt to answer questions in current form

It’s possible to answer these questions in the current format using Pandas, though, we will see how these ways are sub-optimal. Here we find the number of fruit per person by adding one to the count of the pipe characters.

s = df.set_index('name')['fruits']
s.str.count(r'\|') + 1

name
Ana        5
Bill       2
Calvin     2
Dean       2
Elias      3
Felicia    2
George     5
Henry      2
Name: fruits, dtype: int64

The number of people who enjoy banana.

s.str.contains('banana').sum()

3

The people that enjoy both oranges and banana.

s.str.contains('(?=.*orange)(?=.*banana)')

name
Ana         True
Bill       False
Calvin     False
Dean       False
Elias      False
Felicia    False
George      True
Henry       True
Name: fruits, dtype: bool

Finding the number of fruits in-common with both people is particularly difficult and is a clear case for reformatting the data.

Better formatting of the data

All of these questions can be bettered answered if the data is in a different format. The get_dummies string-only method will split all values in a single cell into their own columns creating 0/1 indicator variables. Here, we pass in the pipe character to get_dummies, producing the following DataFrame.

df1 = s.str.get_dummies('|')
df1

We can now answer the same questions as before. Here, we count the number of fruit for each person.

df1.sum(axis=1)

name
Ana        5
Bill       2
Calvin     2
Dean       2
Elias      3
Felicia    2
George     5
Henry      2
dtype: int64

We sum up a single column to count the total number of people who enjoy bananas.

df1['banana'].sum()

3

Here, we use the query method to select each person who likes both oranges and bananas.

df1.query('orange + banana == 2')

Finding the number of fruits in-common with each person is where the largest gain from restructuring comes from. Here, we use the matrix multiplication operator to multiply the DataFrame to itself.

df1 @ df1.T

Master Data Analysis with Python

If you enjoyed this tip and are looking to become an expert with Pandas, then check out my extremely comprehensive book, Master Data Analysis with Python. It is the most comprehensive Pandas book available, comes with 500+ exercises, video tutorials, and certification exams.

Use the Pandas String-Only get_dummies Method to Instantly Restructure your Data was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, you’ll learn the absolute quickest path to create beautiful data visualizations in Python. Specifically, we will be issuing a command from Seaborn, a popular library for creating static two-dimensional plots. We begin by reading in some data from Airbnb listings in Washington D.C.

import pandas as pd
df = pd.read_csv('../../data/airbnb.csv', 
                 usecols=['neighborhood', 'accommodates', 'price'])
df.head()

Create the default box plot

Here, we will use Seaborn to create a box plot of prices by neighborhood splitting by the number of persons the space accommodates.

import seaborn as sns
sns.boxplot(x='neighborhood', y='price', hue='accommodates',  
            data=df);

The set_theme function

While the default plot settings are not bad, they can be improved upon using a single command and that is with the set_theme function in Seaborn. By default, it will use the darkgrid style along with the deep color palette.

sns.set_theme()

After running this command, the same plot will appear substantially different.

sns.boxplot(x='neighborhood', y='price', hue='accommodates', 
            data=df);

Customizing matplotlib run configuration settings

You may call the set_theme function without arguments, but I suggest increasing the DPI (dots per square inch) and reducing the font scale. The default DPI is 100 for matplotlib (the library Seaborn uses for all of its plotting), which is a lower resolution than most modern monitors and is the reason why the default image size (6.4 inches by 4.8 inches) does not match the actual screen inches on your monitor.

Any plot settings (formally, the matplotlib run configuration parameters) are able to changed with the rc parameter. Here, the DPI is increased to 150 and the default figure size decreased.

sns.set_theme(rc={'figure.dpi': 150, 'figure.figsize': (5, 3.75)})
sns.boxplot(x='neighborhood', y='price', hue='accommodates', 
            data=df);

Scaling the font

While the image is sharper and larger, the text is clearly too big. The set_theme function provides a font_scale parameter to decrease the relative size of the text in the image. Here we set it to 0.65.

sns.set_theme(rc={'figure.dpi': 150, 'figure.figsize': (5, 3.75)}, 
              font_scale=0.65)
sns.boxplot(x='neighborhood', y='price', hue='accommodates', 
            data=df);

Choosing other styles and color palettes

There are several built-in styles (white, dark, whitegrid, darkgrid, ticks) as well as color palettes (deep, muted, bright, pastel, dark, colorblind) available as well. Here, we choose the whitegrid style paired with the pastel color palette.

sns.set_theme(style='whitegrid', 
              palette='pastel', 
              rc={'figure.dpi': 150, 'figure.figsize': (5, 3.75)}, 
              font_scale=0.65)
sns.boxplot(x='neighborhood', y='price', hue='accommodates', 
            data=df);

Summary

In summary, use the set_theme function in Seaborn to easily choose a style, color palette, customize the matplotlib run configuration settings and to scale the font.

Master Python, Data Science and Machine Learning

Immerse yourself in my comprehensive path for mastering data science and machine learning with Python. Purchase the All Access Pass to get lifetime access to all current and future courses. Some of the courses it contains:

• Master the Fundamentals of Python — A comprehensive introduction to Python (300+ pages, 150+ exercises, 25 hours of video)
• Master Data Analysis with Python — The most comprehensive course available to learn pandas. (800+ pages and 500+ exercises, 10 hours of video)
• Master Machine Learning with Python — A deep dive into doing machine learning with scikit-learn constantly updated to showcase the latest and greatest tools. (300+ pages)

Get the All Access Pass now!

Use this One Line of Code to Create Beautiful Data Visualizations in Python was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2014, I was first introduced to pandas and had no idea how to use it. By 2017, I had written the 500 page book Pandas Cookbook. This is roughly the path I took to mastering pandas for free:

1. Read the official documentation
2. Practice examples in the documentation
3. Flashcards
4. Share a full data analysis with others
5. Answer old Stack Overflow Questions
6. Answer new Stack Overflow Questions
7. Teach others in-person or online
8. Write pandas blog posts
9. Repeat

Read the official documentation

The official pandas documentation is often one of the best sources when learning any technical skill. The same is true for pandas. Start with user guide that takes you through all of the library.

Practice examples in the documentation

This should be done in tandem with reading. Use a new jupyter notebook for each section of the user guide. Make your own notes and write (not copy) the code. Experiment on your own with each example testing out the different parameter options.

Flashcards

Use a flashcard system such as Anki to help become more familiar with the available attributes and methods. While Stack Overflow is a great resource, you’ll show mastery once you can do an entire data analysis without leaving your notebook.

Share a full data analysis with others

After you’ve played with the toy examples in the docs, find a small real-world dataset. Ask interesting questions and then answer them. Share your work with others to get feedback. Kaggle datasets are great for this.

Answer old Stack Overflow Questions

Stack Overflow has nearly 250k questions tagged as pandas, most of them having solutions. Read the question and answer them on your own. Then compare yours to the other solutions. Answer several hundred. Do not post your solution yet.

Answer new Stack Overflow Questions

Very few people vote on old Stack Overflow questions. Post your solution to new Stack Overflow questions. They will get answered quickly by others, so you may not get upvotes at first. Be patient and keep posting and reviewing other people’s solutions.

Teach others in-person or online

Start mentoring other people who are beginning pandas users. It’s very helpful to see beginners make mistakes and then to correct them.

Write pandas blog posts

Blog posts can really help solidify your knowledge as a whole. Divide up your posts into two categories: Those that focus on a specific pandas topic and those that focus on real world data analysis with pandas.

Repeat

By this point, you should have some pandas expertise, but as any skill in life, you’ll need continual practice to perfect your craft. Each one of these steps will need to be repeated in order to maintain and surpass your current level of expertise. For instance, I’ve lost track of all the times I’ve read through the official documentation. There is an enormous amount of features available to pandas users, and it’s difficult to retain all of it without repeated reference.

Master Data Analysis with Python — A Comprehensive Learning Path

For those that are looking for a more direct path, my book Master Data Analysis with Python is a comprehensive guide with hundreds of exercises, projects, solutions, and video lessons.

How to become an Expert at Pandas for Data Analysis for FREE was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Awesome Pandas Tricks — Advent of Code Problems 1–5

I have a new tutorial where I show interesting, unusual and just plain awesome ways to use the pandas library to solve data problems. Pandas is such a versatile library with so many ways to solve so many different problems.

The Advent of Code is a series of 25 fun problems released every December 1st, each containing some kind of data that require some sort of computing power to solve. While pandas is probably not the goto choice for these challenges, it is absolutely has the ability to solve these problems and often quite impressively. Many of these challenges require advanced and unusual usages of the pandas library, which is why I chose them for this tutorial. Hope you enjoy the tricks.

All of the material is located here on GitHub with the video below covering a detailed presentation of all of the tricks used to solve the problems.

Awesome Pandas Tricks — Advent of Code Problems 1–5 was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve used matplotlib and seaborn to create data visualizations enough, then you’ve probably run into the issue of overlapping text labels on the x-axis.

Let’s take a look at an example that uses Airbnb listings data.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

cols = ['neighborhood', 'accommodates', 'price']
airbnb = pd.read_csv('data/airbnb.csv', usecols=cols)
airbnb.head()

After setting the theme, we create a bar plot of the mean price for the top 10 most frequent neighborhoods split by the number of people the listing accommodates.

sns.set_theme(style='darkgrid', rc={'figure.dpi': 147},              
              font_scale=0.7)
fig, ax = plt.subplots(figsize=(7, 2))
top10 = airbnb['neighborhood'].value_counts().index[:10]
ax.set_title('Mean Price per Neighborhood')
sns.barplot(x='neighborhood', y='price', data=airbnb, ax=ax, 
            order=top10, ci=None, 
            hue='accommodates', hue_order=[1, 2, 3])
ax.legend(bbox_to_anchor=(1, 1), title='accommodates');

Overlapping labels

As you can see, most of the neighborhood names overlap one another making for an ugly graph. One solution is to rotate the labels 90 degrees.

ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
ax.figure

Wrapping the labels

Perhaps a better solution involves wrapping the labels at a given width. The below function uses the textwrap standard library to break strings at spaces at the given width, with the option to break in the middle of words.

import textwrap
def wrap_labels(ax, width, break_long_words=False):
    labels = []
    for label in ax.get_xticklabels():
        text = label.get_text()
        labels.append(textwrap.fill(text, width=width,
                      break_long_words=break_long_words))
    ax.set_xticklabels(labels, rotation=0)

We use the function to wrap the words at or before the 10th character.

wrap_labels(ax, 10)
ax.figure

Master Data Analysis with Python

If you enjoyed this tip, then check out my comprehensive book Master Data Analysis with Python that teaches you how to become an expert at using Python and Pandas to do data analysis.

Automatically Wrap Graph Labels in Matplotlib and Seaborn was originally published in Dunder Data on Medium, where people are continuing the conversation by highlighting and responding to this story.