BeautifulSoup has long been the go-to library for web scraping and parsing HTML in Python. In fact, I love it and use it 90% of the time for my data pulling from HTML.

However, when it comes to parsing HTML tables, I have come across far too many articles on Medium that demonstrate using BeautifulSoup instead of more efficient and easier alternatives. While working with tables can be a great way to learn BeautifulSoup due to their rigid structure, it’s not the most efficient or easiest method for handling <table> tags. Enter Pandas — a versatile library that not only simplifies data manipulation but also provides a powerful alternative to BeautifulSoup for parsing HTML tables.

In fact, Pandas uses BeautifulSoup under the hood for its HTML parsing capabilities. It’s worth noting that majority of the time when parsing table tags, you are going to put the data into a table or dataframe with Pandas anyway. Additionally, if you are looking to convert the data into other structures such as JSON, Pandas can do that as one-liners too. Let’s dive into a comparison of these two approaches, showcasing why you should consider making the switch from straight up BeautifulSoup to Pandas for parsing HTML tables.

Example with BeautifulSoup: First, let’s take a look at how to extract data from an HTML table using BeautifulSoup. For this example, we will use a simple HTML page containing a <table> tag:

BeautifulSoup

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.basketball-reference.com/leagues/NBA_2023.html'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

table = soup.find('table', {'id':'per_game-team'})

header = [th.text.strip() for th in table.find('thead').find_all('th')]
data = []

for row in table.find_all('tr'):
    rowData = [td.text.strip() for td in row.find_all(['th','td'])]
    if len(rowData) == len(header):
        data.append(rowData)

df = pd.DataFrame(data, columns=header)

The same results can be accomplished much easier with simply:

Pandas

import pandas as pd

url = 'https://www.basketball-reference.com/leagues/NBA_2023.html'
df = pd.read_html(url, attrs={'id':'per_game-team'})[0]

To extract the href attribute from the <a> tags within the table, you can modify the code as follows:

BeautifulSoup

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.basketball-reference.com/leagues/NBA_2023.html'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

table = soup.find('table', {'id': 'per_game-team'})

header = [th.text.strip() for th in table.find('thead').find_all('th')]
header.insert(2, 'href')
data = []

for row in table.find_all('tr'):
    rowData = []
    for cell in row.find_all(['th', 'td']):
        # Check if the cell has an <a> tag
        link = cell.find('a')
        if link:
            href = link['href']
            rowData.extend((cell.text.strip(), href))
        else:
            rowData.append(cell.text.strip())

    if len(rowData) == len(header):
        data.append(rowData)

df = pd.DataFrame(data, columns=header)

Furthermore, with Pandas version ≥ 1.5.0, you can utilize the extract_links:

Pandas

import pandas as pd

url = 'https://www.basketball-reference.com/leagues/NBA_2023.html'
df = pd.read_html(url, attrs={'id':'per_game-team'})[0]
df_links = pd.read_html(url, attrs={'id':'per_game-team'}, extract_links='body')[0]

df_links['href'] = df_links['Team'].str[1]
df = pd.merge(df, df_links['href'], how='left', left_index=True, right_index=True)

Both methods yield the output:

Conclusion

While BeautifulSoup is an amazing library for web scraping and parsing HTML in general, when it comes to parsing HTML tables, Pandas offers a more efficient and streamlined approach. With Pandas, you can quickly convert the parsed data into tables, dataframes, or even other formats like JSON. By leveraging the power of Pandas, you can save time, simplify your code, and focus on more important aspects of your data analysis and machine learning projects.

Thank you for reading this article, and I hope you’ll consider making the switch from BeautifulSoup to Pandas for parsing HTML tables in your future projects.

If you have any questions or need help with your web scraping projects, don’t hesitate to reach out. I’d be more than happy to assist and share my expertise.

Additionally, if you’re interested in discussing sports data or the application of machine learning to sports analytics, I’d love to chat and exchange ideas. Together, we can learn, grow, and create better solutions for data-driven decision-making in sports and beyond.

And just like that, our luck can change…

Sometimes, we run into a string of good luck and iregardless of effort we get a positive outcome.

In reality, these experiences may not be as extraordinary as they seem. Rather, they are our observations of rare events occurring based on our perception and how the events relate to one another.

I came across this book by David Hand called The Improbability Principle. In his book, Hand explores how the rare events we sometimes associate with good or bad luck are actually a result of our perception of these occurrences.

Hand outlines five “laws” that help to explain how these seemingly improbable events are not as unusual as they might appear:

The 5 ‘Laws’ are briefly described as:

1. the law of inevitability — Something will happen…(obviously!)
2. the law of truly large numbers — given enough opportunities, the probabilities of even very unlikely events can mount up to be almost certain
3. the law of selection — You can make things as likely as you want if you choose after the event. Like “connecting” the dots AFTER the event occurred…regardless if they actually are related.
4. the law of the probability lever — Slight changes can make highly improbable events almost certain. For example: Your co-worker who you JUST met at the college campus bookstore happens to be in 2 of the exact same classes that you’re enrolled in!! Out of 30,000 possible students and 100s of different courses to enroll in…what a coincidence!! Crazy? Not so much… it also just so happens you and him/her a) go to the same school, b) are entering in the the same academic school year c) pursuing the same/similar degrees, d) need the same credits/courses, etc, etc, etc. So is it really that crazy you’d be in the same classes?
5. the law of near enough — you consider similar events as identical. If you relax your conditions enough, you can make probabilities as high as you like.

The fifth law is particularly intriguing, as it highlights how our perception of coincidences is entirely within our control. Hand shares an example from his own life, when he bumped into his brother in Paris. While the encounter was surprising, he would have been equally surprised to meet other family members, friends, or coworkers during his trip. Moreover, learning about their overlapping travel plans after the fact, or discovering they had been in France or even Europe, and not necessarily in Paris at the same time, would have also been astonishing.

Another aspect to consider is the idea of relativity. Rare positive or negative events happen to us all the time, but we tend not to notice them when they occur within time frames that are fairly equal in length. This is because there’s no disruption in what we have been accustomed to observing in our daily lives — it feels like “business as usual.”

However, rare events do not occur at pre-determined intervals. Sometimes, two events happen closer together, creating a sense of good fortune or misfortune. When this happens, our awareness of these “rare” events is heightened, leading us to question our perceived luck.

Let’s look at visual representation of a time frame of three ‘rare’ events occurring within a time frame. These events can be:

positive — i.e. a promotion at work, a new baby in the family, a relative/friend getting engaged, your favorite sports team winning a championship, you bump into an ex-boyfriend/girlfriend, etc

negative — i.e. going through a divorce, family member passes away, laid off from work, car accident, you bump into an ex-boyfriend/girlfriend (notice bumping into an ex can be positive or negative depending on the circumstances), etc.

Should these events happen to fall within relatively equal distance/time from each other as in Scenario A, they go unnoticed. At the very least, maybe you do make a note of the rare occurrences, but soon it is forgotten and most likely, these events are happening all the time, you just haven’t made note of it.

But events don’t just occur pre-planned in equal increments. One of those events will happen closer to the other event as seen in Scenario B and/or Scenario C. This leaves a big gap between 2 events, while clumping the other 2 events close together. Thus, creating that artificial belief of “Wow! I can’t believe these things are happening! I just have good fortune / I just can’t catch a break!”

Conclusion

Luck may be more a matter of perception than anything else. By understanding the principles behind seemingly improbable events, we can better appreciate the role that perception plays in shaping our experiences. Ultimately, it may be that we create our own luck, simply by adjusting our perspective on the events that unfold around us.