Introduction

In an interview, I was once asked “What is the difference between using join and merge in pandas?” and honestly, I was a bit stumped.

Typically when combining dataframes in pandas, I would have a quick google and head to the documentation to check the syntax — I always got mixed up which to use and whether the function is applied to the left dataframe or if the left dataframe is the first parameter.

These days, merge is my go-to over join and in this article I will explain to you why.

Pandas merge vs join

In pandas, both merge and join are used to combine objects. The main difference between them is that a merge offers more flexibility where you can use column(s) or index(es) to perform a join, whilst join is limited to using index(es) in at least one of the objects.

Let’s explore each in more detail.

Pandas merge

Of the two, merge is the most similar to a SQL JOIN, however it does come with some differences.

When to use merge

• You are joining two dataframes or series objects
• The join is based on column(s) or index(es)
• The columns or indexes can have the same, or different names in the two objects

When not to use merge*

• Unlike in SQL, if you have null values in your join columns, these are matched during the merge which could results in some strange output in your data

Example: left join two dataframes
As an example, you have two dataframes, customers and orders:

customers = pd.DataFrame(
    {
        "id": [1, 2, 3, 4, 5],
        "location": ["London", "New York", "Paris", "London", "Madrid"],
    }
)
orders = pd.DataFrame(
    {
        "id": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
        "customer_id": [1, 2, 3, 4, 1, 1, 2, 3, 3, 3, 1],
        "amount": [100, 150, 100, 200, 150, 300, 600, 250, 250, 400, 500],
        "date": [
            "2024-01-01",
            "2024-01-10",
            "2024-01-10",
            "2024-01-11",
            "2024-01-12",
            "2024-01-12",
            "2024-01-15",
            "2024-01-16",
            "2024-01-16",
            "2024-01-17",
            "2024-01-18",
        ],
    }
)

Let’s say you want to join the two to find the location that has the most orders. Both dataframes have reference to customer ids — this is the column to join on.

If you are interested in all customer locations, even if they have 0 orders, you can use a left join. If you only want the locations with orders then use an inner join. Let’s go for a left join here.

Since the columns have different names in each dataframe, we need to use left_on and right_on. If they were both named customer_id, we could use on with the shared column name specified.

pd.merge(customers, orders, how="left", left_on="id", right_on="customer_id")

The output of this merge is a dataframe with each row containing the customer data and an order from that customer. This is then ready for any aggregation. Since the join columns are named differently, both are retained in the output dataframe.

You might notice that there are also id columns in both dataframes and a suffix of x and y is added to each after the merge. The suffix is ('_x', '_y')by default and this can be changed by specifying your preferred suffix as a parameter.

💡 I find for exploratory data analysis (EDA) work, the indicator flag comes in very handy! It creates a new column _merge to indicate whether the row is left-only, right-only or both. indicator is set to False by default so needs setting to True when calling merge if you want this extra detail in your joined object.

Example: join using columns with null values

✨ Want to wow your interviewers? This example covers the edge case of combining dataframes when null values exist in your join column.

In this example you have two dataframes: users and activity. The user data contains user ids and their registration date, and the activity data includes the user id and number of posts. In both tables there are some cases where the user id has not been captured, resulting in null values.

users = pd.DataFrame(
    {
        "user_id": [np.nan, 1, 2, 3, 4, np.nan],
        "registration_date": [
            "2024-01-01",
            "2024-01-01",
            "2024-01-02",
            "2024-01-02",
            np.nan,
            "2024-01-03",
        ],
    }
)
activity = pd.DataFrame(
    {"user_id": [1, np.nan, 2, 3, 4, np.nan], "posts": [2, 3, 3, 5, 12, 7]}
)

When joining these datasets using merge, both of the empty user ids from users get linked to both of the rows with empty user data in the activity.

users.merge(activity, on=["user_id"], how="left")

⚠️ *You can still use merge in these cases however do so with caution. You may want to filter out the null values or do some additional checks to avoid any unexpected results!

One thing to note here, if you are joining two dataframes that have the same name for the join column, this column only appears once in the output. If there are any other columns with the same name that aren’t used in the join, then they will have a suffix appended.

Advanced Usage

When to use pandas.merge vs. pandas.DataFrame.merge?
Confusingly, both df1.merge(df2) and pd.merge(df1, df2) can be used to perform a merge — so which to use?

Well, this is a matter of preference. pandas.DataFrame.merge, is calling pandas.merge so there is no functional difference between them. Check out the source code for details.

If you are performing multiple merges, then the chaining method df1.merge(df2).merge(df3) is a bit more readable than the wrapping style function pd.merge(pd.merge(df1, df2), df3) as described on StackOverFlow, but ultimately, they are doing the same thing.

Pandas join

Pandas join implements the chaining method using df1.join(df2). Under the hood it actually calls merge (with some additional logic) if you take a look at the source code.

When to use join

• Joining two dataframes based on index(es) or column(s) and index(es)

When not to use join

• When joining to a series that does not have a name
   — this can be solved by naming your series object
• When you want to join on columns, not indexes
• When indexes or MultiIndex levels do not match up between dataframes
• When using large dataframes, join can be less efficient than using merge

Example: join using an index and a column

In this example we have two dataframes: user data and a lookup for location codes.

users = pd.DataFrame(
    {
        "user_id": [1, 2, 3, 4, 5, 6],
        "user_name": ["adder", "cobra", "viper", "mamba", "corn", "grass"],
        "loc_code": ["AU", "GB", "IN", "US", "IN", "AU"],
    }
)
location = pd.DataFrame(
    {
        "code": ["US", "AU", "GB", "IN"],
        "location": ["United States", "Australia", "Great Britain", "India"],
    }
) 

When creating a dataframe in this way, the index is a row label that is a sequence of integers starting at 0 for the first row. However, it can also be set as a column value.

If we are trying to get the location for each user based on the location code, this is the join key and we must set this to be the index for the second dataframe, in this case location. A requirement for using join is having the index of the second dataframe as a join key.

location = location.set_index("code")

We can then join the dataframes then using:

users.join(location, on="loc_code")

The column specified is the column name of the first dataframe and the join type is using the default which is a left join (note: this is different to merge where inner join is the default).

Example: join using indexes

Another way to do the same join is to also set the index of the first dataframe and repeat the join without specifying the join column:

users = users.set_index("loc_code")
users.join(location)

This will then use an index-on-index join if no column is specified. It carries over the index of the location code to the output dataframe, while when only the location dataframe index was set, the index of the user dataframe is used in the output.

It is also possible to join on multiple columns or indexes by using a MultiIndex. For example: df.set_index(["col1", "col2"]).

Summary

Merge and join can be used in pandas to join two data objects. If you are joining based on the index(es), join is a great place to start. If you are joining based on columns, or dealing with large dataframes, merge is the more suitable and efficient option.

💬 I typically perform column based joins, often with multiple columns and it’s rare that the names across dataframes align. This is why merge is my go-to.

If I were to do more index based joins, merge does support this but using df1.merge(df2, left_index=True, right_index=True) vs. df1.join(df2), I know which I’d prefer to write…

So if you ever get asked in an interview what the difference is between the two, hopefully after reading this you can give a much better answer than the waffling I did when I was asked!

💜 One final tip before you go! I don’t know about you, but I love the pandas documentation. Maybe it’s purely the amount of time I’ve spent reading it over the years but I find it so intuitive and well-formatted. The explanations of parameters are clear and further down the page provides examples which often cover my use case or are generic enough that I can easily apply it to my work. I’d highly recommend starting with the docs over StackOverFlow for your pandas syntax queries! Check out merge and join to see what I mean!

Joining data in pandas: merge vs. join was originally published in Data Engineer Things on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s been a while since I’ve shared an article, so hello again!

Disclaimer: This article was written in August 2021, the principles still hold however some of the sites and features mentioned may have changed!

I’m excited to write more articles in 2024 and share my journey as I move from Data Science to Data Engineering!

Applying for jobs can be a time-consuming process, in this article, I’ve pulled together some things that I’ve learned during my recent job application process.

I was interviewing for several roles during this time, each had several stages of interviews — the interview structure varied for each role but they all started with an informal chat with the internal recruiter to find out more about the role, the company and whether it would be a suitable fit.

Following from there, I experienced a combination of — interviews to talk through my experience, take-home tests, presentation of take-home test findings, technical interviews (SQL, quantitative analysis, statistics, data structure, algorithms), case study questions, and behavioural interviews.

Personally, I am not a fan of take-home tests, especially when applying for several jobs. I do enjoy a data challenge however when you are exhausted from a pandemic, your day job and also preparing for other stages of other job applications, I would much prefer to be assessed in an interview setting!

Here are my top takeaways from my recent job hunting!

1. Find a CV buddy

Updating my CV can sometimes delay me when applying for jobs as it’s such a daunting task, I tend to put it off. On top of that, I often find myself getting stuck when trying to recall projects I’ve worked on and highlighting relevant details for the role I’m applying for — I’m too much of a perfectionist.

This time, however, I found myself a CV buddy — a friend who was also looking to update their CV. Over several video calls, we took turns going through our CVs, providing feedback to each other and the process of talking out loud helped to clarify important details. It certainly helped that they were also looking for Data roles so knew the types of skills and experience that would be applicable, but the most valuable part of teaming up was when explaining what we did in each role and having the other person summarise and pull out the relevant points! By discussing my work and being prompted by questions, I was reminded of substantially more details and was able to reword sections more succinctly. It was an iterative process over a few weeks where we would take our notes away, work to improve and come back together to further refine and we both came away with CVs that we were proud of.

2. Start with shared resources

In a few of the application processes, the recruiter provided links to resources that would be helpful when preparing for the interview. I briefly looked through these and spent time on the ones I considered most relevant, however in one case, I left a 1.5 hour YouTube video until later in the preparation as it felt overwhelming how long it was alongside many other resources that had been sent and I had collected myself. When I eventually got around to watching it, the content was incredibly useful and I regretted not watching it sooner!

3. Research the company and job interview process

It is also worth checking out Glassdoor as many people share their interview experience and the types of questions that they were asked in the interview. I made note of many of these questions to make sure that I was able to answer them, and also these came in handy when practising answering questions out loud. There are also many articles on Medium where people have shared their preparation for Data Science interviews, or with specific companies. My list of resources that I wanted to work through was way longer than the time I had to prepare so required some prioritisation. As mentioned above, in hindsight I should have started with the resources shared by the recruiter but I spent more time on the more technical-focused articles and tutorials to ensure I was up to scratch with those skills which, due to the level of detail and understanding required, was very time intensive!

4. Break it down

If, like me, you are preparing for several interviews in parallel, it can be useful to break down the interviews by topic, time required and memory retention. For example, if you have two interviews, both of which will be assessing your coding skills and a case study and one of them also has a behavioural interview, then it may be worth leaving the behavioural practice until last since you may wish to prepare some situations as examples that may come up. I find these types of questions are best to practice a day or two before the interview so that they are fresh in your memory.

For the coding, this is likely to be building on skills you already have and can be broken down to practising a few questions or running through a section of a tutorial each day leading up to the interview. If there is an area that you feel particularly weak in then it is useful to spend more time on this early on and you can then continue to practice questions on this topic to ensure you are confident and able to apply it — again speaking from experience here, I have little experience using SQL window functions and dedicated a fair amount of time to learn and practice these initially so that I felt confident in my first interview however before subsequent SQL interviews I didn’t spend much time practising these and panicked when under pressure. Before a final SQL interview, I knew that this had tripped me up previously so again spent some time practicing but by then I had lost my confidence and having only a day to prepare I didn’t feel as ready as I did in my initial interview!

The case study, for me, was quite a new concept in an interview so required some reading to understand what may be asked and how best to prepare. Personally, speaking through a scenario out loud with someone where they asked follow-up questions was the most helpful method.

5. Find out more

Throughout the recruitment process, I was able to talk to people within the company to ask questions about working for the company and was even offered a chat with someone about interview preparation. I’m not sure if this is usual as I’ve never come across it before but running through a practice interview question and being told what the interviewers would be looking for was incredibly helpful! After having several interviews though, each one was good preparation for the next since the types of questions and interview techniques were similar.

Another good tip would be to use your network, whether getting in touch with a connection who works for the company already or you could ask people you know who have recently got new jobs about their experience with recent interviews — there will be similarities in many of the interview processes within different companies for Data roles so it’s worth finding out more about what it was like for them and any tips they may have.

6. Interview practice questions

Here are a few resources I found useful when preparing for different types of interviews:

Statistics

Brilliant has a wide range of statistics and maths courses, I really enjoyed the style of the questions however some courses do have overlapping content so it is worth being selective about which sections you complete so as not to waste time. I also signed up for the 7-day free trial which was useful for accessing more content but did require a commitment to focusing on stats for those 7 days.

SQL

Mode has an interactive tutorial with access to a SQL database that you can use to practice queries.

Case Studies

Overlapping with SQL, Mode has a few case study examples that are worth working through and also includes potential solutions to compare your answer to.

General Questions

Interview query the blogs contained some useful example SQL, stats and case study questions to run through. I did also try to complete some of the questions on the site however you are limited to running 1 SQL query and viewing only a few solutions, even when signing up for an account so in my opinion the blogs would be more worthwhile.

Medium is full of interview preparation articles for Data Science jobs! Many articles with example questions and topics may come up, I found the articles useful to practice questions however they could often contain too much detail for example, one article contained a long list of Data Science and ML concepts and links to resources, which can be incredibly useful for your understanding but it is unlikely that you will need to learn all of these concepts for an interview.

Be realistic with the preparation time, the foundation of knowledge you already have and what you want from your next role!

7. Practice

As when writing your CV, it can be helpful to work together when preparing for job interviews. If your buddy is applying for a similar type of role, why not practice some of the technical questions together — perhaps choose a topic and ask each other interview-style questions or run through a case study and each prepare your answers — you’d be surprised how much you could learn from each other!

If your buddy is not in a similar industry, it can still be helpful to work through behavioural questions together or find a list of relevant interview questions and get someone to ask you them so that you can practice working through them out loud, communicating your answers (for example, using the STAR method)!

It is also great to have your job-hunting buddies for support when you have interviews, sharing how it went and cheering each other on! I have loved hearing about my friends smashing interviews and finding new roles where they can learn and grow!

8. Growth mindset

Regardless of the outcome of the interview, the time that you have spent researching and learning is helping you to grow! You may not find a suitable role this time but taking the time to reflect on your interview experience, what you have learnt, and requesting feedback from the interviewers will ultimately help you on the path to landing your next role!

Reflections

Whilst interviewing was an intense and busy process, I enjoyed dedicating time to brushing up on concepts that I haven’t used in a while and feel like I am a better Data Scientist because of it. I am grateful that I was offered and have accepted a new role and throughout the applications, I had a clear idea of my goals and expectations for my next role. During this time, I have also been able to reflect on what I have learnt from my current role and habits or projects I’d like to build going forward!

Have you recently been job hunting? What are your top takeaways?

If you enjoyed this article, you can subscribe to my newsletter to get my latest articles, resources I use and other tips directly in your inbox!

8 takeaways from Data Science interviews was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pandas is a python tool used extensively for data analysis and manipulation. Recently I’ve been using pandas with large DataFrames (>50M rows) and through the PyDataUK May Talks and exploring StackOverflow threads have discovered several tips that have been incredibly useful in optimising my analysis.

This tutorial is part 1 of a series and aims to give an introduction to pandas and some of the useful features it offers while exploring the Palmer Penguin dataset.

In this article, we will go through:

• How to install and import pandas
• Data structures in pandas
• How to input and output data
• Inspecting the data
• Getting started with Data Cleaning

Introducing Palmer Penguins

The iris dataset is one that is commonly used for visualisation and pattern recognition in Data Science. Since recently discovering the author’s link with eugenics, there has been an effort to source other datasets for use.
That’s where the penguins come in, the Palmer Penguins dataset has been collected and published by Dr. Kristen Gorman and Allison Horst under the CC-0 licence as an alternative dataset to explore!

It has been published with examples for use in R, but we will be using python package pandas to explore the raw dataset that can be found on GitHub (note the raw CSV file is no longer included in the main branch as it was when I first started looking at the dataset, so I refer here to an old commit in the repository. I will update this if the file gets readded into the main branch).

The data contains observations for three distinct penguin species on islands in the Palmer Archipelago, Antarctica and we can use the data to compare the body mass, flipper lengths or culmen dimensions between the different species or Islands.

Let’s get started!

1. Install and import pandas

Pandas can be installed from PyPI using the python package manager pip: pip install pandas

Or using conda with: conda install pandas

Once installed, we can import pandas and assign an alias of pd with:

import pandas as pd

2. Data structures

There are two data structures in pandas, Series and DataFrames. The structure I most commonly use is a DataFrame, a 2D tabular structure with labels, similar to a spreadsheet or SQL table.

The other structure is a Series which is a 1D labelled array that can hold any data type. Each row is labelled with an index.

A DataFrame can be built in many different ways including as a dictionary of Series, where each Series is a column in the DataFrame.

This article will focus on DataFrames and we will use a CSV as input to the DataFrame.

To explore data structures further check out the pandas documentation.

3. Input and output data

Input

DataFrames can be created in a variety of ways:

A) Create an empty DataFrame: df = pd.DataFrame()

B) Input data: df = pd.DataFrame(data = data), where the input data can be in many different formats, making building a DataFrame flexible and convenient as the data you work with may be in any number of structures including a dictionary of Series and as represented in the diagram above and can be built with the following code:

d = {'student1': pd.Series([85., 72.], index=['maths', 'science']),
   'student2': pd.Series([62., 70., 55.], index=['maths', 'science', 'english']),
   'student3': pd.Series([45., 48., 70.], index=['maths', 'science', 'english'])}
df = pd.DataFrame(d)
print(df.head())

C) Input from a file or data source which we will focus on here looking at the raw CSV of the Palmer Penguin data.

Much like when building a DataFrame using input data, when creating a DataFrame from a file or data source, many input formats are accepted.
This includes:

• excel spreadsheets
• SQL
• CSV
• JSON

Check out the pandas documentation for the full list.

This offers a lot of flexibility in data storage and you can read in files that are stored locally, remotely and it even accepts compressed files.

The Palmer Penguin file we are interested is hosted on GitHub, you can download this file and read the CSV specifying the location on your machine, or we can provide the link to the raw file and open it with one line of code:

df = pd.read_csv('https://raw.githubusercontent.com/allisonhorst/palmerpenguins/1a19e36ba583887a4630b1f821e3a53d5a4ffb76/data-raw/penguins_raw.csv')

Another common data source is an SQL database. There are many different python packages to connect to the database, one example is pymysql where we can first connect to a MySQL database and then use read_sql to load the data from the query into df. This offers the flexibility to connect to remote databases quickly and easily!

Note: This is just an example of how to read in SQL data and is not used in the tutorial

# Example code: how to use read_sql
import pymysql

con = pymysql.connect(host='localhost', user='test', password='', db='palmerpenguins')

df = read_sql(f'''SELECT * FROM penguins''', con)

💡 Python 3’s f-strings are convenient to use when inputting queries.

We might later input a condition that uses a variable we have previously defined and instead of hard-coding the condition we can include the variable directly in the f-string.

The triple quotation also allows single quotes to be used within the f-string without an escape character as in the example below. For example:

# Example code: f-string's and variables
region = tuple('Anvers')

df = read_sql(f'''SELECT * FROM penguins WHERE Region IN {region} AND Date Egg > '2007-11-11' ''', con)

Large DataFrames

When reading in large datasets, it is possible to iterate over the file and read in a limited number of rows per iteration by specifying the option of chunksize to read_csv , read_sql or other input functions. However, it is worth noting that this function now returns a TextFileReader instead of a DataFrame and requires a further step to concatenate the chunks into one DataFrame.

df = read_csv('https://raw.githubusercontent.com/allisonhorst/palmerpenguins/1a19e36ba583887a4630b1f821e3a53d5a4ffb76/data-raw/penguins_raw.csv', chunksize = 10000)

df_list = []
for df in df:
    df_list.append(df)

df = pd.concat(df_list,sort=False)

This step is not necessary to use on the Palmer Penguin dataset as it’s only a few hundred rows, but is shown here to demonstrate how you could use this on a large file to read the data in 10k rows at a time.

Output

Just as we can input the file in many formats, to output a DataFrame to a file is just as flexible and easy!

After some manipulation, we can write our DataFrame to a CSV with the option to compress the file if required:

df.to_csv('output.csv', compression='gzip)

If your data is stored within AWS, there is an AWS SDK for Python, boto3 that can be used to connect to your AWS services.

4. Quick checks to inspect the data

Before we start exploring our data, the first thing to check that it has loaded correctly and contains what we expect.

Dimensions of DataFrame

We can first check that the DataFrame is not empty by checking that the number of rows is greater than 0 and further check the dimensions with the following functions:

• Get the number of rows: len(df)
• Get the number of columns: len(df.columns)
• Get the number of rows and columns: df.shape
• Get the number of elements (rows X columns): df.size

if len(df) > 0:
    print(f'Length of df {len(df)}, number of columns {len(df.columns)}, dimensions {df.shape}, number of elements {df.size}')

else:
    print(f'Problem loading df, df is empty.')

This returns:

Length of df 344, number of columns 17, dimensions (344, 17), number of elements 5848

Our dataset has loaded correctly and has 344 rows and 17 columns. What does this DataFrame contain?

dtypes and memory

We’ve loaded in the DataFrame but we still don’t know what type of data it contains. We can get a summary with df.info()- let's take a look at what details it returns:

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 344 entries, 0 to 343
Data columns (total 17 columns):
studyName              344 non-null object
Sample Number          344 non-null int64
Species                344 non-null object 
Region                 344 non-null object 
Island                 344 non-null object 
Stage                  344 non-null object 
Individual ID          344 non-null object 
Clutch Completion      344 non-null object 
Date Egg               344 non-null object 
Culmen Length (mm)     342 non-null float64 
Culmen Depth (mm)      342 non-null float64 
Flipper Length (mm)    342 non-null float64 
Body Mass (g)          342 non-null float64 
Sex                    334 non-null object 
Delta 15 N (o/oo)      330 non-null float64 
Delta 13 C (o/oo)      331 non-null float64 
Comments               54 non-null object 
dtypes: float64(6), int64(1), object(10) memory usage: 45.8+ K 

df.info() returns:

• the index data type (dtype) and range, in this case, a pandas DataFrame with 344 entries denoted by index values 0–343,
• the name and dtypes of each column, and the number of non-null values,
• the memory usage

The penguin data contains columns with mixed or string datatypes objects, integers int64 and floats float64. The table below summarises dtypes in pandas.

Using df.info() the memory usage by default only gives us an estimation based on the column dtype and number of rows.
We can specify to use deep introspection to calculate the real memory usage which can be particularly useful when dealing with large dataframes:

df.info(memory_usage='deep')

<class 'pandas.core.frame.DataFrame'> 
RangeIndex: 344 entries, 0 to 343 
Data columns (total 17 columns): 
studyName              344 non-null object 
Sample Number          344 non-null int64 
Species                344 non-null object 
Region                 344 non-null object 
Island                 344 non-null object 
Stage                  344 non-null object 
Individual ID          344 non-null object 
Clutch Completion      344 non-null object 
Date Egg               344 non-null object 
Culmen Length (mm)     342 non-null float64 
Culmen Depth (mm)      342 non-null float64 
Flipper Length (mm)    342 non-null float64 
Body Mass (g)          342 non-null float64 
Sex                    334 non-null object 
Delta 15 N (o/oo)      330 non-null float64 
Delta 13 C (o/oo)      331 non-null float64 
Comments               54 non-null object 
dtypes: float64(6), int64(1), object(10) memory usage: 236.9 KB

We can also check the memory usage per column with:

print(df.memory_usage(deep=True))
Index                     80 
studyName              22016 
Sample Number           2752 
Species                31808 
Region                 21672 
Island                 21704 
Stage                  25800 
Individual ID          21294 
Clutch Completion      20604 
Date Egg               23048 
Culmen Length (mm)      2752 
Culmen Depth (mm)       2752 
Flipper Length (mm)     2752 
Body Mass (g)           2752 
Sex                    21021 
Delta 15 N (o/oo)       2752 
Delta 13 C (o/oo)       2752 
Comments               14311 
dtype: int64 

Or the total memory usage with the following:

print(df.memory_usage(deep=True).sum())

242622

We can see here that the numerical columns are significantly smaller than the columns containing objects. Not only that but we aren’t interested in all the columns in our analysis.

This raw file contains all the data that has been collected. If we are to compare the body mass, flipper lengths and culmen dimensions in the different species of penguins and on the different islands, then we can clean the data and keep only the data that is relevant.

5. Data cleaning

To clean the data there are several steps we can take:

Remove columns we aren’t interested in

Let’s take a first glance at the first few rows of data:

print(df.head())

This output doesn’t look quite right. We know that there are 17 columns but we can only see 4 of them plus the index here. Depending on the size of the console you are working in, you might see a few more columns here but probably not all 17.

To view all the columns, we can set the value for display.max_columns to be None.

pd.set_option('display.max_columns', None)
print(df.head())

Looking at a sample of the data, we can identify the columns that we want to use and reassigning a new value to df by specifying only the rows and columns we want to keep. We can use df.loc(rows, cols) to do this. In the rows argument, a colon denotes all values and the columns we can specify the columns we are interested in, which are: Species, Region, Island, Culmen Length (mm), Culmen Depth (mm), Flipper Length (mm), Body Mass (g), Sex

keep_cols = ['Species', 'Region', 'Island', 'Culmen Length (mm)', 'Culmen Depth (mm)', 'Flipper Length (mm)', 'Body Mass (g)', 'Sex']

df = df.loc[:, keep_cols]  

print(df.columns)  

>>> Index(['Species', 'Region', 'Island', 'Culmen Length (mm)',        'Culmen Depth (mm)', 'Flipper Length (mm)', 'Body Mass (g)', 'Sex'],       dtype='object'

We now only have the columns of interest stored in our DataFrame.

Replace or drop null values

We can check again the memory usage of df and we can see it has halved by removing irrelevant columns.

df.info(memory_usage='deep')

<class 'pandas.core.frame.DataFrame'> 
RangeIndex: 344 entries, 0 to 343 
Data columns (total 8 columns): 
Species                344 non-null object 
Region                 344 non-null object 
Island                 344 non-null object 
Culmen Length (mm)     342 non-null float64 
Culmen Depth (mm)      342 non-null float64 
Flipper Length (mm)    342 non-null float64 
Body Mass (g)          342 non-null float64 
Sex                    334 non-null object 
dtypes: float64(4), object(4) memory usage: 104.8 KB 

Looking at the number of non-null values, the size data has 2 null values and the sex has 10. We can also see this with:

print(df.isna().sum())  
Species                 0 
Region                  0 
Island                  0 
Culmen Length (mm)      2 
Culmen Depth (mm)       2 
Flipper Length (mm)     2 
Body Mass (g)           2 
Sex                    10 
dtype: int64

We can either drop these values or replace them with another value, specifying inplace=True in the function to reassign the value.
In this case, we can replace the na values with Unknown for the Sex columns and we can drop the na values for the other columns.

df['Sex'].fillna('Unknown', inplace=True) 

print(df.isna().sum())  

Species                0 
Region                 0 
Island                 0 
Culmen Length (mm)     2 
Culmen Depth (mm)      2 
Flipper Length (mm)    2 
Body Mass (g)          2 
Sex                    0 
dtype: int64  

df.dropna(inplace=True) 

print(df.isna().sum())  

Species                0 
Region                 0 
Island                 0 
Culmen Length (mm)     0 
Culmen Depth (mm)      0 
Flipper Length (mm)    0 
Body Mass (g)          0 
Sex                    0 
dtype: int64

Since there were only a few values we removed, this is unlikely to impact the memory usage but is useful to clean the data and when dealing with large DataFrames with a significant number of null values, it may reduce memory.

Convert objects to Categorical dtypes

The final step in cleaning our data is to check the number of unique values in each column to determine whether a categorical dtype is more suitable in place of using objects.

We already know that there are three penguin species in our data yet the dtype for this column is an object, so let's check this column first.

print(df['Species'].head())  

0    Adelie Penguin (Pygoscelis adeliae) 
1    Adelie Penguin (Pygoscelis adeliae) 
2    Adelie Penguin (Pygoscelis adeliae) 
4    Adelie Penguin (Pygoscelis adeliae) 
5    Adelie Penguin (Pygoscelis adeliae) 
Name: Species, dtype: object  

print(df['Species'].nunique())  

3  

print(df['Species'].unique())  

['Adelie Penguin (Pygoscelis adeliae)' 'Gentoo penguin (Pygoscelis papua)'  'Chinstrap penguin (Pygoscelis antarctica)']

As expected there are only 3 unique values in this column but each row contains a string with the penguin species, taking up significant memory. We can instead convert this column to be a category dtype and check how this changes the memory.

print(df.memory_usage(deep=True))

Index                   2736
Species                31626
Region                 21546
Island                 21575
Culmen Length (mm)      2736
Culmen Depth (mm)       2736
Flipper Length (mm)     2736
Body Mass (g)           2736
Sex                    21213
dtype: int64

df['Species'] = df['Species'].astype('category')

print(df.memory_usage(deep=True))

Index                   2736
Species                  702
Region                 21546
Island                 21575
Culmen Length (mm)      2736
Culmen Depth (mm)       2736
Flipper Length (mm)     2736
Body Mass (g)           2736
Sex                    21213
dtype: int64

This has now gone from 31626 bytes to 702, reducing the size significantly! The same approach can be taken for the other object columns, Region, Island and Sex.

for col in ['Region','Island','Sex']:
    print(f'Column: {col}, number of unique values, {df[col].nunique()}, unique values: {df[col].unique()}')

Column: Region, number of unique values, 1, unique values: ['Anvers']
Column: Island, number of unique values, 3, unique values: ['Torgersen' 'Biscoe' 'Dream']
Column: Sex, number of unique values, 4, unique values: ['MALE' 'FEMALE' 'Unknown' '.']

Region has only one value, this column can be removed since it doesn’t provide any value.

df.drop(columns=['Region'], inplace=True)
print(df.columns)

Index(['Species', 'Island', 'Culmen Length (mm)', 'Culmen Depth (mm)',
       'Flipper Length (mm)', 'Body Mass (g)', 'Sex'],
      dtype='object')

Island and Sex can be converted to categorical dtypes in the same way as Species. Before we do that, there is a strange value in the Sex column of a full-stop, let’s take a look at how many of these there are.

print((df['Sex']=='.').value_counts())

False    341
True       1
Name: Sex, dtype: int64

Since there is only 1 value, let’s replace it with Unknown in the same way that we replaced the null values.

df['Sex'].replace('.','Unknown', inplace=True)
print((df['Sex']=='.').value_counts())

False    342
Name: Sex, dtype: int64

Now we are happy with the columns, let’s convert Sex and Island to categories and check the final memory usage:

df['Sex'] = df['Sex'].astype('category')
df['Island'] = df['Island'].astype('category')

print(df.memory_usage(deep=True))

Index                  2736
Species                 702
Island                  613
Culmen Length (mm)     2736
Culmen Depth (mm)      2736
Flipper Length (mm)    2736
Body Mass (g)          2736
Sex                     610
dtype: int64

print(df.memory_usage(deep=True).sum())

15605

The total memory usage has gone from 242622 to 15605.

df.info(memory_usage='deep')

<class 'pandas.core.frame.DataFrame'>
Int64Index: 342 entries, 0 to 343
Data columns (total 7 columns):
Species                342 non-null category
Island                 342 non-null category
Culmen Length (mm)     342 non-null float64
Culmen Depth (mm)      342 non-null float64
Flipper Length (mm)    342 non-null float64
Body Mass (g)          342 non-null float64
Sex                    342 non-null category
dtypes: category(3), float64(4)
memory usage: 15.2 KB

Reducing the memory from 236.9 KB to 15.2 KB.

Mission accomplished!

We’ve so far covered how to get started with reading a file into pandas and cleaning the file to optimise performance. The penguin data is already well structured, with only minimal cleaning needed compared to the types of data you might come across from other sources. It is also a fairly small dataset starting at 200 KB when pandas can deal with files in the order of GBs, where these quick tips can yield significant improvements in performance.

This is part 1 of a series of tutorials. In the following parts, we will explore our data quantitively and graphically, creating data visualisations using pandas, and some additional python packages — matplotlib and seaborn and we will dive into some other features of pandas that are useful to know!

The full code from this tutorial can be found on GitHub!

Are there any pandas functions for data inspection and cleaning that you’d like to hear more about? Or any I’ve missed? Let me know!

If you enjoyed this article, you can subscribe to my monthly newsletter to get my latest articles, resources I use and other tips directly in your inbox!

Want to learn about web scraping or learning resources, including data science courses in python, check out some of my other articles:

• Data Science Skills: Web scraping using python
• Learn to code. Learn Python.

Data Analysis in python: Getting started with pandas was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you wanting to learn to code but unsure where to start? You’ve come to the right place. I’ll walk you through the reasons to learn Python to help you decide if it’s right for you and provide a list of resources to learn the basics, get your environment set up, find projects to get started with and where you can go for help and support through your coding journey.

Why learn Python?

Python is a versatile programming language that can be used for software development, data analysis, machine learning and even web development! If you’re wanting to program robots, follow a career in Data Science or make pretty visualisations, learning python is a good place to start!

If this is your first experience of learning to code then the syntax (the structure of statements) is similar to English, requires fewer lines of code and once you have the basic grasp of the logic, syntax and structure, it can be easily applied to other programming languages.

However, if you are more focused on web or app development, there may be other programming languages out there that you might want to consider instead. HTML, CSS and JavaScript are also great starting points when learning to code. Don’t stop reading just yet though, many of the resources and tips below are relevant to any new developer!

Resources

Depending on your style of learning, there are different types of courses you can follow. There are written tutorials with code examples, short videos with interactive coding problems, live-coding videos or the simple textbook. I’ve included a range of resources in this list so that you can give them a try and decide which learning style works best for you.

Learn to Code for free

1. FreeCodeCamp

FreeCodeCamp is a dedicated community platform for learning to code. They have courses, tutorials, videos and articles to help you get started and cover many different programming languages. The courses on their website are mainly focussed on web development and are a great starting point if you want to learn HTML, CSS and JavaScript!
For python, they have put together a comprehensive guide to resources, as well as a YouTube course.

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner, Intermediate, Advanced
Content type: Articles, Videos, Interactive Tutorials

2. HiPy

HiPy is an open community of people dedicated to introducing anyone and everyone to coding in Python. They run regular events in Liverpool and have a series of beginner-friendly notebooks which you can work through and an Intro to Python course covering how to get started. They also have a YouTube channel and have recently streamed live-coding events that are definitely worth a watch!

Programming languages: Python
Level: Beginner, Intermediate, Advanced
Type: Tutorials, Videos, Events

3. EdX Courses

EdX has a range of python courses available. The courses are free to complete with an option to pay for a certificate. The courses are often run by companies or Universities and you may need to explore to find a course that is the right style for you. I’ve found the courses very hit and miss!

Check out this Microsoft course for an Intro to Python for absolute beginners!

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner, Intermediate, Advanced
Type: Courses

4. Coursera

Similar to EdX, Coursera has a range of different courses available for free, with payment to receive certification.

If you’re looking for an introduction to Machine Learning, I completed Andrew Ng’s Machine Learning course last year and highly recommend checking it out! It provides a broad overview with weekly assignments and questions alongside the course. I really enjoyed the style of teaching and the assignments were interesting and appropriately challenging! My only warning is that the assignments use a programming language called Octave which could sometimes be a little awkward — there is an intro to Octave within the course however if you are new to programming, you might find it a little tricky at times!

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner, Intermediate, Advanced
Type: Courses

5. Kaggle micro-courses

Kaggle is a data science platform which hosts competitions and data challenges. It also provides micro-courses including an introduction to python alongside a YouTube channel with live-coding demonstrations including concepts in python and data science. Once you have the basics and with a foundation in Data Science, you can get involved with the community, practice your skills with the playground competitions such as the Titanic challenge or explore the Kaggle datasets for use in your own projects.

Programming languages: Python
Level: Intermediate, Advanced
Type: Tutorials, Videos

6. W3Schools

W3Schools is a website for learning web technologies, containing code examples and includes interactive tutorials in Python, HTML, CSS and JavaScript. I often find myself referring back to w3 for documentation and detailed explanations with examples of how to use the function or package.

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner
Type: Documentation, Tutorials

7. Udemy

Udemy offers over 100k online courses from a wide range of topics and also in different languages. There is an overwhelming number of python courses from different instructors however you can narrow this down by specifying only free courses and selecting your level and topics you are looking for. I’d recommend learning the basics with the resources above and using Udemy to dive deeper into a specific topic if you are familiar with an instructor and like their teaching style.

Note: Udemy also contains paid courses but they often have a sale on, reducing the price significantly so it’s best to wait for these!

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner, Intermediate, Advanced
Type: Course

8. Automate the Boring Stuff with Python

If you prefer books, Automate the Boring Stuff with Python is a great place to get started and the content is also available online. It provides practical examples on how to get started and project examples you can work through at the end of each chapter.

Programming languages: Python
Level: Beginner, Intermediate
Type: Book

9. Python for Data Science

If you’ve grasped the basics and are looking to explore Python with data science, Python for Data Science is a valuable resource and again is available for free online, although also handy to have a copy to pick up off the shelf!

Programming languages: Python
Level: Intermediate, Advanced
Type: Book

Paid Resources

There are also resources that offer subscriptions for their learning platforms

1. 365DataScience

365DataScience is a Data Science learning platform with an initial free tier to sample the course material. I’ve not used this resource personally, but they cover a range of topics in data science and have a free tier to give you a taster before you commit to the subscription.

Programming languages: Python, SQL
Level: Intermediate, Advanced
Type: Interactive Tutorials

2. Learn the basics with Codecademy

Codecademy requires a subscription to access the full course content and offers introductory and advanced courses to different programming languages. The content that is free to access has changed significantly over time. Whilst this was my go-to resource when I was first learning Python as I liked the interactive style of the courses, now with the platform, you can be mid-way through a chapter and are unable to progress unless you sign up.

Programming languages: Python, HTML, CSS, JavaScript + more
Level: Beginner, Intermediate
Type: Interactive Tutorials

3. DataQuest

If you’re already familiar with the basics in Python and are looking to apply your knowledge to concepts in data science, check out DataQuest. Like the previous two platforms, DataQuest also requires a subscription however they do offer a free trial so if you’re interested, give it a try!

Programming languages: Python, SQL, R
Level: Intermediate, Advanced
Type: Interactive Tutorials

Troubleshooting

One of the hardest parts when learning to code can be trying to work out what to search for when you’re stuck. This does get easier the more you practice! Here are a few common places where you might find the answers you are looking for:

1. StackOverflow

When searching for a programming question, the top results are usually from StackOverflow. Stackoverflow is an open community where people can ask questions and other developers try to help. Chances are, the problem you’re having, others have already asked on StackOverflow with possible solutions waiting for you!

2. DEV.to

DEV is a community of software developers getting together to help one another out. The platform supports collaboration and networked learning through articles, tutorials and discussions in addition to podcasts and videos shared by the development community! If you sign up, you too can share your coding journey and the things that you learn with the community and meet other people learning to code too!

3. Medium

Medium has a range of publications dedicated to learning to code. TowardsDataScience has many articles and tutorials on Data Science and Python, BetterProgramming and HackerNoon also have content around programming and you may also find company blogs such as Netflix Technology Blog who share technical articles and tutorials.

4. Data Science Stack Exchange

Similar to StackOverflow, Data Science Stack Exchange is a forum to ask technical questions, this time specific to Data Science and Machine Learning. When troubleshooting Python questions, this resource often appears in the top search results and is another site where the community may already have the answer you are looking for!

5. Twitter

There is a large tech community on Twitter. Many people join #100DaysOfCode, committing to tweet every day they code over the course of 100 days, sharing what they’ve been learning and supporting others in their journeys too. If you are stuck, you can also reach out to this community and someone may be able to answer your question or point you to something that could help!

6. Join a Python community

There are also global networks that host meetups and code nights all over the world. There are also many meetup groups focused on putting on technical talks, tutorials and events — check out Meetup or Eventbrite for events in your local area. It’s also worth considering online communities and groups with Slack channels, Telegram groups or virtual meetups where you can join to learn, share and network.

Installing your Python environment

You’ve read this far and have a number of resources to get you started, next, let’s get Python set up on your machine!

There are several different options when installing Python depending on which OS you are using and how you’d like to get set up. For new programmers, I’d recommend installing Anaconda which is quick to get started and available for Windows, MacOS and Linux.

Python 2 or Python 3?

Support for Python 2 was stopped on January 1st 2020. You may still find software written using Python 2 but I’d recommend installing/upgrading to Python 3 if possible!

What is Anaconda?

Anaconda is an open-source distribution of Python and R. This means that it installs all the necessary software and packages that you need to get started. It includes:

• the Python and R programming languages
• pre-installed python packages
• conda, a python package manager which enables you to easily install and update python packages. You may also see pip mentioned, conda is an alternative to pip.
• Spyder is an Integrated Development Environment (or IDE, for short), which includes a text editor — think Microsoft Word but for writing code — in addition to tools for testing, debugging and inspecting your program and many features to making programming easier, including autocomplete!
• (Optional but recommended) PyCharm is an another IDE and you will be prompted on installation whether you’d like to install it alongside Anaconda. I’d recommend doing so! I much prefer PyCharm over Spyder, finding it easier to navigate and get started with.
• Jupyter Lab/Jupyter Notebooks also can be installed through the Anaconda Navigator. These allow you to write code alongside text or visualisations and can be useful when presenting projects or going through tutorials.

If you already have a preferred IDE that is not included above, such as Sublime or Visual Studio Code, check out the Anaconda documentation to find out how to select Anaconda Python as your python interpreter within the IDE.

Python within your browser

You can also access Python within your browser by using Trinket where you can also set up an account if you want to save your code.

Other ways to install Python

If you’d prefer not to use Anaconda, you can download and install Python from the Python Software Foundation.
If you’re using Linux or MacOS, you may already have Python pre-installed but this may not be Python 3. For further installation options, follow the instructions on RealPython.com.

What’s next?

So you’ve installed python, covered the basics and are feeling ready to get started with a project!

Here is a list of possible projects and challenges to give you some inspiration:

1. CodeWars

CodeWars has a number of coding katas, or challenges, where you can put your python skills to the test and compete to solve problems from beginner to incredibly advanced!

2. Project Euler

Feeling like a new challenge? Why not attempt the first 100 Project Euler problems using python. You can follow #ProjectEuler100 on Twitter and join the community to solve the challenges! I started the challenge at the start of 2020 with the aim to complete all 100 by the end of the year… entering the 5th month of the year, I’m way behind target!

3. Get started with a project

Check out Medium and DEV.to for tutorials you can follow to get started on your first project, or if you are looking to get started with a Data Science challenge, check out the Titanic Machine Learning challenge on Kaggle.

Watch this space for my next articles where I will walk through a project, exploring datasets, creating visualisations and providing you with the skills and tools to continue your coding journey.

Thank you for reading! If you enjoyed my article then subscribe to my monthly newsletter where you can get my latest articles and top resources delivered right to your inbox!

You can follow me on Medium for more articles, follow me on Twitter or find out more about what I’m up to on my website.

Learn to code. Learn Python. was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2020, my focus is on reading more non-fiction and whilst reading Atomic Habits by James Clear, I summarised each chapter with one sentence. By taking notes in this way I can easily digest and review the content and then come back to these again and again.

The book is incredibly well written with accompanying diagrams and examples to demonstrate important concepts and at the end of each chapter, James Clear summarises all the key points.

If you are looking to create new habits or break bad ones, check out the chapter by chapter breakdown below for a brief insight into the tools and techniques provided in the book.

The Fundamentals: Why Tiny Changes Make a Big Difference

1. The Surprising Power of Atomic Habits
Build a system of atomic habits by implementing regular routines that are small and easy to do to result in compound growth.

2. How Your Habits Shape Your Identity (and Vice Versa)
Shape your habits around who you want to become.

3. How to Build Better Habits in 4 Simple Steps
Your habits are shaped by the systems in your life and become automated when they take little energy and effort.
The Four laws of behaviour change: make it obvious, make it attractive, make it easy, make it satisfying.

The 1st Law: Make It Obvious

4. The Man Who Didn’t Look Right
Be aware of your habits as some we may do unconsciously, use the pointing and calling technique to consciously acknowledge behaviour.

5. The Best Way to Start a New Habit
Use implementation intention by setting a clear time and place for your habit and add on to an existing habit through habit stacking.

6. Motivation is Overrated; Environment Often Matters More
Habit cues are influenced by environments and context, make them obvious to trigger habits or build habits in a new environment.

7. The Secret to Self-Control
Self-control is a short term strategy; optimise your environment to make good habit cues obvious and bad habit cues invisible.

The 2nd Law: Make It Attractive

8. How to Make a Habit Irresistible
Temptation bundling pairs a habit you need to do with a habit you want to do to make it more attractive where the anticipation of reward is driven by dopamine.

9. The Role of Family and Friends in Shaping Your Habits
We imitate the habits of the close, the many and the powerful. Join a culture with your desired behaviour and a shared interest.

10. How to Find and Fix the Casuses of Your Bad Habits
Habits are associated with emotions, to quit a bad habit make it seem unattractive and to create a motivation ritual by doing something that evokes good feelings before a difficult habit.

The 3rd Law: Make It Easy

11. Walk Slowly, but Never Backward
Motion is useful, action will deliver an outcome and repetition makes an action more automated.

12. The Law of Least Effort
Redesign your life to create less friction to good behaviours and more friction to bad behaviours.

13. How to Stop Procrastinating by Using the Two-Minute Rule
Implement a habit that takes only 2 minutes and then over time build on it.

14. How to Make Good Habits Inevitable and Bad Habits Impossible
Future-proof your habits by using commitment devices to make bad habits difficult in the present and automating your desired habits.

The 4th Law: Make It Satisfying

15. The Cardinal Rule of Behaviour Change
If you enjoy a habit you will repeat it, link an immediate reward to the habit to initially incentivise the habit and over time it becomes your identity and the intrinsic rewards will sustain it.

16. How to Stick with Good Habits Every Day
“The first mistake is never the one that ruins you” — habit trackers can be used to measure progress and make the habit satisfying.

17. How an Accountability Partner Can Change Everything
Make a habit unsatisfying by using techniques such as an accountability partner and/or habit contracts with negative consequences for failing to put your habit into action.

Advanced Tactics: How to Go from Being Merely Good to Being Truly Great

18. The Trulh About Talent (When Genes Matter and When They Don’t)
Tailor your habits around your personality and use explore/exploit trade offs to find areas of strength.

19. The Goldilocks Rule: How to Stay Motivated in Life and Work
Work on tasks at the edge of your current ability, in this zone you can achieve a flow state and also learn to enjoy the task even when you’re bored of it.

20. The Downside of Creating Good Habits
It is important to reflect and review to prevent building mistakes into habits and to avoid holding tightly onto an identity to encourage long term growth.

Conclusion: The Secret to Results That Last

The four laws of behaviour change present a set of tools and strategies to build better systems and better habits.

Small habits can compound to make remarkable changes.

Thank you for reading! If you enjoyed my article then subscribe to my monthly newsletter where you can get my latest articles and top resources delivered right to your inbox!

Over the past few months, I’ve started to use Notion for my digital organisation. Whilst I like to think that I’m quite organised and tidy, I must admit my digital organisation is not up to scratch. On discovering Notion I fully immersed myself in tutorials, articles and using different templates to find a set up that works for me. Since October I’ve been using this tool on a daily basis to monitor and track my tasks, projects, fitness as well as to save documents and articles.

One of the key reasons for using Notion is the amount of flexibility it offers. Used along with web clipping tools and expert tips from people such as Marie Poulin and Maria Aldrey, it is possible to integrate your digital resources all in one place instead of across several apps.

Getting started, I followed all of Marie Poulin’s videos and templates but have now established a workspace that works for me. I’d definitely recommend checking out Notion Office Hours, Marie Poulin’s YouTube channel, Maria Aldrey’s GroovyWink as well as the PARA method from Tiago Forte as a digital organisation system.

In this tutorial I’d like to share with you how I set up my work dashboard to track daily tasks, weekly progress and meetings and I hope you can make use of the template and integrate some of the tips into your own workspaces!

Basic Work Dashboard (with Template)

1. Getting Started with Notion

Once you sign up for a free account at https://www.notion.so/, create a new workspace with Add a Page.
Give your new workspace a name and we’re ready to go!

2. Setting up the template

The dashboard is made of several elements

• Daily Planning database
• Quick Access section
• Good Morning workflow
• Weekly Focus section

You can duplicate the template and work straight from there or in the following steps we will go through how to create each section.

3. Creating a Daily Planning Table

Whilst in many of the resources linked above work from Task databases so that these can be shared with other members of a team (see Advanced Dashboard), I found that I like to see my meetings, tasks and weekly schedule from within one database.
I treat this table as time slots in a daily planner and create different views to see my daily or weekly schedule.

3.1 Create a table
Create a new page within your empty dashboard and select Database > Table. This sets up a basic table for you, with a few fields already created. Give your table a name eg. Daily Planning, and add the following properties:

• Name
• Date
• Timeline
• Description
• Tags — configure options of Weekly Review, Task, Meeting, Recurring
• Type — configure options specific to your tasks eg. Admin, Content, Coding
• Day — configure options with the days of the week, as well as Weekly and Someday
  💡It is useful to index each item with a number to get the order to appear correctly in the kanban board, as shown later
• Priority 1
• Priority 2
• Priority 3
• Productivity — configure options of 1–5 stars
• Status — configure options such as Not Started, On Hold, In Progress, Complete

You might notice that there are two fields containing dates; date and timeline. For me, there are tasks I may be waiting on or may take several days to complete. To ensure that task appears in Today’s Tasks kanban board the date field needs to be the latest date and the timeline the duration that the task has been active for. This is still a work in progress and I’m open to ideas to make this easier!

Also there are some fields that are used exclusively for Weekly Reviews such as Priority and Productivity and have created a template for Weekly Reviews with preset values for some of the properties. You can create templates by clicking the arrow next to the blueNew button and selecting New Template. I also have set up templates for New Tasks and New Meetings again with some fields populated.

3.2 Views

At the moment our database looks like a table however, it is possible to add more views (by selecting Add new View) to help make organisation easier!

• Default View: Table, as created
• Today Kanban: Kanban board* with Filter on Date is today and group by Status
• Weekly Kanban: Kanban sorted by Date descending and group by Day
• Weekly Reviews: Gallery with Filter on Tags contains Weekly Review and sorted by Date descending
  You can also change the properties to be displayed on the cards to show only the fields you want to quickly see
• Calendar: Calendar by Date
• Not Complete: Table with Filter on Status not complete, Filter on tags does not contain Meeting, Filter on tags does not contain Weekly Review

*Kanban refers to a style of project management where tasks are displayed in columns on a board

3.3 Recurring tasks

Notion currently doesn’t have the capability to handle recurring tasks. To deal with this in the table, I have addedRecurring as a tag and for these tasks specifically do not assign a date field. For the recurring task Update Tasks I have set a Day of Friday since this is a weekly recurring task.
For monthly tasks, you could also configure an option of 7 — Monthly as a Dayand change the date to the next day to perform the task once it has been completed each month to ensure it is added to your Today kanban.

I have been using this set up for around a month now and it is incredibly easy to maintain and I work mostly from within the Weekly kanban, unless I have a particularly busy day. It does rely on updating the tasks at the end of the week to remove the Day property but this can serve as a good reflection on your tasks for the week as you are filling in your Weekly review.

4. Quick Access Bar

There are some documents that I like to have easy access to and I’ve added these to the quick access bar on a different colour background. To create this box, you need to first create a toggle list in this case the list is: ✨✨✨

Within the list you can create some Template Buttons such as a New To-do or a New Meeting which creates a meeting notes document created from a Notion inbuilt template.

Finally you can create a title of My Documents and add some useful resources here.

Once this is all set up, select the three dots next to the start of the toggle and select Color and select aBackground Color of choice to make it stand out. Finally, remove the toggle arrow and your quick access bar is ready to go!

5. Good Morning Toggle

Again using a Toggle List you can create a drop down to-do list of tasks to check off Daily or Weekly, you can set this up with your own motivational quotes and checklists.

To get the checklist to align with the Quick Access bar, you can highlight the section and drag the block to the end of the row where two columns will then be created. I have also added a blank line to create a 3rd column where I have adjusted the spacing.

6. Weekly Focus

Finally, we create a Weekly Focus block.

Firstly create the heading of Weekly Focus and underneath this create another toggle list, this time without a title.
Inside the toggle create a placeholder text such as Focus 1 and change the background colour. Duplicate this toggle list depending on how many focus points you want each week.
Again drag each toggle to the end of the row with the Weekly Focus title and add an extra blank line for an extra column at the end, if desired. Adjust the column widths to fit your page.

You can now get started with your own weekly dashboard! If you would prefer to use the template for this dashboard you can access that here and duplicate into your own workspace.

I also have a more advanced template you can try which contains further databases such as Tasks, OKRs & Projects.

Thank you for reading! If you enjoyed my article then subscribe to my monthly newsletter where you can get my latest articles and top resources delivered right to your inbox!

You can follow me on Medium for more articles, follow me on Twitter or find out more about what I’m up to on my website.

Build a Weekly Planning Dashboard in Notion was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post can also be found at https://kaparker.com/posts/technomads-datascience

On Monday 25th November we held our monthly TechNomads meetup where we focussed on Data Science. Joseph Allen & John Carney, Data Scientists based in Manchester, visited us at Liverpool Science Park to give an introduction to Data Science followed by a Q&A session. They provided fascinating insight drawn from their experience and shared plenty of tips and advice with us. The audience was made up of a mix of people working with data, including researchers, students and analysts as well as developers, aspiring data scientists and people working on side projects.

Why do we need Data Science?

The field of Data Science is broad and constantly evolving. As the hype for data science has grown, as has the confusion of what the role of a Data Scientist involves. As experienced Data Scientists and now running a Data Consultancy PDFTA, Joseph & John simplified the definition of Data Science to be: converting data into value.

One point that was highlighted that there is often over-emphasis on using Machine Learning, when a project may be quicker, simpler and cleaner to use other methods.

Data Science: converting data into value.

There are however lots of quick ways to get started with Machine Learning projects, including:

- Image classification: is the image a cat or a dog?

- Forecasting: using a python package fbprophet

- Chatbots: integrating these into social media pages

How do we apply Data Science?

A revised version of the Data Science pyramid was introduced with the foundation of Data Science relying on good Software Engineering practices and how this is something we should all endeavour to improve.

Data Science is 95% exploring data, 5% Machine Learning

There are also many different paths into data science and Joe has summarised these in an article on Medium.

The Data Scientists i’ve met.

Something I took away from the talk was that Data Science can still use the concept of a Minimum Viable Product. It’s better to apply a simple model and deploy to get feedback than to risk months for no value.

What is Data Science?

Joseph and John then went through some of the key terms that you hear in Data Science such as Model, Regression, Classification and gave examples of where these can be used — check out their slides for details!

Case Studies

Some example projects were presented which John and Joseph have worked on and a few of which are a good first Data Science project such as sentiment analysis of WhatsApp messages!

They then introduced Case Studies in Data Science including how Recommendation Engines can be used to search for similar and interesting products for the user. For this, UX teams, particularly in e-commerce, can provide insightful feedback to get example data of how users respond to this. Either Content-based or Behaviour-based recommendation systems can be used and python packages such as prod2vec which is based around word2vec can be used.

Q&A

After a short break, the audience asked John & Joseph questions with some fascinating questions about:

• How often should I model be updated?
• The ethics and bias of data
• Where to get started as a Data Scientist?

Overall, it was a brilliant evening, thanks to John and Joseph for coming along to TechNomads and sharing your expertise and thanks everyone for coming! I certainly learned a lot and if you couldn’t attend, be sure to check out the slides!

You can find John, Joseph and PDFTA on Twitter. They also co-organise PyDataMCR where they have monthly meetups as well as monthly Code Nights to bring your questions and help others with theirs!

During a recent NLP project, I came across an article where word clouds were created in the shape of US Presidents using words from their inauguration speeches. Whilst I had used word clouds to visualise the most frequent words in a document, I’d not considered using this with a mask to represent the topic or subject. This got me thinking...

Several months ago, shortly before the final series of Game Of Thrones and after having just rewatched seasons 1–7, I was eagerly awaiting the finale — so much so that I went looking for any Game Of Thrones data I could find online to create a predictor for who would survive the chaos of season 8 and what would happen within the 7 Kingdoms. Unfortunately, I was out of time but I did find plenty of data and visualisations that people had put together. When I came across the inaugural word clouds I wondered if I could use this Game Of Thrones data, particularly scripts, with image masks of the characters to create some pretty cool visualisations. In this article, as presented at a recent tech meet-up, I will walk through my steps to create Game Of Thrones word clouds using python.

Getting Started

Following a similar method to the inaugural word cloud article, there are several steps involved when dealing with text based data. The steps are:

1. Finding relevant data
2. Cleaning data
3. Creating a mask from the image
4. Generating word clouds

Since I will be using python in this tutorial, there are many libraries to help with the above steps, also Dipanjan Sarkar’s guide to Natural Language Processing provides a comprehensive introduction to techniques in text analytics if you are looking for further reading around the topic.

Before getting started with the first step, I defined the goal of the project to ensure that no important steps were missed and to have a clear vision of the target.

Project goal: To create word clouds for Game Of Thrones characters masked with an image.

1. Finding relevant data

There is plenty of Game Of Thrones data available online — GitHub and Kaggle are where I usually first search for datasets — and I quickly found a dataset containing the scripts.

To achieve the project goal, the lines for each character need to be stated —the first question was: are the character lines are available?

Looking at a snippet from the first episode, the character data is available! However, when exploring further episodes and season, the format for character names varies.

In the case above only the character’s first name is used and this is all in uppercase, whilst in other seasons the characters full name is used with letter case. Stage directions can also be included within the character name.

Th search for all lines by the character name, a regular expressions with the character names can be used:

re.findall(r'(^'+name+r'.*:.*)', line, re.IGNORECASE)

This expression searches for the start of a line (^), followed by the character name which we input as a variable, followed by any text (.*), then a colon to indicate that this is a character’s line (:) and followed by any text (.*). By containing this regular expression within brackets, the full line is returned. Finally, the case is ignored, so the character name can be upper/lower/letter case.

Using the function below will return all the lines for a character:

# final_data taken from:  
# https://github.com/shekharkoirala/Game_of_Thrones

# get data for characters
def get_char_lines(char):    
    output = []          
    print('Getting lines for', char)        

    with open('final_data.txt', 'r') as f:
        for line in f:
            if re.findall(r'(^'+char+r'.*:.*)',line,re.IGNORECASE):
                output.append(line)
    f.close()
    print(char, 'has ', len(output), 'lines')

return output

# get lines using
get_char_lines('arya')

2. Cleaning data

Now that we have the lines for a character, these need to be cleaned.

The following techniques are used for cleaning the lines, these same techniques are also outlined in detail in the NLP Guide referenced above. Again regular expressions are useful here to replace or remove characters:

• Remove line info eg. JON:

re.sub(r'.*:', '', text)

• Remove brackets — remove any stage directions from the character lines

re.sub('[\(\[].*?[\)\]]', ' ', text)

• Remove accented characters and normalise using the unicodedata library

unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')

• Expand any contracted words eg. don’t → do not
  There is a python library for this, copy contractions.py to your working directory from:
  https://github.com/dipanjanS/practical-machine-learning-with-python/

• (Optional): apply lemmatisation where a word is stemmed to a root word that is in the dictionary, eg. is, are → be
  Not currently using lemmatisation as it doesn’t handle some words well eg. Stannis → Stanni
• Convert all text to lowercase

text.lower()

• Remove special characters (*,.!?) with the option to remove numbers too — default is set to false

pattern = r'[^a-zA-Z0-9\s]' if not remove_digits else r'[^a-zA-Z\s]'    re.sub(pattern, '', text)

• Remove stop words, using stop words from the nltk library by first tokenising the text, splitting the string into a list of substrings and then removing any stop words

stopword_list = stopwords.words('english')    
tokens = nltk.word_tokenize(text)    
tokens = [token.strip() for token in tokens]    
' '.join([token for token in tokens if token not in stopword_list])

These cleaning techniques have been based on several different sources and there are many variations of these steps as well as additional cleaning techniques that can be used. These initial cleaning steps have been used in this project.

3. Create a mask from image

Based on the inauguration word clouds, the PIL library is used to open the image, a numpy array is created from the image to create a mask.

char_mask = np.array(Image.open("images/image.jpeg"))    image_colors = ImageColorGenerator(char_mask)

Optionally the numpy array can be used with wordcloud.ImageColorGenerator to then recolor the word cloud to represent the colours from the image, or otherwise. This will be covered in the next section.

After initially testing an image as a word cloud mask, the background in the image created too much noise that the shape of the character was not well defined. To avoid this, it can be useful to remove the image background and replace with a white background instead.

4. Generate word clouds

The final step is to create the word cloud using the generate() function.

wc = WordCloud(background_color="white", max_words=200, width=400, height=400, mask=char_mask, random_state=1).generate(text)

# to recolour the image
plt.imshow(wc.recolor(color_func=image_colors))

The word cloud will be masked with an image and the size of text will be based on word frequency.

The parameters of the word cloud can be adjusted — try increasingmax_words to see some of the less frequent words included, note that this should be less than the number of unique words within your document.

As mentioned in the previous section, the recolor step is optional and here is used to represent the original image colours.

Game of Thrones Word clouds

Using the steps above with images of Game Of Thrones characters, the generated words clouds are presented below:

Arya

Jon Snow

Daenerys

Ser Davos

This has also been extended to generate word clouds based on Houses:

House Lannister

House Stark

With these word clouds the initial project goal has been reached!

Improvements

There are several ways in which these word clouds can be improved.

• Common words: There are recurring words that are common in all the characters scripts. Currently the word clouds are generated based on the word frequency however, an alternative would be TFIDF which weights the terms based on the term frequency in the document and the frequency with respect to the corpus. Alternatively, a custom stop word list could be generated to remove other frequent words
• Lemmatisation/stemming: Lemmatisation has not been used in the examples above as some words unique to Game of Thrones were shortened when testing (Stannis → Stanni), however this does mean that words which come from the same root word are not linked and occur several times in the word cloud eg. say, said. An alternative lemmatisation method or stemming technique could be used.
• Text cleaning: There are further cleaning or data preparation steps that could be used
• Word cloud features: I have tested with different word colours, background colours and word cloud parameters, the grey colour and black background works in this case but some of the parameters could be further optimised
• Further text analysis: many insights can be drawn from this dataset. Much more scope for analysis!

Extension — Stranger Things

Looking into further shows, I eventually found Stranger Things script — although they are missing character lines the data can still be used to generate word clouds….

Summary

In this article I have walked through the basic steps to generate word clouds which are masked with an image.

This is just the start of my experimenting with word clouds! There is a huge scope for further development and improvement in the project, I hope you can follow the steps to create your own projects with word clouds!\

Thank you for reading! If you enjoyed my article then subscribe to my monthly newsletter where you can get my latest articles and top resources delivered right to your inbox!

You can follow me on Medium for more articles, follow me on Twitter or find out more about what I’m up to on my website.

Creating word clouds with python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are different ways of scraping web pages using python. In my previous article, I gave an introduction to web scraping by using the libraries:requests and BeautifulSoup. However, many web pages are dynamic and use JavaScript to load their content. These websites often require a different approach to gather the data.

Data Science Skills: Web scraping using python

In this tutorial, I will present several different ways of gathering the content of a webpage that contains Javascript. The techniques used will be the following:

1. Using selenium with Firefox web driver
2. Using a headless browser with phantomJS
3. Making an API call using a REST client or python requests library

TL;DR For examples of scraping javascript web pages in python you can find the complete code as covered in this tutorial over on GitHub.

Update November 7th 2019: Please note, the html structure of the webpage being scraped may be updated over time and this article initially reflected the structure at the time of publication in November 2018. The article has now been updated to run with the current webpage but in the future this may again change.

First steps

To start the tutorial, I first needed to find a website to scrape. Before proceeding with your web scraper, it is important to always check the Terms & Conditions and the Privacy Policy on the website you plan to scrape to ensure that you are not breaking any of their terms of use.

Motivation

When trying to find a suitable website to demonstrate, many of the examples I first looked at explicitly stated that web crawlers were prohibited. It wasn’t until reading an article about sugar content in yogurt and wondering where I could find the latest nutritional information inspired another train of thought where I could find a suitable website; online supermarkets.

Online retailers often have dynamic web pages that load content using javascript so the aim of this tutorial is to scrape the nutritional information of yogurts from the web page of an online supermarket.

Setting up your environment

Since we will be using some new python libraries to access the content of the web pages and also to handle the data, these libraries will need to be installed using your usual python package manager pip. If you don’t already have beautifulsoup then you will need to install this here too.

pip install selenium
pip install pandas

To use selenium as a web driver, there are a few additional requirements:

Firefox

I will be using Firefox as the browser for my web driver so this means you will either need to install Firefox to follow this tutorial or alternatively you can use Chromium with Chrome.

Geckodriver

To use the web driver we need to install a web browser engine, geckodriver. You will need to download geckodriver for your OS, extract the file and set the executable path location.

You can do this in several ways:
(i) move geckodriver to a directory of your choice and define this the executable path in your python code (see later example),

(ii) move geckodriver to a directory which is already a set as a directory where executable files are located, this is known as your environmental variable path.
You can find out which directories are in your $PATH by the following:

Windows
Go to:

Control Panel > Environmental Variables > System Variables > Path

Mac OSX / Linux
In your terminal use the command:

echo $PATH

(iii) add geckodriver location to your PATH environment variables

Windows
Go to:

Control Panel > Environmental Variables > System Variables > Path > Edit
Add the directory containing geckodriver to this list and save

Mac OSX / Linux
Add a line to your .bash_profile (Mac OSX) or .bash_rc (Linux)

# add geckodriver to your PATH
export PATH="$PATH:/path/to/your/directory"

Restart your terminal and use the command from (ii) to check that your new path has been added.

PhantomJS

Similar to the steps for geckodriver, we also need to download PhantomJS. Once downloaded, unzip the file and move to a directory of choice or add to your path executable, following the same instructions as above.

REST Client

In the final part of this blog, we will make a request to an API using a REST client. I will be using Insomnia but feel free to use whichever client you prefer!

Scraping the web page using BeautifulSoup

Following the standard steps outlined in my introductory tutorial into web scraping, I have inspected the webpage and want to extract the repeated HTML element:

<div data-cid="XXXX" class="listing category_templates clearfix productListing ">...</div>

As a first step, you might try using BeautifulSoup to extract this information using the following script.

# import libraries
import urllib.request
from bs4 import BeautifulSoup

# specify the url
urlpage = 'https://groceries.asda.com/search/yogurt' 
print(urlpage)
# query the website and return the html to the variable 'page'
page = urllib.request.urlopen(urlpage)
# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(page, 'html.parser')
# find product items
# at time of publication, Nov 2018:
# results = soup.find_all('div', attrs={'class': 'listing category_templates clearfix productListing'})

# updated Nov 2019:
results = soup.find_all('div', attrs={'class': 'co-product'})
print('Number of results', len(results))

Unexpectedly, when running the python script, the number of results returned is 0 even though I see many results on the web page!

https://groceries.asda.com/search/yoghurt
BeautifulSoup - Number of results 0

When further inspecting the page, there are many dynamic features on the web page which suggests that javascript is used to present these results.
By right-clicking and selecting View Page Source there are many <script> elements in use and searching for the element above containing the data we are interested in returns no matches.

The first approach to scrape this webpage is to use Selenium web driver to call the browser, search for the elements of interest and return the results.

Scraping the web page using Selenium

1. Selenium with geckodriver

Since we are unable to access the content of the web page using Beautiful Soup, we first need to set up a web driver in our python script.

# import libraries
import urllib.request
from bs4 import BeautifulSoup
from selenium import webdriver
import time
import pandas as pd

# specify the url
urlpage = 'https://groceries.asda.com/search/yogurt' 
print(urlpage)
# run firefox webdriver from executable path of your choice
driver = webdriver.Firefox(executable_path = 'your/directory/of/choice')

As mentioned when installing geckodriver, if the executable file is not in an executable path, we are able to define the path in our python script. If it is in an executable path then the line above becomes:

# run firefox webdriver from executable path of your choice
driver = webdriver.Firefox()

Once set up, we can now connect to the web page and find the elements of interest. When loading the webpage in a browser, results often take a while to load and also may not even load until we scroll down the page.
With this in mind, here we can add some javascript for the web driver to execute to perform such actions. Below is a simple example to get the page to scroll, there will be more efficient ways to do this, why not test your own javascript here and let me know in the comments what works best for you!

We also add a sleep time as another method to wait for the page to fully load.

# get web page
driver.get(urlpage)
# execute script to scroll down the page
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
# sleep for 30s
time.sleep(30)
# driver.quit()

If we run the script now (you can also uncommentdriver.quit() at the end to ensure the browser closes), as your python script runs Firefox will open the url specified and scroll down the page. Hopefully, you should many products load up before the script finishes running.

Next, we want to get the elements of interest. Previously, using Beautiful Soup we have tried to find all elements based on the tag and class attributes, however, in this example we will use a slightly different approach to access the product information. Instead, we can search for the elements by xpath, based on the XML structure or the css selector.

We can inspect the element of interest and within the toolbar, right-click on the highlighted element and Copy > Copy xpath (or Copy Selector). This is another interesting way to understand the structure of the html. In this case we will be using the xpath to find the elements, and we can then print the number of results that match:

# find elements by xpath

# at time of publication, Nov 2018:
# results = driver.find_elements_by_xpath("//*[@id='componentsContainer']//*[contains(@id,'listingsContainer')]//*[@class='product active']//*[@class='title productTitle']")

# updated Nov 2019:
results = driver.find_elements_by_xpath("//*[@class=' co-product-list__main-cntr']//*[@class=' co-item ']//*[@class='co-product']//*[@class='co-item__title-container']//*[@class='co-product__title']")
print('Number of results', len(results))

One of the main reasons for using the xpath rather than using the element as the results have a few elements where the stem of the id is listingsContainer with some additional words, so the contains function has been used to select all of the results but also to exclude any of the other div elements within the container such as for adverts.

Firefox Webdriver - Number of results 38

Now that we have some results from the page, we can loop over each result and save the data of interest. In this case, we can save the product name and link.

Note that there are actually more than 38 results on the web page. This number also may vary depending on how many results load when you connect to the page. All results can be gathered by either changing the javascript we execute as suggested above, alternatively other methods will be explored in the following sections.

# create empty array to store data
data = []
# loop over results
for result in results:
    product_name = result.text
    link = result.find_element_by_tag_name('a')
    product_link = link.get_attribute("href")
    # append dict to array
    data.append({"product" : product_name, "link" : product_link})

Outside of this loop, we can close the browser and as we imported the pandas library, we can make use of that by saving the data we have scraped to a dataframe. We can print the dataframe to view the content.

# close driver 
driver.quit()
# save to pandas dataframe
df = pd.DataFrame(data)
print(df)

In this format, we can very simply write this data to a csv.

# write to csv
df.to_csv('asdaYogurtLink.csv')

Using Selenium with geckodriver is a quick way to scrape the web pages that are using javascript but there are a few drawbacks. I have found that sometimes the page does not load (I’m sure that this could be more efficient by changing the javascript we execute as mentioned above, but I am new to JS so this might require some time), but also loading the browser and waiting for the page to load takes time.

Another option, we can use a headless browser. This should speed up the scraping as we don’t have to wait for the browser to load each time.

2. Selenium with a headless browser

When using PhantomJS as a headless browser instead of geckodriver, the only difference is how the web driver is loaded. This means that we can follow the method above but change the line that initialises the web driver which becomes:

# run phantomJS webdriver from executable path of your choice
driver = webdriver.PhantomJS(executable_path = 'your/directory/of/choice')

Note here that Selenium support for PhantomJS has been depreciated and provides a warning.

It is also possible to use headless mode with geckodriver by using the headless option:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True
driver = webdriver.Firefox(firefox_options=options, executable_path = 'your/directory/of/choice')

By using the headless browser, we should see an improvement in time for the script to run since we aren’t opening a browser but not all results are scraped in a similar way to using firefox webdriver in normal mode.

Making an API request

The final approach we will discuss in this tutorial is making a request to an API. When inspecting the Network page XHR files, as a page loads this page displays the requests that are being made. Within this list is a /search request which calls an API endpoint to get the results that are presented on the page.

We are able to make the same request using either a REST client or with a few lines of python.

If we inspect the search file and look at the headers, the request url containing the keyword and other parameters that are needed to make the request. Below the general details are the response and request headers which we may need later.

To get the response, we can take the request url and as a test enter this into the address bar of your browser. Since the parameters are added in the string we can also try to remove all but the keyword parameter to test whether any further parameters are required. In this case, the keyword query returns the results in the browser, so we can also perform the same request using a REST client or in python.

Insomnia REST client

Using insomnia we can enter the request url and send the request.

This returns a JSON response containing the data that we are looking for!

This example is very straight forward with no headers or security tokens required. For other cases, the REST client allows you to enter any additional response parameters that you can get from the inspect tool when gathering the request details.

Python request

We can also make the same request from python using the urllib.request library in the same way that we connect to a web page before scraping.

The JSON response can be made more readable by adding a few parameters for indenting and sorting the keys so that we can now open the file and see the response data provided to the webpage when a search is made.

# import json library
import json

# request url
urlreq = 'https://groceries.asda.com/api/items/search?keyword=yogurt'

# get response
response = urllib.request.urlopen(urlreq)

# load as json
jresponse = json.load(response)

# write to file as pretty print
with open('asdaresp.json', 'w') as outfile:
    json.dump(jresponse, outfile, sort_keys=True, indent=4)

For now, we will keep all the data. My next tutorial will cover data structures and output in more detail so we can manipulate the JSON and find the relevant data.

Summary

This tutorial has outlined some of the methods we can use to scrape web pages that use javascript. These methods include:

Using a web driver to scrape content

• Using selenium web driver to connect to a web page either with Firefox web driver, PhantomJS, headless browser
• Use the web driver to find the elements of interest
• Loop over the results and saving variables of interest
• Saving data to a dataframe
• Writing to a csv file

Make a HTTP request

• Inspect the web page to find HTTP request details
• Make the GET request using either a browser, REST client, python

Whilst the HTTP request method is quicker to implement in this tutorial and provides all the data we need from one request, this is not always the case. Not all websites will make their requests visible, additional security may be in place with expiring authentication tokens or the output data may require significant cleaning which would be more work than using a web driver with some javascript to enable loading all results and looping over all pages. This tutorial provides a few different alternatives you can try to make it possible to scrape javascript.

In my next tutorial we will explore data structures, manipulating data and writing to output files or databases.

Thank you for reading! If you enjoyed my article then subscribe to my monthly newsletter where you can get my latest articles and top resources delivered right to your inbox, or find out more about what I’m up to on my website.

If you are new to python or want to improve, check out my article with a list of learning resources including courses in data science:

Learn to code. Learn Python.

Data Science Skills: Web scraping javascript using python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.