Why you should use WebSocket to retrieve live crypto data

Life-saving hack in getting ever-changing data!

One of the greatest ways to automate the process of retrieving data from a website is to build a web crawler to handle the work. You can create a web crawler that scrapes data in a variety of methods. For example, if you want to retrieve data from a website once every few hours or once a week, you may write a program that could automatically send the HTTP request or open a browser to do so as it is less frequent. However, if you are thinking of crawling data within seconds, it is still best to connect through a web socket.

Before going through the practical part of coding a web scraper, let us briefly goes through what it is and normally when you are going to use it.

Let me put this in a real-life context…

One day, you told your friend that you will be calling her to discuss a concept that you do not understand. So, when you were calling your friend (handshake), your friend would check who is calling and then decide whether to receive the call (verify). After your friend completed the verification and picked up your call, he will be explaining the concept and keep the line open for your incoming queries (Data streaming from the server).

Normally, you should be thinking of using a web socket to retrieve data in cryptocurrencies, live stock price, live betting, live messaging, trading platform, and many more.

After knowing the concepts, let’s picture a real scenario and work from there. For instance, if you would want to build a live trading bot to trade cryptocurrencies, and you already have a set of rules ready to beat the market.

What’s next?

You will need to get a website that you can retrieve live data. You come across some APIs but are afraid that those API will fail which might eventually lead to losing money. Thus, you decide to find a website that provides live crypto prices. Let’s pick Cryptocompare (a global cryptocurrency market data provider) for illustration purposes.

Exploration

When you navigate to this website using chrome, right-click, and click on inspect, then click on the WS tab. You should be able to see a similar view as the screenshot above now. You will notice the request url for this web socket is wss://streamer.cryptocompare.com/v2?format=streamer .

Next, let’s navigate to the message tab to get the message needed to send to the server in order to complete the verification.

Now, you will see there are two types of arrows -

1. The messages following the green color arrows indicate the outgoing messages that your browser is sending to the server.
2. The messages after the red color arrows are the incoming data received by your browser from the server.

Till now you just need to copy down the green arrow’s message by right-clicking the message and selecting “copy message”. You would need these 5 green arrow messages in the coding part.

Coding

I have attached a code snippet for you to try on. Something that you would need to pay attention to is to change the WebSocket key (“Sec-WebSocket-Key”) to the key that you can get from your browser under the “Headers” tab. Example below:

Here is a summary of what does the code snippet above does:

1. Create a connection with the request url

2. Send the 5 outgoing messages to the server

3. Sit down and relax to wait for the incoming messages, normally is within seconds.

Result

Save the code snippet to a python file and run it. You should see similar output.

Problems You Might Face

Here are some of the problems you might face and how you can solve it.

1. Error Message: Connection is already closed

If you see this message, normally what it means is that the server is trying to stop you from connecting and the fastest way to solve it, is to change your proxies or VPN.

2. HTTP Status Code: 401 (Unauthorized)

A 401 response means that the server does not accept your connection request. If the server is working properly, then there is a problem with your connection request. In short, normally it means you are using the wrong credentials. To solve it, check the request headers and modify them accordingly.

3. HTTP Status Code: 403 (Forbidden)

Similar to 401, but in this case, re-authenticating will make no difference. This might mean that the original application logic change for instance the handshake process has changed. Thus, to make sure it runs smoothly, check your request headers and the messages that you need to send to the server.

Final Thought

This article is written as the previous old snippet code is not working. Thus I combine all the errors and questions that my readers raised and answer them accordingly.

If you find this code not working in the future, feel free to comment down below and I will answer them accordingly.

Thank you for reading until the end. See you in the next post!

Here is the latest crawler which is to crawl crypto exchange announcement news, feel free to check out here.

About the Author

Low Wei Hong - Medium

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more in crawling websites, creating data pipelines,s and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Why you should use websocket to retrieve live crypto data was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Due to the current stay at home policy, I am working from home. When I am browsing through the internet, to find COVID-19 related news article, I am curious how some of the websites are able to fetch the latest news or articles. These websites could retrieve news or articles from various websites.

I am thinking, fantastic! Do they build multiple crawlers on various websites to get the latest news? But that will need a lot of work.

Or maybe there exists some API to retrieve the news for free? However, I find out these API providers only track certain websites.

I am thinking is there an easier way to get the latest google news? As Google bots are scraping most of the websites, it should be a good way to start. Nevertheless, Google is known for its efficiency in restricting bots.

After hours of research, I figure out that there is a way to retrieve the latest google news, which is scraping through Google RSS news feeds.

If you are interested to know how to build a website that will display the latest news? Or maybe you are trying to integrate live news to your website? Or perhaps you are just interested in knowing how to retrieve the news?

This article is for you.

What is RSS?

RSS stands for “Rich Site Summary” or “Really Simple Syndication”, it is a format for delivering regularly changing web content. It allows a user to keep track of many different websites in a single news aggregator. Hence, many news-related sites, weblogs, and other online publishers syndicate their content as an RSS Feed to whoever wants it.

So, you have a better understanding of what an RSS is, let us start with the scraping!

What is the exact URL we need to scrape?

BASE URL: http://news.google.com/news?

Here is the list of parameters that I think is useful to retrieve English related COVID-19 news. If you are interested to find out more, feel free to click here for google official XML API documentation.

Query terms:

1. q: This stands for the query term which I want to make a request to, in this case, it is COVID-19.
2. hl: The host language of your user interface that you are using. I prefer to use en-US.
3. sort: This parameter is optional. I will want to sort the news according to the date, so the value is the date.
4. gl: This is the boosts search results whose country of origin matches the parameter value. The value is the US as this is the default value for my web browser.
5. num: Number of news you would like to get. I will just go for the maximum number which is 100.
6. output: The format of your desired output. I will go for RSS.

Here is the final string which I will be sending requests to.

URL: http://news.google.com/news?q=covid-19&hl=en-US&sort=date&gl=US&num=100&output=rss

Congratulation, you are now left with the code part.

Coding Part

The programming language I am using is Python. Before you are testing the code, do install these two libraries — bs4 and feedparser.

You will notice that I create a class called ParseFeed, with two methods clean and parse.The clean function will extract all the text from the HTML documents and replace the character \xa0 with a space character.

Besides, the parse function will parse the HTML and print some of the fields which I think is important. For instance, the title, description, published date and URL of the news.

Here is a snapshot of the output after running the code above on the Jupyter Notebook.

Final Thought

As you might observe, you are able to retrieve the news through RSS in a much convenient way. If you are interested to track other kinds of news, you can just tweak the parameters to get what you want. However, do note that there are limits to some of the parameters.

Thank you so much for reading until the end. If you have any kind of topics you would like me to discuss, feel free to comment below.

Stay home, Stay safe everyone!

Note from the editors: Towards Data Science is a Medium publication primarily based on the study of data science and machine learning. We are not health professionals or epidemiologists, and the opinions of this article should not be interpreted as professional advice. To learn more about the coronavirus pandemic, you can click here.

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

How to get the latest COVID-19 News using Google News Feed was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to retrieve LIVE cryptocurrency prices effectively

Why do we need to learn how to extract data from WebSocket?

Isn’t it requests already able to scrape almost all kinds of websites?

First of all, let me give a brief introduction to WebSocket, and you will know why do we need to learn how to scrape it.

Websocket protocol is used to provide persistent real-time connection. It means that most of the websites use WebSocket to send real-time data from its server to the web so that you are able to view the ever-changing live data.

You might ask, what kind of websites normally uses WebSocket?

There are a few specific kinds of websites that I know, uses WebSocket. For instance, live betting, cryptocurrency, and stock market websites.

Now, let’s get into the complete picture of how WebSocket works. In order to make it more practical, I will be using Python to scrape Cryptocompare (a global cryptocurrency market data provider) as an illustration purpose.

First, you will need to find the WebSocket URI.

Here I am using Chrome developer tools to inspect. After open Chrome DevTools, click the WS (Web Socket) tab, then you will be able to find the request URL which is bounded by a purple box above.

Request_URL: wss://streamer.cryptocompare.com/v2

Besides, one thing to notice is that the success status code is different from normal requests. Normally when you are requesting a website, the success status code is 200 but the success status code for WebSocket is 101 which is bounded by the red box in the above screenshot.

Before getting to the handshake part, let’s discuss some of the simple yet important concepts when you are connecting through WebSocket. Let’s take a look at the diagram below.

Click on the Messages tab, and you will have the same view as mine. You will notice that there are green and red color arrows. Green arrows indicate that you are sending this message to the website. On the other hand, red arrows are the messages that you will be receiving when connecting through the WebSocket.

Let’s move on to the next part — handshake. Remember that there is a green arrow in the above figure? That’s actually a handshake process.

What does this mean?

You can think of this process as a verification process. After you manage to send this message to the website, you will be able to retrieve the LIVE data. This concludes the whole process and let’s get into the code part.

There are several packages that you can use, but in this case, I will be using websocket-client Python package.

Coding Part

First, you need to copy your web browser’s header to here and use json.dumpsto convert it into the string format.

After that, create the connection to the server by usingcreate_connection.Then, perform the handshake by sending the message, and you will be able to see the data on your side.

Below is a similar output that you should be seeing.

You might be wondering, the message seems meaningless and messy, are you sure that the price is in the information here?

The answer is yes. You can have a look at the text bounded by the purple box in the above screenshot, and that is actually the current price of BTC-USD.

After scraping this website, you will notice how easily you can retrieve information through WebSocket. If you are interested to know why you should scrape through WebSocket, feel free to visit this link.

Final Thought

This article is written to answer one of the questions which I have received through my article’s comment section. If you have any questions which you would like me to clarify, feel free to comment below.

Thank you for reading until the end. See you in the next post!

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

Websocket — Retrieve Live Data was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Previously, when I was studying at university, I wanted to go into the data science field so badly.

I kept on study a lot of statistics with the hope that I could ace my data science interview.

However, I kept failing.

After that, I spent some time, trying to recall is there any common areas that most of the hiring managers are looking for.

I figured out that I actually do not have a nice data science project to showcase.

Hence, I spent some time, tried to craft a data science project, and thought I was going to succeed this time.

However, I still failed. Because I did not pay attention to some of the details of the project.

After making improvements to my projects again and again, I finally managed to secure a data science internship.

In this article, I will walk you through some of the improvements I made that led to a data science internship opportunity.

1. Understand your machine learning models

You may think that implemented some fancy kind of models will increase your chance of scoring an interview. In fact, without really understanding your model, it actually will hurt you.

For instance, you implemented a very complicated deep learning model. However, you do know why it works. You just clone other people’s work from GitHub and got a high accuracy score.

Then, when the interviewer asks you to explain your implemented models, or in which kind of scenarios, your model will not work well, you will be stunned and leave the interview.

Here is the message I want to share.

You need to prepare some questions about your models that hiring managers will question you. If you do not have much time to prepare for your data science interview, but at least, make sure that you understand the models you used clearly.

Below are some examples of questions that you can prepare.

1. Do you have any benchmark performance to compare?
2. In which circumstances that your model will not be able to give accurate results?
3. If you have more time, what kind of improvement you will make to your model?

You can either include this as a side note on your presentation slides or markdown file which you will be presenting. Either way works!

2. Keep your analytics concise

There will always be some insights that you could find interesting when you are doing your project. You will spend some time, plot different kinds of graphs, and hope to impress the hiring manager.

Plotting some graphs are great. However, do plot some important graphs. For example, plot some graph that is related to your result or the features for your model, or the distribution of each category in your training examples.

One thing to note is also to prepare a story for your plot or analysis.

For instance, don’t just say, the sales spikes in May. Instead, you can say the conversion of the sales is much higher across each channel during May, and thus it leads to spikes in sales.

When you find some interesting findings, spend some time to find out the reason behind it.

3. Explain why do you choose those features

Features are extremely vital to our machine learning models. Having the right features, you can produce excellent accuracy without using complicated models.

There will be various features you can choose from. Explain the underlying reason for choosing the subset of available features will show that you know the importance of feature extraction in machine learning. In other words, you are not taking random inputs, and throwing them to the models.

Besides, you may want to include the process of choosing the right feature. For instance, you are trying to predict the housing price. The features you are considering to use include, the minimum distance to the nearest public transportation station, the number of bedrooms, the size of the house, etc.

Let’s say you find that the minimum distance to the nearest transportation station is not an important feature. It hurts the performance of your model. Do include this in your data science project report. Data science is not all about the final result, sometimes, it is the process of finding the solution that matters.

4. Prepare Both Powerpoint slides and Jupyter Notebook

You may ask, why do we need to prepare both?

I don’t have time to prepare both, I thought presenting the code to the interviewers would be enough?

It depends actually on the situation you have. For me, it is better to have both. For instance, while you are presenting using PowerPoint slides, the interviewer interested to know how you process your data and for some reason, you did not put this information on the presentation slide.

If you have prepared your code with markdowns and comments, you will be able to show him or her on the spot.

Another scenario would be you only prepare Jupyter Notebook and expect to explain the code to the interviewer line by line. Base on my experiences, this is not a good way to present. The interviewer just wants to know how you tackle the problem, therefore, summarize your thought process in PowerPoint will be a better idea.

Hence, my suggestion is to use PowerPoint as the main presenting material. At the same time, make your clean and tidy jupyter notebook available as the supporting material.

Final Thought

Having a great data science project to showcase is definitely one of the most important factors to stand out among other candidates.

In the process of preparing the project, you will find that there is a lot of work required to put in. While you already put in a lot of effort into it, don’t overlook some of the important details as it might ruin your presentation.

All these tips are provided based on my experience. If you have any questions or other different thoughts, feel free to comment!

Thank you for reading till the end, and see you in the next post!

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

4 Tips to Boost Your Data Science Project Presentation was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Shared from a data scientist who used to be a business intelligence analyst

According to the US Bureau of Labor Statistics, demand for qualified business intelligence analysts and managers is expected to soar to 14% while demand for data scientists is expected to rise 16 % by 2028.

Given the average of the growth rate for all occupations is about 5%, these two different fields seem to have a better outlook. However, a business intelligence analyst does not seem to be as popular as a data scientist given that both the future outlook is quite similar.

If you are looking to transit from business intelligence analyst to data scientist, but not sure which kind of skills are transferable.

If you are bored with building machine learning models and start to wonder whether a business intelligence analyst would be a better option for you?

This article is for you.

I have worked in these two fields before. Although these two fields seem very different, they are actually quite relatable.

In this article, I am going to share with you the top 3 similarities between these two fields.

1. Analyze and understand data

Nowadays, I find a lot of people called themselves as a data analyst. I think it is because it sounds cool. However, it is not difficult for you to figure out that most of them are just reporting the numbers without understanding why it happens.

For instance, they reported that the sales increase by 30% compares to the previous week. However, they do not try to understand what is the reason which contributes to the increase in sales.

If you want to do well in either field, you must understand the data. Don’t just report for the sake of reporting, but instead try to figure out the fact that data wanted to tell you.

In data science, your understanding of data will determine how successful your model does. For instance, you are building a model to forecast the sale. If you know what is the most important or combination of factors that affect sales, then you can input those factors into your model. Therefore, your model will be able to give reasonable predictions.

In Business intelligence, this is also very important. Every day, you will be flooded with business requests. No matter you are trying to answer business questions or provide insights, you need to have a concrete understanding of data. For example, you can’t give insight into which kind of customers convert more without knowing the customer journey.

2. Using data to solve business problems

The software or technique use varies quite different between these two fields, but they are all used to tackle problems.

For instance, the business intelligence department is required to forecast the sales to allocate budget for each sales team to spend. By performing simple regression on historical data in the excel sheet, and also manually tweak some variables to comprehend future events, it can just be done using some formulas in an excel sheet.

If this problem is required to tackle by the data science team, base on what I observed, the tool used will be very different. For instance, they will try out with a sophisticated machine learning model or a deep learning model that cannot be achieved by just using excel.

Although the tool is different, both of them are just trying to forecast the sales as accurately as possible. A combination of both teams will actually lead to a more efficient result.

Why do I say so?

This is because the tasks of the business intelligence team are answering questions or provide insights. They will understand the data more and therefore if both teams work together, the data science team can save time to find out those important variables to be used to train the model.

3. Storytelling and Visualization

Imagine you are able to find out some valuable insights, but you ended up plotting a graph that is not able to clearly show your insights.

Or maybe you plot the right graph, but you are not able to communicate with your customers or clients for them to understand your insights clearly.

After all the hard work of cleaning and analyzing the data, you need to also make sure that you can present your results in a clear manner. However, the tool to plot the graph could be different. For instance, the business intelligence team uses Tableau, Excel or inhouse built’s business intelligence dashboard to plot the graph.

On the other hand, the data science team will utilize programming languages such as Python or R to visualize by coding.

No matter what tools both teams are using, the end result will be to do PowerPoint slides to show their work.

Final Thought

By now, you will have an overall picture of the similarities between these two professionals. Some of the examples stated above might vary across companies.

If you want to transit from one field to another, you will know what kind of skills are actually transferable.

Therefore, you don’t need to worry so much about the different skillsets between these two jobs, you can either foster transferable skills or work on skills that you think you currently lack.

Don’t be afraid to fail. Be afraid not to try — Michael Jordan

Thank you for reading until the end, and hope to see you in the next article.

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

Story You May Like

How Business Intelligence is different from Data Science

The Top 3 Similarities between Business intelligence and data science was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

When I am still a student at university, I am curious how does the automated information extraction of resume work. I will prepare various formats of my resumes, and upload them to the job portal in order to test how actually the algorithm behind works. I would always want to build one by myself. Thus, during recent weeks of my free time, I decided to build a resume parser.

At first, I thought it is fairly simple. Just use some patterns to mine the information but it turns out that I am wrong! Building a resume parser is tough, there are so many kinds of the layout of resumes that you could imagine.

For instance, some people would put the date in front of the title of the resume, some people do not put the duration of the work experience or some people do not list down the company in the resumes. This makes the resume parser even harder to build, as there are no fix patterns to be captured.

After one month of work, base on my experience, I would like to share which methods work well and what are the things you should take note before starting to build your own resume parser.

Before going into the details, here is a short clip of video which shows my end result of the resume parser.

Data Collection

I scraped multiple websites to retrieve 800 resumes. The resumes are either in PDF or doc format.

The tool I use is Puppeteer (Javascript) from Google to gather resumes from several websites.

One of the problems of data collection is to find a good source to obtain resumes. After you are able to discover it, the scraping part will be fine as long as you do not hit the server too frequently.

After that, I chose some resumes and manually label the data to each field. The labeling job is done so that I could compare the performance of different parsing methods.

Preprocessing data

For the rest of the part, the programming I use is Python. There are several packages available to parse PDF formats into text, such as PDF Miner, Apache Tika, pdftotree and etc. Let me give some comparisons between different methods of extracting text.

One of the cons of using PDF Miner is when you are dealing with resumes which is similar to the format of the Linkedin resume as shown below.

The way PDF Miner reads in PDF is line by line. Thus, the text from the left and right sections will be combined together if they are found to be on the same line. Therefore, as you could imagine, it will be harder for you to extract information in the subsequent steps.

On the other hand, pdftree will omit all the ‘\n’ characters, so the text extracted will be something like a chunk of text. Thus, it is difficult to separate them into multiple sections.

Therefore, the tool I use is Apache Tika, which seems to be a better option to parse PDF files, while for docx files, I use docx package to parse.

Overview of the data extraction flow

Here is the tricky part. There are several ways to tackle it, but I will share with you the best ways I discovered and the baseline method.

Baseline method

Let’s talk about the baseline method first. The baseline method I use is to first scrape the keywords for each section (The sections here I am referring to experience, education, personal details, and others), then use regex to match them.

For example, I want to extract the name of the university. Therefore, I first find a website that contains most of the universities and scrapes them down. Then, I use regex to check whether this university name can be found in a particular resume. If found, this piece of information will be extracted out from the resume.

In this way, I am able to build a baseline method that I will use to compare the performance of my other parsing method.

Best Method

On the other hand, here is the best method I discovered.

Firstly, I will separate the plain text into several main sections. For instance, experience, education, personal details, and others. What I do is to have a set of keywords for each main section’s title, for example, Working Experience, Eduction, Summary, Other Skillsand etc.

Of course, you could try to build a machine learning model that could do the separation, but I chose just to use the easiest way.

After that, there will be an individual script to handle each main section separately. Each script will define its own rules that leverage on the scraped data to extract information for each field. The rules in each script are actually quite dirty and complicated. As I would like to keep this article as simple as possible, I would not disclose it at this time. If you are interested to know the details, comment below!

One of the machine learning methods I use is to differentiate between the company name and job title. The reason that I use the machine learning model here is that I found out there are some obvious patterns to differentiate a company name from a job title, for example, when you see the keywords “Private Limited” or “Pte Ltd”, you are sure that it is a company name.

Where do I get the data to train?

I scraped the data from greenbook to get the names of the company and downloaded the job titles from this Github repo.

After getting the data, I just trained a very simple Naive Bayesian model which could increase the accuracy of the job title classification by at least 10%.

In short, my strategy to parse resume parser is by divide and conquer.

Evaluation

The evaluation method I use is the fuzzy-wuzzy token set ratio. Let’s say

• s = #of common tokens
• s1 = Sorted_tokens_in_intersection
• s2 = Sorted_tokens_in_intersection + sorted_rest_of_str1_tokens
• s3 = Sorted_tokens_in_intersection + sorted_rest_of_str2_tokens

And the token_set_ratio would be calculated as follow:

token_set_ratio = max(fuzz.ratio(s, s1), fuzz.ratio(s, s2), fuzz.ratio(s, s3))

The reason that I am using token_set_ratio is that if the parsed result has more common tokens to the labelled result, it means that the performance of the parser is better.

If you have other ideas to share on metrics to evaluate performances, feel free to comment below too!

Final Thought

Thank you so much to read till the end. This project actually consumes a lot of my time. However, if you want to tackle some challenging problems, you can give this project a try! :)

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

Story You May Like

Brutal truths that NLP data scientists will not tell you

How to build a resume parsing tool was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Three years ago, I was working as a student assistant in the Institutional Statistics Unit at NTU Singapore.

I was required to obtain the Best Global University Ranking by manually copying from the website and pasting into an excel sheet. I was frustrated, as my eyes were tired, from looking at the computer screen continuously for long hours. Hence, I was thinking about whether there is a better way to do it.

At that time, I googled for automation and I found the answer for it — web scraping. Since then, I managed to create 100+ web crawlers and here is my first-ever web scraper that I would like to share.

Previously, what I did was to use requests plus BeautifulSoup to finish the task. However, after three years when I look back to the same website, I found out that there is a way to get the JSON data instead which works way faster.

If you are thinking of automating your boring and repetitive tasks, please promise me you’ll read till the end. You will learn how to create a web crawler so that you can focus on more value-added tasks.

In this article, I would like to share how I build a simple crawler to scrape universities’ rankings from usnews.com.

Inspect website

The first thing to do when you want to scrape the website is to inspect the web element. Why do we need to do that?

This is actually to find whether there exists a more robust way to get the data or to obtain cleaner data. For the former, I did not go into so deep to dig out the API this time. However, I do find a way to extract cleaner data so that I can reduce the data cleansing time.

If you do not know how to inspect the web element, you just need to navigate to any position of the webpage, right-click, click on inspect, then click on the Network tab. After that, refresh your page and you should see a list of network activities appear one by one. Let’s us look at the specified activity that I have been selecting using my cursor in the screenshot above(i.e. the "search?region=africa&...").

After that, please refer to the purple box in the screenshot above, highlighting the URL that the browser will send the request to in order to get the data to be presented to you.

Well, we can imitate the browser behavior by sending the request to that URL and get the data we need right? But before that, why do I choose to call the request URL instead of the original website URL?

Let’s click on the Preview tab, you will notice that all the information we need, including university ranking, address, countries, etc.. are all in the results field which is highlighted in the blue box.

This is the reason why we scrape through the URL. The data return by the URL is in a very nice format — JSON format.

The above screenshot shows a comparison of the code between today and 3 years before. 3 years before when I was a newbie in web scraping, I just use requests, BeautifulSoup, tons of XPATH and heavy data cleaning processes to get the data I need. However, if you compare the code that I have written today, I just need to use httpx to get the data and no data cleaning needed.

For your information, httpx is built on top of requests, but it supports additional functions like provides async APIs and with httpx, you can send an HTTP/2 request. For a more complete comparison, you may refer to this article.

Request URL:

https://www.usnews.com/education/best-global-universities/search?region=africa&subject=agricultural-sciences&format=json

So now, let’s pay attention to the link we are going to use as shown above. You will notice that you can change the values for region and subject as they are parameters to the URL. (For more information on URL parameters, here is a good read) However, do note that the values for them are limited to the region and subjects provided by the website.

For instance, you can change region=africa to region=asia or subjects=agricultural-sciences to subjects=chemistry. If you are interested to know what are the supported regions and subjects, you can visit my repo to check out.

After knowing how to query this URL to obtain the data you need, the left-over part is how many pages you need to query for a particular combination of region and subject.

So, let’s take this URL as an example, copy and paste the URL into your browser and press enter, then use command+f to search for the keyword “last_page”, and you will find a similar screenshot as below.

*Do note that I have installed a chrome extension that could help me to prettify the plain data into JSON format. This is why you can see that the data shown in my browser is nicely printed.

Congratulation, you manage to find the last_page variable as indicated above. Now, the only remaining process is how to go to the next page and get the data if the last_page is larger than 1.

Here is how I figure out the way to navigate to page 2. Take this link as an example.

First, click on the page number 2, and then view on the right panel. Pay attention to the purple box, you will notice there is an addition of page=2 in the Request URL. This means that you just need to append &page={page_number} to the original request URL in order to navigate through different pages.

Now, you have the whole idea of how to create a web scraper to obtain the data from the website.

If you would like to have a look at the full Python code, feel free to visit here.

Final Thought

Thank you so much for reading until the end.

Here is what I want you to get after reading this article.

1. Know that there are many different ways to scrape the data from a website, for instance getting the link to obtain the data in JSON format.
2. Spend some time to inspect the website, if you manage to find the API to retrieve data, this can save you a lot of time.

The reason that I am comparing my code from 3 years before and the code I have written today is to give you an idea of how you can improve your web crawling and coding skills through continuous practice.

Work hard, the result will definitely come. — Low Wei Hong

If you have any questions or ideas to ask or add on, feel free to comment below!

Here is the latest crawler which is to crawl crypto exchange announcement news, feel free to check out here.

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

How to build a simple web crawler was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

According to the report by Tractica, the AI-enabled NLP software market is predicted to increase from $136 million in 2016 to $5.4 billion in 2025. This is approximately 40 times growth in 10 years!

On the other hand, Association for Computational Linguistics (ACL), the sponsor of the journal Computational Linguistics, has been holding an annual conference which attracts dozen of NLP scientists to submit their works. The submission of scientific papers alone in 2019 had increased by 75% compared to ACL 2018.

You might be thinking — Boom! I am going to work my ass off to become an NLP data scientist in the near future! I love NLP!

Does this mean that you should get into this field just because of its bright future ahead?

Here is the thing. Everything you read suggests the future of NLP is bright and it will be great. However, you should not get into the field without knowing what an NLP data scientist does in and out. I want you to be well-informed of the actual situation so that you can make a better decision on whether to enter the NLP field.

In this article, I will share with you the truth about working as an NLP data scientist.

What is NLP?

In short, it involves computational techniques for the analysis of human natural language and speech.

Hmmmm.. what does this mean?🤔

This means that you are going to deal with text in different kinds of forms. Let me share with you some situations that an NLP data scientist will face from time to time.

Situation #1

Let’s take a look at the tweet below.

Imagine that you are analyzing this text. Your aim is to get the meaning out of the text. Let’s say you identify the subject is “my pants” and the verb for it is “missing”. Then, what do you get from this tweet?

“Pants were missing.”..??!

This is obviously not representative of the meaning of the tweet, right?

Situation #2

Now, let’s say you are performing aspect-based sentiment analysis. Aspect base analysis consists of sentiment (positive or negative) and a target (of opinion).

For instance, “the service provided by the restaurant is good”. Here, the aspect is “service” and the sentiment is “positive”.

Why is this important? It is because the owner of the restaurant not only knows the customer’s experience is good but also specifically the service provided by them is great.

Let’s say you saw this review in your dataset.

This vacuum cleaner really sucks.

The subject here is the vacuum cleaner, but what should be the sentiment? The term “sucks” generally represents a negative sentiment but you can see the meaning could change when it paired with a vacuum cleaner.

A vacuum cleaner cleans by creating suction. When you say it really sucks, you are describing this vacuum cleaner works well, isn’t it?

Situation #3

To make things even complicated. Consider the following question.

Hey bro, where to eat nice food ah?

Does this sentence mean the food is nice? Nope, this is actually just a question.

Situation #4

Another situation is when you are working with text which you have no idea what does it mean. For instance, you understand English, Chinese, and Malay. However, your company is working on Japanese text classification projects. Therefore, when you are trying to understand what goes wrong with your model, you will have a hard time.

One of the solutions might be to use google translate to convert the sentence into the language you know. However, google translation is not perfect. There will be some cases where the translation provided does not make sense and you will need to consult your native Japanese speaker.

Situation #5

Besides, let’s say you are working on a text classification task. The goal is to detect whether an item name should belong to either wine or juice.

Maybe you are thinking of using keyword lists as a feature. After you spend a few hours finding some useful keywords to differentiate the two classes, here is what you think:

First, you decide to have a set of keyword list for each class. For instance, the keyword list for class wine is [‘wine’, ‘grape juice’] while the keyword list for class juice is [‘juice’]. If the element in the keyword list is found in the item name, then the element will be fed into the model as a feature.

However, the performance of your model turns out to be lower than the benchmark of 90% accuracy.

What should you do? You go through each of the wrongly classified test data patiently. You are shocked as you figure out that there are many typos for wine: wne, wyne and etc. Therefore, you have to either update the list or performs some typo checking on your training dataset with the hope that the accuracy will increase.

However, this is not the end of the story. The accuracy did increase, but you are still not satisfied with the result. Again, you eyeballing on the misclassified data and find out this interesting item name:

“Pinot Noir which made from fermented grape juice”

The keyword you extracted from the title is “juice” and thus your model classified it into class juice, which is wrong! Pinot Noir is red wine and hence it should be classified into wine!

Conclusion

Text can be fun and disgusting at the same time. Hence, most of the time, an NLP data scientist will be looking at the data, scratching your head, trying to think a way to make your model understand the context.

Therefore, if you think NLP data scientist is just calling .fit() in sklearn, you should change your direction right now.

If you have to take away only one thing from this article, this is the one.

You should expect 80% of your time, no matter whether you are performing analysis, feature engineering, model improvement, looking at the text data. Thus, make sure you understand your data well and can foresee some potential cases that could happen in the future.

Final Thought

Congratulations! You read until the end of the article. I hope I did not reduce your interest in becoming an NLP data scientist. My aim in creating this article is just to help you have a better understanding of some important information before you enter the NLP field.

Of course, this is just my 2 cents and I welcome you to comment below to share your personal experience as an NLP data scientist as well!🤗

Let me end this article with a quote.

I know you’ve heard it a thousand times before. But it’s true — hard work pays off. If you want to be good, you have to practice, practice, practice. If you don’t love something, then don’t do it. — Ray Bradbury

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Brutal truths that NLP data scientists will not tell you was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

3 PRACTICAL ways that you can start with

What kind of project should I present so that I could stand out during the interview?

Having a great project to present is important in an interview. Although it might not be the most vital element to succeed in an interview, it is definitely a plus to stand out among other candidates.

However, before getting into the details on how to find some great ideas for your project, I will tell you why this matters during your interview.

Imagine that you are one of the candidates competing for a data scientist role. You might be worrying about the technical or machine learning knowledge questions to be asked during the interview. Thus, you put a lot of effort to revise your lecture notes or keep googling to answer any doubts that suddenly pop up in your mind. You are trying your best to swallow all the knowledge before stepping into the interview room so that you will feel more confident.

During the interview, the interviewer bombards you with a lot of tough questions, and you start sweating. You are getting more and more anxious and hence, most of the interview questions you are just trying to guess. Eventually, in your mind, you are thinking, oh my gosh! I am going to fail this interview. Thus, you are thinking very hard to figure out what questions to ask before the end of the interview so that you could leave the interviewer a good impression about you.

Now, if I tell you, there is a way to minimize the questions being asked by the interviewer, so that you stand a higher chance to pass the interview. Will you be interested?

Instead of letting the interviewer keep asking you questions that are out of your control, you can let him ask you the questions which you have already well prepared. You may ask, how can I do that?

You will need at least a well-defined project, which is not a school project, and at least it is somehow interesting to be discussed. Therefore, you just need to prepare all the potential questions which they will be asking you during the interview. If you are smart enough, you can intentionally leave out some key points of the project for them to ask you.

If your project is interesting enough and you are well prepared, most of the time of the discussion will be occupied by the questions related to your project. Now, let’s start with a quote.

All ideas grow out of other ideas — Anish Kapoor

Ideas on what problem to solve actually are not that hard to find. Sometimes, you may not notice that some of the problems you are facing in life, or some questions which you are curious to find out. Those questions are already considered great ideas or problems to start with.

In this article, I will be showing some of the ways where you can find some interesting problems to solve.

1. Past projects

If you are interviewing for a fin-tech company and you have done a fraud-related project. Bingo! You can just present the project but do make sure that there are some aspects which you think are value-added to that company.

However, if you think that the specified project you have done is not good enough, or you have some ideas on which area to improve, please do that! This is because this might be a chance for you to shine.

For instance, for your existing fraud project, you just use some of the normally used features on the internet and apply it to your model. But now, you have a feeling that if the person has a profile picture on the platform, he or she will have a lesser chance of making a fraud, do try it! Although most of the time it may not be a great feature, just show the interviewer that you do have some out-of-the-box thinking skills and they will be interested to understand how you do it!

This is because feature engineering is like a combination of experimenting and having the domain knowledge. Thus, if you think the feature makes sense, try it!

2. Existing ideas on the internet

If you haven’t read about my article on Airbnb which is also featured in Tech in Asia, do click on this link.

Here is how I came out with this idea from the internet. There is a website which is so-called InsideAirbnb. I found that most of the projects that can be found on the internet regarding Airbnb are using the dataset from this website. Interestingly, Malaysia’s dataset does not appear there.

Thus, if I were to scrape the data from Airbnb by myself, and perform the analysis, this will be a great idea. I not only can show that I am able to perform the entire flow by myself (from gathering data, analysis, and modeling) but also to provide value.

The gap here that I had identified is the lack of Malaysia’s dataset and the value that I am able to provide. The value here I mean is to provide Airbnb’s investors a way that they could find out which area they should invest in.

In short, if you are able to identify a gap in the problems which you are solving, this will definitely make your solution looks more interesting.

3. Real-Life Problem

This problem may be faced by you or even your friends. The extra value which I provided in my Airbnb article is actually the problem faced by my friend. He asked me if there is a way to solve this problem and I took the challenge.

Besides, you can tackle some doubts you are having. For instance, you are interested to know which company is having the largest market in Malaysia’s car market and what will be the sales for next year for each company. Follow the three steps I mention in my previous article, identify and formulate the problem, then perform analysis and modeling.

Not only that, you are more than welcome to solve the problem you are currently facing. If you are solving the problems that you are facing, you will have a deeper understanding of the problem itself. Therefore, it will be easier for you to come out with some features which will be useful for your model.

Final Thoughts

Ideas are cheap. Ideas are easy. Ideas are common. Everybody has ideas. Ideas are highly, highly overvalued. Execution is all that matters. — Casey Neistat

Ideas are cheap, execution is all that matters. Thank you so much to read till the end and see you next week!

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

How to generate data science projects ideas was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

For Every Data Science Enthusiast

The expert in anything was once a beginner — Helen Hayes

Previously, when I was a student still studying at university, I wanted to get a data science internship. I wanted to get it so badly that I applied to so many companies, but only got a few replies from them.

To be more specific, less than 10% of the companies I applied to replied to my application. Not only that, when I saw the description of the internship postings, the requirements were to have strong technical skills, excellent grades, in-dept machine learning knowledge and etc.

I was lost. I had neither a strong technical background nor previous experience in data science internships. I was just looking for my first data science internship.

Here are some of the questions I received from LinkedIn on how to land a data science internship.

I knew how it felt to be working so hard but still not getting what you want.

But if I told you that experience or technical background may not matter so much to get a data science internship, would you be interested?

In this article, I will share how you could get your first data science internship.

Now, let’s ask yourself the questions below.

Why do you need to get a data science internship?

And then, change to your hiring manager’s perspective.

Why do you need to hire interns?

After asking these two questions, do you find a common answer that you and your hiring manager are looking for?

One of the answers to the first question would be to learn data science, to know what skills a data scientist should have so that you can learn. However, what is the qualifications, skills or etc which you could bring to the table so that the hiring manager will hire you?

Now, let’s picture two scenarios.

First Scenario

Imagine that the hiring manager is looking for interns who can help with some data annotation tasks and also evaluate various machine learning models. Meaning he or she is not looking for a person who has a strong technical background.

She might just want to find a person who is willing to work on some tasks that are not machine learning related. In other words, the hiring manager is looking for a person who has the personality that she is looking for.

For instance, given a data labeling task, you may think this is a very manual process. When your mentor is training models on your dataset and finds out that most of the labels annotated by you are incorrect, she will feel that she has wasted her time on trusting on your work.

But what if she is looking for a person who has a great personality to work with? Once she’s satisfied with the task given to you, I am sure she will be willing to teach you what you are interested in.

Second Scenario

Let’s say the hiring manager is searching for someone to assist her in solving some of the hardest problems she is facing. In other words, she wants to bring in someone who has strong technical skills and machine learning knowledge.

In this case, the candidate she required will have to have the ability to read research papers, coming out with great ideas to solve problems, clean codes, make the current codes more efficient and etc.

Therefore, the hiring manager would have higher expectations for the candidate. In a nutshell, the candidate is expected to equip with strong technical skills.

Personality Vs Technical Expertise

Now, I guess you have a clearer picture of what a hiring manager is expecting. Therefore, let me share how you should prepare yourself in order to get your first data science internship!

If you are a person, who does not have a strong technical background, here are some of my suggestions. First, you should demonstrate a good personality during your interview. What is considered a good personality to have? For instance, hard-working, willing to learn, passionate about machine learning, independent and etc.

Think of this as your hiring manager perspective, if you do not have a technical background, but as long you are willing to learn and put in your hard work, most of the time, he or she will be willing to give you a chance. This is provided that you are not applying to the internship which requires a strong technical background.

Besides, don’t be picky about the company or job scope. If you got the data science internship, maybe most of your time will be spent on manual tasks. However, if you manage to clear the task faster, and you show that your technical skills really improve a lot in a short period of time, your manager will give you the chance to try to train some models. On the other hand, don’t forget that this internship will bring you more internship opportunities after that.

Moreover, having a great project to demonstrate is also very important. What I am saying here is not school projects, but interesting real-world projects. If you are interested to know which kind of projects to show, comment below!

You may ask your coursemates or friends to assist you. Publishing your code to GitHub is not the end of the story. You should get someone who knows nothing about data science, and you present the project to your friend in such a way that he or she knows what you are talking about.

On the other hand, spend some time on networking events. If you are able to show them a great first impression, you will have a high chance to get an interview.

If you are someone who does have the technical expertise, but still can’t find an internship, here are some ways you can take note of. If you are aiming for those top companies, practice your programming skills.

There are many websites which you could refer to such as Leetcode, Topcoder, Coderbyte and etc. If you are a lazy type of person, ask your seniors, friends, or even check out glassdoor to see whether there is anyone who shares the interview question. You will be shocked that sometimes you could even find the answer to the interview question there.

Besides, practice, practice and practice your presentation. Don’t let your presentation screw up your great project. I couldn’t emphasize this enough because I saw too many interns screwed up because of their presentation during the actual interview. They might have interesting projects to show, but after they finish their presentation, I still can’t catch what they are trying to show. Remember, data science is not only about coding, but also to present your findings clearly.

If you know you are always unable to answer questions related to the machine learning knowledge, do spend time brushing up your knowledge.

Not only that, show interesting projects. Remember to clean up your codes and insert neat comments. Besides, do pay some attention to your resume. Once when I interviewed an intern, his or her portfolio link showed in the resume can’t be accessed.

Personality is still very important in this case. Show that you have a great personality to work with so that your potential boss will take you.

Final Thoughts

There is no perfect formula that exists to get a data science internship, but there are ways you can work out to get it!

You get what you work for, not what you wish for — Daniel Milstein

Thank you so much for reading until the end. If you have any questions, feel free to comment below!

Believe me, you will get one if you follow my advice!

All the best to get your first data science internship and wish you an early Merry Christmas! 🎄

About the Author

Low Wei Hong is a Data Scientist at Shopee. His experiences involved more on crawling websites, creating data pipeline and also implementing machine learning models on solving business problems.

He provides crawling services that can provide you with the accurate and cleaned data which you need. You can visit this website to view his portfolio and also to contact him for crawling services.

You can connect with him on LinkedIn and Medium.

Low Wei Hong - Medium

Story You May Like

• How you can learn machine learning
• 5 Things You Should Know Before Getting into Data Science

How to get a data science internship — Technical Expertise vs Personality was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.