Part 1: Data Acquisition - Gathering Your Ingredients (Data) for Success
Welcome back to the Data Mining Series!
You can view my introduction to the series by clicking the link below: Welcome to the Data Mining Series: Transforming Raw Data into Powerful Insights
Introduction
Welcome to the first part in our Data Mining Series, where we go in-depth into some of the most important initial steps in the whole process of data mining: Data Acquisition.
Think of it this way: if you are going to cook, you need to get all of the ingredients together first. Similarly, you will need to gather the right sort of data to make something out of it. Without strong data, the insights will remain hidden.
The Cooking Analogy
Now, picture that you are going to make a complex dish. You haven’t even started cooking, yet you’re running from one store to another gathering all the ingredients. Fresh vegetables are bought at a local market, dairy products at the supermarket, and spices at some sort of specialty store. Once you’ve gathered everything, it’s crucial to store them properly: dairy in the fridge, spices in the pantry, and fresh produce on the counter.
Data acquisition follows the same principle. It is the process of gathering raw data from several sources, and much like cooking, your output depends on the quality of the ingredient (data) that you put in. There are different varieties of data, such as structured or semi-structured, that would require different kinds of storage and processing; much like the various types of food you have in your kitchen.
Data Types: The Three Categories
Not all data is created equal, and knowing the different types will inform your choices on what methods are appropriate for collection and processing. So, here is a breakdown of the three main data types:
1. Structured Data
Think of structured data as the contents of a pantry that is highly organized: each item labeled and in its place, say boxes of pasta or cans of soup. Structured data fits perfectly into rows and columns and therefore is easy to store and analyze.
- Examples: Customer information, product inventories, sales records
- When to Use: When you want clean, organized, and accessible data, such as financial analysis or if you are running a CRM system.
- Stored In: Spreadsheets (e.g., Excel), relational databases (e.g., MySQL, PostgreSQL).
2. Unstructured Data
Unstructured data is like mystery ingredients strewn about in your kitchen. Think of opening the fridge to pull out a jar of sauce or leftovers without the pieces matching together well. It’s messy, scattered, and will not sit in neat rows or columns. It can be text, image, or video.
- Examples: Social media posts, product reviews, video content.
- When to Use: You work with free-text data, for example, to perform sentiment analysis on social media or extract key phrases from customer reviews.
- Stored In: Document stores (e.g., NoSQL databases like MongoDB), cloud storage (e.g., AWS S3 for media files).
3. Semi-Structured Data
Semi-structured data are the labeled spice jars: the Spice jars have some organization, but their contents can vary. It doesn’t fit neatly into traditional tables, nor is it completely unregulated. Semi-structured data generally consists of tags or key-value pairs.
- Examples: XML, JSON files, or email metadata
- When to Use: Semi-structured data is best to work with when using web data exchanges, such as with APIs, or if you’re using hierarchical data formats.
- Stored In: NoSQL databases (e.g., MongoDB), key-value stores, or document-based systems.
Methods of Data Collection: How to Gather Your Ingredients (aka Data)
Just as you source your ingredients from various stores when cooking, in data acquisition you may collect data from a variety of locations. Here’s how:
1. Web Scraping
Web scraping is like walking into several stores and choosing exactly what is needed. It is one way of automatic accessing of data on websites.
# Web Scraping Example: Scraping Product Titles from Amazon (for educational purposes only)
import requests
from bs4 import BeautifulSoup
# Function to scrape product titles from Amazon search results
def scrape_amazon(url):
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
# Scraping product titles
titles = soup.find_all('span', class_='a-size-medium a-color-base a-text-normal')
product_titles = [title.get_text().strip() for title in titles]
return product_titles
# Example usage
url = 'https://www.amazon.com/s?k=laptops' # Replace with any Amazon search URL
amazon_product_titles = scrape_amazon(url)
print("Amazon Product Titles:", amazon_product_titles)- Explanation: In this case, we’re “picking out” product titles from an Amazon search results page, much like selecting the best items to add to our shopping cart.
- Use Case: Ideal for pulling data from websites like product reviews, stock prices, or tracking social media trends.
- Tools: Popular tools include BeautifulSoup, Scrapy, and Selenium.
2. APIs (Application Programming Interfaces)
Get Farhat Shaikh’s stories in your inbox
Join Medium for free to get updates from this writer.
APIs are actually a lot like ordering groceries online: you send a request to a website-the “grocery store”-and it sends you back the data in a clean, structured format.
# API Data Collection Example: Fetching weather data from OpenWeather API
import requests
# Function to fetch weather data from OpenWeather API
def fetch_weather(city, api_key):
api_url = f'http://api.openweathermap.org/data/2.5/weather?q={city}&appid={api_key}'
response = requests.get(api_url)
if response.status_code == 200:
return response.json()
else:
return None
# Example usage (replace with your OpenWeather API key)
api_key = 'YOUR_API_KEY' # Sign up at openweathermap.org
city = 'London'
weather_data = fetch_weather(city, api_key)
print("Weather Data:", weather_data)- Explanation: Here, we’re sending a request to the OpenWeather API to fetch weather data, which is like ordering weather data directly from an online store.
- Use Case: Use APIs when you need either real-time or historical data from such platforms as Twitter, Reddit, or weather services.
- Tools: Examples include the Twitter API, Reddit API, or Google Maps API.
3. Manual Collection
Manual collection is going down to the market and picking items out for yourself. It’s a very traditional method of data gathering where, for the most part, manually collecting can be extremely time-consuming.
# Manual Data Collection Example: Creating a manual DataFrame using Pandas
import pandas as pd
# Create manual data
def create_manual_data():
data = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [24, 30, 29],
'City': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)
return df
# Example usage
manual_data = create_manual_data()
print("Manual Data Collection:\n", manual_data)
## Or you can fill out the form, excel, etc by yourself- Explanation: Here, we manually create a dataset using Pandas, much like how you might manually create a shopping list based on your preferences.
- Use Case: Great to gather specific data via survey or interview.
- Tools: Google Forms, Typeform, and SurveyMonkey are great to create forms and collect responses.
4. Open Data Sources
Open data sources are like visiting a farmer’s market; everything is free. These are publicly available datasets that anyone can access.
# Access Kaggle Dataset
!pip install kaggle# Open Data Source Example: Loading data from Kaggle using Kaggle API
import pandas as pd
# Download Titanic dataset from Kaggle
!kaggle competitions download -c titanic
# Unzip the dataset
!unzip titanic.zip
# Load the dataset
titanic_data = pd.read_csv('train.csv')
print("Titanic Dataset (first 5 rows):\n", titanic_data.head())- Explanation: We’re downloading a dataset from Kaggle (like going to a public market) and using it in our project.
- Use Case: It is ideal for practice in data analysis or projects where there is no need for private data.
- Sources: Refer to Kaggle, UCI Machine Learning Repository, or government data portals.
5. Data Augmentation
It’s like putting spice into a dish. You have your data, but now you’re spicing it up, making it more variable, improving the quality.
# Data Augmentation Example: Augmenting image data using TensorFlow
import tensorflow as tf
import matplotlib.pyplot as plt
# Load a sample image
(image, label), _ = tf.keras.datasets.cifar10.load_data()
image = image[0]
# Data augmentation function
def augment_image(image):
image = tf.image.random_flip_left_right(image)
image = tf.image.random_flip_up_down(image)
image = tf.image.random_brightness(image, max_delta=0.5)
image = tf.image.random_contrast(image, lower=0.2, upper=0.5)
return image
# Original and augmented images
augmented_image = augment_image(image)
# Display the original and augmented images
plt.subplot(1, 2, 1)
plt.imshow(image)
plt.title("Original Image")
plt.subplot(1, 2, 2)
plt.imshow(augmented_image)
plt.title("Augmented Image")
plt.show()- Explanation: Here, we are “spicing up” an image dataset by augmenting it with random transformations, such as flipping or adjusting brightness.
- Use Case: Very useful in the case of machine learning projects where you would want to have more data with slight modification in already available data.
- Tools: Python libraries like imgaug for images and NLTK for text are excellent while augmenting your dataset.
Challenges in Data Acquisition: The Unexpected Hurdles
Not all ingredients go bad or are hard to find; data collection itself has its problems. Some of those to watch out for include:
1. Data Privacy and Ethics
- Problem: That wouldn’t be cool at all, right — just picking ingredients out of somebody’s garden when they’re not asking you to? The same thing with data: collecting private or sensitive data without permission will get some serious legal consequences.
- Solution: Always follow data protection regulations such as the GDPR (General Data Protection Regulation) and the CCPA (California Consumer Privacy Act). Consent should be explicitly given by a user, and ethical policies must be followed while handling his or her information.
2. Quality of Data
- Problem: Ever bought ingredients to prepare a dish and later found them stale? Just like that, poor-quality data screws up your analysis.
- Solution: Cleanse your data prior to analysis. Data that is complete, accurate, and consistent will not lead to incorrect results.
3. Accessibility of Data
- Problem: Sometimes, a store would require a special membership or may simply be closed. In the world of data, restrictions to access stand in the way of your getting the data that you want.
- Solution: If the data provider provides API access, look for open datasets. Be sure you have legal rights to data you are collecting.
Final Thoughts
Just like preparing a delicious meal begins with sourcing the right ingredients, successful data mining starts with solid data acquisition. Understanding the different types of data and knowing how to collect it properly is key to uncovering valuable insights.
So, let’s cook and brew the data together, transforming raw ingredients into powerful insights!
For more hands-on practice in Data Acquisition, check out these two lab exercises (Credits: Data Mining Class @ASU — Dr. Hua Wei):
🔗 Beautiful Soup & Selenium for IMDb Review Crawling
🔗 Tweepy Twitter API
Stay tuned for the next part of our series, where we dive into ‘Data Preprocessing’ — the step where we transform our raw ingredients (data) into something ready for analysis.
