How Data Is Generated and Collected in Data Analytics
By: Asfand Gul Kasi
Sources of Data
Data is generated every second across the world through countless human activities and digital interactions. Every email sent, online search performed, social media post shared and video watched contributes to the ever-growing volume of data. This enormous amount of information has become one of the most valuable resources for businesses, governments, researchers and organizations. Understanding how data is generated and collected is an essential skill for every data analyst because it provides valuable context for interpreting and analyzing information accurately.
Methods of Data Generation
Data is created whenever information is recorded or captured. In today’s digital world, smartphones, websites, social media platforms, and online services continuously generate data. Even a single digital photograph contains multiple layers of information, including image size, color values, location data and the time it was taken.
Data can also be generated intentionally through structured collection methods. Organizations frequently gather information using surveys, questionnaires, interviews, observations and official forms. For example, governments conduct national population censuses to collect demographic information that supports decisions about funding schools, hospitals, transportation and other public services. Businesses also collect customer feedback through surveys to improve products and services based on real user experiences.
Collecting Data Responsibly
Collecting data involves more than simply gathering information. Data must be collected ethically while respecting privacy, security, and individual rights. Organizations should ensure that personal information is protected and that participants understand how their data will be used. Ethical data collection builds trust and improves the quality of the information gathered.
Another common method of collecting online data involves the use of cookies. Cookies are small files stored on users’ devices that remember preferences, browsing behavior and website settings. Although cookies help improve user experience and provide businesses with valuable insights into customer behavior, they generally do not personally identify users and should always be managed responsibly.
The Importance of Understanding Data Sources
Knowing where data comes from helps analysts evaluate its quality, reliability and relevance. Different collection methods may introduce different strengths and limitations into a dataset. Survey responses, interviews, observations, and automatically generated digital records each provide unique insights but may also contain biases or missing information.
Key Considerations for Collecting Data in Data Analytics
Choosing the Right Data for Analysis
Data is generated continuously through countless activities, making it one of the most valuable resources for organizations. However, a data analyst cannot use every available dataset. Instead, analysts must carefully select data that is relevant to the business problem they are trying to solve. The business objective or project requirements usually help narrow down the choices, allowing analysts to focus on the most useful and reliable information. Selecting appropriate data is the foundation of accurate analysis and informed decision-making.
Different Data Sources
One of the first considerations in data collection is identifying where the data comes from. First-party data is collected directly by an individual or organization using its own resources. Because the source is known, this type of data is generally the most reliable and preferred for analysis.
Second-party data is obtained from another organization that collected the information directly from its own audience. Although the analyst did not gather it personally, the data can still be highly trustworthy if it comes from a reputable source.
Third-party data is collected and distributed by organizations that did not originally gather the information. Since it may pass through multiple sources before reaching the analyst, it requires careful evaluation for accuracy, credibility, bias and reliability before being used.
Population, Sampling and Data Types
Another important consideration is deciding how much data should be collected. A population includes every possible data point relevant to a study, while a sample represents a smaller subset of that population. Because collecting data from an entire population is often expensive and time-consuming, analysts commonly use representative samples that accurately reflect the larger population. Proper sampling allows analysts to draw meaningful conclusions while using fewer resources.
Selecting the correct data type is equally essential. For example, when analyzing traffic patterns, storing information in date and time formats allows analysts to identify trends such as peak traffic hours, busy weekdays or seasonal variations.
Determining the Time Frame
Analysts must also decide whether to use historical or newly collected data. Historical data is useful when immediate analysis is required because it already exists and is readily available. However, if the objective is to monitor future trends or long-term changes, analysts may need to collect new data over an extended period.
Data Types and Formats in Data Analytics
Why Data Types Matter
Data analysts work with a wide variety of information to solve business problems and support decision-making. To analyze data effectively, they must first understand the different types and formats of data available. Identifying whether data is qualitative or quantitative, structured or unstructured, and internal or external helps analysts select the appropriate analytical methods and produce accurate insights. A strong understanding of data types improves data organization, interpretation and overall analysis quality.
Qualitative and Quantitative Data
Data can be broadly classified into qualitative and quantitative categories. Qualitative data describes qualities or characteristics and cannot easily be measured using numbers. Examples include movie titles, actor names, product categories, or customer comments. This type of data is usually represented by names, labels or descriptions.
Quantitative data, on the other hand, represents measurable values expressed as numbers. Examples include movie budgets, box office revenue, sales figures, or customer ages. Quantitative data can be analyzed mathematically to identify trends, averages and patterns.
Quantitative data is further divided into discrete and continuous data. Discrete data consists of countable values with fixed increments, such as the number of products sold or the budget of a movie. Continuous data represents measurements that can take any value within a range, including decimal values, such as movie runtime, temperature or distance.
Nominal, Ordinal and Data Sources
Qualitative data can also be classified as nominal or ordinal. Nominal data consists of categories without any specific order, such as “Yes,” “No,” or “Not Sure.” Ordinal data, however, follows a meaningful sequence, such as customer satisfaction ratings from 1 to 5 or movie rankings.
Another important distinction is between internal and external data. Internal data is generated and stored within an organization’s own systems, making it easier to access and generally more reliable. External data originates outside the organization and may come from government agencies, research institutions, or third-party providers. Although external data expands analytical possibilities, it should always be evaluated for accuracy and credibility.
Structured and Unstructured Data
Most data analysts work with structured data, which is organized into rows and columns within spreadsheets, tables, or relational databases. Structured data is easy to search, manage and analyze efficiently.
In contrast, unstructured data does not follow a predefined format. Examples include audio recordings, videos, emails, social media posts, and images. While unstructured data often contains valuable insights, it requires specialized tools and techniques for processing and analysis.
Structured and Unstructured Data in Data Analytics
The Importance of Data Structure
Data exists in many forms and is generated continuously through digital activities, business operations, and human interactions. To analyze information effectively, data analysts must understand how data is organized. The two primary categories are structured data and unstructured data. Each type has unique characteristics, advantages and challenges. Choosing the appropriate data structure helps analysts store, search, process and interpret information more efficiently while supporting accurate business decisions.
Structured Data
Structured data is organized into a predefined format, typically using rows and columns in spreadsheets or relational databases. Each data field has a defined data type, making it easy to store, search, sort and analyze. Structured data is usually quantitative and follows consistent formatting standards, allowing analysts to perform calculations and generate reports quickly.
Common examples of structured data include customer records, financial transactions, sales reports, phone records, inventory lists, and spreadsheets created with applications such as Microsoft Excel, Google Sheets, or SQL databases. Because the structure remains consistent even when the data is exported, structured data is highly reliable for business reporting and decision-making.
Unstructured Data
Unlike structured data, unstructured data does not follow a predefined format and cannot easily be organized into rows and columns. It often consists of qualitative information that requires specialized tools for processing and analysis. Examples include text messages, emails, social media posts, phone call recordings, images, videos, audio files, PDF documents, websites and open-ended survey responses.
Although unstructured data is more difficult to search and analyze, it provides valuable insights into customer opinions, behaviors, and experiences. Modern technologies such as artificial intelligence (AI) and machine learning have significantly improved the ability to process and analyze large volumes of unstructured data, making it increasingly valuable for businesses.
Fairness and Data Analysis
As organizations rely more heavily on artificial intelligence and machine learning to analyze unstructured data, ensuring fairness becomes increasingly important. If datasets are incomplete, unbalanced, or biased, analytical models may produce inaccurate or unfair results. Certain groups or viewpoints may become overrepresented while others are overlooked, leading to misleading conclusions and poor business decisions. Data analysts must carefully evaluate data quality, representativeness and potential sources of bias before performing analysis.
Data Modeling: Levels, Techniques and Importance
What is Data Modeling?
Data modeling is the process of creating visual diagrams that represent how data is organized, stored, and connected within a system. These diagrams, known as data models, serve as blueprints that help organizations understand the structure and relationships of their data. Just as architects use blueprints to guide the construction of a building, data professionals use data models to design and manage databases effectively. Although junior data analysts are not typically responsible for creating data models, understanding them is essential for interpreting and working with organizational data.
Levels of Data Modeling
Data modeling is commonly divided into three levels, each providing a different degree of detail.
Conceptual Data Model
The conceptual data model offers a high-level overview of an organization’s data and how different business entities relate to one another. It focuses on business requirements rather than technical details. This model helps stakeholders understand the overall structure of the data without considering database implementation.
Logical Data Model
The logical data model provides a more detailed representation of the data by defining entities, attributes, relationships, and unique identifiers. It explains how data is logically organized within the database but does not specify actual table names or database-specific structures. This level serves as a bridge between business requirements and technical implementation.
Physical Data Model
The physical data model is the most detailed level. It describes how the database is actually implemented, including table names, column names, data types, indexes and storage details. Database administrators and developers use this model to build and maintain database systems.
Common Data Modeling Techniques
Organizations use various techniques to create data models, with Entity Relationship Diagrams (ERDs) and Unified Modeling Language (UML) diagrams being the most common.
Get Asfand Gul Kasi’s stories in your inbox
Join Medium for free to get updates from this writer.
An Entity Relationship Diagram (ERD) visually illustrates the relationships between different entities in a database, making it easier to understand how data is connected. A UML diagram provides a more comprehensive view by describing entities, their attributes, operations and relationships within a complete system. While junior data analysts may not design these diagrams, they often work with existing models created by their organizations.
Importance of Data Modeling
Data modeling plays a critical role in maintaining data consistency, improving collaboration, and supporting efficient data analysis. Well-designed data models help analysts understand how information flows across an organization, making it easier to locate, interpret and analyze data accurately. They also promote better communication among analysts, database developers, business stakeholders, and management by providing a shared understanding of data structures.
Data Types in Spreadsheets for Data Analytics
What Are Data Types?
Data types are one of the fundamental concepts in data analytics. A data type is a specific classification that defines the kind of value stored in a cell or database field. Understanding data types helps analysts organize data correctly, perform accurate calculations, and reduce errors during analysis. While programming languages and database systems may support many different data types, spreadsheets commonly use three primary types: numbers, text (strings) and Boolean values.
Number Data Type
The number data type represents numerical values that can be used in mathematical calculations. Examples include sales figures, temperatures, percentages, budgets, and search interest scores. In spreadsheets, number data can also be formatted as currency, percentages, dates or decimals without changing the underlying value.
For example, a dataset tracking worldwide Google searches for sweets may assign a value between 0 and 100 to indicate the popularity of search terms such as cupcakes, ice cream, and candy. These values can be averaged, summed or compared using spreadsheet formulas, making numerical data essential for statistical analysis.
Text (String) Data Type
The text, or string, data type contains letters, words, symbols, and punctuation used to describe information rather than calculate it. Examples include customer names, product names, addresses, city names and categories.
Although text values may contain numbers, such as phone numbers or postal codes, they are treated as text because they are identifiers rather than values for mathematical operations. This distinction helps analysts avoid calculation errors and maintain data accuracy.
Boolean Data Type
The Boolean data type has only two possible values: TRUE or FALSE. It is commonly produced by logical formulas that evaluate whether a condition has been met.
For instance, a spreadsheet may check whether a search interest score is greater than or equal to 50. If the condition is satisfied, the result is TRUE; otherwise, it returns FALSE. Boolean values are especially useful for filtering data, creating conditions, automating tasks and supporting decision-making.
Avoiding Data Type Errors
One common mistake in spreadsheets is confusing data types with cell values. Spreadsheet formulas are designed to work with specific data types. Mathematical functions such as SUM or AVERAGE require numerical data, while attempting to calculate text values usually produces error messages. Understanding the correct data type before performing calculations helps prevent errors and improves analysis accuracy.
Boolean Logic in Data Analytics
What Is Boolean Logic?
Boolean logic is a fundamental concept in data analytics, programming, and database management. It is a system of logical reasoning that uses operators to evaluate whether conditions are true or false. Data analysts use Boolean logic to filter data, write search queries, create formulas, and automate decision-making processes. The three primary Boolean operators are AND, OR, and NOT. By combining these operators, analysts can create powerful conditions that retrieve only the data relevant to a specific task.
The AND Operator
The AND operator requires that all specified conditions be true before a result is returned. It narrows the search by selecting only records that satisfy every condition.
For example, suppose you want to buy shoes that are both grey and pink. The Boolean statement would be:
IF (Color = “Grey”) AND (Color = “Pink”)
Only shoes containing both colors meet this condition. If either color is missing, the result is false. In data analysis, the AND operator is commonly used to apply multiple filters simultaneously, ensuring highly specific results.
The OR Operator
The OR operator returns a result if at least one of the specified conditions is true. This operator broadens the search by accepting records that meet either condition.
For example:
IF (Color = “Grey”) OR (Color = “Pink”)
This statement selects shoes that are grey, pink, or both. The OR operator is useful when analysts need data that matches any one of several possible conditions, making searches more flexible.
The NOT Operator
The NOT operator excludes records that match a particular condition. It reverses the logical value of a condition, making true become false and false become true.
For example:
IF (Color = “Grey”) AND (NOT Color = “Pink”)
This statement returns only grey shoes that do not contain any pink. The NOT operator is valuable when analysts need to remove unwanted records from a dataset while preserving all other matching data.
Combining Multiple Conditions
One of Boolean logic’s greatest strengths is the ability to combine multiple operators into a single statement. Parentheses help group conditions and control the order of evaluation.
For example:
IF ((Color = “Grey”) OR (Color = “Pink”)) AND (Waterproof = TRUE)
This statement selects waterproof shoes that are either grey or pink. Such combined conditions allow analysts to build complex filters that accurately answer business questions.
Data Tables: Records, Fields, and Values in Data Analytics
Introduction to Data Tables
Data tables are one of the most common ways to organize and store structured information. Whether you are viewing an email inbox, a music playlist, a calendar, or a business spreadsheet, you are interacting with data arranged in a table. A data table, also known as tabular data, organizes information into rows and columns, making it easy to search, analyze, and manage. Understanding the structure of data tables is a fundamental skill for every data analyst because it provides the foundation for working with databases, spreadsheets and analytical tools.
Records and Fields
In data analytics, rows are commonly referred to as records, while columns are known as fields. Although the terms rows and columns are widely used in spreadsheets, records and fields are more common when discussing databases and structured data systems.
A record represents a complete entry containing related information. For example, in a music playlist, each song is a record. Similarly, in a customer database, each customer represents one record. Every record contains the same set of fields arranged in the same order.
A field represents a specific characteristic or attribute of a record. In a playlist, fields may include the song title, artist, album, duration and favorite status. Each field stores a particular type of information that describes the record.
Data Types in Fields
Every field is associated with a specific data type, ensuring that information is stored consistently. Text fields contain names or descriptions, numerical fields store values used for calculations, and Boolean fields contain only two possible values, such as True or False.
For example, the song title and artist fields use text data types, while the song length may use a numerical or time-based data type. A favorite field typically uses a Boolean data type because it simply indicates whether a song is marked as a favorite.
Values in Data Tables
The individual pieces of information stored within each cell of a table are called values. Every value belongs to both a record and a field. For example, a customer’s address, an invoice amount or a product price is a value stored within the appropriate field. These values are the actual data that analysts examine, calculate and interpret during analysis.
Wide and Long Data Formats in Data Analysis
Data organization is a fundamental aspect of data analytics because the way information is structured directly affects how it can be analyzed and interpreted. Two of the most common data structures used in spreadsheets and databases are wide data and long data. Each format serves a different purpose and helps analysts answer different types of questions more efficiently.
Wide data is a format where each subject or entity occupies a single row, while different attributes or measurements are stored in separate columns. In the population dataset for Latin American and Caribbean countries from 2010 to 2019, each country is listed on one row and the population for each year is displayed in its own column. This layout makes it easy to compare the same attribute across multiple subjects. For example, you can quickly compare Argentina’s population with Brazil’s, the Bahamas’, or any other country’s population for a specific year. Spreadsheet tools such as sorting further enhance analysis. By sorting the 2010 population column from highest to lowest, Brazil appears at the top because it had the largest population. Similarly, sorting the 2013 population column from lowest to highest reveals the British Virgin Islands as the country with the smallest population for that year.
In contrast, long data stores each observation as a separate row. Instead of having a different column for each year, all years are combined into a single “Year” column, while the corresponding population values appear in another column. As a result, each country is represented by multiple rows one for every year in the dataset. For instance, Argentina’s population from 2010 through 2019 occupies ten consecutive rows rather than a single row.
The long data format is particularly useful for analyzing changes over time, identifying trends, and creating visualizations such as line charts. Because every observation follows the same structure, it is easier to filter, group, summarize and perform statistical analyses across different years and countries.
Understanding when to use wide or long data is an essential skill for data analysts. Wide data is best suited for comparing values across subjects at a specific point in time, while long data is ideal for tracking changes, analyzing trends, and performing advanced analytical tasks. Mastering both formats enables analysts to organize data effectively and answer a broader range of business and research questions with confidence.
Wide and Long Data Formats in Data Analytics
Data can be organized in different ways depending on the type of analysis being performed. Two of the most common formats used by data analysts are wide data and long data. Although the terms “wide” and “long” are often used to describe physical objects, they also describe how information is structured in datasets. Understanding these formats is essential because each serves different analytical purposes.
Wide data is a format in which each data subject occupies a single row, while multiple columns store different attributes or measurements. For example, in a population dataset of Latin American and Caribbean countries, each row represents one country and each column represents the population for a different year. This structure makes it easy to compare values across different subjects. Analysts can quickly examine the population of Argentina, Brazil or the Bahamas for a particular year simply by reading across the corresponding row. Wide data is also convenient for sorting and comparing values. By sorting the population column for 2010, it becomes clear that Brazil had the highest population, while sorting the 2013 data reveals that the British Virgin Islands had the lowest population that year.
In contrast, long data organizes information differently. Instead of assigning a separate column to each year, all years are stored in a single “Year” column, while the corresponding population values are placed in another column. As a result, each country appears in multiple rows, with each row representing one observation at a specific point in time. For example, Argentina’s population from 2010 to 2019 is recorded in ten separate rows. This structure is especially useful for analyzing trends, monitoring changes over time and handling datasets with multiple variables.
One major advantage of long data is its flexibility. If a new variable, such as the average age of the population, needs to be added, only one additional column is required. In a wide dataset, however, adding the same variable for ten years would require ten new columns, making the spreadsheet larger and more difficult to manage. Long data keeps datasets compact, organized, and easier to analyze using statistical tools and visualization software.
Choosing between wide and long data depends on the specific analytical task. Wide data is ideal for comparing values across subjects at a single point in time, while long data is better suited for tracking changes and identifying trends over time. Since data analysts frequently transform data between these two formats, mastering both structures is an essential skill. Understanding how data is organized allows analysts to work more efficiently, perform accurate analyses and make informed, data-driven decisions.
