AWS data pipeline services
Let’s dive a bit deeper into the AWS services used in a typical AWS data pipeline.
Data pipelines for ETL processes
Both AI/ML and traditional data analytics need clean and accessible data in a format that’s usable by analytics tools and AI algorithms. ETL processes are used for this purpose. With ETL, you perform the following steps:
- Extract the data from various sources and store it.
- Transform it into a consistent, usable format for downstream tools to consume.
- Load it into a destination system, like a data warehouse or analytics platform.
Data pipelines are automated assembly lines used to make the ETL process efficient and repeatable. AWS has a suite of integrated services so you can build your own data pipelines.
1. Data ingestion services
Data ingestion involves moving data from source systems into your chosen storage solution. Use real-time ingestion when the data is needed immediately. Use batch ingestion when some latency is tolerable.
Amazon Kinesis Data Streams
You can use Kinesis Data Streams for real-time ingestion of terabytes of data from applications, streams, and sensors. This serverless service even provides automatic provisioning and scaling in on-demand mode.
Amazon Data Firehose
Firehose is an option for data ingestion in near real-time. This fully managed service provides automatic provisioning and scaling. It also delivers data within seconds to data lakes, warehouses, and analytics services.
2. Data storage services
Data can come from many different sources. To gain insights, data is commonly consolidated into a single location. There are two storage options for this. Flexible data lakes store vast amounts of raw data. Alternatively, the more structured data warehouses are optimized for business intelligence.
Amazon S3
Amazon S3 is a popular choice for data lakes. This object storage service can securely house virtually any amount of structured or unstructured data. Amazon S3 is also fully elastic, automatically scaling as you add and remove data.
Amazon Redshift
Amazon Redshift is a fully managed data warehouse service that can store petabytes of structured or semistructured data. With the scalability and pay-as-you-go pricing model, organizations can cost-effectively analyze large datasets.
3. Data cataloging services
Cataloging your data with metadata provides an inventory of your organization’s data.
Get Divyapriya’s stories in your inbox
Join Medium for free to get updates from this writer.
AWS Glue Data Catalog
AWS Glue Data Catalog provides a centralized, scalable, and managed metadata repository that enhances data discovery. It improves the overall efficiency of data pipelines by delivering metadata to various data stores and analytics services.
4. Data processing services
Data processing services clean and transform your data so it’s ready to be analyzed.
AWS Glue
AWS Glue is a fully managed ETL service that makes data preparation simpler, faster, and cost effective. AWS Glue ETL jobs can use the AWS Glue Data Catalog to access metadata about data sources, which can help inform transformations defined in the ETL script.
Amazon EMR
Amazon EMR is ideal for large-scale data processing and organizations with existing big data expertise. It automatically handles infrastructure provisioning, cluster management, and scaling. Amazon EMR supports popular big data frameworks like Apache Spark, Apache Hadoop, and Apache Hive.
5. Data analysis and visualization services
Queries and visualization tools help you to develop important insights about your data.
Amazon Athena
With Athena, you can run SQL queries to analyze data in relational, nonrelational, object, and custom data sources. This fully managed serverless service can access data hosted on Amazon S3, on premises, or even in multi-cloud environments. It offers a cost-effective solution for data analysis because you only pay for the queries you run.
Amazon Redshift
Amazon Redshift is a fully managed data warehouse solution. Its columnar storage and massively parallel processing architecture make it ideal for analyzing large datasets. You can use it to perform complex SQL queries on large datasets for frequent, high-performance analytical workloads.
Amazon QuickSight
With QuickSight, both technical and non-technical users can quickly create modern interactive dashboards and reports from various data sources without managing infrastructure. Amazon Q in QuickSight provides natural language queries so business analysts and users can build, discover, and share meaningful insights in seconds.
Amazon OpenSearch Service
With OpenSearch Service, you can search for relevant content through precise keyword matching or natural language queries. Unified dashboards provide real-time data visualization as you analyze and monitor logs, traces, and metrics for various applications.
Real-world Example
Delivering customer data for analysis and ML model training
Let’s review the solution discussed in the video. An e-commerce company uses an automated data pipeline to ingest, process, and deliver data to multiple stakeholders. As a result, data scientists and ML engineers can use the same data set for analysis and ML model training.
