Comparison: Azure Blob Storage vs Azure Data Lake Storage (ADLS) Gen2.

TL;DR:

Azure Data Lake Store Gen2 is a massively scalable, secure data lake functionality built on Azure Blob Storage which is designed for big data analytics and offers a hierarchical file system.

Azure Blob Storage

Azure Blob Storage is an object storage solution for the cloud. It is optimized for storing massive amounts of unstructured data, such as text or binary data. You can store large amounts of unstructured data (no restrictions on the kinds of data it can hold), in a single hierarchy, also known as a flat namespace.

Blob Storage can manage thousands of simultaneous uploads, enormous amounts of video data, constantly growing log files, and can be reached from anywhere with an internet connection via HTTP/HTTPS.

Blobs aren’t limited to common file formats. A blob could contain gigabytes of binary data streamed from a scientific instrument, an encrypted message for another application, or data in a custom format for an application. Azure takes care of the physical storage needs on your behalf.

Azure Data Lake Store (ADLS) Gen2

Azure Data Lake Storage is a comprehensive, scalable, and cost-effective data lake solution for high-performance big data analytics built into Azure.

Azure Data Lake Storage Gen1 is an enterprise-wide hyper-scale repository for big data analytic workloads. It enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage. It converges the capabilities of Azure Data Lake Storage Gen1 with Azure Blob storage.

ADLS Gen2 = Azure Blob Storage + ADLS Gen1

ADLS Gen2 provides file system semantics, file-level security, and scale, which are inherited from ADLS Gen1. All these capabilities are built on Blob storage resulting in low cost, tiered access, high security, high availability, and durability.

It is designed to manage and process multiple petabytes of information with hundreds of gigabits of throughput. A hierarchical namespace mechanism allows ADLS Gen2 to provide file system performance at object storage scale (optimizes I/O of high-volume data) and prices.

On Feb 29, 2024 Azure Data Lake Storage Gen1 will be retired. Migrate Azure Data Lake Storage from Gen1 to Gen2.

Key Differences

Structure
Blob: Flat namespace object store.
ADLS: Hierarchical namespaces (much like a File System).

Purpose
Blob: General purpose object store for a wide variety of storage scenarios, including big data analytics.
ADLS: Optimized storage for big data analytics workloads.

Performance (Analytics Workload)
Blob: Good storage retrieval performance.
ADLS: Better storage retrieval performance.

Cost
Blob: High cost for Analysis.
ADLS: Low cost for Analysis.

Hierarchical namespaces organize blob data into directories and stores metadata about each directory and the files within it. They keep the data organized, which yields better storage and retrieval performance for an analytical use case and lowers the cost of analysis. This structure allows operations, such as directory renames and deletes, to be performed in a single atomic operation.
Flat namespaces, by contrast, require several operations proportionate to the number of objects in the structure.

Use Cases

Blob storage is ideal for:

• Serving images or documents directly to a browser.
• Storing files for distributed access, such as installation.
• Streaming video and audio.
• Storing data for backup and restore, disaster recovery, and archiving.
• Writing to log files.
• Any type of text or binary data, such as application backend, backup data, and general purpose data.

ADLS is ideal for:

• Creating a modern data warehouse.
• Advanced analytics against big data.
• Creating a real-time analytical solution.
• Hadoop compatible access (HDFS, ABFS) is required. Access it through compute technologies including Azure Databricks, Azure HDInsight, and Azure Synapse Analytics without moving the data between environments.
• ACL and POSIX permissions along with some extra granularity support is required.
• Batch, interactive, streaming analytics, and machine learning data such as log files, IoT data, click streams, and large datasets.

Summary

ADLS Gen2 is built on Azure Blob storage capabilities to optimize it specifically for analytics workloads with hierarchical namespace support.

View more from Awesome Azure

• Difference between Traffic Manager and Front Door Service
• Difference between Azure Load Balancer and Application Gateway
• Difference between Azure Private Links and Azure Service Endpoints
• Difference between Azure SQL Database and Azure SQL Managed Instance
• Difference between Azure SQL Database and SQL Server on VM
• Difference between Azure Storage Queue and Service Bus Queue

Happy Clouding!

Azure — Difference between Azure Blob Storage and Azure Data Lake Storage (ADLS) was originally published in Awesome Azure on Medium, where people are continuing the conversation by highlighting and responding to this story.

Comparison — Azure Service Bus Queue vs Topic.

TL;DR:

Queues and Topics are similar when a sender sends messages, but messages are processed differently by a receiver. A queue can have only one consumer, whereas a topic can have multiple subscribers.

Difference between Azure Storage Queue and Service Bus Queue

Azure Service Bus Queue

A queue provides a one-to-one relationship between each message and its consumer. A queue allows processing of a message by a single consumer only.

Queues offer First In, First Out (FIFO) message delivery to one or more competing consumers. That is, receivers typically receive and process messages in the order in which they were added to the queue, which means the first message delivered into the Queue will be the first one to be retrieved by the receiver. And, only one message consumer receives and processes each message.

Azure Service Bus Topic

In contrast to queue, a topic (and subscriptions) provide a one-to-many relationship between each message and consumers. It follows publish and subscribe (pub/sub) pattern which is useful for scaling to large numbers of recipients. Each published message is made available to each subscription registered with the topic.

Publishers send messages to a topic in the same way that they send messages to a queue, but it varies on a slight factor where Topics can have multiple, independent subscriptions. A subscriber to a Topic can receive a copy of each message sent to that Topic.

Consumers don’t receive messages directly from the topic. Instead, consumers receive messages from subscriptions of the topic.

If you do not want all subscribers to read all the messages, then you can apply rules to specify additional actions to filter or modify message properties.

Subscriptions

A topic subscription resembles a virtual queue that receives copies of the messages that are sent to the topic. Consumers receive messages from a subscription identically to the way they receive messages from a queue.

The message-sending functionality of a queue map directly to a topic and its message-receiving functionality maps to a subscription.

Important: Topic itself is not enough, you also need Subscriptions.

Key Differences

Consumer options
Queue: It is possible to have multiple receivers, but each message will be received only by one of those receivers.
Topic: There are multiple receivers (subscribers) to receive the messages, each message copy can be sent to any number of subscribers associated with the Topic.

Message Filtering
Queue: Queue does not essentially require any filters as the messages are being received by only one user.
Topic: Each message that is sent over a Topic can have a collection of properties. Properties are used when a custom filter per subscription is applied.

Topic subscriptions can enable powerful filtering capabilities, such that you can define certain parameters that messages must meet in order to be copied into the subscriptions virtual queue. This can be handy if you need to handle different types of messages or messages with various data structures on the same topic.

Consumer Scalability
Queue: If you need to scale a queue, you’re still limited to having one consumer listening.
Topic: Topics can be more scalable than queues, as more than one consumer can listen to messages. We don’t need to recreate the Topic. Once a new subscription is created, all the new messages that are sent to that Topic will be received by the new subscription also.

Message Auto-forwarding
Queue: Messages can be auto-forwarded to a specified Queue or Topic. But the destination cannot be a Topic Subscription.
Topic: Messages can be auto-forwarded from a Topic to a Queue or Topic. Also, it is possible to forward the messages from a Topic Subscription to a Queue or Topic.

Message Removal
Queue: The first receiver that completes the reading of the message also removes it from the Queue so that the message cannot be processed by other readers.
Topics: The message is removed only after every receiver has processed the message. When the message is read from all the Subscribers, the message is removed from the Topic too.

Message Locking and Deleting — Both queues and topic subscriptions support PeekLock and ReceiveAndDelete modes, so it ensures a message is processed before being dismissed if required.

Use Cases

Choose Service Bus Queues:

• When there is a need to pass the message in the one-to-one system.

Choose Service Bus Topics:

• When there is a need to send the message to multiple systems.

Summary

Azure Service Bus Queues and Topics can be used to send and receive messages to implement asynchronous communication and improve durability. A queue is often used for point-to-point communication, whereas a topic is useful in publish/subscribe scenarios.

Download sample source: azure-servicebus-labs

View more from Awesome Azure

• Difference between Azure Blob Storage and Azure Data Lake Storage (ADLS)
• Difference between Azure Load Balancer and Application Gateway
• Difference between Traffic Manager and Front Door Service
• Difference between Azure Private Links and Azure Service Endpoints
• Difference between Azure SQL Database and Azure SQL Managed Instance (SQL MI)
• Difference between Azure SQL Database and SQL Server on VM
• Difference between Azure Storage Queue and Service Bus Queue

Happy Clouding!!!

Azure — Difference between Azure Service Bus Queues and Topics was originally published in Awesome Azure on Medium, where people are continuing the conversation by highlighting and responding to this story.