Databricks

->Databricks is a cloud platform mainly used for data engineering, big-data processing, analytics, and AI/ML.

Some imp terms in Databricks :
1) Data warehouse : A centralized system that stores structured, processed data for reporting and analytics.

2) ETL: Extract, Transform, Load — takes data from different sources, cleans/transforms it, and loads it into a data system.

3) Data Lake: A large storage system that keeps raw data in many formats such as CSV, JSON, logs, images, etc.

4) AI/ML : Uses data to build models that can predict, classify, recommend, or automate decisions.

5) BI dashboards : Converts data into charts, graphs, reports, and dashboards so business users can understand it.

6) Governence : Defines rules, security, ownership, quality, privacy, and access for data.

7) Orchestration : Coordinates and schedules data workflows so tasks happen in the correct order and at the correct time.

SECOND CHALLENGE:

-> Now there is a situtation in a company where we are using propreitary solution which means a software solution which will give us the benefit of data warehousing

-> and we have to store data in their propreitary format which means they would encode the data in their format.

-> now we cannot take out the data without using using thier data engine.

-> this process is called as vendor locking .

-> To resolve this problem databricks will use open source platform where data will lie in our data cloud platform.( can be any format)

-> Delta lake will be used here it will communicate with our data and also provide us the output .

-> Now in future if we don’t like Delta lake we can remove this vendor and can also use another vendor ( openness is great advantage for us) .

THIRD CHALLENGE:

-> AI/ ML amd ETL helps us to get the data out of the data lake
-> BI tools are dealing with data warehousing solutions
-> now we have the same copy of data in data lake and in data warehouse
-> owners of the data are also different
-> To overcome this problem databricks has merged both of this and crerated a new thing which is called data wehouse .

This is how it should work

Complete Architecture of the Databricks

->Data Intelligence = Data Lakehouse + Generative AI
->Data lakehouse = Data warehouse + Data Lake

HIGH LEVEL ARCHITECTURE OF DATABRICKS:

Control Plane : Databricks manage your backend services

Get Afaqueplayer’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Data Plane : Data is processed

Architecture of Control plane and Data plane

Roles in a Databricks platform :

1. Account administrator :
   -> creates the workspcae
   -> also works with metastore
   ->Users / permissions
2. Metastore Administrator :
   -> Catalog
   -> Data Objects
3. Workspace Administrator :
   -> admin of a workspace
   -> Manage users at workspace level
   ->Workspace assets
4. Owner
   -> tables
   -> schema

Basic architecture of data engineers :

Data Sources , Data Producers ->Data Integration or Ingestion(Amazon Glue, Kafka, Azure data factory)->Apache Spark

Different types of layers are there:
1) Bronze layer : keeps raw data
2)silver layer : tranforming , preprocessing data , improve high quality data
3)Gold layer : filling you results in desired data models

Two types of data collection is there :
1) Batch -> we collect data at regular interval daily, 30min
2)Stream-> collect data as quickly as possible -> Reporting analytics ( BI reports , ML models)

Apache Spark:

• a tool used to process and analyze very large amounts of data quickly.
• an engine for executing data engineering , stream processing and machine learning on distributed clusters
• Capabilities :
  -> ANSI SQL
  ->Batch Processing API
  ->Stream Processing API
  ->Machine learning API
• Benefits :
  -> Abstraction: hides all the complexities of distributed data processing(it means data is abstracted in backend and send to the spark framework to compute clusters and stored in S3 )
  ->Easy to use can use any language (eg: Spark SQL )
  -> Unified Platform : in single platform we can perform multiple jobs such as , data frames , streaming, graph X
• Cons:

->Data Storage Infrastructure : no data storage paltform like S3
->ACID Transaction capabilities
->Metadata Catalog
->Cluster Management
-> Automation API and tools

Some baic features of Databricks:

1. Spark as cloud native technology (Spark was initially designed to be executed on hadoop)
2. Secure cloud storage integration
3. ACID Transaction via Delta lake integration
4. Unity catalog for metadata management
5. Cluster management
6. Photon Query Engine
7. Notebooks and Workspaces
8. Administration Controls
9. Optimized Spark Runtime
10. Automation tools