Databricks Practical Terms

Databricks notebooks :
->An interactive document that combines live code , outputs and rich text all in one place . similar to jupyter notebook.

• > Databricks noteooks are deeply integrated with apache spark , cloud storage, and databricks platform which makes them powerful for large scale data engineering

Magic commands in databricks:

• > A special keyword that start with a special sign that allows us to add additional features and capabilities to notebook environment.
• It allows to switch the language of the cell in notebook
  eg: %sql , %scala ,
  %run — run child notebook from the parent notebook,
  %sh — linux commands
• /ls to check directories in file system

DButils:

a collection of built-in utilities that helps you interact with the Databricks environment, manage files, access secrets, work with notebook workflows, and perform other common tasks.

dbutils.fs -> File system operations
dbutils.secrets -> Access secrets securely
dbutils.widgets -> Create notebook input parameters
dbutils.notebook -> Run or exit notebooks
dbutils.jobs -> Access job-related utilities where supported

Widgets :

When we want to parameterize the notebooks and return the value from the notebook . The entire notebook behave like function

• > we can do that using two tools widgets and notebooks .

Databricks compute :

they are of two types :
1)Serverless compute :

• > On demand
• faster startup
• automatic scaling
• lowercost

2)Classic compute:
Standard
-> anyone with access can manage permission

• > collaborative project
• scala support

Dedicated:
-> dedicated for single user group

• >Scala R and RDD support
• GPU instances

DELTA LAKE:

Get Afaqueplayer’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Delta Lake as a storage layer on top of your data lake.

1. No Atomicity:
   Two pipeline jobs write to the same storage simutaneously. A query runnign mid-write sees a pratial corrupted.
2. No history:
   A bad pipeline run overwrites your data.There is no built in rollback . You need a backup — ofter missing expensive and slow
3. Performance Decay:
   Every small batch write creates a new Parqet fuile . Over time thousand of tiny files accumulate . Scanning small files are slow

Contains two types of files in that :

1. Parquet Data files :
   -> any tool which can read parqet file can read the data table also
2. Delta_log:
   -> also called as transaction log
   -> every insert , update add a new json file to the delta log

Features of Delta Lake:
1) ACID :
ACID transactions ensure that data changes are committed consistently and prevent incomplete transactions from becoming a valid table version.

-> when write starts it is invisible to the reader till it is fully committed tot he log , anyone who wants to read data from delta table will first look at the log and read only those file from parqet data files taht are comitted as log

2) Time Travel :
Time Travel allows you to access previous versions of a Delta table.

Time Travel is mainly useful for auditing, debugging, data recovery, and accessing historical versions of a Delta table.

Every successfull change create a new version.

Version 0
↓
Version 1
↓
Version 2
↓
Version 3

3) Data skipping

Data skipping reduces the amount of data that needs to be read by skipping files that cannot contain the requested data.

Different layers of Delta lake :

1. Storage layer : ( Amazon s3 , Azure ADLS):

Raw obejct storage -> holds parqet files , delta log JSON files

2. Format layer : (Delta Lake)

how data is stored , Manages transaction logs , provides ACID,data skipping , time travel parqet files + delta_log

3. Unity Catalog : (Governence Layer)

Gives table name , control access , tracls lineage and etc

## Different ways of creating data tables:

1. by using SQL query
2. by using CTAS
3. creating using spark code

TYPES OF DELTA TABLES IN DATABRICKS:

1. Managed Tables :
   -> databricks owns everything
   -> owns metadata entry in unity catalogue
   -> owns data files on storage bucket as S3.
2. External Tables:
   -> databricks only owns the metadata in unity catalogue
   -> data files lives on storage bucket on the S3 that we own or control
   -> after dropping it removes files from Unity catalogue and also it only deletes the metadata.
   -> but never touches the S3 or your storage account.
   -> we have to specify the location explicity then it will become external .