DATABRICKS

What is Databricks-Databricks is the powerfull and efficient tool to process the data and the ETL concept mostly used in databricks for Extract,Transform,Load.

Basic 4 modules in ETL pipeline desingning using databricks platform(with AI )

1. Sensitive data auto classification.
2. Data quality editor
3. Data masking
4. Data governance

Now see briefly about all

1. sensitive data auto classification

Explaination:- First read a data from a data lake to using the tool like a (Data bricks+python)after inside the python using the gemini AI and give prompt to gemini to move the sensive file to a sensitive folder and move the non sesitive file in non sensitive folder so first to create the sensive and non-sensitive folder in the data lake if we give the sample for the sensitive files like contacts,address,card number to the AI .The AI move the files to the sensitive folders it is also have a unquality data like duplicates,null values,formating,ect…but companys didn’t like to share the sensitive data files to a gemini like AI.

Get Babuloob’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

2.Quality editor :-

Explaination:-In the data lake the non-sensisitive folders are read by the (data bricks+python) to give a prompt to a (gemini Ai) just giving a roll like you are a Data quality auditor. AI is processing and check the null values,remove duplicates,formatting and finish that work after load to a (data lake house) data lake house is the architecture in the data lake it is organise and separate the data in tables(structured) and files(unstructured like json,csv) in the data lake.

3.Data masking:-

Explanation:-Data masking is the method of masking the data and the (Data bricks+python) read the sensitive folder in lake and inside the data bricks we will give the prompt to a gemini using python and it will mask the sensitive folder like

EG:If i prompt like mask the number the output is

1. 638*******69 it will mask and move to a data lake this is called as the Data masking

4)Data Governance:-To set a rules,permission to data like Grant,revoke.

#Grant means — TO give a permission to who can only access the data.

If HR only able to see the confidential data to give a (so the administrator give a permission to aHR like (grant).

#Revoke means

Employs does not able to see the confidential data (so the administator give a revoke to the employs).