APACHE SPARK

1. What is apache spark:-Apache spark is a big data processing engine to process the data effieciently it is a (software) use this spark through the APIS like(pyspark,spark java,sparkscala,sparkR) this are called as (application program interfaces)we will use this programs to access the spark engine(software)if i use the pyspark and the pyspark program will go and use the spark engine and process and read the data quiekly from the database or else.

#how its actually works

In our computer or desktops contain minimum number of CPU cores and threads if example our computer contains (6)CPU core+12 threads each CPU core contain the (2) threads. And each threads do one task. If i use the spark .The spark process the data across multiple CPU core.

EG:-

*so each threads handle one task like if i use the (Youtube)it is processing in the one thread so one (task1) same time I use the (Instagram)it will process in the one threads so (task2).

Get Babuloob’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

*IF i give a take to a spark to read the data. It will process the data across multiple computers(CPU cores)Simultaniosly.If i give the 100GB of data. the spark separate the data (25GB),(25GB),(25GB),(25GB) and each and every threads do one task so it works likes

— — *Task1=25GB

— — *Task2=25GB

— — *Task3=25GB

— — *Task4=25gb

so its split the works so it is so efficient to process the data.

THE sparkSQL only is not a application program interface(API)becoz it is a spark components now you ask what is the reason it is only a component inside a spark. Becoz all companies are mosly using the SQL TO QUERY so whenever we use the query in the spark it will goes to a spark . The spark inside having the component named sparkSQL it is executing the queries from the database.