<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sravan Reddy on Medium]]></title>
        <description><![CDATA[Stories by Sravan Reddy on Medium]]></description>
        <link>https://medium.com/@sravansainath7980?source=rss-2bd123571a9f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ketzUs1kC1mbmYVm</url>
            <title>Stories by Sravan Reddy on Medium</title>
            <link>https://medium.com/@sravansainath7980?source=rss-2bd123571a9f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:00:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sravansainath7980/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Hive: A Complete Guide from Basics to Advanced]]></title>
            <link>https://medium.com/@sravansainath7980/apache-hive-a-complete-guide-from-basics-to-advanced-1c46bde240c9?source=rss-2bd123571a9f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c46bde240c9</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[beeline]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Sravan Reddy]]></dc:creator>
            <pubDate>Sun, 11 Jan 2026 15:30:35 GMT</pubDate>
            <atom:updated>2026-01-11T15:32:51.787Z</atom:updated>
            <content:encoded><![CDATA[<p>Master Hive Architecture, Partitioning, Bucketing, and Integration with Spark Everything you need to know about Apache Hive for data warehousing and analytics<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*Sm4oAWw_FjZKX-g9smdhvw.png" /><figcaption>Hive</figcaption></figure><h4>Introduction</h4><p>Apache Hive is a data warehouse system built on top of Hadoop that provides SQL-like querying capabilities for large-scale data analysis. This comprehensive guide covers everything from basic concepts to advanced optimization techniques.</p><p>What you’ll learn:</p><ul><li>Hive architecture and components</li><li>Table types and storage formats</li><li>Partitioning and bucketing strategies</li><li>Integration with Spark</li><li>Performance optimization</li><li>Real-world ETL patterns</li></ul><h3>Part 1: Understanding Hive Architecture</h3><h4>What is Hive?</h4><p>Hive transforms SQL queries (HiveQL) into MapReduce, Tez, or Spark jobs that run on Hadoop. It’s designed for:</p><ul><li>Batch processing of large datasets</li><li>Data warehousing and analytics</li><li>Schema-on-read flexibility</li><li>SQL interface for non-programmers</li></ul><h3>Core Components</h3><pre>User Query (HiveQL)<br>    ↓<br>Hive Compiler &amp; Optimizer<br>    ↓<br>Execution Engine (MapReduce/Tez/Spark)<br>    ↓<br>Hadoop YARN<br>    ↓<br>HDFS (Data Storage)</pre><h4>1. Metastore</h4><p>The metastore is a relational database (Derby, MySQL, PostgreSQL) that stores:</p><ul><li>Table schemas and column definitions</li><li>Partition information</li><li>Table locations in HDFS</li><li>Storage formats and SerDe details</li></ul><p>Why important? Without metastore, Hive would need to scan all files to understand data structure!</p><pre>-- When you create a table<br>CREATE TABLE employees (<br>    id INT,<br>    name STRING<br>)<br>LOCATION &#39;/user/hive/warehouse/employees&#39;;</pre><pre>-- Metadata stored in metastore, data stored in HDFS</pre><h4>2. HiveServer2</h4><p>A service that allows remote clients to execute queries:</p><ul><li>Supports JDBC/ODBC connections</li><li>Handles authentication and authorization</li><li>Enables multi-user access</li><li>Port: 10000 (Thrift), 10002 (Web UI)</li></ul><h4>3. Beeline</h4><p>Modern CLI client that connects to HiveServer2:</p><pre>beeline -u jdbc:hive2://localhost:10000 -n username</pre><p>Why not Hive CLI?</p><ul><li>Hive CLI is deprecated</li><li>Direct metastore access (not scalable)</li><li>No authentication support</li></ul><h3>Part 2: Table Types</h3><h4>1. Managed (Internal) Tables</h4><p>Hive controls both data and metadata:</p><pre>CREATE TABLE employees (<br>    id INT,<br>    name STRING,<br>    salary DOUBLE<br>)<br>STORED AS PARQUET;</pre><pre>-- Data stored at: /user/hive/warehouse/employees/</pre><p>Characteristics:</p><ul><li>Dropping table deletes both metadata and data</li><li>Data managed by Hive</li><li>Use for: Hive-exclusive data</li></ul><h4>2. External Tables</h4><p>Hive controls only metadata, data stays independent:</p><pre>CREATE EXTERNAL TABLE logs (<br>    timestamp BIGINT,<br>    message STRING,<br>    level STRING<br>)<br>LOCATION &#39;/data/application_logs/&#39;;</pre><pre>-- Dropping table deletes only metadata, data remains!</pre><p>Characteristics:</p><ul><li>Data can be shared with other tools</li><li>Safe drops (data preserved)</li><li>Use for: Shared data sources, external data</li></ul><h4>3. Partitioned Tables</h4><p>Divide large tables into smaller, manageable parts:</p><pre>CREATE TABLE sales (<br>    order_id INT,<br>    product STRING,<br>    amount DOUBLE<br>)<br>PARTITIONED BY (year INT, month INT)<br>STORED AS PARQUET;</pre><pre>-- Directory structure:<br>-- /sales/year=2024/month=1/<br>-- /sales/year=2024/month=2/<br>-- /sales/year=2024/month=3/</pre><p>Benefits:</p><ul><li>Query Performance: Only scan relevant partitions</li><li>Data Management: Easy to drop old data</li><li>Cost Savings: Scan less data (important in cloud)</li></ul><p>Partitioning Best Practices:</p><p>✅ Good partition keys:</p><ul><li>Date/time (year, month, day, hour)</li><li>Geography (country, region, city)</li><li>Category (department, product_type)</li><li>Status (active, inactive)</li></ul><p>❌ Bad partition keys:</p><ul><li>High cardinality (user_id, transaction_id)</li><li>Continuous values (timestamp, price)</li><li>Too many partitions (&gt;10,000 becomes slow)</li></ul><p>Example: Multi-level Partitioning</p><pre>CREATE TABLE web_logs (<br>    user_id STRING,<br>    page_url STRING,<br>    session_id STRING<br>)<br>PARTITIONED BY (year INT, month INT, day INT, country STRING)<br>STORED AS PARQUET;</pre><pre>-- Efficient query (partition pruning)<br>SELECT COUNT(*) <br>FROM web_logs<br>WHERE year=2024 AND month=1 AND country=&#39;US&#39;;<br>-- Scans only: /web_logs/year=2024/month=1/day=*/country=US/</pre><pre>-- Inefficient query (full table scan)<br>SELECT COUNT(*) <br>FROM web_logs<br>WHERE user_id = &#39;abc123&#39;;<br>-- Scans ALL partitions!</pre><h4>4. Bucketed Tables</h4><p>Distribute data into fixed number of buckets based on hash:</p><pre>CREATE TABLE users (<br>    user_id INT,<br>    username STRING,<br>    email STRING<br>)<br>CLUSTERED BY (user_id) INTO 10 BUCKETS<br>SORTED BY (username)<br>STORED AS ORC;</pre><pre>-- Creates 10 files:<br>-- /users/000000_0 (bucket 0)<br>-- /users/000001_0 (bucket 1)<br>-- ...<br>-- /users/000009_0 (bucket 9)</pre><p>Benefits:</p><ul><li>Efficient sampling</li><li>Faster joins (bucket join)</li><li>Better data distribution</li></ul><p>When to use Bucketing?</p><pre>-- Two tables bucketed on same key = fast join!<br>CREATE TABLE orders (...) <br>CLUSTERED BY (customer_id) INTO 10 BUCKETS;</pre><pre>CREATE TABLE customers (...) <br>CLUSTERED BY (customer_id) INTO 10 BUCKETS;</pre><pre>-- Bucket join (no shuffle needed!)<br>SET hive.optimize.bucketmapjoin=true;</pre><pre>SELECT o.*, c.name<br>FROM orders o<br>JOIN customers c ON o.customer_id = c.customer_id;</pre><h3>Part 3: Storage Formats</h3><h4>Storage Format Comparison</h4><p>Parquet:</p><ul><li>Type: Columnar</li><li>Compression: Excellent</li><li>Use Case: Analytics, BI</li><li>Schema Evolution: Yes</li></ul><p>ORC (Optimized Row Columnar):</p><ul><li>Type: Columnar</li><li>Compression: Best</li><li>Use Case: Hive-optimized queries</li><li>Schema Evolution: Yes</li></ul><p>Avro:</p><ul><li>Type: Row-based</li><li>Compression: Good</li><li>Use Case: Streaming, schema changes</li><li>Schema Evolution: Yes</li></ul><p>Text/CSV:</p><ul><li>Type: Row-based</li><li>Compression: Poor</li><li>Use Case: Simple data, human-readable</li><li>Schema Evolution: No</li></ul><h4>Parquet (Recommended for Spark)</h4><pre>CREATE TABLE users_parquet (<br>    id INT,<br>    name STRING,<br>    email STRING,<br>    age INT<br>)<br>STORED AS PARQUET<br>TBLPROPERTIES (&#39;parquet.compression&#39;=&#39;SNAPPY&#39;);</pre><p>Advantages:</p><ul><li>Columnar storage (perfect for analytics)</li><li>Excellent compression ratios</li><li>Cross-platform (Spark, Hive, Impala, Presto)</li><li>Efficient predicate pushdown</li></ul><p>Example Query Optimization:</p><pre>-- Only reads &#39;name&#39; and &#39;age&#39; columns from disk<br>SELECT name, age <br>FROM users_parquet <br>WHERE age &gt; 25;</pre><pre>-- Columnar format reads only 2 columns, ignoring id and email!</pre><pre>CREATE TABLE transactions_orc (<br>    txn_id BIGINT,<br>    amount DOUBLE,<br>    timestamp BIGINT<br>)<br>STORED AS ORC<br>TBLPROPERTIES (&#39;orc.compress&#39;=&#39;ZLIB&#39;);</pre><p>Advantages:</p><ul><li>Best compression (better than Parquet)</li><li>Built-in indexes</li><li>Bloom filters for faster lookups</li><li>Optimized for Hive</li></ul><h4>Text Files with SerDe</h4><pre>CREATE TABLE csv_data (<br>    id INT,<br>    name STRING,<br>    salary DOUBLE<br>)<br>ROW FORMAT DELIMITED<br>    FIELDS TERMINATED BY &#39;,&#39;<br>    LINES TERMINATED BY &#39;\n&#39;<br>STORED AS TEXTFILE;</pre><pre>-- Load data<br>LOAD DATA LOCAL INPATH &#39;/tmp/data.csv&#39; INTO TABLE csv_data;</pre><p>Complex Types with Delimiters:</p><pre>CREATE TABLE complex_data (<br>    id INT,<br>    name STRING,<br>    departments ARRAY&lt;STRING&gt;,<br>    skills MAP&lt;STRING, INT&gt;<br>)<br>ROW FORMAT DELIMITED<br>    FIELDS TERMINATED BY &#39;,&#39;<br>    COLLECTION ITEMS TERMINATED BY &#39;:&#39;<br>    MAP KEYS TERMINATED BY &#39;=&#39;<br>    LINES TERMINATED BY &#39;\n&#39;;</pre><pre>-- Input file:<br>-- 1,Alice,Eng:Sales,Java=5:Python=3<br>-- 2,Bob,Marketing,Excel=4</pre><pre>-- Query:<br>SELECT <br>    name,<br>    departments[0] as primary_dept,<br>    skills[&#39;Java&#39;] as java_level<br>FROM complex_data;</pre><h3>Part 4: Partitioning Deep Dive</h3><h4>Static vs Dynamic Partitioning</h4><p>Static Partitioning (Manual):</p><pre>-- Insert into specific partition<br>INSERT INTO TABLE sales PARTITION (year=2024, month=1)<br>VALUES (1, &#39;Laptop&#39;, 999.99);</pre><pre>-- Load file into partition<br>LOAD DATA INPATH &#39;/tmp/jan_sales.csv&#39;<br>INTO TABLE sales PARTITION (year=2024, month=1);</pre><p>Dynamic Partitioning (Automatic):</p><pre>-- Enable dynamic partitioning<br>SET hive.exec.dynamic.partition=true;<br>SET hive.exec.dynamic.partition.mode=nonstrict;<br>SET hive.exec.max.dynamic.partitions=1000;</pre><pre>-- Hive automatically creates partitions<br>INSERT INTO TABLE sales PARTITION (year, month)<br>SELECT <br>    order_id,<br>    product,<br>    amount,<br>    YEAR(order_date) as year,<br>    MONTH(order_date) as month<br>FROM raw_sales;</pre><pre>-- Hive creates partitions based on data:<br>-- /sales/year=2023/month=12/<br>-- /sales/year=2024/month=1/<br>-- /sales/year=2024/month=2/</pre><h4>Managing Partitions</h4><pre>-- Show all partitions<br>SHOW PARTITIONS sales;</pre><pre>-- Add partition manually<br>ALTER TABLE sales ADD PARTITION (year=2024, month=4)<br>LOCATION &#39;/data/sales/2024/04/&#39;;</pre><pre>-- Drop partition<br>ALTER TABLE sales DROP PARTITION (year=2023, month=1);</pre><pre>-- Describe partition<br>DESCRIBE FORMATTED sales PARTITION (year=2024, month=1);</pre><h4>MSCK REPAIR TABLE</h4><p>The Problem: Files exist in HDFS, but Hive doesn’t know about them!</p><pre># Files exist<br>$ hdfs dfs -ls /user/hive/warehouse/sales/<br>/sales/year=2024/month=1/<br>/sales/year=2024/month=2/<br>/sales/year=2024/month=3/</pre><pre>-- But Hive doesn&#39;t see them<br>SHOW PARTITIONS sales;<br>-- Empty!</pre><pre>SELECT * FROM sales WHERE year=2024 AND month=1;<br>-- No data!</pre><p>The Solution:</p><pre>-- Discover and add partitions<br>MSCK REPAIR TABLE sales;</pre><pre>-- Now Hive sees them!<br>SHOW PARTITIONS sales;<br>-- year=2024/month=1<br>-- year=2024/month=2<br>-- year=2024/month=3</pre><p>When to use:</p><ul><li>External data copied to Hive warehouse</li><li>Partitions created by Spark/Pig</li><li>Manual file uploads</li><li>Data migration</li></ul><h3>Part 5: Integrating Hive with Spark</h3><h4>Setup Spark with Hive Support</h4><pre>from pyspark.sql import SparkSession</pre><pre>spark = SparkSession.builder \<br>    .appName(&quot;SparkHiveIntegration&quot;) \<br>    .master(&quot;local[*]&quot;) \<br>    .config(&quot;spark.sql.warehouse.dir&quot;, &quot;/user/hive/warehouse&quot;) \<br>    .enableHiveSupport() \<br>    .getOrCreate()</pre><h4>Read from Hive Tables</h4><pre># Method 1: Spark SQL<br>df = spark.sql(&quot;SELECT * FROM employees WHERE salary &gt; 70000&quot;)<br>df.show()</pre><pre># Method 2: Table API<br>df = spark.table(&quot;employees&quot;)</pre><pre># Method 3: With database<br>df = spark.sql(&quot;SELECT * FROM mydb.employees&quot;)</pre><h4>Write to Hive Tables</h4><pre># Create DataFrame<br>data = [<br>    (1, &quot;Alice&quot;, 75000, &quot;Engineering&quot;),<br>    (2, &quot;Bob&quot;, 65000, &quot;Sales&quot;),<br>    (3, &quot;Charlie&quot;, 80000, &quot;Engineering&quot;)<br>]<br>df = spark.createDataFrame(data, [&quot;id&quot;, &quot;name&quot;, &quot;salary&quot;, &quot;dept&quot;])</pre><pre># Write as new table<br>df.write.mode(&quot;overwrite&quot;) \<br>    .format(&quot;parquet&quot;) \<br>    .saveAsTable(&quot;employees&quot;)</pre><pre># Append to existing table<br>df.write.mode(&quot;append&quot;).saveAsTable(&quot;employees&quot;)</pre><pre># Partitioned table<br>df.write.mode(&quot;overwrite&quot;) \<br>    .partitionBy(&quot;dept&quot;) \<br>    .format(&quot;parquet&quot;) \<br>    .saveAsTable(&quot;employees_partitioned&quot;)</pre><h4>Complex ETL with Spark + Hive</h4><pre># Read from Hive<br>source_df = spark.table(&quot;staging.raw_transactions&quot;)</pre><pre># Transform with Spark<br>from pyspark.sql.functions import col, year, month, when</pre><pre>cleaned_df = source_df \<br>    .filter(col(&quot;amount&quot;) &gt; 0) \<br>    .withColumn(&quot;year&quot;, year(col(&quot;txn_date&quot;))) \<br>    .withColumn(&quot;month&quot;, month(col(&quot;txn_date&quot;))) \<br>    .withColumn(&quot;category&quot;, <br>        when(col(&quot;amount&quot;) &lt; 100, &quot;small&quot;)<br>        .when(col(&quot;amount&quot;) &lt; 1000, &quot;medium&quot;)<br>        .otherwise(&quot;large&quot;)<br>    )</pre><pre># Write back to Hive (partitioned)<br>cleaned_df.write.mode(&quot;overwrite&quot;) \<br>    .partitionBy(&quot;year&quot;, &quot;month&quot;) \<br>    .format(&quot;parquet&quot;) \<br>    .saveAsTable(&quot;warehouse.transactions&quot;)</pre><pre># Verify<br>spark.sql(&quot;SELECT year, month, category, COUNT(*) as count \<br>           FROM warehouse.transactions \<br>           GROUP BY year, month, category&quot;).show()</pre><h3>Part 6: Real-World ETL Patterns</h3><h4>Pattern 1: Daily Incremental Load</h4><pre>from pyspark.sql.functions import current_date, col</pre><pre># Read today&#39;s data<br>daily_data = spark.read.csv(&quot;hdfs:///raw/sales_2024-01-11.csv&quot;, <br>                             header=True, inferSchema=True)</pre><pre># Add metadata columns<br>enriched = daily_data \<br>    .withColumn(&quot;ingestion_date&quot;, current_date()) \<br>    .withColumn(&quot;year&quot;, year(col(&quot;sale_date&quot;))) \<br>    .withColumn(&quot;month&quot;, month(col(&quot;sale_date&quot;)))</pre><pre># Append to partitioned table<br>enriched.write.mode(&quot;append&quot;) \<br>    .partitionBy(&quot;year&quot;, &quot;month&quot;) \<br>    .format(&quot;parquet&quot;) \<br>    .saveAsTable(&quot;warehouse.sales&quot;)</pre><h4>Pattern 2: Slowly Changing Dimension (SCD Type 2)</h4><pre>from pyspark.sql.functions import lit, current_timestamp</pre><pre># Read existing dimension<br>existing = spark.table(&quot;warehouse.dim_customer&quot;) \<br>    .filter(col(&quot;is_current&quot;) == True)</pre><pre># Read new data<br>new_data = spark.table(&quot;staging.customer_updates&quot;)</pre><pre># Find changes<br>changes = new_data.alias(&quot;new&quot;).join(<br>    existing.alias(&quot;old&quot;),<br>    col(&quot;new.customer_id&quot;) == col(&quot;old.customer_id&quot;),<br>    &quot;left&quot;<br>).filter(<br>    (col(&quot;new.address&quot;) != col(&quot;old.address&quot;)) |<br>    (col(&quot;new.phone&quot;) != col(&quot;old.phone&quot;))<br>)</pre><pre># Expire old records<br>expired = changes.select(<br>    col(&quot;old.*&quot;)<br>).withColumn(&quot;is_current&quot;, lit(False)) \<br>  .withColumn(&quot;end_date&quot;, current_timestamp())</pre><pre># Insert new records<br>new_records = changes.select(<br>    col(&quot;new.*&quot;)<br>).withColumn(&quot;is_current&quot;, lit(True)) \<br>  .withColumn(&quot;start_date&quot;, current_timestamp()) \<br>  .withColumn(&quot;end_date&quot;, lit(None))</pre><pre># Write updates<br>expired.write.mode(&quot;append&quot;).saveAsTable(&quot;warehouse.dim_customer&quot;)<br>new_records.write.mode(&quot;append&quot;).saveAsTable(&quot;warehouse.dim_customer&quot;)</pre><h4>Pattern 3: Data Quality Checks</h4><pre># Read data<br>df = spark.table(&quot;staging.transactions&quot;)</pre><pre># Quality checks<br>total = df.count()<br>null_checks = df.filter(<br>    col(&quot;customer_id&quot;).isNull() | <br>    col(&quot;amount&quot;).isNull()<br>).count()<br>invalid_amounts = df.filter(col(&quot;amount&quot;) &lt;= 0).count()</pre><pre># Log results<br>print(f&quot;Total records: {total}&quot;)<br>print(f&quot;Null values: {null_checks}&quot;)<br>print(f&quot;Invalid amounts: {invalid_amounts}&quot;)</pre><pre># Quarantine bad data<br>bad_data = df.filter(<br>    col(&quot;customer_id&quot;).isNull() | <br>    col(&quot;amount&quot;) &lt;= 0<br>)<br>bad_data.write.mode(&quot;append&quot;) \<br>    .saveAsTable(&quot;quarantine.bad_transactions&quot;)</pre><pre># Load only good data<br>good_data = df.filter(<br>    col(&quot;customer_id&quot;).isNotNull() &amp; <br>    col(&quot;amount&quot;) &gt; 0<br>)<br>good_data.write.mode(&quot;append&quot;) \<br>    .saveAsTable(&quot;warehouse.transactions&quot;)</pre><h3>Part 7: Performance Optimization</h3><h4>1. Partition Pruning</h4><pre>-- Good: Scans only one partition<br>SELECT * FROM sales<br>WHERE year=2024 AND month=1;</pre><pre>-- Bad: Scans all partitions<br>SELECT * FROM sales<br>WHERE amount &gt; 1000;</pre><h4>2. Columnar Format Benefits</h4><pre>-- Only reads 2 columns from Parquet<br>SELECT name, salary <br>FROM employees_parquet;</pre><pre>-- vs Text format: reads entire row<br>SELECT name, salary <br>FROM employees_text;</pre><h4>3. Predicate Pushdown</h4><pre>-- Filter pushed to storage layer<br>SELECT * FROM large_table<br>WHERE date = &#39;2024-01-01&#39; <br>  AND status = &#39;active&#39;;</pre><pre>-- Parquet/ORC readers apply filters before loading data!</pre><h4>4. Bucketing for Joins</h4><pre>-- Both tables bucketed on join key<br>CREATE TABLE orders (...) <br>CLUSTERED BY (customer_id) INTO 10 BUCKETS;</pre><pre>CREATE TABLE customers (...) <br>CLUSTERED BY (customer_id) INTO 10 BUCKETS;</pre><pre>-- Enable bucket join<br>SET hive.optimize.bucketmapjoin=true;</pre><pre>-- Fast join (no shuffle!)<br>SELECT o.*, c.name<br>FROM orders o<br>JOIN customers c ON o.customer_id = c.customer_id;</pre><h4>5. Vectorization</h4><pre>-- Enable vectorized query execution<br>SET hive.vectorized.execution.enabled=true;<br>SET hive.vectorized.execution.reduce.enabled=true;</pre><pre>-- Processes data in batches of 1024 rows<br>-- Much faster than row-by-row processing</pre><h4>6. Cost-Based Optimization</h4><pre>-- Collect statistics<br>ANALYZE TABLE employees COMPUTE STATISTICS;<br>ANALYZE TABLE employees COMPUTE STATISTICS FOR COLUMNS;</pre><pre>-- Hive uses stats for better query plans<br>SET hive.cbo.enable=true;<br>SET hive.compute.query.using.stats=true;</pre><h3>Part 8: Best Practices</h3><h4>Data Organization</h4><p>Layer your data warehouse:</p><pre>Bronze Layer (Raw)<br>├── Exact copy of source data<br>├── No transformations<br>└── Partitioned by ingestion_date<br>Silver Layer (Staging)<br>├── Cleaned and validated<br>├── Business rules applied<br>└── Deduplicated<br>Gold Layer (Warehouse)<br>├── Aggregated facts<br>├── Dimensions<br>└── Ready for BI tools</pre><h4>Naming Conventions</h4><pre>Databases:<br>- raw_&lt;source&gt;        (e.g., raw_salesforce)<br>- staging_&lt;domain&gt;    (e.g., staging_finance)<br>- warehouse_&lt;domain&gt;  (e.g., warehouse_sales)</pre><pre>Tables:<br>- fact_&lt;entity&gt;       (e.g., fact_orders)<br>- dim_&lt;entity&gt;        (e.g., dim_customer)<br>- &lt;entity&gt;_snapshot   (e.g., inventory_snapshot)</pre><h4>Partitioning Guidelines</h4><pre>-- Good: Date-based partitioning<br>CREATE TABLE events (...)<br>PARTITIONED BY (year INT, month INT, day INT);</pre><pre>-- Good: Hierarchy of granularities<br>CREATE TABLE sales (...)<br>PARTITIONED BY (region STRING, year INT, month INT);</pre><pre>-- Bad: Too granular (creates too many partitions)<br>CREATE TABLE logs (...)<br>PARTITIONED BY (timestamp BIGINT);  -- DON&#39;T DO THIS!</pre><h4>Add Metadata Columns</h4><pre>from pyspark.sql.functions import current_timestamp, lit</pre><pre>df = df.withColumn(&quot;ingestion_timestamp&quot;, current_timestamp()) \<br>       .withColumn(&quot;source_system&quot;, lit(&quot;SAP&quot;)) \<br>       .withColumn(&quot;etl_batch_id&quot;, lit(batch_id))</pre><h3>Conclusion</h3><p>Apache Hive is a powerful tool for data warehousing on Hadoop. Key takeaways:</p><p>Architecture:</p><ul><li>Understand metastore, HiveServer2, and execution engines</li><li>Use Beeline instead of deprecated Hive CLI</li><li>Choose the right execution engine (Tez/Spark over MapReduce)</li></ul><p>Tables:</p><ul><li>Use external tables for shared data</li><li>Partition large tables by date/category</li><li>Use columnar formats (Parquet/ORC) for analytics</li><li>Implement bucketing for optimized joins</li></ul><p>Integration:</p><ul><li>Spark + Hive is a powerful combination</li><li>Leverage Spark’s speed with Hive’s metadata management</li><li>Use Spark for complex transformations</li></ul><p>Optimization:</p><ul><li>Always filter on partition columns</li><li>Use appropriate file formats</li><li>Collect statistics for CBO</li><li>Enable vectorization</li></ul><p>Production:</p><ul><li>Layer your data (Bronze → Silver → Gold)</li><li>Implement data quality checks</li><li>Add metadata columns</li><li>Monitor and log ETL metrics</li></ul><p>Start with basic tables and queries, then gradually adopt advanced features like partitioning, bucketing, and Spark integration as your needs grow!</p><p><em>Questions or experiences with Hive? Share in the comments!</em></p><p>Tags: #ApacheHive #BigData #DataEngineering #Hadoop #Spark #DataWarehouse #SQL #ETL</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c46bde240c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Complete Big Data Development Environment Setup on Ubuntu 24.XX]]></title>
            <link>https://medium.com/@sravansainath7980/complete-big-data-development-environment-setup-on-ubuntu-24-04-b2deeb243c18?source=rss-2bd123571a9f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2deeb243c18</guid>
            <category><![CDATA[setup]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sravan Reddy]]></dc:creator>
            <pubDate>Sun, 11 Jan 2026 15:01:17 GMT</pubDate>
            <atom:updated>2026-01-11T15:01:49.596Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up Hadoop, Spark, Hive, Kafka, and Jupyter from Scratch</h3><p><em>A comprehensive step-by-step guide to build a production-grade data engineering environment on your local machine</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDaqJmVVVCY1RxsjR0fJhw.png" /></figure><h3>Introduction</h3><p>In this guide, we’ll build a complete big data ecosystem on Ubuntu 24.04, including Hadoop HDFS, YARN, Spark, Hive, Kafka, and Jupyter notebooks. By the end, you’ll have a fully functional data engineering environment for learning and development.</p><p><strong>What we’ll install:</strong></p><ul><li>✅ Java 21 (JDK)</li><li>✅ Hadoop 3.4.2 (HDFS + YARN)</li><li>✅ Apache Spark 4.0.1</li><li>✅ Apache Hive 4.2.0</li><li>✅ Apache Kafka 3.9.0</li><li>✅ Jupyter Notebook with PySpark</li></ul><p><strong>System Requirements:</strong></p><ul><li>Ubuntu 24.04 (or similar Linux distribution)</li><li>8GB RAM minimum (16GB recommended)</li><li>50GB free disk space</li><li>Sudo access</li></ul><h3>Part 1: Prerequisites</h3><h3>Update System</h3><pre>sudo apt update &amp;&amp; sudo apt upgrade -y</pre><h3>Install Java 21</h3><pre># Install OpenJDK 21<br>sudo apt install openjdk-21-jdk -y</pre><pre># Verify installation<br>java -version<br># Output: openjdk version &quot;21...&quot;</pre><pre># Find Java home<br>readlink -f $(which java)<br># Output: /usr/lib/jvm/java-21-openjdk-amd64/bin/java</pre><h3>Install SSH (Required for Hadoop)</h3><pre># Install SSH<br>sudo apt install openssh-server openssh-client -y</pre><pre># Start SSH service<br>sudo service ssh start</pre><pre># Generate SSH key (press Enter for all prompts)<br>ssh-keygen -t rsa -P &quot;&quot;</pre><pre># Enable passwordless SSH to localhost<br>cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys<br>chmod 640 ~/.ssh/authorized_keys</pre><pre># Test SSH (type &#39;yes&#39; when prompted)<br>ssh localhost<br>exit</pre><p><strong>Why SSH?</strong> Hadoop uses SSH to communicate between nodes, even on a single machine.</p><h3>Part 2: Hadoop Installation</h3><h3>Download and Extract Hadoop</h3><pre>cd ~</pre><pre># Download Hadoop 3.4.2<br>wget <a href="https://downloads.apache.org/hadoop/common/hadoop-3.4.2/hadoop-3.4.2.tar.gz">https://downloads.apache.org/hadoop/common/hadoop-3.4.2/hadoop-3.4.2.tar.gz</a></pre><pre># Extract<br>tar -xzvf hadoop-3.4.2.tar.gz<br>mv hadoop-3.4.2 hadoop</pre><h3>Configure Environment Variables</h3><pre># Edit .bashrc<br>nano ~/.bashrc</pre><p><strong>Add at the end:</strong></p><pre># Java<br>export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64</pre><pre># Hadoop<br>export HADOOP_HOME=/home/$USER/hadoop<br>export HADOOP_INSTALL=$HADOOP_HOME<br>export HADOOP_MAPRED_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_HOME=$HADOOP_HOME<br>export HADOOP_HDFS_HOME=$HADOOP_HOME<br>export YARN_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native<br>export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin<br>export HADOOP_OPTS=&quot;-Djava.library.path=$HADOOP_HOME/lib/native&quot;</pre><p><strong>Apply changes:</strong></p><pre>source ~/.bashrc<br>echo $HADOOP_HOME</pre><h3>Configure Hadoop</h3><p><strong>1. hadoop-env.sh:</strong></p><pre>nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh</pre><p>Add:</p><pre>export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64</pre><p><strong>2. core-site.xml:</strong></p><pre>nano $HADOOP_HOME/etc/hadoop/core-site.xml</pre><p>Replace content:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;fs.defaultFS&lt;/name&gt;<br>        &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;<br>        &lt;value&gt;/home/sravan/hadoop/tmpdata&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hadoop.proxyuser.sravan.hosts&lt;/name&gt;<br>        &lt;value&gt;*&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hadoop.proxyuser.sravan.groups&lt;/name&gt;<br>        &lt;value&gt;*&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p><strong>3. hdfs-site.xml:</strong></p><pre>nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml</pre><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;dfs.replication&lt;/name&gt;<br>        &lt;value&gt;1&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;<br>        &lt;value&gt;/home/sravan/hadoop/dfsdata/namenode&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;<br>        &lt;value&gt;/home/sravan/hadoop/dfsdata/datanode&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p><strong>4. mapred-site.xml:</strong></p><pre>nano $HADOOP_HOME/etc/hadoop/mapred-site.xml</pre><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;mapreduce.framework.name&lt;/name&gt;<br>        &lt;value&gt;yarn&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;mapreduce.application.classpath&lt;/name&gt;<br>        &lt;value&gt;$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p><strong>5. yarn-site.xml:</strong></p><pre>nano $HADOOP_HOME/etc/hadoop/yarn-site.xml</pre><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;<br>        &lt;value&gt;mapreduce_shuffle&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;yarn.nodemanager.env-whitelist&lt;/name&gt;<br>        &lt;value&gt;JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><h3>Format HDFS and Start Hadoop</h3><pre># Format NameNode (only once!)<br>hdfs namenode -format</pre><pre># Start HDFS<br>start-dfs.sh</pre><pre># Start YARN<br>start-yarn.sh</pre><pre># Verify<br>jps</pre><p><strong>Expected output:</strong></p><pre>NameNode<br>DataNode<br>SecondaryNameNode<br>ResourceManager<br>NodeManager<br>Jps</pre><p><strong>Access Web UIs:</strong></p><ul><li>HDFS: <a href="http://localhost:9870/">http://localhost:9870</a></li><li>YARN: <a href="http://localhost:8088/">http://localhost:8088</a></li></ul><h3>Part 3: Spark Installation</h3><h3>Download and Extract Spark</h3><pre>cd ~</pre><pre># Download Spark 4.0.1 (pre-built for Hadoop 3)<br>wget <a href="https://downloads.apache.org/spark/spark-4.0.1/spark-4.0.1-bin-hadoop3.tgz">https://downloads.apache.org/spark/spark-4.0.1/spark-4.0.1-bin-hadoop3.tgz</a></pre><pre># Extract<br>tar -xzvf spark-4.0.1-bin-hadoop3.tgz<br>mv spark-4.0.1-bin-hadoop3 spark</pre><h3>Configure Spark</h3><pre># Add to .bashrc<br>nano ~/.bashrc</pre><p>Add:</p><pre># Spark<br>export SPARK_HOME=/home/$USER/spark<br>export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin<br>export PYSPARK_PYTHON=/usr/bin/python3</pre><p>Apply:</p><pre>source ~/.bashrc</pre><p><strong>Create spark-env.sh:</strong></p><pre>cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh<br>nano $SPARK_HOME/conf/spark-env.sh</pre><p>Add:</p><pre>export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64<br>export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop<br>export SPARK_MASTER_HOST=localhost<br>export PYSPARK_PYTHON=/usr/bin/python3</pre><p><strong>Test Spark:</strong></p><pre>spark-shell<br># You should see Spark shell. Type :quit to exit</pre><p><strong>Start Spark History Server (optional):</strong></p><pre># Create HDFS directory for logs<br>hdfs dfs -mkdir -p /spark-logs</pre><pre># Start history server<br>$SPARK_HOME/sbin/start-history-server.sh</pre><pre># Access at <a href="http://localhost:18080/">http://localhost:18080</a></pre><h3>Part 4: Hive Installation</h3><h3>Download and Extract Hive</h3><pre>cd ~</pre><pre># Download Hive 4.2.0<br>wget <a href="https://downloads.apache.org/hive/hive-4.2.0/apache-hive-4.2.0-bin.tar.gz">https://downloads.apache.org/hive/hive-4.2.0/apache-hive-4.2.0-bin.tar.gz</a></pre><pre># Extract<br>tar -xzvf apache-hive-4.2.0-bin.tar.gz<br>mv apache-hive-4.2.0-bin hive</pre><h3>Configure Hive</h3><pre># Add to .bashrc<br>nano ~/.bashrc</pre><p>Add:</p><pre># Hive<br>export HIVE_HOME=/home/$USER/hive<br>export PATH=$PATH:$HIVE_HOME/bin</pre><p>Apply:</p><pre>source ~/.bashrc</pre><p><strong>Create hive-env.sh:</strong></p><pre>cp $HIVE_HOME/conf/hive-env.sh.template $HIVE_HOME/conf/hive-env.sh<br>nano $HIVE_HOME/conf/hive-env.sh</pre><p>Add:</p><pre>export HADOOP_HOME=/home/$USER/hadoop<br>export HIVE_CONF_DIR=/home/$USER/hive/conf<br>export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64</pre><p><strong>Create hive-site.xml:</strong></p><pre>nano $HIVE_HOME/conf/hive-site.xml</pre><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;<br>&lt;configuration&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;<br>        &lt;value&gt;jdbc:derby:;databaseName=/home/sravan/hive/metastore_db;create=true&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hive.metastore.warehouse.dir&lt;/name&gt;<br>        &lt;value&gt;/user/hive/warehouse&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hive.exec.local.scratchdir&lt;/name&gt;<br>        &lt;value&gt;/home/sravan/hive/tmp&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hive.downloaded.resources.dir&lt;/name&gt;<br>        &lt;value&gt;/home/sravan/hive/tmp/resources&lt;/value&gt;<br>    &lt;/property&gt;<br>    &lt;property&gt;<br>        &lt;name&gt;hive.execution.engine&lt;/name&gt;<br>        &lt;value&gt;mr&lt;/value&gt;<br>    &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p><strong>Create HDFS directories:</strong></p><pre>hdfs dfs -mkdir -p /user/hive/warehouse<br>hdfs dfs -mkdir -p /tmp<br>hdfs dfs -chmod g+w /user/hive/warehouse<br>hdfs dfs -chmod g+w /tmp</pre><p><strong>Create local directories:</strong></p><pre>mkdir -p $HIVE_HOME/tmp<br>mkdir -p $HIVE_HOME/tmp/resources</pre><p><strong>Initialize Hive Metastore:</strong></p><pre>$HIVE_HOME/bin/schematool -dbType derby -initSchema</pre><p><strong>Start HiveServer2:</strong></p><pre>nohup hiveserver2 &gt; /tmp/hiveserver2.log 2&gt;&amp;1 &amp;</pre><p><strong>Connect with Beeline:</strong></p><pre>$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n $USER</pre><p><strong>If you found any errors please wait for 2 to 3mins and re run the Beeline<br>Access Web UI:</strong> <a href="http://localhost:10002/">http://localhost:10002</a></p><h3>Part 5: Kafka Installation</h3><h3>Download and Extract Kafka</h3><pre>cd ~</pre><pre># Download Kafka 3.9.0<br>wget <a href="https://downloads.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz">https://downloads.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz</a></pre><pre># Extract<br>tar -xzvf kafka_2.13-3.9.0.tgz<br>mv kafka_2.13-3.9.0 kafka</pre><h3>Configure Kafka</h3><pre># Add to .bashrc<br>nano ~/.bashrc</pre><p>Add:</p><pre># Kafka<br>export KAFKA_HOME=/home/$USER/kafka<br>export PATH=$PATH:$KAFKA_HOME/bin</pre><p>Apply:</p><pre>source ~/.bashrc</pre><p><strong>Create data directories:</strong></p><pre>mkdir -p $KAFKA_HOME/data/zookeeper<br>mkdir -p $KAFKA_HOME/data/kafka</pre><p><strong>Update ZooKeeper config:</strong></p><pre>nano $KAFKA_HOME/config/zookeeper.properties</pre><p>Change:</p><pre>dataDir=/home/sravan/kafka/data/zookeeper</pre><p><strong>Update Kafka config:</strong></p><pre>nano $KAFKA_HOME/config/server.properties</pre><p>Change:</p><pre>log.dirs=/home/sravan/kafka/data/kafka</pre><h3>Start Kafka</h3><pre># Start ZooKeeper<br>nohup $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &gt; /tmp/zookeeper.log 2&gt;&amp;1 &amp;</pre><pre># Wait 5 seconds<br>sleep 5</pre><pre># Start Kafka<br>nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &gt; /tmp/kafka.log 2&gt;&amp;1 &amp;</pre><p><strong>Test Kafka:</strong></p><pre># Create topic<br>kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1</pre><pre># List topics<br>kafka-topics.sh --list --bootstrap-server localhost:9092</pre><h3>Part 6: Jupyter Notebook with PySpark</h3><h3>Create Virtual Environment</h3><pre># Install prerequisites<br>sudo apt install python3-full python3-venv python3-pip -y</pre><pre># Create virtual environment<br>mkdir -p ~/data-engineering-env<br>cd ~/data-engineering-env<br>python3 -m venv jupyter-env</pre><pre># Activate<br>source jupyter-env/bin/activate</pre><h3>Install Jupyter and Libraries</h3><pre># Install Jupyter and data science packages<br>pip install jupyter findspark pandas matplotlib seaborn</pre><pre># Create notebooks directory<br>mkdir -p ~/notebooks</pre><h3>Create Startup Script</h3><pre>nano ~/start_jupyter.sh</pre><p>Add:</p><pre>#!/bin/bash</pre><pre># Activate virtual environment<br>source ~/data-engineering-env/jupyter-env/bin/activate</pre><pre># Check Hadoop<br>jps | grep -q NameNode || start-dfs.sh<br>jps | grep -q ResourceManager || start-yarn.sh</pre><pre># Start Jupyter<br>cd ~/notebooks<br>jupyter notebook</pre><p>Make executable:</p><pre>chmod +x ~/start_jupyter.sh</pre><h3>Test PySpark in Jupyter</h3><p>Start Jupyter:</p><pre>~/start_jupyter.sh</pre><p><strong>Create new notebook and test:</strong></p><pre># Cell 1: Setup<br>import findspark<br>findspark.init(&#39;/home/sravan/spark&#39;)</pre><pre>from pyspark.sql import SparkSession</pre><pre>spark = SparkSession.builder \<br>    .appName(&quot;Test&quot;) \<br>    .master(&quot;local[*]&quot;) \<br>    .enableHiveSupport() \<br>    .getOrCreate()</pre><pre>print(f&quot;Spark {spark.version} is ready!&quot;)</pre><pre># Cell 2: Create DataFrame<br>data = [(&quot;Alice&quot;, 25), (&quot;Bob&quot;, 30), (&quot;Charlie&quot;, 35)]<br>df = spark.createDataFrame(data, [&quot;name&quot;, &quot;age&quot;])<br>df.show()</pre><pre># Cell 3: Query Hive<br>spark.sql(&quot;SHOW DATABASES&quot;).show()</pre><h3>Part 7: Starting and Stopping Services</h3><h3>Start All Services</h3><p>Create a master startup script:</p><pre>nano ~/start_all_services.sh</pre><pre>#!/bin/bash</pre><pre>echo &quot;Starting Hadoop...&quot;<br>start-dfs.sh<br>start-yarn.sh</pre><pre>echo &quot;Starting HiveServer2...&quot;<br>nohup hiveserver2 &gt; /tmp/hiveserver2.log 2&gt;&amp;1 &amp;</pre><pre>echo &quot;Starting Spark History Server...&quot;<br>$SPARK_HOME/sbin/start-history-server.sh</pre><pre>echo &quot;Starting Kafka...&quot;<br>nohup $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &gt; /tmp/zookeeper.log 2&gt;&amp;1 &amp;<br>sleep 5<br>nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &gt; /tmp/kafka.log 2&gt;&amp;1 &amp;</pre><pre>echo &quot;All services started!&quot;<br>echo &quot;Check with: jps&quot;</pre><p>Make executable:</p><pre>chmod +x ~/start_all_services.sh</pre><h3>Stop All Services</h3><pre>nano ~/stop_all_services.sh</pre><pre>#!/bin/bash</pre><pre>echo &quot;Stopping Kafka...&quot;<br>$KAFKA_HOME/bin/kafka-server-stop.sh<br>$KAFKA_HOME/bin/zookeeper-server-stop.sh</pre><pre>echo &quot;Stopping Spark History Server...&quot;<br>$SPARK_HOME/sbin/stop-history-server.sh</pre><pre>echo &quot;Stopping HiveServer2...&quot;<br>pkill -f hiveserver2</pre><pre>echo &quot;Stopping Hadoop...&quot;<br>stop-yarn.sh<br>stop-dfs.sh</pre><pre>echo &quot;All services stopped!&quot;</pre><p>Make executable:</p><pre>chmod +x ~/stop_all_services.sh</pre><h3>Part 8: Verification and Testing</h3><h3>Check All Services</h3><pre>jps</pre><p><strong>Expected output:</strong></p><pre>NameNode<br>DataNode<br>SecondaryNameNode<br>ResourceManager<br>NodeManager<br>RunJar          (HiveServer2)<br>HistoryServer   (Spark)<br>QuorumPeerMain  (ZooKeeper)<br>Kafka</pre><p>ServiceURLDescriptionHDFS NameNode<a href="http://localhost:9870/">http://localhost:9870</a>HDFS file systemYARN ResourceManager<a href="http://localhost:8088/">http://localhost:8088</a>YARN applicationsSpark History<a href="http://localhost:18080/">http://localhost:18080</a>Completed Spark jobsSpark UI<a href="http://localhost:4040/">http://localhost:4040</a>Running Spark jobsHiveServer2<a href="http://localhost:10002/">http://localhost:10002</a>Hive web interface</p><h3>Test HDFS</h3><pre># Create directory<br>hdfs dfs -mkdir /test</pre><pre># Upload file<br>echo &quot;Hello HDFS&quot; &gt; /tmp/test.txt<br>hdfs dfs -put /tmp/test.txt /test/</pre><pre># List files<br>hdfs dfs -ls /test/</pre><pre># Read file<br>hdfs dfs -cat /test/test.txt</pre><h3>Test Spark</h3><pre># Run example<br>spark-submit --class org.apache.spark.examples.SparkPi \<br>  --master local[2] \<br>  $SPARK_HOME/examples/jars/spark-examples*.jar \<br>  10</pre><h3>Test Hive</h3><pre>$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n $USER</pre><pre>CREATE DATABASE test_db;<br>USE test_db;</pre><pre>CREATE TABLE employees (<br>    id INT,<br>    name STRING,<br>    salary DOUBLE<br>);</pre><pre>INSERT INTO employees VALUES (1, &#39;Alice&#39;, 75000);<br>INSERT INTO employees VALUES (2, &#39;Bob&#39;, 65000);</pre><pre>SELECT * FROM employees;</pre><h3>Test Kafka</h3><pre># Producer (in one terminal)<br>kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092<br>&gt; Hello Kafka<br>&gt; Message 2</pre><pre># Consumer (in another terminal)<br>kafka-console-consumer.sh --topic test-topic --bootstrap-server localhost:9092 --from-beginning</pre><h3>Troubleshooting</h3><h3>Common Issues</h3><p><strong>1. Port already in use:</strong></p><pre># Check what&#39;s using a port<br>sudo lsof -i :9000<br># Kill process<br>kill -9 &lt;PID&gt;</pre><p><strong>2. HDFS Safe Mode:</strong></p><pre># Leave safe mode<br>hdfs dfsadmin -safemode leave</pre><p><strong>3. Permission denied:</strong></p><pre># Fix HDFS permissions<br>hdfs dfs -chmod -R 777 /user</pre><p><strong>4. SSH connection refused:</strong></p><pre># Restart SSH<br>sudo service ssh restart</pre><pre># Re-test<br>ssh localhost</pre><p><strong>5. Hive connection refused:</strong></p><pre># Check if HiveServer2 is running<br>jps | grep RunJar</pre><pre># Check logs<br>tail -100 /tmp/hiveserver2.log</pre><h3>Conclusion</h3><p>Congratulations! You now have a complete big data development environment with:</p><ul><li>✅ Hadoop HDFS for distributed storage</li><li>✅ YARN for resource management</li><li>✅ Spark for fast data processing</li><li>✅ Hive for SQL-based data warehousing</li><li>✅ Kafka for streaming data</li><li>✅ Jupyter for interactive development</li></ul><p><strong>Next Steps:</strong></p><ul><li>Build ETL pipelines with Spark</li><li>Create data warehouses with Hive</li><li>Implement streaming applications with Kafka</li><li>Explore advanced features of each technology</li></ul><p><strong>Useful Resources:</strong></p><ul><li><a href="https://hadoop.apache.org/docs/">Hadoop Documentation</a></li><li><a href="https://spark.apache.org/docs/latest/">Spark Documentation</a></li><li><a href="https://hive.apache.org/">Hive Documentation</a></li><li><a href="https://kafka.apache.org/documentation/">Kafka Documentation</a></li></ul><p><em>Have questions or run into issues? Leave a comment below!</em></p><p><strong>Tags:</strong> #BigData #Hadoop #Spark #Hive #Kafka #DataEngineering #Ubuntu #Tutorial</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2deeb243c18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark With Yarn Architecture Explained]]></title>
            <link>https://medium.com/@sravansainath7980/spark-with-yarn-architecture-explained-fffc150921c9?source=rss-2bd123571a9f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fffc150921c9</guid>
            <category><![CDATA[hadoop-cluster]]></category>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Sravan Reddy]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 17:30:08 GMT</pubDate>
            <atom:updated>2025-01-10T17:31:01.363Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article i will explain in detail in layman terms about the Spark and Yarn Architecture. We will go step by step first Hadoop then Yarn at last Spark.</p><blockquote>All Three Applications i.e, HDFS, Yarn and Spark work with Master-Slave Architecture.</blockquote><blockquote>Master-slave architecture involves a single central unit, referred to as the “master,” that governs and guides the activities of several slaves, or subordinate units. The master node in this configuration controls and assigns tasks to the slave nodes</blockquote><h3><strong>Apache Hadoop</strong></h3><p>Apache Hadoop is software framework which help in distributed processing of large datasets across cluster of computers.It mainly contains of three components.</p><ol><li>Hadoop Distributed File System (HDFS)</li><li>Hadoop Yarn (Yet Another Resource Manager)</li><li>Hadoop MapReduce</li></ol><p>If you download the hadoop from official website , by default all these three applications will also be included.</p><h3><strong>HDFS ( Hadoop Distributed File System</strong>)</h3><p>Hadoop comes with HDFS which is s unique file system helps in storing and managing large data across a cluster of computers. If You read the definition again I have mentioned it as unique because by far all the file systems which you may have encountered stores the data in a single computer. But what&#39;s the limit of the single machine ? upto how much space can you keep on adding! There will be a limit at some point in future.</p><p>What if you want to store and process files with size of 10Tb or 20Tb or more than that in single computer and have to make backup of files to provide fault tolerant ? For Eg : If you want to count the number of lines in text file of size around 10Tb , I will take around 26.5 hours approx in single node machine and with the help of Hadoop we can do it in less than a hour.</p><p>Now let us see how HDFS stores the data by distributing it across a cluster of computers.As I have mentioned earlier HDFS works in Master Slave Architecture which contains mainly three components</p><ol><li>NameNode -&gt; Master Node</li><li>DataNode -&gt; Slave Node</li><li>Secondary NameNode ( Backup of Master Node i.e, NameNode)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*56D1eJwyKlwGVjaf9AlieQ.png" /><figcaption>HDFS Architecture Overview</figcaption></figure><p><strong>NameNode</strong> (Master Node) Will manage all the DataNodes (Slave Nodes) and stores all details about the metadata of the files ,no of blocks and block ids ,location of the files , location of partition of files.Namenode is built with costly commodity hardware unlike datanodes.</p><p><strong>DataNodes</strong> (Slave Nodes) will store the actual data which we are storing . They will continuously communicate with NameNode via heartbeat and Block Report.Heart Beat indicates that DataNode is fine and Block Report contains details for the block which are stored in that datanode.</p><p>This is Small Overview of Hadoop there are lot of underneath things which i haven&#39;t mentioned here. I will write in detail about them in separate article.</p><h3>Hadoop Yarn (Yet Another Resource Manager)</h3><p>There are some things which you need to know before going into what is YARN and why we are using it.</p><p>In order to perform any type of action on data like reading , writing , processing we need some computational power i.e, Memory(RAM) , CPU Power (Cores).In order to process vast amount of data we will be needing some much of computational power . Even if you have all the resources to use these resources upto there full capacity we need a manager on top of the resource to manage the resources and scheduling of jobs.Here is the place where Hadoop YARN will come to picture.</p><p>Hadoop Yarn is a distributed resource management system for Hadoop clusters.It is used for mainly for resource management and job scheduling and monitoring.YARN was introduced by Yahoo in 2012 and being used in hadoop form 2.0 version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*Hxw7TOyrXEHSf4Xvi8epFA.png" /></figure><p>Hadoop YARN also follows master-slave architecture , it mainly have two components</p><ol><li>Resource Manager (Master Node)</li><li>Node Manager ( Slave Nodes)</li></ol><h3><strong>Resource Manager</strong></h3><p>Resource Manager is the heart and master daemon of YARN.It manages the cluster resources and scheduling the applications based upon the capacities and queues.It has mainly two parts Scheduler and Application Manager</p><p><strong>1.Scheduler</strong> : The Scheduler is responsible for allocating resources to the applications. It perform the scheduling function based on resources (the memory, cpu, disk, network etc) available in Node Managers .</p><p>Cluster can be divided into various queues i.e, in any big data organization the cluster resources will divided into various sections like for batch processing , stream processing , machine learning , graph processing , flink etc .It keep tracks of the available resources in all available sections in the clusters.</p><p>The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application</p><p>2.<strong>Application Manager</strong></p><p>Application Manager is the first point of contact for any job application entering into cluster. It is responsible for the accepting the job submissions and it will negotiate with the Scheduler to start the first container for the job application which is known as <strong>Application Master . </strong>Each application in cluster will have its own application master.</p><p>Don&#39;t get confused with the terms <strong>Application Manager</strong> and <strong>Application Master</strong> , they might sound similar but they are entirely different.</p><p>Application Manager monitors all the applications running in the cluster. The per-application Application Master is responsible for negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.</p><h3><strong>Node Manager</strong></h3><p>Node Managers are the slave demons of the YARN application. The are responsible for launching and managing containers on a node they are running.</p><p>A<strong> Container</strong> represents a portion of a Node Manager’s capacity and is used by the client to run a program. Each NodeManager, operating on a single node, follows the Resource Manager’s instructions and manages the containers it hosts.</p><p>Node Managers registers with the Resource Manager and periodically sends heartbeats to convey the node’s health. If the resource manager didn&#39;t receive the heat beat from any Node Manager it will stop requesting the containers from that node. It monitors resource usage, manages logs, and terminates containers when instructed by the ResourceManager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*ZQvG1KIoW8mX-yWEEw2B8w.png" /></figure><p>This diagram shows a YARN setup where the ResourceManager (on the master node) coordinates the distribution of workloads across multiple NodeManagers (on slave nodes). Each NodeManager has 16 CPU cores and 64GB RAM, and it receives instructions from the ResourceManager — specifically, the Scheduler and the Application Manager. The Scheduler decides how to allocate resources among different applications, while each NodeManager launches and manages the containers that actually run the tasks.</p><h3><strong>How Applications Run on YARN</strong></h3><p>Let us see how a spark application runs in YARN Cluster. Let us assume that jobs are running in client mode i.e, spark driver will be deployed in the local machine.</p><p>The below picture shows detailed overview of steps happening in application life cycle</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*12VoZfNed27K9WLChoxwPg.png" /></figure><ol><li><strong>Client → ResourceManager</strong><br>The client submits a new application (job) to the ResourceManager, requesting the resources needed to run.</li><li><strong>Creating Application Master</strong><br>The ResourceManager (via the Application Manager ) picks a suitable NodeManager on which to launch the <strong>first container</strong> — the one that will run the <strong>Application Master</strong> for this application.<br>The chosen NodeManager (in this example, NM-1) is instructed to spin up a container that will become the Application Master (AM).</li><li><strong>Application Master Requests Resources From Scheduler</strong><br>Once the Application Master is up and running, it requests more containers from Scheduler for the actual tasks.</li><li><strong>Scheduler Allocates Some Resources to the Application Master</strong><br>The Scheduler will allocate some resources based on the availability of resources in the cluster .Here it tells the Application Master to use resource in Node Managers (e.g., NM-3, NM-8) to create containers (“container-01,” “container-02,” etc.) where the application’s tasks will run.</li><li><strong>Application Master ↔ NodeManager</strong><br>The Application Master registers with the Node Managers which are allocated to it by Scheduler and requests to launch containers <br>(e.g. NM-3, NM-8) in it to start, monitor, and manage the tasks. <br>Node Managers will launch requested containers and will be in touch with Resource Manager and Application Master about the status of the tasks it is running. <br>The ResourceManager (specifically, the Scheduler) decides if and when to grant additional containers.</li><li><strong>Application Master → NodeManagers (Task Execution)</strong><br>Once the tasks running in the container finishes , The Application Master collects all the job results and will return it to driver and containers will exit on there own. The NodeManager then cleans up resources and informs the Resource Manager about the container status and Node status.</li><li><strong>ResourceManager → Client (Completion Status)</strong><br>When the application finishes (successfully or otherwise), the ResourceManager notifies the client of the final job status and resource usage.<br>Application Manager in the Resource Manager will hold the applications details during and after the application is ran . We can go through the logs and execution plan happened for that application.</li></ol><p>This the basic overview of how applications run inside YARN . There still lot more steps involved in this process i just made sure that only important steps are included.</p><h3>Combination Of Hadoop HDFS and Hadoop Yarn</h3><p>In Most of the Organizations who uses HDFS with YARN will host both the Data Node and Node Manager on a Single Node .Hadoop HDFS and Hadoop YARN are two different frameworks where one is used for storage and another one for processing that data stored in HDFS. So having a single node with storage ,ram and cpu cores can host Data Node in HDFS and Node Manager in YARN. <br>This would benefit the overall architecture in various ways like below</p><ul><li><strong>Data Locality<br></strong>When a NodeManager runs tasks, they can access data on the same node (via the local DataNode), reducing network traffic.This improves performance, as containers do not need to pull data across the network.</li><li><strong>Efficient Use of Hardware<br></strong>Having both a NodeManager and a DataNode on the same node ensures you’re utilizing CPU, RAM, and disk resources effectively in a single machine, rather than underutilizing separate machines for each role.</li><li><strong>Reduced I/O Bottlenecks<br></strong>Containers can directly read/write to the local disk where the data is stored, rather than involving external data transfers.</li><li><strong>Scalability and Simpler Cluster Design<br></strong>It’s simpler to add new nodes to the cluster: each node contributes both compute (NodeManager) and storage (DataNode) capacity, making horizontal scaling easier .</li><li><strong>Standard Hadoop Best Practice<br></strong>The original Hadoop design encourages this co-location model to get the best out of the “moving compute to data” philosophy, which continues to be a common practice in production clusters in most of the organizations.</li></ul><p>Combined would look like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-BN87epZC43luyyJKe20Q.png" /><figcaption>Spark Yarn Architecture</figcaption></figure><h3>Conclusion</h3><p>Hadoop YARN and HDFS form a strong foundation for large-scale data processing. YARN’s distributed resource management and scheduling capabilities enable Spark (and other engines) to run diverse workloads efficiently. Meanwhile, HDFS provides reliable, fault-tolerant storage of massive datasets. By co-locating storage and compute on the same nodes, organizations benefit from faster data access, better resource utilization, and simplified scaling.</p><p>This article gave you a solid idea of how Spark works with YARN and HDFS. But there’s more to learn! In upcoming articles, I’ll talk about <strong>edge nodes</strong>, which act like gateways for users to interact with the cluster, and explain <strong>client mode</strong> vs. <strong>cluster mode</strong> in Spark. These topics will help you set up and manage a production environment more effectively. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fffc150921c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Irregular CSV Files with Spark]]></title>
            <link>https://towardsdev.com/spark-working-on-csv-files-c2f3bc3794d5?source=rss-2bd123571a9f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2f3bc3794d5</guid>
            <category><![CDATA[csv]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[dataengineering-future]]></category>
            <dc:creator><![CDATA[Sravan Reddy]]></dc:creator>
            <pubDate>Wed, 25 Dec 2024 10:25:12 GMT</pubDate>
            <atom:updated>2025-08-19T17:10:19.949Z</atom:updated>
            <content:encoded><![CDATA[<p>CSV known as comma separated file is widely used format in Big Data world. It is plain text, so it’s easy to open and understand and you can use nearly any editor or spreadsheet software to open the csv file.</p><p>Another Reason why it is being used most is , present database tools can easily integrate with csv weather it is reading or writing or exporting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*79K48qwVdjkxKFHTh6GicQ.png" /></figure><p>In order to understand how reading csv , writing csv and controlling happens in Spark you need understand what are quote , escape , escapeQuotes , QuoteAll , Multiline , header</p><p>you can find all these options in spark documentation link below</p><p><a href="https://spark.apache.org/docs/3.5.3/sql-data-sources-csv.html">CSV Files</a></p><p>Let us take a scenarios where you extracting data from database into dataframe in Pyspark</p><p>I have loaded the same in a dataframe</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AGJARIZjbt0LG3IE7bnFbQ.png" /></figure><p>You clearly observe the data we have used we have columns having qoutes , new line characters and commas in middle of values.Let us try various ways on to convert this data into csv and load again correctly into temporary table.</p><blockquote>By Default quote , escape , escapequote will be true</blockquote><h4><strong>1.Delimiter (Read/Write)</strong></h4><p>A delimiter is a character (or sequence of characters) used to separate individual columns within a csv file. Its default value is comma(,) but you can change it to what even character you want like | , ^ , $ ,# , |%| etc.</p><p>Whatever the character you are setting as delimiter make sure it support encoding standards otherwise you will end up getting all null values</p><pre>Delimiter = ,<br>John Doe,30,Software Engineer<br><br>Delimiter = |<br>John Doe|30|Software Engineer</pre><h4>2. Quote (Read/Write)</h4><p>Quote is used to enclose fields that may contain special characters such as commas, newlines(\n) or any special characters.Default Value is “ ( double Quotes)</p><p>Why Quotes Are Used</p><p><strong>Prevent Misinterpretation</strong>: If a field contains a delimiter (e.g., a comma when using CSV), enclosing that field in quotes ensures that the delimiter is treated as part of the field’s text rather than a separator between fields.</p><p><strong>Handle Special Characters</strong>: Fields containing newlines (\n), tabs (\t), or commas can be preserved in their entirety if they’re enclosed in quotes.</p><p><strong>Escape Mechanism</strong>: When a field already has a quote character inside it, systems often use an “escape” or doubled quote (&quot;&quot;) to let the parser know the quote is part of the text and not the end of the field.</p><blockquote>You should remember that Quote doesn&#39;t handle columns which have only quotes in it , those column will be handle by <strong>escapeQuote</strong> option</blockquote><p>I know it might sound confusing I will explain it with an example later</p><pre>John Doe,30,Software Engineer,&quot;He Know Python,Java&quot;<br><br>The Last column is qouted in double quotes because it have comma (delimiter in it)</pre><h4>3. Escape ( Read / Write )</h4><p><strong>Escape</strong> refers to the character used to indicate that a special character (like a quote) should be treated as part of the field’s text rather than a delimiter for that field.</p><ul><li><strong>Example Scenario</strong>: You have a field containing a double quote inside the text, such as:</li></ul><pre>John Doe,30,Software Engineer,He Know &quot;Python&quot; ,Java</pre><p>By Default CSV uses &quot; as the quote character and \ as the escape character to write the data</p><pre>John Doe,30,Software Engineer,&quot;He Know \&quot;Python\&quot; ,Java&quot;</pre><p>Here, the <strong>\</strong> (backslash) <strong>escapes</strong> the double quotes, telling the parser that these quotes are part of the field’s value, not the end of the field</p><h4>4. EscapeQuotes (Write Option Only)</h4><p>EscapeQuotes is a boolean (true/false) configuration option, commonly seen in Spark’s CSV handling, that determines whether quote characters <strong>inside</strong> a field should be automatically escaped when you <strong>write</strong> CSV data.</p><ul><li><strong>When </strong><strong>escapeQuotes is </strong><strong>true</strong>:<br>Spark will look for any quote characters within the field and prepend the <em>escape</em> character (usually a backslash \) before them.</li><li><strong>When </strong><strong>escapeQuotes is </strong><strong>false</strong>:<br>Spark will <strong>not</strong> escape quote characters inside the field. If your data contains quotes and you don’t handle them in some other way (e.g., manually escaping them or using a different format), the resulting CSV might be ambiguous or cause parsing errors later.</li></ul><p>By Default escapeQuotes is true , You should disable it only when have perfect data with only text or number but i would not dare to do it.</p><p>Quote , Escape , EscapeQuote are interlinked to each other . Let me Explain it via example</p><p>In the below example</p><ol><li>In first column we have quotes only</li><li>4th column we have extra delimiter</li><li>5th Column we have extra delimiter , new line character and Quotes also</li></ol><pre>| Name           | Age | Role                | Description                    | Hobbies                |<br>|----------------|-----|---------------------|--------------------------------|------------------------|<br>| John &quot;le&quot; Doe  | 30  | Software Engineer   | He Know Python, Java and C++   | Movies , and <br>                                                                                &quot;Cricket&quot;              |</pre><ol><li><strong>Quote = “ , Escape = \ and EscapeQuotes = True (Default Options)</strong></li></ol><p>When you write the dataframe into csv with this options , you will get the below output</p><pre>&quot;John \&quot;le\&quot; Doe&quot;,30,Software Engineer,&quot;He Know Python,Java and C++&quot;,&quot;Movies <br>and \&quot;Cricket\&quot; &quot;</pre><p>2.<strong>Quote = “ , Escape = \ and EscapeQuotes = False</strong></p><pre>John &quot;le&quot; Doe,30,Software Engineer,&quot;He Know Python,Java and C++&quot;,&quot;Movies <br>and \&quot;Cricket\&quot; &quot;</pre><p>If you observe the difference between first and second output you will find difference for the first column.</p><p>In the First option ,</p><p>The job of the quote is to handle the column fields which have extra delimiters or new line characters. Here we had a new line char in Last column field , so it will surround the field in quotes telling the files praser to consider that as single column.</p><pre>Field with New Char<br><br>Movies <br>and &quot;Cricket&quot;<br><br>Quote will surround the field due to new line character<br><br>&quot; Movies <br>and &quot;Cricket&quot; &quot;</pre><p>But what if there are quotes already in columns which have extra delimiters or new line characters , <strong>Escape Character handles this issues</strong>.</p><p>In the second option we have mentioned that EscapeQuotes=False , when it was true the columns values which contain only quotes will also be quoted but when we have mentioned it as False , it didn&#39;t quotes the 1st column</p><h4>5.QuoteAll ( Write Option Only)</h4><p>QuoteAll is a boolean (true/false) configuration option, which specifies whether <strong>every field</strong> in the output CSV file should be enclosed in quotes regardless of whether it contains special characters.</p><p><strong>quoteAll=False</strong> (default):Only fields containing special characters (such as the delimiter, quote character, or new line) get enclosed in quotes.Other fields, without special characters, remain unquoted.</p><p><strong>quoteAll=true </strong>All<strong> </strong>fields are surrounded by quotes, even if they do not contain a delimiter, quote, or newline.This can be helpful if you want a very consistent output format, but it can also make the CSV size larger because of the extra quotes.</p><pre>&quot;John Doe&quot;,&quot;30&quot;,&quot;Software Engineer&quot;,&quot;He Know \&quot;Python\&quot; ,Java&quot;</pre><h4>6.MultiLine ( Read Option )</h4><p>Multiline is a Boolean setting that controls how Spark handles <strong>multiline rows</strong> in a CSV file. By default, Spark expects <strong>one record per line</strong> (multiline = false ). However, if your CSV file has fields that span multiple lines (e.g., a quoted field with line breaks inside), you need to enable multiline mode (multiline = true) while reading the file.</p><p>You should keep in mind that this option will degrade the jobs performance . When multiline parsing is enabled in Spark, it typically forces Spark to process each CSV file with <strong>a single executor</strong> or a single partition.</p><p>The reason is that Spark’s default parallelization strategy for CSV relies on <strong>line-by-line splitting</strong> (each line is treated as one row) , it can spilt the file and run them in different executors but In multiline mode a single CSV row can span multiple lines, so Spark cannot reliably split the file at arbitrary line boundaries. Instead, it must process the file <strong>sequentially</strong> to correctly identify where each record starts and ends.</p><p>That’s it for this one!</p><p>Please Comment below on this article if you found mistakes</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2f3bc3794d5" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/spark-working-on-csv-files-c2f3bc3794d5">Handling Irregular CSV Files with Spark</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>