<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yogesh Srivastava on Medium]]></title>
        <description><![CDATA[Stories by Yogesh Srivastava on Medium]]></description>
        <link>https://medium.com/@Ysrivast?source=rss-187ef0168297------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y-GFdg8AwUJmm9VY4gVhAw.jpeg</url>
            <title>Stories by Yogesh Srivastava on Medium</title>
            <link>https://medium.com/@Ysrivast?source=rss-187ef0168297------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:23:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Ysrivast/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Parquet- Colum Based Store]]></title>
            <link>https://medium.com/xebia-engineering/apache-parquet-colum-based-store-501e039788a0?source=rss-187ef0168297------2</link>
            <guid isPermaLink="false">https://medium.com/p/501e039788a0</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[columnar-storage]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Yogesh Srivastava]]></dc:creator>
            <pubDate>Wed, 03 Sep 2025 20:00:38 GMT</pubDate>
            <atom:updated>2025-09-03T20:00:38.123Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Parquet- Column Based Store</h3><p><a href="https://parquet.apache.org/docs/overview/"><em>Apache Parquet</em></a><em> is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming language and analytics tools.</em></p><figure><img alt="How Data Flows The application sends data to the writer. The writer stores data in the Parquet file (fileSummary: This diagram provides a high-level overview of how data moves through the Apache Parquet system — from its origin in an application, through writing and storing in a Parquet file, reading the file, and finally being consumed for analytics or other purposes. internals**). Later, a reader loads data from" src="https://cdn-images-1.medium.com/max/184/1*sOAjyAroRhKTo10epmo2qA.png" /><figcaption>Apache Parquet Architecture</figcaption></figure><p><em>An analytics application sends data to a Parquet </em><strong><em>writer</em></strong><em>, which organizes values by column into row groups and pages and records schema and statistics in a footer for efficient access. Inside the file, this columnar layout (row groups → column chunks → pages) lets engines read only needed columns and skip entire groups using min/max metadata. A Parquet </em><strong><em>reader</em></strong><em> loads the footer, prunes to requested columns and matching row groups, then decodes selected pages to reconstruct rows. The reconstructed data then flows to analytics tools, BI dashboards, or ML/Arrow-based pipelines for fast processing and interoperability.</em></p><blockquote>How Parquet Works</blockquote><p><em>Unlike traditional row-based formats (like CSV), Parquet stores data by columns, which allows for better compression and faster query performance especially when only a subset of columns is needed. It also supports complex nested data structures and is optimized for both read and write operations, making it ideal for large-scale data workflows.</em></p><p><strong>Columnar Storage</strong> parquet stores data by columns instead of rows, which is a key difference from traditional formats like CSV or JSON. This columnar layout allows for better compression and faster query performance because similar data types are stored together. When a query only needs a few columns, Parquet can read just those columns, significantly reducing disk I/O and speeding up processing.</p><p><strong>Schema and Metadata e</strong>ach Parquet file is self-describing, meaning it contains its own schema definition. This schema is stored using Apache Thrift and includes information about column names, data types, and the structure of nested fields. This makes it easy for different systems to read and interpret the data without needing external</p><p><strong>File Structure </strong>a parquet file is composed of several key components. The file begins with a header that includes a magic number (PAR1) to identify the file format. The data is organized into row groups, which are blocks of rows stored together. Each row group is further divided into column chunks, one for each column, and these chunks are split into pages—the smallest units of storage. Finally, the footer contains metadata about the schema, row groups, and statistics, enabling efficient data access and filtering during queries.</p><p><strong>Compression and Encoding </strong>parquet supports multiple compression algorithms such as Snappy (the default), GZIP, and Brotli. It also employs various encoding techniques like dictionary encoding, run-length encoding (RLE), and bit-packing. These methods help reduce file size and improve read/write performance by minimizing redundancy and optimizing how data is stored. This makes Parquet highly efficient for both storage and processing.</p><p><strong>Predicate Pushdown </strong>one of Parquet’s most powerful features is predicate pushdown, which allows query engines to skip reading irrelevant data based on query filters. For example, if a query filters for age &gt; 30, Parquet can use metadata to skip row groups where all values are 30 or less. This significantly improves query performance by reducing the amount of data read from disk and processed in memory.</p><p><strong>Support for Nested Data </strong>parquet supports complex and nested data structures such as structs, lists, and maps. This makes it suitable for storing semi-structured data like JSON, allowing for rich data modeling while maintaining the benefits of a columnar format. It is especially useful in modern data applications that require flexibility in data representation.</p><p><strong>Tool and Language Support </strong>parquet is widely supported across many tools and programming languages. It integrates seamlessly with big data tools like Apache Spark, Hive, Drill, and Presto. In Python, libraries such as pyarrow and fastparquet allow for easy reading and writing of Parquet files, making it a popular choice for data scientists and engineers working with Pandas and Dask. Its broad compatibility makes it a go-to format for scalable data workflows.</p><blockquote><strong>Parquet File Structure</strong></blockquote><p>Apache Parquet files are organized to enable efficient storage, compression, and query performance. The structure consists of several key components:</p><p><strong>File Header </strong>contains a “magic number” (PAR1) identifying the file as a Parquet file.</p><p><strong>Row Groups </strong>the file is divided into one or more row groups. Each row group contains a subset of rows from the dataset. Row groups allow parallel processing and efficient reading.</p><p><strong>Column Chunks </strong>inside each row group, data is stored in column chunks — one per column. Each column chunk contains all the data for a single column in that row group. Enables columnar storage, compression, and encoding.</p><p><strong>Pages </strong>column chunks are further divided into pages (the smallest unit of storage). Each page typically contains data, repetition/definition levels, or dictionary values. Pages allow fine-grained compression and encoding.</p><p><strong>File Footer </strong>contains metadata, including schema (column names, types, structure), row group information (locations, sizes) and column statistics (min, max, null counts). Footer also has a magic number (PAR1) for verification.</p><p><strong>Parquet File Layout Diagram</strong></p><pre>+-------------------+<br>|   File Header     |   (magic number)<br>+-------------------+<br>|   Row Group 1     |<br>|  +-------------+  |<br>|  | Column A    |  |<br>|  | Column B    |  |<br>|  +-------------+  |<br>+-------------------+<br>|   Row Group 2     |<br>|  +-------------+  |<br>|  | Column A    |  |<br>|  | Column B    |  |<br>|  +-------------+  |<br>+-------------------+<br>|   ...             |<br>+-------------------+<br>|   File Footer     |   (metadata, schema, stats)<br>+-------------------+</pre><blockquote><strong>Summary</strong></blockquote><pre>Header: Identifies file type.<br>Row Groups: Partition data for processing.<br>Column Chunks: Store data column-wise.<br>Pages: Fine-grained storage/compression.<br>Footer: Stores metadata and schema.</pre><p>This structure is what enables Parquet’s high performance and interoperability for big data analytics.</p><blockquote><strong>Parquet Example</strong></blockquote><p>Suppose you have a simple table of users:</p><pre>| id  | name   | age  |<br>| - - | - - - -| - - -|<br>| 1   | Alice  | 30   |<br>| 2   | Bob    | 25   |<br>| 3   | Carol  | 27   |</pre><p>In a Parquet file, these values would be stored grouped by columns, not by rows:</p><pre>- id:   [1, 2, 3]<br>- name: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Carol&quot;]<br>- age:  [30, 25, 27]</pre><p>This format is especially efficient for analytic queries like “What is the average age?” because only the age column needs to be read.</p><blockquote>Advantages of Parquet</blockquote><ol><li>Efficient Columnar Storage: Great for queries that scan only a few columns.</li><li>Compression: Reduces disk I/O and storage costs.</li><li>Schema Evolution: Supports changes to schema over time.</li><li>Interoperability: Supported by many big data tools (Spark, Hive, Presto, etc.).</li><li>Splitting and Parallel Processing: Columnar format allows splitting files for distributed processing.</li></ol><blockquote>Disadvantages of Parquet</blockquote><ol><li>Not Optimized for Row-based Access: Retrieving entire rows can be slower compared to row-based formats (like CSV, JSON).</li><li>Complexity: Requires more setup and understanding of schemas and data types.</li><li>Tools and Libraries Needed: You need specific libraries to read/write Parquet files, unlike CSV or JSON.</li><li>Overkill for Small Datasets: Parquet’s benefits shine with large datasets. For small data, simple formats may suffice.</li><li>Binary Format: Not human-readable — requires tooling to inspect or debug.</li></ol><blockquote>Building a RESTful Parquet Data Service in Java: Exploring parquet</blockquote><p>Data engineering often demands fast, efficient storage formats for big data. Parquet — a columnar storage format — has become a favourite in this space. But what if you want to work with Parquet files in your own Java application, and even expose them via REST APIs? That’s the problem solved by the [<a href="https://github.com/ysrivast/parquet-demo">ysrivast/parquet-demo</a>] project.</p><p><strong><em>What is parquet-demo?</em></strong></p><p>parquet-demo is a Spring Boot application that demonstrates how to read and write Parquet files in Java. The project exposes RESTful endpoints for basic CRUD (Create, Read, Update, Delete) operations, making it easy to interact with Parquet data over HTTP. The idea is to serve as a practical starter for anyone looking to integrate Parquet handling into Java microservices, with future plans to support XML input, S3 storage, and a simple web UI.</p><p><strong><em>How Does It Work?</em></strong></p><p>At the core, the project uses classes like CustomParquetWriter and CustomParquetReader, which wrap Apache Parquet’s APIs for file operations. The configuration is flexible, letting you specify schema and output paths in application.properties.</p><p>The main application ParquetDemoApplication initializes example data and writes it to Parquet files via the service layer. Reading from Parquet is performed using custom logic that converts raw Parquet column data into Java objects.</p><p><strong><em>Why Use This Project?</em></strong></p><p>If you need a quick, extendable Java backend for handling Parquet files — whether for analytics, ETL, or as a microservice — parquet-demo offers a clean starting point. Its modular design and Spring Boot foundation make it easy to customise or integrate into larger systems.</p><p>Explore the code and contribute: [<a href="https://github.com/ysrivast/parquet-demo">ysrivast/parquet-demo on GitHub</a>]</p><p>Feel free to open issues or feature requests!</p><blockquote>Parquet is a powerful format for efficient, scalable columnar data storage — ideal for analytics, but may be unnecessary for small/simple data needs.</blockquote><p>Let me know if you want more examples, diagrams, or a deeper dive into any aspect!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=501e039788a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xebia-engineering/apache-parquet-colum-based-store-501e039788a0">Apache Parquet- Colum Based Store</a> was originally published in <a href="https://medium.com/xebia-engineering">Xebia Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prototype design pattern]]></title>
            <link>https://medium.com/xebia-engineering/prototype-design-pattern-cad550f94745?source=rss-187ef0168297------2</link>
            <guid isPermaLink="false">https://medium.com/p/cad550f94745</guid>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[prototype]]></category>
            <dc:creator><![CDATA[Yogesh Srivastava]]></dc:creator>
            <pubDate>Thu, 04 Jun 2020 03:59:36 GMT</pubDate>
            <atom:updated>2020-06-04T03:59:36.401Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-t2-XvLC_1SjmnXu0eoymw.jpeg" /></figure><p><strong>What is the design pattern ?</strong></p><p>A design pattern is an efficient way to solve a particular class of problem. It is like how a lot of people work and find the best solution for a problem.</p><p><strong>Lets talk about prototype design pattern</strong></p><p>The Prototype pattern is basically the creation of new instances through cloning existing instances. By creating a prototype, new objects are created by copying this prototype.</p><p>It is useful when the object component is already configured once and there is a need for a similar object. The concept of copying an existing object rather than creating a new instance from scratch is a very costly operation. Using a prototype saves costly resources and time especially when the object is very costly to create. An existing object acts as a prototype which contains the state of the object. It is possible that the newly created object may require changes to some of its property.</p><p><strong>Let’s have a look on UML of prototype</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*Iu9UeoAr_8Xu0pXO" /></figure><p>In this diagram</p><p>the prototype is a an interface that defines a function clone()</p><p>A real prototype has to implement this interface and implement the clone() function to return a copy of itself.</p><p>Lets understand with example :-</p><p>Suppose we have a book and bookshop object where the bookshop is fetching all books from the database. So every time we create a new instance of bookshop will result in a query to the database for fetching the record of all books. So to improve the performance will use a prototype pattern .</p><pre><em>package com.xebia.prototype.example;</em></pre><pre><em>public class Book implements Cloneable{</em></pre><pre><em>private Long id;<br>private String name;<br>private String author;</em></pre><pre><em>public Book(Long id, String name, String author) {<br>  this.id = id;<br>  this.name = name;<br>  this.author = author;<br>}</em></pre><pre><em>//setters getters</em></pre><pre><em>@Override<br>protected Book clone() throws CloneNotSupportedException {</em></pre><pre><em>  Book book = new Book(this.id, this.name, this.author);<br>  return book;<br>  }</em></pre><pre><em>}</em></pre><pre><em>package com.xebia.prototype.example;</em></pre><pre><em>import java.util.ArrayList;<br>import java.util.List;</em></pre><pre><em>public class BookShop implements Cloneable {</em></pre><pre><em>  private String name;<br>  private List&lt;Book&gt; books;</em></pre><pre><em>  public BookShop() {<br>   this.books = new ArrayList&lt;&gt;();<br>  }</em></pre><pre><em>  public void initialize() {<br>  // let’s suppose list is coming from database</em></pre><pre><em>  for (int i = 0; i &lt; 10; i++) {<br>   Book book = new Book(Long.valueOf(i), “Book-” + i, “Author-” + i);<br>  this.books.add(book);<br>   }<br>  }</em></pre><pre><em>@Override<br>protected BookShop clone() throws CloneNotSupportedException {<br>  BookShop shop = new BookShop();<br>  for (Book book : this.getBooks()) {        shop.getBooks().add(book.clone());<br>  }<br>return shop;<br>}</em></pre><pre><em> //setters getters<br>}</em></pre><pre><em>public class BookTest {</em></pre><pre><em>public static void main(String[] args) throws CloneNotSupportedException {<br>  BookShop bookShop = new BookShop(“Xebia Shop India”);<br>  bookShop.initialize();<br>  System.out.println(bookShop);</em></pre><pre><em>  BookShop bookShop2 = bookShop.clone();<br>  bookShop2.setName(“Xebia Shop DUbai”);<br>  System.out.println(bookShop2);<br>  }<br>}</em></pre><p>After looking into the code above we can see for the first time when we are creating a bookshop it’s fetching all the books from the database. Next time when we create a new instance by cloning the bookshop object it is fetching data from the first object. So that is the way we can create multiple instances of the bookshop.</p><p>Object cloning can be done in two ways:-</p><ol><li>Shallow cloning</li><li>Deep cloning</li></ol><p>When we use the default clone method, it will create a shallow copy of the object. It means the cloned object will have the same state as the source object. In shallow cloning only memory addresses are copied if any changes made in the field of the object will reflect to the cloned object as well. It works similar to deep cloning in case of primitives</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*Aj3pOb5s6wOfaHPr" /></figure><p>A deep cloning object will have the same state as shallow cloning but the original and cloned object will be fully disjoint. They will be independent of each other. If there is any change made to the clone object it will not be reflected in the original object or vice-versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/0*XzhOKFt_hkIumoy6" /></figure><p>In the above example we have used deep cloning of book to ensure that every bookshop can be independent of each other.</p><p>Where to use</p><ul><li>When a system needs to be independent of how its objects are created, composed, and represented.</li><li>When adding and removing objects at runtime.</li><li>When specifying new objects by changing an existing objects structure.</li><li>When configuring an application with classes dynamically.</li><li>When trying to keep the number of classes in a system to a minimum.</li><li>When state population is an expensive or exclusive process.</li></ul><p>Benefits</p><ul><li>Speeds up instantiation of large, dynamically loaded classes.</li><li>Reduced subclassing.</li></ul><p>Drawbacks/consequences</p><ul><li>Each subclass of Prototype must implement the Clone operation.</li><li>Could be difficult with existing classes which do not support copying.</li></ul><p>That’s it for this blog post. I hope you found it useful and stay tuned for more 😀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cad550f94745" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xebia-engineering/prototype-design-pattern-cad550f94745">Prototype design pattern</a> was originally published in <a href="https://medium.com/xebia-engineering">Xebia Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>