<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pratichi on Medium]]></title>
        <description><![CDATA[Stories by Pratichi on Medium]]></description>
        <link>https://medium.com/@pratichi3?source=rss-95d5850e185e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j7qMCNc_AehFwyfFZE4n_w.jpeg</url>
            <title>Stories by Pratichi on Medium</title>
            <link>https://medium.com/@pratichi3?source=rss-95d5850e185e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:34:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pratichi3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon SDE I Full Loop Interview Experience]]></title>
            <link>https://pratichi3.medium.com/amazon-sde-i-full-loop-interview-experience-b199ad972356?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b199ad972356</guid>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[faang]]></category>
            <category><![CDATA[interview-experience]]></category>
            <category><![CDATA[new-grad]]></category>
            <category><![CDATA[sde-1]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 04:05:15 GMT</pubDate>
            <atom:updated>2026-11-20T04:30:10.424Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Recruiter email → OA → 4-round virtual loop → decision</em></blockquote><blockquote><strong>Role:</strong> Software Development Engineer I (new grad, 2026), USA</blockquote><h3>The Process</h3><ol><li><strong>Recruiter outreach:</strong> Amazon emailed me, <em>“Still interested?”</em> I reapplied to the new grad SDE role through Amazon.jobs.</li><li><strong>Online Assessment:</strong> I got the OA invite soon after and completed every section.</li><li><strong>Full loop:</strong> 4 rounds of 60 minutes each on Zoom, with coding in Amazon’s LiveCode editor. Most rounds had Leadership Principle (behavioral) questions first, then a technical question.</li><li><strong>Decision:</strong> within two business days.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sGXxgiSPaMFMZ3Fg4ulIEg.png" /></figure><h3>Timeline</h3><ul><li>Recruiter email → OA invite: ~1 week</li><li>OA invite → OA completed → within a week</li><li>OA completed → full loop invite: ~3–5 days</li><li>Full loop invite → virtual full loop: ~2.5 weeks</li><li>Full loop → decision: 2 business days</li></ul><h3>The 4 Rounds</h3><p><strong>Round 1: Hiring Manager</strong></p><ul><li><em>Behavioral:</em> I was asked how I use AI and about a time it gave me wrong results. There were a lot of follow-up questions.</li><li><em>Coding:</em> It was a real-world-style problem built around hierarchical access control: checking whether a user has access through direct, team-level, or organization-level permissions. The task was to check whether a user has access to a resource at any of these levels. I didn’t spot the pattern right away, so I started with a brute-force approach and explained my reasoning. Looking back, it’s a <strong>tree traversal / recursion</strong> problem: start at the user and walk up the hierarchy.</li></ul><p><strong>Round 2: SDE IIs</strong></p><ul><li><em>Behavioral:</em> I was asked about a challenge I faced. I shared a technical story, and the follow-ups went into deadlines and prioritization.</li><li><em>Coding:</em> It was a frequency-counting hashmap problem. I wrote a working solution, tested it with different test cases, and explained the time and space complexity. I was asked to optimize the SC , but time ran out.</li></ul><p><strong>Round 3: Bar Raiser</strong></p><ul><li>This round was fully behavioral. I was asked about looking back on what I could have done differently, and about moving forward without clear requirements. Every answer led to more follow-ups. This was my best round. I had prepared around 12 stories from my work experience and academics, so I felt calm and confident, and the follow-ups felt more like a conversation.</li></ul><p><strong>Round 4: Object-Oriented Design</strong></p><ul><li><em>Behavioral:</em> I was asked about a time I took initiative on something nobody had asked me to do.</li><li><em>Design:</em> I was asked to design a pizza-ordering system with customizable toppings and a way to calculate the total cost. I started by listing the requirements, then defined the classes and functions, and built a brute-force approach that calculated the cost based on the selected toppings. The interviewer was very supportive throughout. He gave me suggestions and told me my brute-force approach was correct, then asked how I would scale it.</li></ul><h3>Result</h3><p>Not selected.</p><h3>My Learnings</h3><ol><li><strong>Recognizing patterns is harder when the problem is a story.</strong> The coding questions didn’t look like standard LeetCode problems. They were framed as real-world scenarios, and I couldn’t spot the underlying pattern in time.</li><li><strong>Keep practicing every day.</strong> Taking a year off DSA hurt me. Doing even one or two problems a day keeps your pattern recognition sharp, so you don’t have to rebuild it right before an interview.</li><li><strong>Low-level design needs its own prep.</strong> I only had time for last-minute practice on standard questions like a parking lot design, and it wasn’t enough. LLD deserves dedicated practice, separate from DSA.</li><li><strong>Practice patterns together with real-world use cases.</strong> For each pattern, think of real situations where it applies. Then when the same pattern shows up inside a new story, you’ll recognize it.</li><li><strong>Know your stories in depth.</strong> My behavioral prep helped a lot, and I felt confident, but every round pushed hard with follow-up questions.</li></ol><h3>Final Thoughts</h3><p>I’m not disappointed. I went into this interview with nothing to lose. I wanted to find out what a FAANG-level SDE interview really feels like. I used my three weeks to prepare, revise, and give it my best.</p><p>In the end, it comes down to how well you’re in the flow and how much of your best you can bring in those 60 minutes. Now I know what that looks like, and I know exactly what to work on next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TUjyKSukmfyaB0Z1o3dZUw.png" /></figure><p>If you’re preparing for Amazon, I hope this helps. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b199ad972356" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Before You Start LLD: The Basics That Actually Matters]]></title>
            <link>https://pratichi3.medium.com/part-1-before-you-start-lld-the-basics-that-actually-matters-2ed0c1345a9a?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ed0c1345a9a</guid>
            <category><![CDATA[lld]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[low-level-design]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Sun, 21 Dec 2025 19:31:52 GMT</pubDate>
            <atom:updated>2025-12-21T19:31:52.261Z</atom:updated>
            <content:encoded><![CDATA[<p>If you want to start <strong>Low-Level Design (LLD)</strong> but feel lost before even picking a question, this is for you. This blog is not about mastering LLD, it’s about knowing just enough to start solving LLD problems without fear.</p><h3>Step1 : Learn how to <em>think</em> in classes</h3><p>Pick something familiar, like a <strong>Student</strong>, and answer three questions:</p><ul><li>what is the <strong>entity</strong> name? Student</li><li>what <strong>data</strong> does it have? id, name, marks</li><li>what <strong>actions</strong> can it do? get_average()</li></ul><pre>class Student:<br>  def __init__(self,name,id,marks):<br>    self.name = name<br>    self.id = id<br>    self.marks = marks<br><br>  def get_average(self):<br>    return sum((self.marks) / len(self.marks))</pre><h3>Step 2: Understand Relationships Between Classes</h3><p>Once you can create a single class, the next step is understanding <strong>how classes relate to each other</strong>.</p><blockquote>Just ask one simple question each time: <strong>Can one object exist without the other?</strong></blockquote><h3>1. IS-A → Inheritance</h3><h4><em>One thing </em><strong><em>is a type of</em></strong><em> another thing.</em></h4><p>Ask: <em>Is a Dog an Animal? YES</em></p><p>When to use: When classes share behavior , child can replace parent</p><pre>class Animal:<br>    def sound(self):<br>        return &quot;Some sound&quot;<br><br>class Dog(Animal):  <br>    def sound(self):<br>        return &quot;Bark&quot;</pre><h3>2. HAS-A → Composition (Strong ownership)</h3><h4><em>One thing </em><strong><em>owns</em></strong><em> another thing.</em></h4><p>Ask: <em>Can an Engine exist meaningfully without a Car? NO</em></p><p>Key idea: Engine is created inside Car , If Car is destroyed → Engine is gone</p><pre>class Engine:<br>    def start(self):<br>        return &quot;Engine started&quot;<br><br>class Car:<br>    def __init__(self):<br>        self.engine = Engine() </pre><h3>3. HAS-A → Aggregation (Weak ownership)</h3><h4><em>One thing </em><strong><em>has</em></strong><em> another thing, but does NOT own it fully.</em></h4><p>Ask: <em>Can a Book exist without a Library? YES</em></p><p>Books can exist independently and can even belong to multiple libraries.</p><pre>class Book:<br>    def __init__(self, title):<br>        self.title = title<br><br>class Library:<br>    def __init__(self, books):<br>        self.books = books </pre><p>Here, library has books and library does not control the lifetime of books. This is <strong>aggregation</strong>.</p><h3>4. USES-A → Association</h3><h4>One thing <strong>uses</strong> another thing temporarily.</h4><p>Ask: <em>Does a Driver own a Car? → </em><strong><em>NO , </em></strong><em>Does a Driver use a Car? → </em><strong><em>YES</em></strong></p><p>Key idea: No ownership , Object is passed as a parameter, Very common in real systems</p><pre>class Driver:<br>    def drive(self, car):<br>        print(&quot;Driving the car&quot;)</pre><h3>A Simple Way to Remember Relationships</h3><ol><li>Is it a type of? → Inheritance</li><li>Can it exist alone? → No , Composition</li><li>Can it exist alone? → Yes, Aggregation</li><li>Is it temporary usage? → Association</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ed0c1345a9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Think About File Formats During Data Ingestion]]></title>
            <link>https://pratichi3.medium.com/how-to-think-about-file-formats-during-data-ingestion-d91d8118e306?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d91d8118e306</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[file-format]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 22:42:25 GMT</pubDate>
            <atom:updated>2025-11-05T22:42:25.301Z</atom:updated>
            <content:encoded><![CDATA[<p>When you build a data pipeline, you’re not just “collecting data.” You’re deciding <strong>how data will flow, be stored, and be read</strong> across every system in your stack.</p><p>That’s why your choice of <strong>file format</strong> is <em>not</em> trivial. It decides how much your pipeline costs, how fast it runs, and even whether it scales when the data volume explodes.</p><h3>Step 1: Think About the Type of Source and Ingestion Mode</h3><p>Before worrying about Parquet or Avro, pause and ask yourself two questions:</p><ol><li><strong>Where is my data coming from?</strong></li></ol><ul><li>File dumps (e.g., daily CSV reports)</li><li>APIs or logs (JSON)</li><li>Databases (via connectors like JDBC)</li><li>Streaming platforms (Kafka, Kinesis)</li></ul><p><strong>2. How is it coming in?</strong></p><ul><li><strong>Batch:</strong> Loaded in chunks every few hours or days.</li><li><strong>Streaming:</strong> Constant real-time flow of events.</li></ul><blockquote><strong>Why this matters:</strong><br> → Batch = more data at once, latency is okay → focus on <em>compression, parallel reads</em>.<br> → Streaming = small frequent data, latency-sensitive → focus on <em>schema, compact size, serialization speed</em>.</blockquote><h3>Step 2: Row-Based vs Column-Based — The Core Divide</h3><p>Imagine your dataset like a spreadsheet:</p><ul><li><strong>Row-based</strong> means storing one full record after another. Great for writing or transmitting data — like how logs are appended line by line.</li><li><strong>Column-based</strong> means storing each column’s data together. Great for reading selectively — like running a query that only needs “price” and “date.”</li></ul><blockquote><strong>Row-based formats</strong> → CSV, JSON, Avro</blockquote><blockquote><strong>Column-based formats</strong> → Parquet, ORC</blockquote><h3>Rule of thumb:</h3><blockquote><em>Row-based = better for </em><strong><em>write-heavy / streaming</em></strong><em><br>Column-based = better for </em><strong><em>read-heavy / analytics</em></strong></blockquote><h3>Step 3: The 6 Big Factors That Influence File Format Choice</h3><p>Let’s now understand how to think like an engineer, not just memorize file names.</p><h3>1. Schema Enforcement</h3><p>Schema defines structure, what fields exist, what types they have.</p><ul><li><strong>CSV / JSON:</strong> No enforced schema → flexible but risky (can break pipelines if columns change).</li><li><strong>Avro / Parquet / ORC:</strong> Schema is stored with the data → strong typing, supports schema evolution.</li></ul><blockquote>💡 <em>If your upstream systems change often (adding/removing fields), go with Avro or Parquet.</em></blockquote><h3>2. Compression &amp; Storage Cost</h3><p>Big data = expensive storage. The format determines how much you save.</p><ul><li><strong>CSV / JSON:</strong> Almost no compression → large files, costly on S3.</li><li><strong>Parquet / ORC:</strong> Built-in compression (Snappy, ZSTD, GZIP) → 5x–10x smaller.</li><li><strong>Avro:</strong> Medium compression — smaller than JSON but not as tight as Parquet.</li></ul><blockquote><strong>💡 <em>Always compress — your cloud bill will thank you.</em></strong></blockquote><h3>3. Splittability (Parallel Processing)</h3><p>Can multiple workers read the same file simultaneously?</p><ul><li><strong>CSV, JSON:</strong> Not splittable — one worker reads the full file.</li><li><strong>Parquet, ORC, Avro:</strong> Splittable — each worker processes a part in parallel (critical for Spark, Hadoop, EMR).</li></ul><blockquote><strong>💡 <em>If your dataset is large (GBs+), splittability = speed.</em></strong></blockquote><h3>4. Throughput and Latency</h3><p>This is about <strong>how fast</strong> your data moves through the system.</p><ul><li><strong>Throughput:</strong> Amount of data processed per second.</li><li><strong>Latency:</strong> Time between data generation → availability.</li></ul><p>Now connect this to formats:</p><ul><li><strong>Row-based (Avro)</strong> → High throughput for writes; low latency for real-time systems (Kafka).</li><li><strong>Column-based (Parquet)</strong> → Optimized for reads; lower throughput when writing frequently.</li></ul><blockquote><strong>💡 <em>Streaming = low latency → Avro; Analytics = high read throughput → Parquet.</em></strong></blockquote><h3>5. Ecosystem Compatibility</h3><p>Different tools “prefer” different formats.</p><ul><li><strong>CSV / JSON:</strong> Universally readable.</li><li><strong>Avro:</strong> Natively integrates with Kafka, Hadoop, Confluent Schema Registry.</li><li><strong>Parquet / ORC:</strong> Best with Spark, Hive, Athena, Databricks, and cloud warehouses (Snowflake, BigQuery).</li></ul><blockquote><strong>💡 <em>If you’re using Spark or Databricks — Parquet is the default champion.</em></strong></blockquote><h3>6. Human Readability &amp; Debugging</h3><p>Sometimes you just want to open the file and “see” the data.</p><ul><li><strong>CSV / JSON:</strong> Human-friendly, easily editable.</li><li><strong>Avro / Parquet / ORC:</strong> Binary — compact but not human-readable.</li></ul><blockquote><strong>💡 <em>During debugging or small POCs, use CSV/JSON. For production-scale data, go binary.</em></strong></blockquote><h3>Step 4: Real-World Use Cases</h3><p><strong>Example 1: Real-Time Data Ingestion (Kafka → S3) : </strong>I’d use Avro because it’s row-based, compact, supports schema evolution, and integrates directly with Kafka producers/consumers. Later, I can convert to Parquet for analytical queries.”</p><p><strong>Example 2: Batch Analytics (S3 → Spark / Athena) : </strong>I’d store data in Parquet because it’s columnar, splittable, and compressed, perfect for selective queries and cost-efficient scans.</p><p><strong>Example 3: Simple Reports or APIs : </strong>For small datasets or REST API ingestion, JSON is fine — human-readable and widely supported.”</p><h3>Step 5: Quick Framework for Choosing</h3><p>When deciding in a project or explaining in an interview:</p><ol><li>Is the data streaming? → Use <strong>Avro</strong></li><li>Is it large-scale analytical data? → Use <strong>Parquet</strong> or <strong>ORC</strong></li><li>Is it small or human-readable? → Use <strong>CSV / JSON</strong></li><li>Does schema evolve often? → Prefer <strong>Avro / Parquet</strong></li><li>Need parallel reads and cost-efficiency? → <strong>Parquet</strong></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d91d8118e306" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OLTP(Online Transaction Processing)- The “Everyday Transaction” System]]></title>
            <link>https://pratichi3.medium.com/oltp-online-transaction-processing-the-everyday-transaction-system-53018baaf732?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/53018baaf732</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[oltp]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[oltp-vs-olap]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Sat, 25 Oct 2025 20:09:14 GMT</pubDate>
            <atom:updated>2025-10-25T20:09:14.551Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. What is OLTP?</h3><p><strong>OLTP is a type of system architecture designed to manage real-time transactional data.</strong></p><p><strong>Core idea:</strong> OLTP systems handle day-to-day business operations where speed, accuracy, and reliability are critical.</p><blockquote><strong><em>Analogy:</em></strong><em> Think of OLTP as a </em><strong><em>fast, reliable cashier, </em></strong><em>handling millions of transactions accurately, instantly, and concurrently.</em></blockquote><h3>2. Why do we use OLTP?</h3><p>OLTP is required for systems that need:</p><ol><li><strong>Real-time processing: </strong>actions must be recorded instantly</li><li><strong>High concurrency: </strong>multiple users transacting simultaneously</li><li><strong>Data integrity: </strong>no overselling, duplication, or inconsistencies</li><li><strong>Reliability: </strong>transactions must either fully succeed or rollback</li></ol><h3>3. OTLP Principles</h3><p>OLTP systems follow <strong>core principles</strong> to ensure reliability, speed, and data integrity:</p><h4>A. ACID Properties</h4><ul><li><strong>Atomicity:</strong> All steps of a transaction succeed or none do</li><li><strong>Consistency:</strong> Database remains valid after each transaction</li><li><strong>Isolation:</strong> Concurrent transactions do not interfere</li><li><strong>Durability:</strong> Committed transactions persist permanently</li></ul><h4>B. Normalization</h4><ul><li>Reduces data redundancy and prevents anomalies</li><li>Example: Separate tables for Orders, Order_Items, Inventory, and Payment</li></ul><h4>C. Row-Oriented Storage</h4><ul><li>Optimized for <strong>fast single-row operations</strong></li><li>Only the relevant rows are read/written</li></ul><h4>D. High Concurrency</h4><ul><li>Handles millions of simultaneous transactions</li><li>Uses <strong>locking or MVCC</strong> to avoid conflicts</li></ul><h4><strong>E. Real-Time Processing</strong></h4><ul><li>Ensures <strong>instant transaction updates</strong>, supporting live applications</li></ul><h3>4. OLTP Database Architecture</h3><p>Think of OLTP as a <strong>well-oiled machine</strong> that handles millions of live transactions without breaking. The architecture has several <strong>key components</strong>:</p><h4>Core Components</h4><ol><li><strong>Application Layer</strong></li></ol><ul><li><strong>Role:</strong> Receives requests from users (web apps, mobile apps, APIs)</li><li><strong>Example:</strong> When a customer clicks “Buy Now” on Amazon, the app sends a transaction request to the backend.</li><li><strong>Why it matters:</strong> It’s the entry point; the faster and cleaner the request handling, the smoother transactions are.</li></ul><p><strong>2. OLTP Database</strong></p><ul><li><strong>Role:</strong> Stores all transactional data in a <strong>normalized, row-based format</strong></li><li><strong>Example Tables:</strong> Orders, Order_Items, Inventory, Payment</li><li><strong><em>Why row-based &amp; normalized: </em></strong>Row-based storage allows <strong>fast inserts and updates, </strong>Normalization reduces redundancy and prevents anomalies</li></ul><p><strong>3. Transaction Manager</strong></p><ul><li><strong>Role:</strong> Ensures <strong>ACID compliance</strong> for every transaction</li><li><strong>Example:</strong> If a payment fails mid-checkout, the transaction manager ensures the order and inventory updates are <strong>rolled back</strong></li><li><strong>Why it matters:</strong> Guarantees data integrity and reliability</li></ul><p><strong>4. Concurrency Control</strong></p><ul><li><strong>Role:</strong> Manages multiple users performing transactions at the same time</li><li><strong>Techniques:</strong></li><li><strong>Locking:</strong> Prevents two users from updating the same row simultaneously</li><li><strong>MVCC (Multi-Version Concurrency Control):</strong> Allows reads while writes happen</li><li><strong>Why it matters:</strong> Without it, simultaneous transactions could <strong>overwrite each other</strong> or create inconsistent data</li></ul><p><strong>5. Backup &amp; Recovery</strong></p><ul><li><strong>Role:</strong> Ensures that committed transactions are durable and recoverable</li><li><strong>Example:</strong> Daily snapshots, transaction logs, or point-in-time recovery</li><li><strong>Why it matters:</strong> Protects against <strong>hardware failures, crashes, or human errors</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTjogUUKuUlDnfk86b4VrQ.png" /></figure><pre>-- Example of OLTP: a customer buying a product online<br> <br>-- 1. Create a new order<br>INSERT INTO orders (order_id, customer_id, order_date, total_amount)<br>VALUES (100, 515, NOW(), 194.99);<br><br>-- 2. Add items to the order<br>INSERT INTO order_items (order_id, product_id, quantity, price)<br>VALUES (101, 2001, 1, 189.99);<br><br>-- 3. Update inventory<br>UPDATE products<br>SET stock = stock - 1<br>WHERE product_id = 1001;<br><br>-- 4. Record payment<br>INSERT INTO payments (order_id, amount, status)<br>VALUES (101, 199.99, &#39;SUCCESS&#39;);</pre><h3>5. OLTP Database Examples</h3><ul><li><strong>Relational Databases:</strong> PostgreSQL, MySQL, Oracle, SQL Server</li><li><strong>Managed Cloud DBs:</strong> Amazon Aurora, RDS</li><li><strong>NoSQL (for high-concurrency / key-value operations):</strong> DynamoDB, MongoDB</li></ul><h3>6. Scaling OLTP Systems</h3><p><strong>Challenges:</strong></p><ul><li>High write volume</li><li>Millions of concurrent users</li><li>Maintaining ACID under scale</li></ul><p><strong>Solutions:</strong></p><ol><li><strong>Sharding / Partitioning: </strong>Split DB by region, customer ID, or product category</li><li><strong>Replication: </strong>Read replicas for read-heavy queries</li><li><strong>Caching: </strong>Frequently accessed data like product details</li><li><strong>Optimized Indexing: </strong>Primary + composite indexes for faster lookups</li><li><strong>Queue-based Decoupling: </strong>Asynchronous processing for non-critical operations</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53018baaf732" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 4: Unity Catalog in Databricks]]></title>
            <link>https://pratichi3.medium.com/part-4-unity-catalog-in-databricks-ab13799c70f2?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab13799c70f2</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 21:41:41 GMT</pubDate>
            <atom:updated>2025-09-26T21:41:41.773Z</atom:updated>
            <content:encoded><![CDATA[<p>Table of content:</p><ol><li>What is unity catalog?</li><li>Structure of unity catalog</li><li>key features of unity catalog</li></ol><p>When we work with data, it’s not enough to just store it. We also need to organize it, secure it and keep track of who is using it. In Databricks, this job is done by <strong>Unity Catalog</strong>.</p><h3>What is Unity Catalog?</h3><p>Unity catalog is a central governance system for all data in Databricks. Think of it as a <strong>librarian</strong> and <strong>security guard</strong> combined:</p><ul><li>The <strong>librarian</strong> part organizes all the data (catalogs, schemas, tables).</li><li>The <strong>security guard</strong> part controls who can see or change that data.</li></ul><p>It makes sure data is organized and secure across all workspaces.</p><h3>How data is organized ? Structure of Unity Catalog</h3><p>Unity Catalog follows a simple three-level hierarchy:</p><ol><li><strong>Catalog</strong> → The top container for data (like a library building).</li><li><strong>Schema</strong> → A logical group within a catalog (like a floor inside the library).</li><li><strong>Table/View</strong> → Actual datasets (like bookshelves with books).</li></ol><pre># Standard naming format in Unity Catalog is : <strong>catalog.schema.table</strong><br>Example: finance_catalog.sales_schema.transactions </pre><p>Here, <strong>finance_catalog = Catalog (library)</strong> , <strong>sales_schema = Schema (floor)</strong> and <strong>transactions = Table (bookshelf). That means </strong>inside the Finance catalog, under the Sales schema, we have a table called Transactions.</p><h3>Key Features of Unity Catalog</h3><ol><li><strong>Centralized Permissions</strong></li></ol><ul><li>Manage access across all workspaces in one place.</li><li><em>Example: Allow Finance team to read “finance_catalog”, but block “HR data”.</em></li></ul><p><strong>2. Fine-Grained Access Control</strong></p><ul><li>Permissions can be applied at Catalog level, Schema level, Table level, Even column/row level</li><li><em>Example: Hide the salary column from interns while keeping other data visible.</em></li></ul><p><strong>3. Data Lineage</strong></p><ul><li>Automatically tracks how data moves and transforms.</li><li><em>Example: If you create monthly_salary from transaction, the relationship is recorded.</em></li></ul><p><strong>4. Audit Logging</strong></p><ul><li>Logs who accessed what and when.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab13799c70f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 3 : Databricks Architecture — The Engine Behind the Lakehouse]]></title>
            <link>https://pratichi3.medium.com/part-3-databricks-architecture-the-engine-behind-the-lakehouse-4999d2d34e3c?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4999d2d34e3c</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 23:29:19 GMT</pubDate>
            <atom:updated>2025-09-15T23:29:19.180Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 3 : <strong>Databricks Architecture — The Engine Behind the Lakehouse</strong></h3><p><strong>Table of Contents:</strong></p><ol><li>Databricks Architecture — Key Components</li><li>Medallion Architecture</li></ol><p>Databricks powers the modern <strong>data lakehouse</strong>, making it easy to store, process, and analyze massive amounts of data.</p><h3><strong>Databricks operates on two main planes:</strong></h3><h4>A. Control Plane (Managed by Databricks)</h4><ul><li>Manages workspace, users, permissions, job scheduling, and cluster requests.</li><li>Fully managed, meaning we don’t need to handle infrastructure manually.</li><li>Example: When we create a cluster or schedule a job, it’s orchestrated here.</li></ul><h4>B. Compute Plane (Where Your Jobs Run)</h4><ul><li>Executes your <strong>actual ETL or Spark jobs</strong>.</li><li>Cluster Types: <strong>Classic Compute:</strong> Manual setup, more control, more maintenance &amp; <strong>Serverless Compute:</strong> Managed, auto-scaling, less overhead</li><li>Example: PySpark transformations, SQL queries, and Delta Lake operations run on this plane.</li></ul><h3>Medallion Architecture (Bronze → Silver → Gold)</h3><p>A <strong>best practice for structuring pipelines</strong>:</p><ul><li><strong>Bronze Layer:</strong> Raw, ingested data (CSV, JSON, logs)</li><li><strong>Silver Layer:</strong> Cleaned and transformed data (joins, filtering, standardization)</li><li><strong>Gold Layer:</strong> Aggregated data for analytics, dashboards, or ML models</li></ul><p><strong>Benefits:</strong></p><ul><li>Reliable, structured pipelines</li><li>Easier debugging and scaling</li><li>Supports batch &amp; streaming analytics</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4999d2d34e3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2: Delta Lake : Backbone of Lake house]]></title>
            <link>https://pratichi3.medium.com/part-2-delta-lake-backbone-of-lake-house-9bdf235d92c6?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bdf235d92c6</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Mon, 08 Sep 2025 22:47:07 GMT</pubDate>
            <atom:updated>2025-09-08T22:47:07.052Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2: Delta Lake : Backbone of Lake house</h3><p>In<a href="https://pratichi3.medium.com/part-1-data-lake-52dffbb10553"> Part 1</a>, we discussed about Data Lake and how Data Lake solved the 3Vs problem.</p><blockquote><strong>But , Data Lake had their own problem</strong></blockquote><p>Data Lake can store raw data (no strict schema), therefore it can becomes messy which can lead to data quality issues, inconsistent updates, no ACID transactions.</p><p>To fix this, <strong>Delta Lake</strong> was introduced as a storage layer <strong>on top of Data Lakes</strong></p><p>Delta Lake is a layer made on the top of data lakes to provide the properties same like SQL warehouse (ACID transactions, schema enforcement, and time-travel queries (commit, rollback)).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bdf235d92c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1 : Data Lake]]></title>
            <link>https://pratichi3.medium.com/part-1-data-lake-52dffbb10553?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/52dffbb10553</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[3vs-of-data]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Sat, 06 Sep 2025 22:23:51 GMT</pubDate>
            <atom:updated>2025-09-06T22:23:51.617Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1 : Data Lake</h3><p>Table of contents:</p><ol><li>What is Data Lake and why do we need it ?</li><li>What are 3Vs ? How did Data Lake solved the problem of 3Vs?</li></ol><blockquote><strong>What is Lake house ?</strong></blockquote><p>In year 1980s — 1990s , companies stored data in a SQL Warehouse (oracle, IBM, Teradata ). This was designed to handle structured data (rows, columns) , perform ETL Batch jobs (load data at the end of day/week) , Business Intelligence Reporting.</p><p>But in year 2000s, big data boomed and we started facing 3Vs problem.</p><h3>3Vs</h3><blockquote><strong>Volume</strong> : Data exploded to terabytes/ petabytes (social-media , e-commerce platform).</blockquote><blockquote><strong>Variety</strong> : Different formats of data like structured(tables) , unstructured (audio, video, images )and semi-structured (xmls , json, logs)</blockquote><blockquote><strong>Velocity</strong> : Real time data streams (early 2000 — youtube era)</blockquote><h3>Hence, the concept of Data Lake came into the picture and eventually solved the problem of 3Vs. <strong>But the question is how?</strong></h3><h4><strong>1. Volume — Scalable storage</strong></h4><p>Problem: Warehouses were expensive to scale for huge data.</p><p>Solution: <strong>Distributed, cheap storage systems</strong> emerged:</p><ul><li><em>Hadoop HDFS (storing data across many cheap commodity servers)</em></li><li><em>Later, cloud storage like AWS S3, Azure Data Lake Storage, GCP Storage</em></li></ul><blockquote><strong>Intuition: Instead of one expensive machine, you use hundreds of cheap servers to store petabytes of data.</strong></blockquote><h4><strong>2. Variety → Schema-on-Read</strong></h4><p>Problem: Warehouses only accepted structured tables (rows/columns).</p><p>Solution: Data lakes store raw data as-is it : structured, semi-structured (JSON, logs), and unstructured (images, videos, audio).</p><blockquote><strong>Intuition: You don’t force the data to fit a table upfront. You decide the structure later when you query it (schema-on-read).</strong></blockquote><h4><strong>3. Velocity → Real-Time &amp; Batch</strong></h4><p>Problem: Warehouses were batch-oriented (ETL → nightly/weekly loads).</p><p>Solution: Big data frameworks and streaming engines were developed: Apache Spark, Kafka for real-time data processing Data lakes could now ingest both batch and streaming data</p><blockquote><strong>Intuition: Data can flow continuously into the lake, analytics can happen on-demand or in real-time.</strong></blockquote><p>With the data lake , it was possible to analyze structured, semi-structured, and unstructured data efficiently and enabled ML/AI use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52dffbb10553" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your WordPress Deployment on AWS Using Terraform]]></title>
            <link>https://pratichi3.medium.com/automate-your-wordpress-deployment-on-aws-using-terraform-bca4c9aea30a?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bca4c9aea30a</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 22:10:45 GMT</pubDate>
            <atom:updated>2025-04-25T22:18:57.021Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Have you ever wished that deploying a WordPress website on AWS could be just a few lines of code away? That’s exactly what I set out to do in this project — <strong>automate</strong> the entire deployment process using <strong>Terraform</strong>. From containerizing WordPress to setting up a scalable backend with RDS and routing traffic through an ALB, this blog walks you through how to build a cloud-native, secure, and cost-efficient setup, with no manual clicking in the AWS console.</p><blockquote><strong>Access the project here : </strong><a href="https://github.com/FnuPratichi/Automating-WordPress-Hosting-with-Terraform-and-AWS"><strong>GitHub repository</strong></a></blockquote><h3>Objectives</h3><p>Here’s what we aimed to achieve with this deployment:</p><ol><li><strong>Deploy WordPress on ECS</strong>: Use ECS with Fargate to run the WordPress container, freeing us from managing the underlying infrastructure.</li><li><strong>Configure ALB (Application Load Balancer)</strong>: Distribute incoming web traffic across multiple ECS containers for better performance and redundancy.</li><li><strong>Set up RDS with MariaDB</strong>: Use Amazon RDS for database management, making it easy to scale and back up data.</li><li><strong>Security Groups and IAM Roles</strong>: Securely control access to services, ensuring that only the right resources can communicate with each other.</li><li><strong>Automate Deployment with Terraform</strong>: Write infrastructure code in Terraform to ensure that the setup is repeatable and easily manageable.</li></ol><h3>Step-by-Step Breakdown</h3><p>Here’s how the setup was done:</p><ol><li><strong>Provisioning the ALB (Application Load Balancer)</strong>:<br> The ALB is the first point of contact for any incoming traffic. We set up an ALB to route HTTP requests to the ECS containers running WordPress. To ensure that traffic is only directed to healthy containers, we configured health checks.</li><li><strong>Creating the ECS Cluster &amp; Task Definition</strong>:<br> We used ECS with Fargate to run the WordPress containers. Fargate allows us to focus on the app rather than managing the infrastructure, which is one of the key benefits of this approach. We defined the task with resources like CPU and memory to ensure smooth performance. IAM roles were added to give the ECS tasks the permissions they needed to interact with other AWS services.</li><li><strong>Configuring the ECS Service</strong>:<br> Next, we set up the ECS service to launch the WordPress containers and connect them to the ALB. This ensures that traffic is automatically distributed across containers. We also made sure the networking configurations (subnets and security groups) were in place to secure communication between ECS and other resources.</li><li><strong>Setting up the RDS MariaDB Database</strong>:<br> WordPress needs a database, so we used Amazon RDS to provision a MariaDB instance. By doing this, we can easily scale the database, set up automated backups, and ensure high availability. Security groups were configured to allow ECS containers to connect securely to the RDS instance.</li><li><strong>Security Groups &amp; Networking</strong>:<br> Security is critical. We set up security groups and VPC configurations to ensure the resources were isolated and only accessible by the necessary services. This prevents unauthorized access and ensures that only the ECS containers can interact with the RDS database.</li><li><strong>Finalizing the Terraform Configuration</strong>:<br>To give the ECS tasks the necessary permissions, we made a small but essential change to the ecs.tf file. By adding these lines, we ensured that ECS tasks could securely access the ALB and RDS without needing additional IAM policies:</li></ol><pre>execution_role_arn = &quot;arn:aws:iam::1**********:role/LabRole&quot;<br>task_role_arn = &quot;arn:aws:iam::1***********:role/LabRole&quot;<br>Note: The ARN(hidden by *) used here is from the AWS learner’s lab account.</pre><p><strong>7. Accessing the WordPress Application</strong>:<br>Once everything was deployed, we used the DNS name of the ALB to access the WordPress application via a public URL. This URL pointed to the ECS containers, allowing users to interact with WordPress.</p><h3>Testing the Deployment</h3><p>Once the infrastructure was up and running, it was time for testing:</p><ol><li><strong>WordPress Accessibility</strong>:<br> The WordPress site was accessible via the public DNS name of the ALB, confirming that traffic was correctly routed to the ECS containers.</li><li><strong>Database Connectivity</strong>:<br>A successful connection between WordPress and the MariaDB database was established, proving that the database was set up and functioning as expected.</li><li>All tests passed without any issues, meaning the deployment was successful and met the objectives.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDBO-isTYGlLoz5Up9YZDg.png" /><figcaption>All tests passed without any issues, meaning the deployment was successful and met the objectives.</figcaption></figure><h3>How You Can Try It Too</h3><blockquote>If you want to deploy WordPress using Terraform on AWS yourself, you’re in luck! I’ve made all the Terraform files available on my <a href="https://github.com/FnuPratichi/Automating-WordPress-Hosting-with-Terraform-and-AWS">GitHub repository</a>. You can download the files, tweak them as needed, and follow the steps to set up your own WordPress app on AWS.</blockquote><blockquote>Terraform makes it incredibly easy to automate and manage your infrastructure, and with AWS, the possibilities are endless. So, give it a try, and let me know how it goes!</blockquote><p>Ready to give it a shot? Clone the repo, make it your own, and feel free to drop your questions or share feedback in the comments or GitHub issues!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bca4c9aea30a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Dimensionality Reduction Techniques: A Comparative Analysis Using the Kaggle Digit…]]></title>
            <link>https://pratichi3.medium.com/exploring-dimensionality-reduction-techniques-a-comparative-analysis-using-the-kaggle-digit-fcc33738a75e?source=rss-95d5850e185e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcc33738a75e</guid>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[t-sne-visualization]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[pca-analysis]]></category>
            <category><![CDATA[lda]]></category>
            <dc:creator><![CDATA[Pratichi]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 05:50:10 GMT</pubDate>
            <atom:updated>2025-01-08T05:50:10.510Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring Dimensionality Reduction Techniques: A Comparative Analysis Using the Kaggle Digit Recognizer Dataset​</h3><p>Dimensionality reduction is crucial in machine learning, particularly when dealing with high-dimensional datasets. This blog delves into four key techniques named <strong>Principal Component Analysis (PCA)</strong>, <strong>t-distributed Stochastic Neighbor Embedding (t-SNE)</strong>, <strong>Linear Discriminant Analysis (LDA)</strong>, and <strong>Autoencoders </strong>analyzed using the Kaggle Digit Recognizer dataset, which is based on the MNIST dataset.</p><h3>The Dataset: Kaggle Digit Recognizer</h3><p>The dataset provides a CSV-based representation of handwritten digits, with each image represented by 784 features (28x28 pixels). Most values are zeros (background), while non-zero values form the digit. The primary goal is to reduce the dimensionality of these 784 features and analyze the performance of the mentioned techniques in data visualization and classification.</p><blockquote>source: <a href="https://www.kaggle.com/competitions/digit-recognizer/data?select=train.csv">https://www.kaggle.com/competitions/digit-recognizer/data?select=train.csv</a></blockquote><h3><strong>1. Implementation of PCA</strong></h3><p>We implemented PCA to reduce the dimensionality of the MNIST dataset and enable 2D and 3D visualization. We identify and remove constant features (those with zero variance)to eliminate redundant information. Next, we standardize the data so that each feature contributes equally, ensuring that PCA effectively captures the variance across features. Finally, we apply PCA projects the data into two and three dimensions, allowing us to visualize and explore underlying patterns and clusters while retaining the most significant variance in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SknlN9KEApwQ4wyrTEP4Tw.png" /></figure><h3>2. Implementation of TSNE</h3><p>In this code, we are applying t-distributed Stochastic Neighbor Embedding (t-SNE) to reduce the original 784-dimensional MNIST dataset down to lower dimensions for visualization and analysis. Specifically, we perform two reductions: first to 2D (with n_components =2) and then to 3D (with n_components=3). Parameters like perplexity (which controls the balance between local and global aspects) and n_iter (number of iterations) are tuned to optimize the visualization quality, with perplexity=30 helping manage the neighborhood size and n_iter ensuring convergence. By reducing from 784 dimensions to 2D and 3D, t-SNE allows us to visualize high-dimensional clusters and patterns that would otherwise be challenging to detect in the original dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Bq7WozrTo_GpJTez7LSWw.png" /></figure><h3><strong>3. Implementation of LDA</strong></h3><p>In this code snippet, we perform dimensionality reduction using Linear Discriminant Analysis (LDA) to condense the original 784-dimensional input into both 3-dimensional and 2-dimensional spaces. The goal of LDA here is to project the data into a lower-dimensional space that maximizes the separability of the digit classes, guided by the labels provided in y.</p><p>In the first step, LDA (2D), we project the data into two dimensions, creating a representation that is especially useful for visualization. Unlike PCA, which is unsupervised and aims to retain maximum variance without regard to class, LDA is a supervised method, focusing specifically on maximizing the distance between digit classes. By reducing from 784 dimensions to 2 or 3, we achieve a more interpretable dataset structure while retaining critical information in class.</p><p>In the second step, LDA (3D), we specify three components, transforming the data into a 3D space that preserves class distinctions as much as possible, allowing us to explore how different digit classes are spread across three dimensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrwzMeOWabpit1CjFvdHsA.png" /></figure><h3>4. Implementation of Autoencoders</h3><p>In this code, we implemented an autoencoder for dimensionality reduction using the MNIST dataset. The autoencoder consists of two parts: an encoder and a decoder. The encoder compresses the input data, reducing its dimensions to a specified size (2D or 3D in this case), while the decoder reconstructs the original data from the compressed form. We first trained the model with a 2-dimensional encoding (encoding_dim = 2) and then repeated the process with a 3-dimensional encoding (encoding_dim = 3). After training, the autoencoder, we used the encoder to generate the reduced 2D and 3D representations of the original data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBdMZ4H03lRmWZFDTJ8DuA.png" /></figure><h3>Experimental Results :</h3><blockquote><strong><em>1. Reduced 784 Dimensions TO 2-Dimensions: Comparative Data Visualization of PCA, T-SNE, LDA, and autoencoder in 2D</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jxnzvzHu_FG9tuKX2_xGRA.png" /></figure><ul><li><strong>t-SNE</strong>: Showed the clearest clustering, effectively capturing complex, non-linear relationships. Ideal for visualizing high-dimensional data, it provides excellent separation of digit classes.</li><li><strong>LDA</strong>: Performed exceptionally well with provided class labels, ensuring distinct separation of digit classes.</li><li><strong>PCA</strong>: Preserved global structure and linear variance but struggled with separating complex, non-linear structures.</li><li><strong>Autoencoders</strong>: Demonstrated flexibility in capturing non-linear features, though the separation quality depended on hyperparameter tuning.</li></ul><blockquote><strong><em>2. Reduced 784 Dimensions to 3 Dimensions</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CrnVZR854KbRB3AXJGfu2A.png" /></figure><ul><li><strong>t-SNE</strong>: Maintained its strength in revealing intricate non-linear structures, showing clear clustering of digit classes.</li><li><strong>LDA</strong>: Offered excellent separation by maximizing class separability using labels.</li><li><strong>PCA</strong>: Preserved variance but struggled with class separation in this context.</li><li><strong>Autoencoders</strong>: Continued to capture non-linear patterns but were slightly less effective in providing clear clustering without fine-tuning.</li></ul><blockquote><strong><em>3. Comparision of accuracy after reducing 784 dimensions to 9 dimensions</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DneOApP82TOfqTNhT-OUBg.png" /></figure><ul><li><strong>LDA</strong>: Shows the highest accuracy of 88.24 %, as LDA is supervised and retains class-discriminative features.</li><li><strong>PCA:</strong> Performs similarly to the autoencoder of 78.30%, preserving variance in the data but not as class-specific as LDA.</li><li><strong>Autoencoder</strong>: Performs reasonably well with 76.15% accuracy, as it preserves significant data structure in lower dimensions.</li><li><strong>T-SNE</strong>: Converting data into lower dimensions using t-SNE proved to be computationally expensive. Due to memory constraints, we limited the reduction to 2D, which still required significant processing time. The resulting classification accuracy was only 42.5%, likely because t-SNE is not well-suited for dimensionality reduction in combination with a linear classifier. As a tool primarily designed for visualization, it struggles to retain features that are critical for classification tasks in high-dimensional spaces.</li></ul><blockquote><strong><em>4. Comparision of accuracy after reducing 784 dimensions to 9,50,100,200 dimensions using PCA</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qL0Fty4L-iEVNPYoOHbfg.png" /></figure><p>We can observe that as the number of dimensions increases, variance is preserved more. Hence for 200D, variance preserved is 87.93%.</p><blockquote><strong><em>5. Comparision of accuracy after reducing 784 dimensions to 9,50,100,200 dimensions using Autoencoder</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCx0T3RHicC3u9PHER-SPw.png" /></figure><p>We observed that as the encoding dimension increases from 9 to 200, the accuracy increases, and the reconstruction error on the training set decreases. This suggests that the autoencoder is learning better representations of the data with more encoding capacity. The validation error reduces slightly with higher encoding dimensions but doesn’t improve dramatically beyond 100. This suggests that increasing the encoding dimension further may lead to diminishing returns on generalization. If the validation error were to increase, that might indicate overfitting, where the model becomes too specialized for the training data.</p><p>In <em>conclusion, this research highlights the significant impact of the dimensionality reduction method on classification accuracy. Supervised methods such as LDA outperform others by preserving class-discriminative features, making them ideal for classification tasks. Unsupervised methods, including Autoencoders and PCA, demonstrate strong potential in capturing meaningful patterns within the data, contributing to robust performance. On the other hand, t-SNE, designed primarily for visualization, exhibits limited effectiveness in classification, underscoring the importance of selecting dimensionality reduction techniques that align with the specific objectives of the analysis.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcc33738a75e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>