<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinay Kumar Reddy on Medium]]></title>
        <description><![CDATA[Stories by Vinay Kumar Reddy on Medium]]></description>
        <link>https://medium.com/@kapuluru.vinay07?source=rss-5fd2f22eaf1f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*eVRqLmuhs9IpUvmh</url>
            <title>Stories by Vinay Kumar Reddy on Medium</title>
            <link>https://medium.com/@kapuluru.vinay07?source=rss-5fd2f22eaf1f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:47:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kapuluru.vinay07/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Databricks Interview Questions]]></title>
            <link>https://medium.com/@kapuluru.vinay07/databricks-interview-questions-f8f48656f829?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8f48656f829</guid>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Mon, 24 Nov 2025 16:51:46 GMT</pubDate>
            <atom:updated>2025-11-24T16:53:52.058Z</atom:updated>
            <content:encoded><![CDATA[<p>What is the Databricks Lakehouse Architecture?<br>Answer:<br>Databricks Lakehouse combines the best of data lakes (low-cost, scalable storage like ADLS/S3) and data warehouses (ACID transactions, schema enforcement, BI performance) It is built on the Delta Lake format which brings reliability to data lakes.<br>Key features:<br>· ACID transactions<br>. Schema enforcement &amp; evolution<br>. DML operations (MERGE/UPDATE/DELETE)<br>. Time Travel<br>. High-performance caching</p><p>2. Scenario: You have late-arriving data every day. How do you handle it using Databricks?<br>Answer:<br>Use MERGE INTO with Delta Lake to upsert.<br>Code:<br>MERGE INTO target t<br>USING source s<br>ON t.id = s.id<br>WHEN MATCHED THEN UPDATE SET *<br>WHEN NOT MATCHED THEN INSERT *<br>Delta Lake ensures ACID compliance and avoids duplicates.</p><p>3. What is Autoloader? When do you use it?<br>Answer:<br>Autoloader is a high-performance ingestion feature in Databricks used for incremental file ingestion from cloud storage. <br>Why use it?<br>· Automatically detects new files<br>Supports schema inference &amp; evolution<br>. Scales to billions of files<br>. Works best for streaming ingestion</p><p>Example:<br>df = spark.readstream.format(&quot;cloudFiles&quot;)\<br>.option(&quot;cloudFiles.format&quot;,&quot;json&quot;)\<br>.load(&quot;/mnt/raw/&quot;)</p><p>4. You have schema changes in your raw data. How do you manage this in Databricks?</p><p>Answer:<br>. Use Schema Evolution with Delta Lake.</p><p><strong>“ALTER TABLE users SET TBLPROPERTIES<br>&#39;delta.schema.autoMerge.enabled’=’true ); “</strong></p><p>. Manage schema changes in Databricks using a Bronze–Silver–Gold model with Delta Lake.<br>In the Bronze layer, we ingest raw files using Auto Loader with mergeSchema and autoMerge enabled, so new columns are handled automatically without job failures.<br>We maintain a metadata-driven design – only approved columns are selected into Silver/Gold, and we avoid SELECT *, so schema drift in raw doesn’t break curated tables.<br>When new columns arrive, we first update metadata, then explicitly add them into Silver/Gold and optionally backfill. For breaking changes like type/rename, we do controlled migrations and use Delta’s time travel for safety.</p><p>How do you handle schema changes in a normal environment, not Databricks?<br>You can answer:<br>&gt; still follow a layered approach.<br>Raw / Staging always stores data as close to source as possible. When a new column comes, first extend the raw or staging (ALTER TABLE or updated schema), then control promotion of that column into transformed/reporting tables using metadata or explicit mappings.<br>Old data simply has NULLs in new columns. For breaking changes like datatype or renames, use controlled migrations instead of direct changes.</p><p>5. How does Delta Lake support Time Travel?<br>Answer:<br>You can query older versions of data using VERSION AS OF or TIMESTAMP AS OF.<br>It is enabled by Delta’s underlying transaction log (_delta_log), which tracks every change made to the table.</p><p>#Query by version number:<br>SELECT * FROM customers VERSION AS OF 3;<br>#Query by timestamp:</p><p>SELECT * FROM customers TIMESTAMP AS OF &#39;2025-11-25T10:00:00&#39;;<br>#Restore table to an older version:<br>RESTORE TABLE customers TO VERSION AS OF 3;<br>Or:<br>RESTORE TABLE customers TO TIMESTAMP AS OF &#39;2025-11-25&#39;;</p><p>Useful for audit, rollback, debugging.</p><p>6. What is Delta Live Tables (DLT)? Why is it used?<br>Answer:<br>DLT is a framework to build reliable &amp; declarative ETL pipelines.<br>Features:<br>. CDC support<br>. Auto-recovery<br>. Quality rules (EXPECT statements)<br>. Lineage tracking<br>Example:<br>@dlt.table<br>def cleaned_orders():<br>return alt.read(&quot;raw_orders&quot;).where(&quot;amount &gt;0&quot;)</p><p>7. Scenario: How do you implement Change Data Capture (CDC) in Databricks?<br>Answer:<br>Use Delta MERGE+ SCD Type 2 design.<br>MERGE INTO target t<br>USING changes o<br>ON t.id =c.id<br>WHEN MATCHED AND c.is_deleted = true THEN DELETE<br>WHEN MATCHED THEN UPDATE SET ··<br>WHEN NOT MATCHED THEN INSERT …</p><p>8. Explain Unity Catalog. Why is it important?<br>Answer:<br>Unified governance layer for Databricks<br>Features:<br>. Central metadata management<br>. Fine-grained access control<br>. Lineage visualization<br>. Cross-workspace sharing</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8f48656f829" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the Primary Purpose of Vector Embeddings in Large Language Models (LLMs)?]]></title>
            <link>https://medium.com/@kapuluru.vinay07/what-is-the-primary-purpose-of-vector-embeddings-in-large-language-models-llms-ed32ba89d7d9?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed32ba89d7d9</guid>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Thu, 22 May 2025 15:53:23 GMT</pubDate>
            <atom:updated>2025-05-22T15:53:23.901Z</atom:updated>
            <content:encoded><![CDATA[<p>Vector embeddings help LLMs understand the meaning of words, sentences, or documents by converting them into numbers (vectors) in a way that captures their meaning and relationships.</p><h3>Breaking It Down:</h3><p>Imagine you&#39;re learning a new language, like Spanish. You come across words like:</p><ul><li>&quot;dog&quot;</li><li>&quot;puppy&quot;</li><li>&quot;cat&quot;</li><li>&quot;car”</li></ul><p>You know from experience that &quot;dog&quot; and &quot;puppy&quot; are more related than &quot;dog&quot; and &quot;car.&quot; But how can a computer figure that out?</p><p>That&#39;s where vector embeddings come in.</p><p>What Are Embeddings?</p><p>Embeddings are a way to turn words or phrases into numbers not just any numbers, but numbers that capture meaning.</p><p>For example:</p><p>Word | Vector (simplified example)</p><p>Dog | [10.8, 0.2.0.1]</p><p>Puppy | [0.79, 0.21, 0.11]</p><p>Car | [10.1.0.9.0.7]</p><p>You can see that &quot;dog&quot; and &quot;puppy have similar vectors, while &quot;car&quot; is different. This helps the model understand that dog and puppy are similar, even if you didn&#39;t spell them the same.</p><p>How LLMs Use Vector Embeddings?</p><p>Large Language Models (like GPT or those in Azure OpenAI) don&#39;t understand language like humans do.</p><p>They rely on math. So embeddings help by:</p><ul><li>Representing meaning in numbers</li><li>Helping the model understand context (e.g., &quot;bank&quot; can mean riverbank or money bank)</li><li>Supporting tasks like search, recommendation, question-answering, and translation</li></ul><h3>Azure Al and Vector Embeddings:</h3><p>Here&#39;s how this connects with the Azure Al-900 certification:</p><p>1. AI Concepts</p><p>Embeddings are a key part of Natural Language Processing (NLP)</p><p>Azure uses embeddings in services like:</p><ul><li>Azure Cognitive Services</li><li>Azure OpenAl Service</li></ul><p>2. Azure OpenAl Use</p><p>When you use an Azure OpenAl model for:</p><ul><li>Text generation</li><li>Search (semantic search)</li><li>Chatbots</li><li>Recommendations</li></ul><p>Embeddings are used behind the scenes to understand and process text.</p><p>3. Semantic Search (Key for Al-900)</p><ul><li>In traditional search, you search for exact words.</li><li>With embeddings, Azure can do semantic search finding content that&#39;s related in meaning, not just exact matches.</li></ul><p>Keywords:</p><ul><li>Embeddings: Numeric representation of text</li><li>Semantic meaning: Similar meanings have similar embeddings</li><li>Vector. A list of numbers</li><li>Natural Language Processing (NLP): Embeddings are used here</li><li>Azure Services: Azure OpenAl, Cognitive Search use embeddings</li></ul><p>How Are Embeddings Calculated?</p><p>1. Training the Embedding Model</p><p>To calculate these numbers (vectors), an algorithm is trained on a large amount of text data. Some popular algorithms used for creating embeddings include:</p><ul><li>Word2Vec</li><li>Glove (Global Vectors for Word Representation)</li><li>FastText</li></ul><p>These algorithms learn to associate words with similar meanings by observing how words appear in different contexts across large datasets (like books, articles, websites, etc.).</p><p>2. Contextual Relationships</p><p>The model analyzes the context in which each word appears. For example:</p><p>Words like &quot;dog&quot; and &quot;puppy&quot; often appear in similar contexts (like &quot;pet,&quot; &quot;animal,&quot; &quot;bark&quot;).</p><p>Words like &quot;dog&quot; and &quot;car&quot; don&#39;t share much context, so their embeddings will be different.</p><p>3. Mathematics Behind It</p><ul><li>When training the embedding model, it uses mathematical techniques like neural networks or matrix factorization to figure out which words should be close together in the vector space (those with similar meanings) and which should be far apart.</li></ul><p>4. Vector Representation (The Numbers)</p><p>Once the model is trained, it can convert a word into a vector (a list of numbers). These vectors are not random; they reflect relationships between words based on context.</p><p>Example:</p><ul><li>&quot;dog&quot; and &quot;puppy&quot; have similar meanings, so their vectors are close to each other.</li><li>dog — [0.8, 0.2, 0.1]</li><li>puppy — [0.79, 0.21, 0.11]</li><li>&quot;dog&quot; and &quot;car&quot; have very different meanings, so their vectors are far apart.</li><li>dog (0.8, 0.2, 0.1]</li><li>car [0.1, 0.9, 0.7]</li></ul><p>5. Example: How Words Are Embedded</p><p>Think of embeddings as coordinates in a multi-dimensional space. Let&#39;s say we&#39;re working in a 3-dimensional space for simplicity. Here&#39;s how we calculate the vectors (these are simplified numbers for illustration):</p><ul><li>The first value of the vector could represent a dimension like &quot;animal-ness&quot; (how much the word is related to animals).</li><li>The second value could represent &quot;size&quot; (how big the thing is).</li><li>The third could represent something like &quot;functionality&quot; (what the object is used for).</li></ul><p>So, if you look at the embeddings:</p><ul><li>dog (0.8, 0.2, 0.1] might indicate it&#39;s very animal-like (0.8), somewhat small/medium (0.2), and has a low &quot;car&quot; functionality (0.1).</li><li>puppy 10.79, 0.21, 0.11] is very similar to dog, meaning both are close in meaning.</li><li>car [0.1, 0.9, 0.7] shows that it&#39;s very different from &quot;dog&quot; it&#39;s not related to animals (0.1), but is large (0.9) and used for transportation (0.7).</li></ul><p>6. What Happens During Training?</p><p>During training, the model:</p><ul><li>Looks at neighboring words in sentences (e.g., &quot;I love my dog&quot; or &quot;The puppy runs fast&quot;).</li><li>Adjusts the vectors so that words with similar meanings are close together.</li><li>Words with different meanings will have vectors that are far apart.</li></ul><p>It uses backpropagation and optimization techniques to adjust the vectors so that the model gets better at predicting and understanding word meanings.</p><p>How Does This Help?</p><p>Once we have these vectors, we can:</p><ul><li>Compare words (find similarities and differences).</li><li>Cluster words into groups (eg., &quot;dog&quot; and &quot;puppy&quot; cluster together, while &quot;dog&quot; and &quot;car&quot; are far apart).</li><li>Use embeddings for tasks like:</li></ul><ol><li>Search engines (matching queries with relevant documents).</li><li>Recommendation systems (finding similar items).</li><li>Text generation (predicting the next word based on context).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed32ba89d7d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Views and Indexes in SQL]]></title>
            <link>https://medium.com/@kapuluru.vinay07/views-and-indexes-in-sql-56af63017c4b?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/56af63017c4b</guid>
            <category><![CDATA[sql-view]]></category>
            <category><![CDATA[clustered-index]]></category>
            <category><![CDATA[non-clustered-index]]></category>
            <category><![CDATA[sql-index]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 16:09:02 GMT</pubDate>
            <atom:updated>2024-01-08T16:09:02.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>Views</h3><p>A view is stored SQL query. You can think view as a virtual table. View are also permanent objects in the database which means they are also reusable.</p><p><strong>why do views exists and what is purpose of them?</strong></p><p>Imagine! If you are in a SQL developer in a company and manager comes to you saying that, need to give table level access to someone in the finance department, that person wants to see only 3 columns from 4 or 5 prod tables. That 4 or 5 prod table consists of 300 columns. Now it might be a security risk to give that one person access to those prod tables. Instead of doing that we can create a view that joins the tables together and only selects that 3 columns need by that employee of the finance department. Then we can give that employee access to the view only, this means there is no need for us to give that employee the underlying tables anymore. This is a great way to uphold the data security for our company&#39;s data and cleaner solution for an employee because now all they have to do is to query a view without having to write from table name in a join and write all those joins it. This is why view can be very helpful.</p><h3><strong>What is Index?</strong></h3><h3><strong>Index:</strong></h3><p>Think that index is a virtual tables. Then what is difference between table and an index, so table with no indexes, its actually called as heap. Heap is a database object with no order to the data, think like pile of data. So, SQL sever have to scan the all the records in a table that has no indexes, because SQL server has no idea where the exact data is. When we create indexes on these tables. Then suddenly table is no longer a heap and now the table has order to it. Now SQL server does not have to scan the data, now it will seek data. which is lot more efficient and optimal for searching. So queries can run faster.</p><p><strong>Index is a database object that holds ordered data on a column or set of columns.</strong></p><h3><strong>Clustered Index:</strong></h3><p>Clustered indexes are the indexes that order tables data based on a column or set of columns. Clustered index will always have the columns in a table stored in it. Let’s say a table has 300 columns, that means clustered index will have all 300 columns will store within the index, but keep in mind that all those columns are ordered based on one or more columns. If a table with no clustered index is also called as heap as discussed above. When we create a clustered index on a table, the concept of heap does not exists anymore for that table. Since the clustered index replaces the heap, no additional storage in taken up.</p><p>Suppose a table that was storing about 2 GB of data, if table has no indexes. Then we create a clustered index on that table and that clustered index also taken up same 2 GB of data, because its just replaces current 2 GB. So there is no additional space taken up.</p><p><strong>Note:</strong></p><p>If we create a Primary key on table, that automatically creates clustered index for that table as well based whatever the primary key is.</p><p>Also a single table cannot have a more than one clustered index, the reason behind a table can have only one clustered index is that it sorts the data in an order and the data can be sorted either way asc/ desc.</p><h3>Non-Clustered Index</h3><p>Non- clustered index is an index structure that contains specific columns from a table and the data is ordered based on one or more those columns.</p><p>Imagine! you have a query that is causing performance issue in prod, and the query needs only four columns in a table with 300 columns. SQL server takes all those columns in to a memory to process that query. This is not efficient, if we create non clustered index using only those 4 columns needed. The query will then use that index structure and only takes those 4 columns into memory. So query will do better performance.</p><p>And non clustered also have what we call as <strong>pointers </strong>in the index structure, pointers will tell to the SQL server where to find the data for the other columns that are not included in that index.</p><p>Suppose if a query needed data from a column that was not in the non-clustered index, then normally use the pointer in the non clustered index to find the information is located in the source table which it could be a clustered index or a heap. The table can have many non-clustered indexes associated with it. This is because non-clustered indexes are stored as a separate database objects unlike clustered indexes.</p><h3><strong>Unique Index</strong></h3><p>Not going deep it in.</p><p>A unique index just means that the index key contains no duplicate values. You can have unique clustered indexes and unique non clustered indexes.</p><h3><strong>Others available Index types :</strong></h3><p>Single column index</p><p>Multi column index (composite index)</p><p>Bitmap index</p><p>Full text index</p><p>Spatial index</p><p>Keep Learning…🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56af63017c4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is NoSQL Database]]></title>
            <link>https://medium.com/@kapuluru.vinay07/what-is-nosql-database-504240f2ac0f?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/504240f2ac0f</guid>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Fri, 29 Dec 2023 18:33:45 GMT</pubDate>
            <atom:updated>2023-12-29T18:33:45.055Z</atom:updated>
            <content:encoded><![CDATA[<p>Before going to know about NoSQL, will just recap what is Relational Database.</p><p><strong>What is Relational Database?</strong></p><p>A relational database is a collection of items with pre-defined relationships between them. These items are organized as a set of tables with columns and rows. Tables are used to hold information about the objects to be represented in the database.<br>Ex: MySQL, SQL Server, PostgreSQL, Oracle, MariaDB etc…</p><p><strong>Example for Beginners:</strong></p><p>Imagine you have a notebook where you want to organize information about your favorite movies, including details like the title, release year, and director. Instead of jotting everything down randomly, you decide to create a structured system.</p><ol><li>Pages as Tables:</li></ol><ul><li>Think of your notebook as the entire database. Instead of having a chaotic mix of information on each page, you decide to dedicate each page to a specific type of information — one page for movie titles, another for release years, and so on.</li></ul><p>2. Table Columns:</p><ul><li>Now, each page (table) is organized into columns. For the page about movie titles, you have columns like “Title,” “Release Year,” and “Director.” Each column has a specific type of information.</li></ul><p>3. Rows as Entries:</p><ul><li>Each row on these pages represents information about a specific movie. So, if “The Matrix” is one of your favorite movies, you’d have a row under the “Movie Titles” page with details filled in for “The Matrix,” its release year, and the director.</li></ul><p>4. Relationships:</p><ul><li>Here’s where the relational part comes in. Suppose you have another page dedicated to your favorite actors. If Keanu Reeves is one of your favorite actors and he starred in “The Matrix,” you can create a connection. You don’t need to rewrite all the details; you can reference the same information on both pages.</li></ul><p>In this analogy, your notebook is like a relational database. Each page represents a table, columns organize specific types of information, and rows contain details about individual items. The beauty is in the relationships — connecting information seamlessly across different pages.</p><p>So, a relational database is like your well-organized notebook, making it easy to find, update, and connect information about your favorite movies and actors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_ADpW2kt5J6XdPWwgcOaA.png" /><figcaption>Relational Database and Table relation</figcaption></figure><h3><strong>NoSQL Database (Not Only SQL)</strong></h3><p>. NoSQL database concept was emerged to overcome the limitations of traditional Relational Databases.<br>. When compared to relational database like Sql Server, MySQL etc…, NoSQL databases are highly scalable, flexible and faster, because NoSQL is Schemaless and non relational structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBMqMStQFwqLdlExbQLfrg.png" /><figcaption>Difference between SQL and NoSQL</figcaption></figure><p><strong>Please refer above image while reading below points:</strong></p><ol><li>Relational database creates the table as “create table &lt;tableName&gt; “ and need to store values as predefined columns. But in NoSQL created using Json/XML format.</li><li>Suppose if we need to update table with new record, in relational database if i want to add EmpId as “E5” — we cannot do it, because we already defined it as integer. But in NoSQL we can do that, here there is no relation between each node.</li><li>In Relational database if you want to add new column without changing table schema as “DeptName” as shown in above image, we cannot do that. But in NoSQL we can create.</li></ol><p>To Over come all these limitations NoSQL came into picture.</p><p><strong>NoSQL Advantages:</strong></p><p>. Enables easy updates to schema and fields.<br>. Stores unstructured, semi-structured and structured data.<br>. Handles large volumes of data at high speed with scale-out architecture.<br>. developer-friendly.<br>. Take full advantage of the cloud to Deliver Zero Downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cUXxci62uoq6C6fUKVio9A.png" /><figcaption>Examples of NoSQL databases</figcaption></figure><p><strong>NoSQL Database Models:</strong></p><p>Each databases has different types of models</p><ol><li>Document -based Database</li><li>Graph-based Database.</li><li>Wide column based Database.</li><li>key-value database.</li></ol><p><strong>Remember NoSQL is never replacement of Relational database, each one has their own advantages.</strong></p><p>Keep Learning…🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=504240f2ac0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Architecture]]></title>
            <link>https://medium.com/@kapuluru.vinay07/spark-architecture-ffb8ea78703c?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffb8ea78703c</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[dağ]]></category>
            <category><![CDATA[spark-architecture]]></category>
            <category><![CDATA[data-engineer]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Wed, 27 Dec 2023 11:01:59 GMT</pubDate>
            <atom:updated>2023-12-27T11:01:59.304Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction: </strong>Spark, an open-source distributed computing system, has become a go-to choice for processing large-scale data. Understanding its architecture sheds light on how it efficiently transforms user code into actionable insights. Let’s embark on a journey through the key steps in the Spark architecture.</p><p><strong>Step 1: Code Submission and DAG Creation:</strong></p><ul><li>The client submits Spark user code.</li><li>Upon submission, the driver converts user code containing transformations and actions into a logical plan called the DAG (Directed Acyclic Graph).</li></ul><p><strong>Step 2: DAG to Physical Execution Plan:</strong></p><ul><li>The logical plan is then transformed into a physical execution plan, breaking it down into tasks within each stage.</li><li>These tasks are bundled and sent to the Spark cluster.</li></ul><p><strong>Step 3: Resource Negotiation and Executor Launch:</strong></p><ul><li>The driver communicates with the cluster manager, negotiating resources for the Spark job.</li><li>The cluster manager launches executors on worker nodes, acting on behalf of the driver.</li><li>Executors, upon initiation, register themselves with the driver.</li><li>The driver sends tasks to the executors based on data placement, gaining a complete overview of executing tasks.</li></ul><p><strong>Step 4: Dynamic Execution Monitoring:</strong></p><ul><li>Throughout task execution, the driver monitors the set of running executors.</li><li>It also schedules future tasks based on data placement.</li></ul><p><strong>Conclusion:</strong> In essence, Spark’s architecture orchestrates a symphony of tasks, efficiently utilizing resources and optimizing data processing. By delving into this process, we gain insights into how Spark transforms user code into a dynamic and scalable data processing pipeline.</p><p><strong>Now will understand same with Treasure Hunt Story 😊</strong></p><p>Imagine you’re on a treasure hunt. The hunt involves multiple clues and steps, each leading to the next until you find the treasure.</p><ol><li><strong>User Code Submission:</strong></li></ol><ul><li>You decide the steps you’ll take in your treasure hunt, just like a Spark programmer creates code.</li><li>You write down these steps as a list, representing your plan.</li></ul><p>2. <strong>Logical Plan (DAG):</strong></p><ul><li>Before starting the hunt, you organize your list into a logical sequence, making sure each step follows the other logically.</li><li>This organized list is like the Directed Acyclic Graph (DAG) in Spark. It’s a roadmap for Spark to understand the order of tasks.</li></ul><p>3. <strong>Physical Execution Plan (Tasks):</strong></p><ul><li>Now, imagine if someone else (a driver) needed to follow your treasure map and execute the steps. They would convert your logical plan into specific actions — tasks.</li><li>These tasks could include digging in specific spots, following clues, or solving puzzles. Each task contributes to reaching the final goal.</li></ul><p>4. <strong>Cluster Launch and Execution:</strong></p><ul><li>To make the hunt more efficient, you decide to collaborate with others (executors). Each collaborator (executor) handles different tasks based on their skills.</li><li>The driver communicates with these collaborators, launches them to various locations (worker nodes), and ensures they’re ready to follow the map.</li></ul><p>4. <strong>Execution Monitoring and Scheduling:</strong></p><ul><li>As the hunt progresses, you (the driver) keep an eye on the collaborators (executors). If some are faster or encounter obstacles, you might adjust the plan for future tasks.</li><li>You schedule upcoming tasks based on where each collaborator is and the best route to the next step.</li></ul><p>In essence, the DAG is like your well-thought-out treasure map, guiding Spark through a sequence of tasks to achieve a computational goal. It ensures efficient and organized task execution, much like a successful treasure hunt!</p><p>Hope you enjoyed the story 😜</p><p>Keep Learning …🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffb8ea78703c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OLAP — Online Analytical Processing]]></title>
            <link>https://medium.com/@kapuluru.vinay07/olap-online-analytical-processing-fa7c25e558c7?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa7c25e558c7</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[oltp]]></category>
            <category><![CDATA[olap]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Tue, 26 Dec 2023 13:31:23 GMT</pubDate>
            <atom:updated>2023-12-26T13:31:23.499Z</atom:updated>
            <content:encoded><![CDATA[<h3>OLAP — Online Analytical Processing</h3><p>. Its basically consists of historical data.</p><p>. It is more de-normalized tables.</p><p>. Mostly data is read — means update frequency is less compared to OLTP. (The data is written once and read multiple times)</p><p>. Data is majorly filtered and cleaned.</p><p>. It is supported with row/ columnar database.</p><p>. Analytical reports — means more complex queries.</p><p>Why we need OLAP over OLTP</p><p>Ex: 1. It has all the historical data, but in OLTP it has some period of data only.</p><p>i) In OLTP, as it has live data — some millions of read/write operations will happens per millisecond and it has normalized tables. So, when we need to do analytics on this, it will be more complex (due to normalized — have to do more joins) and already R/W operations going on, same time on top of that we are doing analytics it will be a huge load (and also it do not have historical data).</p><p>Ex:2. In OLTP — Suppose in eCommerce there is a input column, named as country — where users needs to fill the filed while ordering. Assume Different users will fill in different way like IND or INDN or INDIA or Hindustan. When business need to check how many orders happens on India then it is very difficult to find. Then by using ETL load data in OLAP and clean the data make country filed to standard value as INDIA and can do analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa7c25e558c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery Architecture Explanation]]></title>
            <link>https://medium.com/@kapuluru.vinay07/bigquery-architecture-explanation-2e2c616d71ad?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e2c616d71ad</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[column-oriented]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 08:56:02 GMT</pubDate>
            <atom:updated>2024-01-08T18:06:08.993Z</atom:updated>
            <content:encoded><![CDATA[<p>Before discuss about architecture will discuss few things. If you take any query engines, generally we need two things.</p><p>1. Storage — To store all the data</p><p>2. Compute — Means processing data — to run some queries</p><p>BigQuery also has these two, in some query engines these two (storage and compute) will be in coupled manner (means cannot splits these two). By that it will be more expensive for infrastructure. In BigQuery, storage and compute are separated so, it is inexpensive means we no need to pay two charges by using single service (called as cost management).</p><h3><strong>Big Query Architecture:</strong></h3><p>It has different layers</p><p><strong>1. Colossus:</strong></p><p>a. This is where BigQuery stores the data.</p><p>b. It is cluster level file system and distributed management (so there is no single point of failure).</p><p>c. It will also maintain replication for us, which data we have been stored.</p><p>d. It is successor of Google file system. (Colossus like HDFS in Hadoop).</p><p>e. It stores the data in column oriented format.</p><p><strong>2. Dremel:</strong></p><p>a. It is compute engine used by BigQuery to compute our queries.</p><p>b. Dremel internally has</p><p><strong>i. Leaf Nodes:</strong> It helps to read the data from the storage i.e. from <strong>Colossus</strong></p><p><strong>ii. Mixers:</strong> This is the layer where aggregations will happen, which we have written in our queries.</p><p><strong>iii. Root Server:</strong> It coordinates with Mixers and leaf nodes.</p><p><strong>3. Jupiter:</strong> Something need to connect Storage and Compute to perform our operations, that will do by Google Jupiter.</p><p>i. Actually, Jupiter is high speed network protocol connects Dremel and Colossus.</p><p>ii. It is, In-house project in Google from long time. It usually call it as <strong>Petabit Jupiter network.</strong></p><p><strong>4. Borg:</strong> The whole architecture is orchestrated by using Borg component.</p><p>Suppose, when we trigger any query or job, this query should communicate internally with Dremel, Colossus etc., to complete end to end query. This complete scheduling process will be done by Borg. This also an In-house project in GCP, use for orchestration scheduling the jobs.</p><p><strong>Bonus Tip:</strong></p><p><strong>Advantage of Column oriented format:</strong></p><p>We have discussed Colossus is column oriented.</p><p>All will say BigQuery, Hive, Redshift (AWS), Cassandra(No SQL) , Hbase(No SQL) etc., are column oriented. What it actually means.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/1*ijfxthQzulx0jq_sQIv8cQ.png" /><figcaption>Dataset</figcaption></figure><p><strong>In Row oriented format:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*C11XKZQf7Ax-1Buhnaxo6A.png" /><figcaption>Row Oriented</figcaption></figure><p>When we are doing some aggregation on Salary column, it should read entire row and will give results. Reading unnecessary data while processing the query on Salary column.</p><p><strong>In Column Oriented:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*yBfYKmWqQKhKHnJQwe8yuw.png" /><figcaption>Column Oriented</figcaption></figure><p>When we are doing aggregations on Salary column, it will go directly on salary column and perform the operations, instead of reading unnecessary data.</p><p>-&gt; In BigQuery when we store in column oriented it called as <strong>Capacitor</strong>. It is terminology used in BigQuery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e2c616d71ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interview questions and answers on AWS EMR?]]></title>
            <link>https://medium.com/@kapuluru.vinay07/interview-questions-and-answers-on-aws-emr-7ad832fde76f?source=rss-5fd2f22eaf1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ad832fde76f</guid>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-emr]]></category>
            <dc:creator><![CDATA[Vinay Kumar Reddy]]></dc:creator>
            <pubDate>Tue, 12 Dec 2023 17:25:51 GMT</pubDate>
            <atom:updated>2023-12-13T04:38:33.496Z</atom:updated>
            <content:encoded><![CDATA[<p>Interview questions and answers on AWS EMR?</p><h3>1. What is AWS EMR?</h3><p>AWS EMR (Elastic MapReduce) is a cloud-based big data processing service provided by Amazon Web Services. It simplifies the processing of large datasets using popular frameworks such as Apache Hadoop, Apache Spark, Apache Hive, Apache HBase, and others.</p><h3><strong>2. What is the difference between AWS EMR and Glue?</strong></h3><p><strong>AWS EMR (Elastic MapReduce):<br>Purpose:</strong></p><p>EMR is primarily designed for processing and analyzing large-scale data using distributed frameworks such as Apache Hadoop, Apache Spark, Apache Hive, and more. It is suitable for big data processing, batch processing, and complex analytics.<br><strong>Managed Clusters:</strong></p><p>Users create and manage clusters, which consist of master and core nodes. EMR allows fine-tuning of cluster configurations based on specific workloads.</p><p><strong>Processing Frameworks:</strong></p><p>Supports various processing frameworks, including Hadoop, Spark, Hive, HBase, and others. Users have flexibility in choosing the right framework for their specific use case.<br><strong>Control Over Environment:</strong></p><p>Provides more control over the underlying infrastructure and software configurations. Users can install additional software and customize the environment.<br><strong>Use Cases:</strong></p><p>Ideal for organizations with specific big data processing needs, custom algorithms, or existing Hadoop/Spark workflows.</p><h3>AWS Glue:</h3><p><strong>Purpose:</strong></p><p>AWS Glue is a fully managed ETL (Extract, Transform, Load) service. It is designed for data integration, transformation, and loading from various sources into data lakes or data warehouses.<br><strong>Serverless ETL:</strong></p><p>Operates in a serverless environment. Users define and run ETL jobs without managing the underlying infrastructure.<br>Crawlers and Catalog:</p><p>Includes crawlers that can automatically discover and catalog metadata about data stored in various sources. The AWS Glue Data Catalog serves as a central metadata repository.</p><p>Transformations:</p><p>Offers a visual interface for building transformations. Users can define transformations using a script or use the built-in ETL job authoring features.<br><strong>Integration with Other AWS Services:</strong></p><p>Integrates seamlessly with other AWS services, such as Amazon S3, Amazon Redshift, and AWS Lambda, making it part of an end-to-end data processing pipeline.<br><strong>Use Cases:</strong></p><p>Suited for organizations looking for a managed ETL service without the need for extensive infrastructure management. Ideal for data integration and data preparation tasks.</p><h3><strong>3. What is the Significance of Bootstrap Actions in EMR?</strong></h3><p>Bootstrap Actions are scripts that run on cluster nodes before Hadoop starts. They are used to install additional software, configure settings, or set up custom environments on EMR instances.</p><h3><strong>4. Explain the Difference Between On-Demand and Spot Instances in EMR.</strong></h3><p><strong>On-Demand Instances:</strong></p><p>These are instances you pay for on an hourly basis with no upfront commitment.</p><ul><li>Think of On-Demand Instances like a pay-as-you-go service. You use these instances for as long as you need, and you pay for the time they are running.</li><li>It’s similar to renting a car hourly. You pay for the hours you use it, and you can keep it for as long as you need without any long-term commitment.</li><li>On-Demand Instances are reliable, and once you start them, they will continue running until you decide to stop them.</li></ul><p><strong>Spot Instances:</strong></p><p>These are spare Amazon EC2 instances available at a lower price. However, they can be terminated by AWS if the capacity is needed elsewhere.</p><ul><li>Spot Instances are like getting a great discount on a hotel room, but there’s a catch. You can get these instances at a lower price, but they are &quot;spare&quot; or extra capacity that Amazon has available.</li><li>The catch is that Amazon can ask you to check out (terminate) if they need that capacity for something else. It’s like staying in a hotel room at a discounted rate, but the hotel might ask you to leave if they suddenly get more guests.</li><li>Spot Instances can be a cost-effective option, especially if you are flexible with your computing needs and can handle the possibility of instances being terminated.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ad832fde76f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>