<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by RK Iyer on Medium]]></title>
        <description><![CDATA[Stories by RK Iyer on Medium]]></description>
        <link>https://medium.com/@rkniyer999?source=rss-c05b49979e2e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cGLRW79OKrWZPAxLhATyyA.jpeg</url>
            <title>Stories by RK Iyer on Medium</title>
            <link>https://medium.com/@rkniyer999?source=rss-c05b49979e2e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:46:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rkniyer999/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Infrastructure to Intent: How Databricks Abstracts Spark Complexity]]></title>
            <link>https://rkniyer999.medium.com/from-infrastructure-to-intent-how-databricks-abstracts-spark-complexity-fe15c2cf0652?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe15c2cf0652</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sun, 11 Jan 2026 06:16:19 GMT</pubDate>
            <atom:updated>2026-01-11T06:16:19.229Z</atom:updated>
            <content:encoded><![CDATA[<h4>SparkContext in Databricks</h4><h4>DATABRICKS SERIES 01: FUNDAMENTALS — WHY YOU DONT’T NEED TO INITIALIZE SPARKCONTEXT IN DATABRICKS</h4><h3>❑ Overview</h3><p>If you’ve worked with Apache Spark outside Databricks (especially like me 😀), you’re used to starting with something like:</p><pre><br>from pyspark.sql import SparkSession<br>spark = SparkSession.builder.appName(&quot;MyApp&quot;).getOrCreate()</pre><p>So, when you first open a Databricks notebook, a natural question arises:</p><p><strong><em>Where is the Spark initialization? And why don’t we need it?</em></strong></p><p>The short answer: <strong>Databricks already did it for you.</strong><br>The longer answer reveals an important design philosophy of the platform. Let’s get into in….</p><h3>❑ Spark in the “Vanilla” World</h3><p>In a traditional Spark setup (on‑prem or DIY cloud clusters), your application is responsible for:</p><ul><li>Creating a SparkContext</li><li>Connecting to the cluster</li><li>Managing executors, memory, and configs</li></ul><p>Since Spark 2.x, SparkSession became the unified entry point, wrapping SparkContext, SQLContext, and more—but <strong>you still had to create it yourself</strong>.</p><h3>❑ Core components of Spark</h3><p>Below are the core components of the programming model and data management system in Apache Spark -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwiQJXsFUkPw20EBHO3rUg.png" /><figcaption>SparkSession Mental Model</figcaption></figure><h4>1. SparkSession</h4><p>Introduced in Spark 2.0, the <strong>SparkSession</strong> is the unified entry point for all Spark functionalities. It consolidates previously separate contexts (SparkContext,SQLContext, HiveContext) into a single interface.</p><ul><li><strong>Usage:</strong> It is used to create DataFrames, register temporary views, and execute SQL queries.</li><li><strong>Access:</strong> Created via SparkSession.builder.getOrCreate()</li></ul><h4>2. SparkContext</h4><p>The <strong>SparkContext</strong> is the lower-level entry point that represents the connection to a Spark cluster. While the SparkSession is now the primary interface, the SparkContext remains active “under the hood” to manage cluster resources and RDD-level operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*h0SR6Slb2aauCXVqu7g56g.png" /><figcaption>SparkContext</figcaption></figure><p>As shown above, the SparkContext can connect to several types of <em>cluster managers</em> (either Spark’s own standalone cluster manager, YARN or Kubernetes), which allocate resources across applications. Once connected, Spark acquires <em>executors</em> on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends <em>tasks</em> to the executors to run.</p><ul><li><strong>Usage</strong>: Connects your application to a Spark cluster. Manages executors, task scheduling, memory.</li><li><strong>Access:</strong> You can access it from a session via spark.sparkContext.</li></ul><h4>3. SQL Engine (Spark SQL)</h4><p>The <strong>SQL Engine</strong> is the specialized module for processing structured data. It provides the optimization layer (Catalyst Optimizer) that converts high-level SQL queries or DataFrame operations into efficient RDD-level execution plans.</p><p><strong>Capability:</strong> It allows you to run SQL queries directly on various data sources like Parquet, JSON, and JDBC.</p><h4>4. Catalog</h4><p>The <strong>Catalog</strong> is the interface for managing metadata about the data environment, such as databases, tables, and functions.</p><ul><li><strong>Functionality:</strong> It allows users to list tables, check if a table exists, and manage temporary views.</li><li><strong>Access:</strong> It is accessed directly through the SparkSession using spark.catalog.</li><li><strong>Common Methods:</strong></li><li>listTables(): Returns a list of tables in the current database.</li><li>currentDatabase(): Shows the name of the active database.</li><li>dropTempView(viewName): Removes a temporary view from the catalog.</li></ul><p><strong>Think of it this way:</strong></p><ul><li><strong>SparkContext</strong> is the <strong>engine room</strong></li><li><strong>SparkSession</strong> is the <strong>control panel</strong></li><li>In Databricks, the control panel is already switched <strong>ON</strong></li></ul><h3>❑ Databricks Flips the Model</h3><p>Databricks is not just Spark — it’s a <strong>managed Spark runtime and execution environment</strong>.</p><p>When you:</p><ol><li>Start a Databricks cluster, and</li><li>Attach a notebook to it</li></ol><p>Databricks automatically:</p><ul><li>Starts the driver and executors</li><li>Initializes a <strong>single, shared </strong><strong>SparkContext</strong></li><li>Injects a ready‑to‑use <strong>SparkSession (</strong><strong>spark)</strong> into your notebook</li></ul><p>By the time your first cell runs, Spark is already alive and connected.</p><p><strong><em>No boilerplate. No setup.</em></strong></p><h3>❑ The Role of Databricks Notebooks</h3><p>A Databricks notebook is <strong>not</strong> just a script runner.</p><p>It is:</p><ul><li>Bound to a <strong>live Spark session</strong></li><li>Executed inside a <strong>managed driver process</strong></li><li>Isolated per user + per notebook</li></ul><blockquote>Think of it as: <em>“Your notebook is already inside a running Spark application.”</em></blockquote><h3>❑ What Happens Behind the Scenes</h3><p>When a Databricks cluster starts:</p><ol><li>Databricks spins up a <strong>driver VM</strong></li><li>Spark runtime is bootstrapped</li><li>SparkContext is initialized once</li><li>A default SparkSession is created</li><li>The session is injected into every attached notebook</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*Hvy91SCxi3xGw9s-ynMR3Q.png" /><figcaption>Databricks Cluster Startup flow</figcaption></figure><h3>❑ Antipattern</h3><p>Creating another Spark session would be like:</p><ul><li>Spawning a second driver (not allowed)</li><li>Competing for the same cluster resources</li><li>Fighting Databricks’ lifecycle management</li></ul><p>So Databricks <strong>intentionally discourages</strong> manual context creation.</p><h3>❑ Why Databricks Does This (Intentionally)</h3><p>This is a <strong>design decision</strong>, not a shortcut.</p><ul><li><strong>Stability &amp; safety</strong>: Only one SparkContext per cluster prevents conflicts in multi‑user environments</li><li><strong>Governance</strong>: Cluster policies, security, and configs are enforced centrally</li><li><strong>Performance</strong>: Resource management is predictable and optimized</li><li><strong>Productivity</strong>: Users focus on data and analytics — not cluster plumbing</li></ul><p>Creating your own SparkSession inside a Databricks notebook is at best redundant—and at worst rejected—because the platform already manages the lifecycle.</p><h3>❑ What You <em>Should</em> Do Instead</h3><p>Use Spark directly:</p><pre>df = spark.read.parquet(&quot;/mnt/data/sales&quot;)</pre><p>If you need tuning:</p><ul><li>Set configs at the <strong>cluster level</strong></li><li>Use <strong>cluster policies</strong> or <strong>init scripts</strong></li><li>Apply runtime configs safely via spark.conf.set() when appropriate</li></ul><h3>❑ Plain Spark vs Databricks (Quick Comparison)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCWYRWvQTnZ84ZCBydDUrA.png" /></figure><h3>❑ The Bigger Picture</h3><p>Databricks abstracts Spark initialization <strong>on purpose</strong>.<br> It reflects the platform’s philosophy:</p><blockquote><strong>Move from infrastructure to intent.</strong><br> Write analytics, not cluster code.</blockquote><h3>❑ References</h3><p><a href="https://spark.apache.org/docs/latest/cluster-overview.html">Cluster Mode Overview — Spark 4.1.0 Documentation</a></p><p><a href="https://spark.apache.org/docs/latest/job-scheduling.html">Job Scheduling — Spark 4.1.0 Documentation</a></p><h3>❑ Conclusion</h3><p>I hope this blog helped you to understand on “How Databricks Abstracts Spark Complexity”. We will surely explore many of databricks topics in detail, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe15c2cf0652" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Microsoft Built a NoSQL Engine on Relational Foundations]]></title>
            <link>https://medium.com/microsoft/how-microsoft-built-a-nosql-engine-on-relational-foundations-997926432408?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/997926432408</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-cosmos-db]]></category>
            <category><![CDATA[documentdb]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sat, 13 Dec 2025 06:31:17 GMT</pubDate>
            <atom:updated>2026-01-12T17:08:17.974Z</atom:updated>
            <content:encoded><![CDATA[<h4>DocumentDB on PostgreSQL</h4><h4>Relational NoSQL Revolution: PostgreSQL as DocumentDB’s Secret Engine</h4><p>There has been a lot of excitement in the NoSQL world recently with Microsoft’s decision to open-source their DocumentDB project under the Linux Foundation. This initiative has quickly gained broad support and participation from industry leaders such as AWS, Cockroach Labs, Google, Snowflake, Supabase, Yugabyte, and many more. For those interested in the journey of Azure DocumentDB, I highly recommend checking out my detailed blog on the topic.</p><p>What fascinates me most is the architectural innovation behind DocumentDB, specifically how it leverages the extensibility of PostgreSQL as the storage engine for a NoSQL document database.</p><blockquote><strong>Yes, you heard it right: a relational database powering a NoSQL database!</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmOcUO1Jdbtt1fq8YFDX3w.png" /><figcaption>Azure DocumentDB</figcaption></figure><p>The video talk by<strong> Vinod Sridharan</strong>, a key contributor to Microsoft DocumentDB and Partner Software Engineer at Microsoft, titled “<a href="https://www.youtube.com/watch?v=OL0YO1jOSX0">Designing for Document Databases in PostgreSQL</a>,” clarifies the sophisticated design choices and extensibility features that make this possible. In this blog, I aim to demystify this design and explain how Microsoft’s DocumentDB harnesses the power of PostgreSQL to deliver a scalable, flexible, and powerful document database solution.</p><p>Setting context on Document Databases and the history of DocumentDB -forgive me for boring you, but hey, even superheroes have origin stories!</p><h3>What Are Document Databases?</h3><p>Document databases are a <strong>type of</strong> <strong>NoSQL database</strong> that store data as <strong>schema-free documents</strong>, typically in formats like <strong>JSON</strong> or <strong>BSON</strong>. This flexibility makes them ideal for applications where data structures evolve frequently. They are designed for <strong>horizontal and vertical scaling</strong>, enabling performance across small datasets and massive workloads. While they traditionally offer fewer SQL features (such as joins and complex transactions), modern document databases increasingly incorporate these capabilities as the technology matures.</p><h3>History of DocumentDB</h3><p>The <strong>DocumentDB project</strong> began in <strong>2024</strong> as a <strong>pair of PostgreSQL extensions</strong> created by Microsoft to bring support for popular <strong>BSON data models</strong> and <strong>document queries</strong> into the PostgreSQL ecosystem. Over time, it evolved into a <strong>complete, user-friendly, MIT-licensed open-source document database solution</strong> built on PostgreSQL. Today, DocumentDB is fully <strong>MongoDB-compatible</strong>, leveraging the widely adopted MongoDB drivers for seamless integration.</p><blockquote>DocumentDB was built by Microsoft with a simple goal: give developers an open document database with the flexibility of NoSQL and the power, reliability, openness, and ecosystem of Postgres</blockquote><h3>Design principles for DocumentDB</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OU4p8UGD6CzFGr2Hx1VCCw.png" /><figcaption>DocumentDB Design principles</figcaption></figure><p>The following design principles guided the development of DocumentDB -</p><ul><li>DocumentDB Project, recently open-sourced, was used as the backend for Azure DocumentDB (formerly MongoDB vCore). When DocumentDB was open-sourced, MongoDB API semantics were mapped to PostgreSQL query semantics, and interfaces were built on similar contracts to ensure ease of use — no table creation or schema management required.</li><li>Inspiration was drawn from other NoSQL databases for features like easy index management (wildcard, root, sub-path), filters, sorts, and TTL for automatic document expiry. BSON was chosen as the storage layer to maintain type fidelity and reduce transcoding costs, which is critical for OLTP workloads demanding low latency and high throughput.</li></ul><h3>Why PostgreSQL as a Document Database Backend?</h3><p><strong>PostgreSQL’s extensibility</strong> allows for <strong>custom data types, operators, indexes, and query planning</strong>. DocumentDB leverages this by:</p><ul><li>Using BSON as the storage format for <strong>better type fidelity</strong> and <strong>reduced transcoding overhead</strong> compared to JSONB (PostgreSQL’s binary JSON format enabling fast indexing, querying, and native operators).​ This is super important when you’re dealing with things like OLTP workloads, which are <strong>low-latency and high-throughput</strong>.</li></ul><blockquote>Type fidelity is the accurate preservation of original data types end to end, so an integer stays an integer, a date stays a date, and a boolean stays a boolean rather than everything becoming generic types like strings. Maintaining strong type fidelity in serialization and database mappings prevents subtle bugs (for example, broken numeric comparisons) and makes querying and analysis more reliable.</blockquote><ul><li>Abstracting away <strong>schema and table creation for ease of use</strong>, mimicking managed NoSQL services.</li><li>Mapping collections to PostgreSQL tables, with a mapping layer for naming differences.​</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EdM-1O8JQOtlPdtLyxEWxA.png" /><figcaption>DocumentDB to PostgreSQL Mapping (Source — Vinod Sridharan Posette event slides)</figcaption></figure><h3>Key PostgreSQL Extensibility Features Used</h3><h3>1. Custom Data Types and Operators</h3><ul><li>BSON is modeled similarly to JSONB, supporting <a href="https://www.postgresql.org/docs/9.2/storage-toast.html">TOAST</a> (storing large column values (up to 1GB) efficiently) and compression.</li><li>Custom operators and functions are created for document operations.</li></ul><h3>2. Planner hook</h3><ul><li>As MongoDB query semantics had to be converted to SQL queries, planner hook available in PostgreSQL was used to replace the default planner and the specific function was used to<em> </em>generate the query and push it to PostgreSQL to improve performance based on the table metadata information, index information etc.</li></ul><h3>3. Indexing for Document Databases</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3RtZnZun9cWkI2dXIhHnow.png" /><figcaption>PostgreSQL Indexes (Source — Vinod Sridharan Posette event slides)</figcaption></figure><ul><li>GIN Index: Has good operator extensibility<em> &amp; s</em>upports multi-value indexing, useful for arrays and wildcard indexes but <em>could not support unique constraints.</em></li><li>RUM Index (Next generation GIN): Extends GIN, adding support for unique constraints and positional information, critical for advanced filtering and ranking.​For more info read <a href="https://postgrespro.com/blog/pgsql/4262305">Indexes in PostgreSQL — 8 (RUM) : Postgres Professional</a></li><li>B-tree and GiST: Used for simpler cases but lack multi-value or wildcard support.</li></ul><h4>Index Extensibility Points</h4><p>PostgreSQL’s index extensibility includes:</p><ul><li>Extract Value: Called on insert/update to produce terms for the index.</li><li>Extract Query: Maps query predicates to index terms.</li><li>Compare Partial: Optional check for range scans.</li><li>Consistency Check: Ensures the index result matches the query.</li></ul><p>These points allow building custom indexes for document databases that handle arrays, wildcards, and complex filtering.</p><h4>Performance (JSONB vs BSON) and Optimization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*RU6mB5mZE5CNJOysUDqU2w.png" /><figcaption>JSON vs BSON ( (Source — Vinod Sridharan Posette event slides)</figcaption></figure><ul><li>Point Reads: Comparable between JSONB and BSON.</li><li>Filters: Latency overhead is due to the planning overhead and scenario being measured</li><li>Index Pushdown: Avoids unnecessary document decompression and projection, reducing query time from 30s to 136ms in real-world scenarios.</li></ul><h3>Summary</h3><p>PostgreSQL was a natural choice for the foundation, thanks to its <strong>popularity, robustness, and extensibility</strong>, making DocumentDB both powerful and developer friendly.</p><p>Document databases on PostgreSQL leverage its extensibility for custom storage, indexing, and query planning. Key features include:</p><ul><li>Custom BSON storage and operators.</li><li>RUM/GIN indexes for multi-value and wildcard support.</li><li>Planner and executor hooks for efficient query execution.</li><li>Index condition pushdown and support functions for advanced filtering.</li></ul><p>This approach allows PostgreSQL to mimic the ease and scalability of managed document databases while retaining the power and flexibility of SQL.</p><h3>❑ Special Thanks</h3><p>Special Thanks to <strong>Vinod Sridharan</strong> for bringing clarity through the talk <a href="https://www.youtube.com/watch?v=OL0YO1jOSX0">Designing for Document Databases in PostgreSQL</a> &amp; reviewing the blog.</p><h3>❑ References</h3><p><a href="https://www.youtube.com/watch?v=OL0YO1jOSX0">Designing for Document Databases in PostgreSQL | POSETTE: An Event for Postgres 2025 — YouTube</a></p><p><a href="https://stackoverflow.com/questions/14446684/could-one-use-postgres-as-a-document-database">postgresql — Could one use Postgres as a document database? — Stack Overflow</a></p><p><a href="https://postgrespro.com/docs/enterprise/current/rum">Postgres Pro Enterprise : Documentation: 17: F.73. rum — an access method to work with the RUM indexes : Postgres Professional</a></p><p><a href="https://dzone.com/articles/hooks-the-secret-feature-powering-the-postgres-ecosystem">Hooks: The Secret Powering the Postgres Ecosystem</a></p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through Azure DocumentDB internals. We will surely explore and continue with other topics of Azure DocumentDB in upcoming blogs, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=997926432408" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/how-microsoft-built-a-nosql-engine-on-relational-foundations-997926432408">How Microsoft Built a NoSQL Engine on Relational Foundations</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure DocumentDB: Open source, MongoDB-Compatible Database for Modern Apps]]></title>
            <link>https://medium.com/microsoft/azure-documentdb-open-source-mongodb-compatible-database-for-modern-apps-0ace448ec927?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ace448ec927</guid>
            <category><![CDATA[azure-database]]></category>
            <category><![CDATA[cloud-database]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[documentdb]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 19:30:28 GMT</pubDate>
            <atom:updated>2025-12-30T05:46:18.783Z</atom:updated>
            <content:encoded><![CDATA[<h4>ALL ABOUT AZURE DOCUMENT DB’S JOURNEY</h4><h4>SERIES: FUNDAMENTALS OPEN SOURCE POWER MEET CLOUD INNOVATION</h4><h3>❑ Overview</h3><p>The world of cloud databases is evolving rapidly, and Microsoft has been at the forefront of this transformation. What started as <strong>Azure Cosmos DB for MongoDB vCore</strong> has now matured into <strong>Azure DocumentDB</strong>, a fully managed, open-source, AI-ready document database designed for modern workloads. This journey reflects a shift toward openness, flexibility, and innovation — empowering developers to build scalable, intelligent applications without vendor lock-in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSneZoLpUlic-O2LxMSzOg.png" /><figcaption>Key milestones-Azure Cosmos DB for MongoDB (vCore) to Azure DocumentDB</figcaption></figure><p>In this blog, we’ll start by exploring the categories of NoSQL databases and then dive into <strong>DocumentDB</strong> — a fully open-source, MongoDB-compatible document database with contributions from Microsoft, AWS, and Yugabyte. We’ll also explain how DocumentDB powers <strong>Azure DocumentDB</strong>, enabling a fully managed, MongoDB-compatible database service for building modern, scalable applications.</p><h3>❑ Categories of NoSQL Databases?</h3><p>NoSQL databases can be classified into four main categories based on their data models and use cases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCSglAPpERpxs_En83VpiQ.png" /><figcaption>NoSQL DB classification</figcaption></figure><ol><li><strong>Document databases</strong> — Store data as documents, usually in JSON, BSON, or XML formats. e.g. Mongo DB, Couch DB.</li><li><strong>Key-value stores</strong> — Store data as simple key-value pairs where each unique key maps to a value, which can be any data type. e.g. Redis, Aerospike.</li><li><strong>Column-oriented databases</strong> <strong>(Wide-column stores)</strong> — Store data in columns rather than rows, optimizing for queries and analytics on large datasets. Apache Cassandra, HBase.</li><li><strong>Graph databases</strong> — Store data as nodes (entities) and edges (relationships) to efficiently manage and query relationships.</li></ol><p>Now let’s dive into Document Databases.</p><h3>❑ What are Document Databases in NoSQL Databases?</h3><ul><li>A<strong> </strong>document database is a <strong>type/category of NoSQL</strong> database designed to store, retrieve, and manage data as documents, usually in formats like JSON, JSONB or BSON.</li><li>Instead of organizing data in tables and rows like a relational database, document databases group related data into flexible, self-contained documents, which can have varying fields and structures across records.</li><li>They typically scale up &amp; scale out.</li><li>They typically provide a subset of Traditional SQL features e.g. Transactions, Cross-document joins, Richer Isolation semantics. (New features are added day-by-day)</li></ul><blockquote>BSON (Binary JSON) is a <strong>binary-encoded serialization format</strong> designed primarily for MongoDB and some other NoSQL/document databases. JSONB is <strong>PostgreSQL’s binary storage format for JSON,</strong> used when you declare a column as type JSONB.</blockquote><p>Now that we know about different categories of NoSQL DB &amp; Document databases, let&#39;s get in DocumentDB which is core topic for this blog.</p><h3>❑ What is DocumentDB?</h3><p><a href="https://documentdb.io/">DocumentDB</a>is a <strong>MongoDB compatible open-source document</strong> database <strong>built on PostgreSQL</strong>. It offers a native implementation of a document-oriented NoSQL database, enabling seamless CRUD (Create, Read, Update, Delete) operations on BSON (Binary JSON) data types within a PostgreSQL framework. Beyond basic operations, DocumentDB empowers users to execute complex workloads, including full-text searches, geospatial queries, and vector search, delivering robust functionality and flexibility for diverse data management needs.</p><p>Key reasons to consider DocumentDB include:</p><ul><li><strong>Open source and MIT licensed</strong>: Encourages community innovation and unrestricted use.</li><li><strong>MongoDB compatibility</strong>: Seamlessly supports MongoDB APIs, tools, and drivers with minimal changes.</li><li><strong>Built on PostgreSQL</strong>: Leverages proven stability, extensibility, and rich features from a mature RDBMS.</li><li><strong>Robust Query Capabilities</strong>: Supports complex workloads including CRUD operations on BSON data, full-text search, geospatial queries, and vector search.</li><li><strong>Advanced indexing capabilities:</strong> Provides advanced indexing capabilities including single field, multi-key, compound, text, and geospatial indexes.</li><li><strong>On-premises and cloud flexibility</strong>: Can be deployed in diverse environments depending on your infrastructure and compliance needs.</li><li><strong>Enhanced scalability and performance</strong>: Supports scalable document operations with PostgreSQL’s performance optimizations.</li><li><strong>Community-driven development: Can </strong>reach out to help using <a href="https://discord.com/invite/vH7bYu524D">Discord</a> or <a href="https://github.com/documentdb/documentdb/discussions">Github discussions</a>.</li></ul><h3>❑ Components of DocumentDB</h3><p>Below three core components combine to allow MongoDB apps to seamlessly work in harmony whether in cloud or in your own infrastructure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lsRvu-2wNjN2iPNNT72K9A.png" /><figcaption>DocumentDB Architecture components</figcaption></figure><p>DocumentDB consists of three main components working in harmony to deliver its capabilities:</p><h4><strong>pg_documentdb_gw (Gateway)</strong></h4><p>Functions as a gateway &amp; translation layer converting MongoDB API calls into native PostgreSQL queries, enabling compatibility with MongoDB drivers and applications without major changes. It is responsible for <strong>routing client requests</strong> (coming from MongoDB drivers or APIs) to the underlying PostgreSQL engine that powers DocumentDB. Below are some of the key responsibilities —</p><ul><li><strong>Protocol Translation — </strong>Converts <strong>MongoDB wire protocol</strong> commands (e.g., find, insert, update) into equivalent PostgreSQL operations. Ensures compatibility with MongoDB drivers without requiring application changes.</li><li><strong>Connection Management — </strong>Handles client connections, pooling, and session lifecycle. Provides secure authentication and authorization integrated with Azure Identity.</li><li><strong>Query Parsing &amp; Dispatch — </strong>Parses BSON-based queries and maps them to PostgreSQL functions exposed by pg_documentdb and pg_documentdb_core.Optimizes query execution paths for performance.</li><li><strong>Feature Mediation — </strong>Acts as an intermediary for advanced features like <strong>vector search</strong>, <strong>text indexing</strong>, and <strong>transaction support. </strong>Ensures consistent behavior across MongoDB-compatible APIs and PostgreSQL backend.</li><li><strong>Scalability &amp; Load Balancing — </strong>Distributes requests across compute nodes in a <strong>vCore architecture</strong>. Supports horizontal scaling for high-throughput workloads.</li></ul><h4><strong>pg_documentdb</strong></h4><p><em>Optional only if you don’t need MongoDB-compatible CRUD API; core still required for BSON support.</em></p><p>Acts as the public API layer providing CRUD functionality (Create, Read, Update, Delete) specifically tailored for document operations, simplifying development for MongoDB-savvy developers.</p><ul><li><strong>CRUD Operations (Create/Read/Update/Delete)</strong><br> Provides functions and handlers that implement the expected behavior of MongoDB operations (e.g., insertOne, find, updateMany, deleteOne) against collections stored in PostgreSQL.</li><li><strong>Query Execution &amp; Planning</strong><br> Parses MongoDB‑compatible filters, projections, sorts, limits, and translates them into calls that leverage PostgreSQL indexes and the BSON capabilities from pg_documentdb_core.</li><li><strong>Index Management</strong><br>Implements creation and maintenance of <strong>single‑field, compound, text, and geospatial</strong> indexes, ensuring queries remain performant as collections grow.</li><li><strong>Transaction &amp; Session Semantics</strong><br> Coordinates multi‑document operations using PostgreSQL transactional guarantees, exposing familiar behaviors to MongoDB‑savvy developers.</li><li><strong>Feature Mediation</strong><br> Bridges higher‑level features (e.g., <strong>text search</strong> and <strong>geospatial</strong>, and where available, <strong>vector search</strong>) by orchestrating calls to extensions provided at the core/engine level.</li></ul><h4><strong>pg_documentdb_core</strong></h4><p>At the bottom of the stack is the <strong><em>pg_documentdb_core</em></strong> layer — a fully homegrown and customized Postgres extension to optimize support for the BSON data type. This extension provides the following capabilities:</p><ul><li>The ability to parse and manipulate BSON documents in the Postgres layer of the database engine, at all levels of nesting within the BSON document.</li><li>The ability to index fields in the BSON document — including single field indexes, multi-key indexes, compound indexes to optimize query filtering criteria on multiple fields, text indexes as well as geospatial indexes leveraging the capabilities of the PostGIS extension.</li><li>A fully functional authentication mechanism including SCRAM (Salted Challenge Response Authentication Mechanism) authentication.</li></ul><p>These features at the bottom of the stack will serve as the foundation for building an end-to-end NoSQL database user experience. A protocol translation layer can be built at the top of this stack to convert the inbound NoSQL database protocol of choice into the underlying Postgres protocol.</p><h4>Typical Request flow —</h4><ol><li><strong>Client</strong> issues find via MongoDB driver.</li><li><strong>Gateway (</strong><strong>pg_documentdb_gw)</strong> authenticates, parses the command, and routes it.</li><li><strong>pg_documentdb</strong> interprets filters/projection/sort, chooses indexes, and orchestrates the scan.</li><li><strong>pg_documentdb_core</strong> provides BSON parsing, storage, and low‑level operators; results are marshaled back up.</li></ol><h3>❑ What is Azure DocumentDB?</h3><p>Azure DocumentDB is the <strong>new name</strong> for vCore-based Azure Cosmos DB for MongoDB, a fully managed, multi-cloud-ready, enterprise-grade NoSQL document database service built on the open-source <a href="https://azure.microsoft.com/en-us/products/cosmos-db">DocumentDB engine</a>. It combines the flexibility of a document database with high performance, instant scalability, and advanced features like vector search, making it suitable for modern and AI-powered applications. It can be run on Azure, on-premises, or in other clouds, offering multi-cloud flexibility.</p><p>Below are some of the key features of Azure DocumentDB -</p><ul><li><strong>MongoDB-Compatible API</strong> — Seamlessly supports MongoDB drivers and tools for easy migration and development.</li><li><strong>Open-Source Foundation</strong> — Built on PostgreSQL with the DocumentDB engine, fully open-sourced under MIT license.</li><li><strong>Native BSON Support</strong> — Stores and processes BSON documents efficiently inside PostgreSQL.</li><li><strong>Advanced Indexing</strong> — Includes text, compound, and geospatial indexing for fast queries.</li><li><strong>Vector Search for AI</strong> — Enables hybrid search combining structured, unstructured, and vector data for AI-driven apps.</li><li><strong>Flexible Scaling</strong> — Independent compute and storage scaling for cost optimization.</li><li><strong>Enterprise Security</strong> — Integrated with Azure security, compliance, and identity management.</li><li><strong>Integrated Azure Services</strong> — Works seamlessly with Azure AI, Functions, and other cloud services for modern app development.</li></ul><h3>❑ References</h3><p><a href="https://www.linuxfoundation.org/press/linux-foundation-welcomes-documentdb-to-advance-open-developer-first-nosql-innovation">Linux Foundation Welcomes DocumentDB to Advance Open, Developer-First NoSQL Innovation</a></p><p><a href="https://learn.microsoft.com/en-us/documentdb/overview">DocumentDB Overview — DocumentDB (in Azure) | Microsoft Learn</a></p><p><a href="https://learn.microsoft.com/en-us/azure/documentdb/overview">Introduction/overview — Azure DocumentDB | Microsoft Learn</a></p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through Azure DocumentDB’s journey. We will surely explore and continue with advance topics of Azure DocumentDB in upcoming blogs, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ace448ec927" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/azure-documentdb-open-source-mongodb-compatible-database-for-modern-apps-0ace448ec927">Azure DocumentDB: Open source, MongoDB-Compatible Database for Modern Apps</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Privacy by Design: PII Detection and Anonymization with PySpark on Microsoft Fabric]]></title>
            <link>https://rkniyer999.medium.com/privacy-by-design-pii-detection-and-anonymization-with-pyspark-on-microsoft-fabric-f4e079fee989?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4e079fee989</guid>
            <category><![CDATA[pii-data]]></category>
            <category><![CDATA[pii]]></category>
            <category><![CDATA[pii-compliance]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sat, 04 Oct 2025 05:19:16 GMT</pubDate>
            <atom:updated>2025-10-04T05:19:16.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Whether you’re building analytics pipelines or conversational AI systems, the risk of exposing sensitive data is real. AI models trained on unfiltered datasets can inadvertently memorize and regurgitate PII, leading to compliance violations and reputational damage. This blog explores how to build scalable, secure, and compliant data workflows using PySpark, Microsoft Presidio, and Faker — covering hands-on examples of detection, masking, hashing, and synthetic data generation that apply equally to data engineering and AI use cases.</p><h3>Data Anonymization</h3><p>Data anonymization is the process of transforming personal or sensitive data in such a way that the individuals to whom the data pertains can no longer be identified — either directly or indirectly. This is a critical step in ensuring compliance with privacy regulations like GDPR, PDPA, and HIPAA, and in enabling safe data sharing for analytics and AI model training.</p><p>Unlike encryption, which is reversible with a key, anonymization aims to be irreversible, ensuring that once data is anonymized, it cannot be traced back to an individual.</p><h3>Why PII Anonymization matters?</h3><p>PII such as names, emails, phone numbers, and national IDs can be inadvertently exposed during data processing. Anonymizing this data ensures:</p><ul><li>Compliance with regulations like GDPR and PDPA</li><li>Reduced risk of data breaches</li><li>Fairness in AI/ML models by removing bias-inducing attributes</li></ul><h3>Common Data Anonymization Techniques</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*9EOm6HlRJyEqCwOB.png" /></figure><p>Effective data anonymization requires the application of techniques suited to the nature of the data and its intended use. Below are the <strong>most widely used methods</strong>:</p><h3>Masking –</h3><p>Masking involves replacing original data values with specified characters, either partially or completely. For example, sensitive information like contact numbers might appear as “XXX-XXX-XXXX.” This technique is particularly useful for safeguarding data in testing or development environments.</p><pre>Original:jack@example.com -&gt; masked_email = &quot;j***@example.com&quot;</pre><h3>Hashing –</h3><p>Applies a one-way cryptographic function to convert data into a fixed-length string of characters (hash value). Useful for consistent anonymization across datasets.</p><pre>Original (Customer Id): 234568 -&gt; hash value: c9e1c6a7b5e2e3e9b8a7c2e6e1a5e8a7b8e1a5c0e6e1a5e8a7b8e1a5c0e6e1a5e8a7b8e1</pre><h3>Encryption –</h3><p>Encryption secures data by encoding it with sophisticated algorithms, rendering it unreadable without authorized decryption keys. While ideal for securing data in transit and storage, encryption may not align with datasets intended for public or external sharing.</p><pre>Original (Customer Id): 234568 -&gt; Encrypted value: VGVzdFN0cmluZw==</pre><h3>Generalization –</h3><p>This technique minimizes the specificity of data to reduce the risk of identification. For instance, sharing only the birth year rather than the complete date of birth is a form of generalization, often employed in demographic studies.</p><pre>Original: 1985-07-23 → Generalized: 1985</pre><h3>Suppression –</h3><p>Suppression eliminates sensitive information entirely from a dataset. While effective at maintaining privacy, the utility of the resulting data may be diminished due to the loss of critical details.</p><pre>Original: &quot;Her SSN number is 123-45-6789&quot; → Suppressed: &quot;Her SSN number is ***-**-****&quot;</pre><h3>Perturbation –</h3><p>Perturbation introduces noise or intentional modifications to data, creating uncertainty about individual records. Techniques such as differential privacy are categorized under perturbation, offering mathematical assurances of privacy preservation.</p><pre>Original: &quot;Her age is 35&quot; -&gt; Suppressed: &quot;Her age is 36&quot;</pre><h3>Synthetic Data Generation –</h3><p>Creates artificial datasets that mimic the patterns and characteristics of real data but contain no actual personal information.</p><pre>Original: Her SSN number is 123-45-6789 -&gt; Synthetic: Her SSN number is 987-65-4321</pre><h3>Pseudonymization –</h3><p>Pseudonymization replaces identifiable data with unique pseudonyms or identifiers. Unlike complete anonymization, pseudonymized data can be reverted to its original form if the linking key is retained, making it highly suitable for controlled environments.</p><pre>Original: Her SSN number is 123-45-6789 -&gt; Pseudonymized: Her SSN number is TOKEN-ABCD-EFGH</pre><h3>Fabric Implementation: Privacy by design at scale</h3><p>To operationalize privacy in modern data ecosystems, the Fabric implementation brings together the power of Microsoft Fabric’s Lakehouse, Data Engineering and Data Factory with the precision of PySpark and Microsoft Presidio. This setup enables automated, scalable, and standardized PII protection across your data pipelines and AI workflows. Whether you’re masking emails, hashing IDs, texts, comment columns or generating synthetic personas, this stack ensures your data stays useful — without compromising privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*DzNvAr_GzY6ojk5c.png" /></figure><p>By integrating open-source tools like <strong>Presidio</strong> with <strong>PySpark</strong>, we can implement robust PII detection and anonymization strategies at scale that align with privacy-by-design principles.</p><p><strong>Platform</strong>Microsoft Fabric (Lakehouse + Data Engineering + Data Pipelines)<strong>Processing Engine</strong>PySpark<strong>PII Detection</strong>Microsoft Presidio<strong>Anonymization Techniques</strong>Masking, Hashing, Encryption<strong>Synthetic Data</strong> <strong>generation</strong>Faker libraryTech Stack Overview</p><p>While there are multiple approaches to implementing data privacy at scale — including third-party tools, built-in solutions like MIP labels and Microsoft Purview, or custom PySpark pipelines with AI functions — this blog focuses specifically on the following three approaches using PySpark as processing engine:</p><h3>1. Identify and anonymizing PII in Structured and Unstructured Data Using Presidio</h3><p><a href="https://microsoft.github.io/presidio/">Microsoft Presidio</a> is an open-source framework developed by Microsoft for detecting and anonymizing sensitive data. Presidio <em>(Origin from Latin praesidium ‘protection, garrison’)</em> helps to ensure sensitive data is properly managed and governed. It provides fast <strong><em>identification</em></strong> and <strong><em>anonymization</em></strong> modules for private entities in text and images such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more. It supports both structured and unstructured data, making it ideal for use cases like:</p><ul><li>Detecting and anonymizing PII in <strong>unstructured </strong>text, pdf, image files.</li><li>Detecting and anonymizing PII in free-text fields in <strong>structured </strong>data such as <strong>comments</strong>, <strong>feedback</strong>, or <strong>survey responses</strong>.</li><li>Identifying sensitive entities like names, emails, phone numbers, and credit card numbers using NLP and regex-based recognizers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*GbWZwjH7Zkmeo-Rn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*9q2z3_s6BH0OGSPM.png" /></figure><p><strong>Sample Use Case</strong>: Use Presidio in a PySpark UDF to scan customer profile stored in a file and flag PII entities.</p><h3>2. Generate Synthetic Data Using Faker for Anonymization</h3><p>Once PII is detected, you can replace it with <strong>synthetic but realistic data</strong> using libraries like Faker. This is especially useful for:</p><ul><li>Creating <strong>safe test datasets</strong>.</li><li>Preserving data utility while ensuring privacy.</li></ul><p><strong>Example</strong>: Replace detected names with fake names, emails with dummy addresses, and phone numbers with randomly generated ones.</p><p>This approach ensures that downstream analytics can continue without exposing real user data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*9C7q8I6YCPRfpw1d.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*gD-WfH3Kxv5d2ymW.png" /></figure><h3>3. Use Built-in PySpark Functions for Hashing and Masking</h3><p>For structured data like customer tables, PySpark provides native functions to anonymize data:</p><ul><li><strong>Masking</strong>: Use complete or partially hide/mask data (e.g., mask all but the last 4 digits of a phone number).</li><li><strong>Hashing</strong>: Uses PySpark’s built-in sha2 function to compute the SHA-256 hash (hexadecimal string) for each value in the given column. (e.g. Customer ID is hashed).Since SHA-256 is a deterministic hashing algorithm, it ensures consistent output for the same input, making it suitable for anonymizing columns that need to be joined across multiple tables.</li></ul><p>These techniques are efficient and scalable, making them ideal for large datasets processed in Fabric pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/0*nINcly_7LHRKG4l3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*WhVoeDw2_NaQzmIX.png" /></figure><h3>Implementation Guide</h3><p>The <a href="https://github.com/microsoft/fabric-samples/tree/main/community-samples/PII%20SparkShield">PII-SparkShield</a> repository contains sample implementations and code for the three approaches discussed in the blog.</p><h3>Conclusion</h3><p>Data anonymization plays a pivotal role in safeguarding privacy while enabling the productive use of data. Employing techniques such as masking, perturbation, and pseudonymization allows organizations to navigate the delicate balance between privacy preservation and data utility. However, challenges such as re-identification risks and compliance intricacies highlight the need for continuous innovation and vigilance. As privacy concerns grow globally, organizations must prioritize anonymization practices to ensure trust and compliance in the digital landscape.</p><h3>Acknowledgment</h3><p>My sincere appreciation goes to <strong>Omri Mendels </strong>— Thanks for helping me whenever needed. This would not have been possible without your help.</p><p>Sincere thanks to <strong>Abhishek Narain</strong>, <strong>Noelle Li</strong>, <strong>Santosh Kumar Ravindran</strong>, <strong>Ron Shakutai </strong>&amp; <strong>Gyani Sinha </strong>for their inputs, right from ideation to implementation.</p><p><em>Same Blog can be found on </em><a href="https://blog.fabric.microsoft.com/en-us/blog/privacy-by-design-pii-detection-and-anonymization-with-pyspark-on-microsoft-fabric"><em>Privacy by Design: PII Detection and Anonymization with PySpark on Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4e079fee989" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Database explained]]></title>
            <link>https://rkniyer999.medium.com/vector-database-explained-bfdb66d37944?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfdb66d37944</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[vector]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sat, 13 Sep 2025 17:23:13 GMT</pubDate>
            <atom:updated>2025-09-15T04:27:10.870Z</atom:updated>
            <content:encoded><![CDATA[<h4>Deep dive into Vector DB essentials</h4><h3>Vector Databases Demystified: The Backbone of Modern AI Applications</h3><h4>MAKE IT SIMPLE AI SERIES — UNDERSTANDING FUNDAMENTALS OF VECTORS, EMBEDDINGS &amp; DATABASES: A MODERN DATA REVOLUTION</h4><h3>❑ Overview</h3><p>In the AI era, traditional databases fall short in handling complex, high-dimensional data. <strong>Vector databases</strong> offer smarter, faster search by storing data as multi-dimensional vectors. In this blog, I will help you learn <strong>what vectors are</strong> and how <strong>embeddings</strong> transform raw data into searchable formats, explore how vector databases power <strong>semantic search</strong> <strong>&amp; compare vector vs. traditional databases</strong> to understand when each is best used.</p><p>Whether you’re building AI apps or exploring intelligent search, this guide is your starting point.</p><h3>❑ What is Vector?</h3><p>A vector is a mathematical object which has both magnitude and direction(dimension) and can be represented by an ordered list of numbers.</p><h3>Real-Life Example:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLj10Pu4vuKSR9vS7adNcQ.png" /><figcaption>Vector Real life examples</figcaption></figure><ol><li>Imagine a car moving at <strong>40 miles per hour toward the East</strong>.</li></ol><ul><li><strong>Magnitude</strong>: 40 miles per hour (this tells us <em>how fast</em> the car is moving)</li><li><strong>Direction</strong>: East (this tells us <em>where</em> the car is heading)</li></ul><p>So, the car’s <strong>velocity</strong> is a <strong>vector</strong> because it combines:</p><p>Velocity=40 mph East\text {Velocity} = 40 \text {mph East} Velocity=40 mph East</p><p><em>If you only said, “40 mph,” that would be a </em><strong><em>scalar</em></strong><em> (speed), because it lacks direction.</em></p><p><strong>2. Airplane in Wind (Multi-Dimensional) — </strong>Because the airplane’s motion is influenced by two components (North and East), forming a <strong>vector in 2D space</strong>.</p><p>An airplane is flying with:</p><ul><li><strong>Air Speed</strong>: 500 mph toward the North</li><li><strong>Wind Speed</strong>: 100 mph toward the East</li></ul><h3>❑ What is a high-dimensional vector?</h3><p>Think of a vector as an arrow that points somewhere. In math and AI, this arrow doesn’t just point in one direction — it can have <strong>many directions at once</strong>, each representing a feature.</p><p>For example:</p><ul><li>A <strong>simple vector</strong> for a dog picture might only include <strong>color</strong> and <strong>type</strong>.</li><li>A <strong>high-dimensional vector</strong> could include <strong>hundreds or thousands of features</strong>: ear shape, eye color, fur texture, background details, and more.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dngEZhYJGuNIHRVLS_5Mw.png" /><figcaption>High dimensional vector of a dog picture</figcaption></figure><p>So, a <strong>high-dimensional vector</strong> is like a <strong>super-detailed description packed into one long arrow</strong>.</p><p>[Ear Shape=0.8, Eye Color=0.3, Fur Texture=0.9, Tail Length=0.6, Background Color=0.2, …]</p><blockquote>Question — So now how is vector related to AI &amp; machine learning?</blockquote><blockquote>These vectors help machine learning models understand complex things and the relationships between them.</blockquote><h3>❑ Why Vector? Why has it become more relevant in Generative AI?</h3><p>So, coming back to <strong>WHY</strong> Vectors are Famous in Generative AI?</p><ol><li><strong>Efficient Data Representation</strong> — Vectors have become increasingly relevant in generative AI because they allow complex, unstructured data — like text, images, and audio — to be <strong>converted into high-dimensional numerical arrays that capture semantic meaning, relationships, and context.</strong> This representation enables</li></ol><ul><li><strong><em>Semantic Representation</em></strong> — Vectors (embeddings) encode the essence and “meaning” of data, allowing AI to understand concepts and relationships. For instance, the vector for “king” is close to “queen” in vector space, meaning they are related semantically</li><li><strong>Efficient Similarity Search</strong>: Generative models need to find related content quickly. Vectors make it possible to compare vast sets of data using mathematical metrics (like cosine similarity) to retrieve the most relevant information.</li><li><strong>Data Manipulation &amp; Latent Space Navigation</strong>: Vectors allow AI models to interpolate and manipulate data by simple arithmetic. For example, calculating vector(“king”) — vector(“man”) + vector(“woman”) produces a vector close to “queen”. This is critical for creative generation, style transfer, and controlled output.</li></ul><p>2. <strong>Optimized for Modern Hardware</strong> —</p><p><em>Vectors are ideal data structures for machine learning, as modern CPUs and GPUs are optimized for the mathematical operations they require.</em></p><ul><li>Matrix multiplications, dot products, and cosine similarity are core to neural network operations.</li><li>GPUs excel at parallelizing these operations across large batches of vectors, enabling real-time inference and training.</li></ul><h3>❑ What is embedding?</h3><blockquote>Embedding is numeric representation of objects within highly dimensional spaces. Example of an embedding model is the popular <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a>, which is regularly used for a wide variety of text-based tasks. Also look at TensorFlow’s <a href="https://projector.tensorflow.org/">projector</a> tool, which makes it easy to visualize embeddings.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIXWSOxR4phTsfH1flg33w.png" /><figcaption>Raw data to vector embedding conversion by embedding model</figcaption></figure><p>Consider a simplified two-dimensional vector space where words are represented as points. For instance, the point (2, 3) represents the word <em>joy</em> while the point (3, 4) represents <em>happy</em>. Because these words have similar meanings, their vectors are located close to each other in the vector space. On the other hand, the point (-5, -8) represents the word <em>sad</em>. Since it has the opposite meaning of <em>joy</em>, and their vectors are situated far apart from each other.</p><p>➟<strong>Types of Embedding Models</strong></p><p>Below are types on embedding model available -</p><p><strong>1. Text Embedding Models —</strong></p><p>These models convert textual data into vector representations.</p><p>OpenAI</p><ul><li>text-embedding-ada-002: 1536 dimensions, widely used for semantic search and RAG.</li><li>text-embedding-3-small and text-embedding-3-large: Faster and cheaper successors to ada-002.</li><li>Cohere</li></ul><p><strong>Cohere</strong></p><ul><li>embed-english-v3.0, embed-multilingual-v3.0: Known for normalized vectors and multilingual support.</li></ul><p><strong>Google</strong></p><ul><li>textembedding-gecko, universal-sentence-encoder: Versatile models supporting various similarity metrics.</li></ul><h4><strong>2. </strong>Image and Document Embeddings</h4><ul><li><strong>CNN-based models</strong>: Used for image embeddings.</li><li><strong>Doc2Vec</strong>: Extends Word2Vec to entire documents.</li></ul><h3>❑ Vector database (Vector DB)</h3><p><strong>Vector databases</strong> is a <strong>specialized database</strong> designed to <strong>handle highly dimensional vectors</strong> primarily generated from embeddings of complex data type such as <strong>text, images, audio</strong>. It provides capabilities to store and index unstructured data and enhance searches, as well as retrieval capabilities as a service.</p><p>Below are some of the <strong>key factors/characteristics</strong> of a vector Database -</p><ol><li><strong>Indexing </strong>— Indexing refers to the process of organizing highly dimensional vectors in a way that allows for efficient similarity searches and retrievals.</li></ol><ul><li><strong>Flat </strong>— index structure that stores vectors in a flat, unstructured format. Flat index doesn’t have any precomputed structures, which means each query needs to go through every single vector in the database to find its nearest neighbors.</li><li><strong>IVF — stands</strong> for Inverted File. In an IVF system, the dataset is first partitioned into several clusters. For each cluster, an inverted file (a list) is created that keeps track of the vectors that belong to this cluster. The advantage is during a search; you only need to search within the relevant clusters instead of the entire dataset. IVF index can significantly reduce the computational cost and improve search efficiency.</li><li><strong>HNSW (Hierarchical Navigable Small World) — </strong>Builds a graph where each node (vector) connects to its neighbors. Search starts from a high-level graph and navigates down. Very fast and accurate; excellent for high-dimensional data. Memory-intensive; slower to build.</li><li><strong>Product Quantization (PQ) — </strong>Compresses vectors by splitting them into sub-vectors and quantizing each part separately. Great for reducing memory usage &amp; enables fast approximate search. It has lower accuracy compared to HNSW.</li></ul><p>2.<strong> Search &amp; retrieval</strong> — Instead of relying on exact matches as traditional DB’s, Vector DBs provide vector search capabilities to quickly find vectors that are roughly closest to a give input. Vector DB uses <strong>similarity measures</strong> which are the foundation of how a vector database compares and identifies the most relevant results for a given query.</p><p>Here are the most common ones:</p><ul><li><strong>Cosine Similarity</strong><br> Measures the angle between two vectors. <strong>1</strong> = identical ,<strong>0</strong> = completely unrelated, <strong>-1</strong> = opposite direction</li><li><strong>Euclidean Distance</strong><br> Measures the straight-line distance between vectors. <strong>0</strong> = identical, larger values = more different</li><li><strong>Dot Product</strong><br> Combines vector magnitudes and their directional alignment. <strong>Positive</strong> = same direction, <strong>0</strong> = no relation, <strong>Negative</strong> = opposite direction</li></ul><p>3. <strong>Create, Read, Update &amp; Delete (CRUD)</strong> — A vector DB manages highly dimensional vectors and offers create, read, update &amp; delete operations to vectorized data.</p><p>4. <strong>Easy Integration -</strong> Vector DB’s provide seamless integration to LLM orchestration frameworks and SDK such as Langchain and Semantic Kernel.</p><h4>Vector DB Limitations</h4><ol><li><strong>Accuracy vs Speed Trade-off</strong> — While dealing with high dimensional data, vector DB’s face a tradeoff between speed and accuracy for similarity searches.</li><li><strong>Embedding Quality</strong> — The effectiveness of vector db is dependent on quality of the vector embedding used. Poorly designed embeddings can lead to inaccurate search results or missed connections.</li></ol><h3>❑ Vector Database Query Process Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Blp5fEhUhrNZFTEmFB83A.png" /><figcaption>Ingestion &amp; Retrieval Process flow in a Vector DB</figcaption></figure><p>Based on the diagram, here’s a clear <strong>process flow</strong> for how a vector database ingests raw data (Step 1–4) &amp; retrieve results from a query (Step 5–8) using vector embeddings from a vector database:</p><ol><li><strong>Raw Data/Content — </strong>Original data (e.g., text, image, audio) that needs to be stored.</li><li><strong>Embedding Model — </strong>A machine learning model (e.g. OpenAI Embeddings) converts raw data into high-dimensional vector representations.</li><li><strong>Vector Embedding — </strong>The output of the embedding model: a numerical vector (e.g., [0.45, -1.7, 1.33, …, -0.03, 1.42]) representing the semantic meaning of the data.</li><li><strong>Vector Database — </strong>Stores all vector embeddings and supports efficient similarity search using indexing algorithms (e.g., HNSW, IVF).</li><li><strong>Query — </strong>A user or system submits a query (e.g., a sentence or image) to find similar items.</li><li><strong>Query Gets Embedded — </strong>The query is passed through the same embedding model to generate its vector representation.</li><li><strong>Embedded Query Gets Fired in Vector DB — </strong>The query vector is used to search the vector database for nearest neighbors (similar vectors).</li><li><strong>Query Result Returned — </strong>The database returns the most similar items based on vector proximity (e.g., cosine similarity or Euclidean distance).</li></ol><h3>❑ Vector database VS Traditional Database</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNRP16AaLCL7jBKf_HsbMA.png" /><figcaption>Vector Databases VS Traditional Databases</figcaption></figure><h3>➟ Purpose &amp; Use Case</h3><ul><li><strong>Vector Databases</strong> are designed for <strong>similarity search</strong> and <strong>AI-driven applications</strong>. They store high-dimensional data like embeddings from text, images, or audio, making them ideal for <strong>semantic search</strong>, <strong>recommendation systems</strong>, and <strong>machine learning</strong>.</li><li><strong>Traditional Databases</strong> (e.g., MySQL, PostgreSQL) are optimized for <strong>structured data</strong>, supporting <strong>transactional systems</strong>, <strong>reporting</strong>, and <strong>analytics</strong> with well-defined schemas and SQL queries.</li></ul><h3>➟ Data Type &amp; Structure</h3><ul><li><strong>Vector DBs</strong> store <strong>unstructured or semi-structured data</strong> as vectors in high-dimensional space.</li><li><strong>Traditional DBs</strong> store <strong>structured data</strong> in tables with rows and columns, using relational models.</li></ul><h3>➟ Query Mechanism</h3><ul><li><strong>Vector DBs</strong> use <strong>nearest neighbor algorithms</strong> (e.g., k-NN, ANN, DiskANN) to find similar vectors.</li><li><strong>Traditional DBs</strong> use <strong>SQL or NoSQL</strong> queries for exact matches and predefined conditions.</li></ul><h3>➟ Performance &amp; Scalability</h3><ul><li><strong>Vector DBs</strong> are optimized for <strong>fast similarity searches</strong> and scale horizontally across distributed systems.</li><li><strong>Traditional DBs</strong> excel in <strong>transactional consistency</strong> and complex relational queries, with scalability depending on architecture.</li></ul><h3>➟ Examples</h3><ul><li><strong>Vector DBs</strong>: Milvus, Pinecone, Weaviate, FAISS (Facebook AI Similarity Search), Azure Cosmos DB (with vector indexing)</li><li><strong>Traditional DBs</strong>: SQL Server, MySQL, PostgreSQL, Oracle, MongoDB</li></ul><h3>➟ When to Use What</h3><ul><li><strong>Use Case</strong> — Semantic search, AI/ML, image/text/audio similarity</li><li><strong>Best Fit </strong>— Vector Database</li><li><strong>Use Case</strong> — Financial records, inventory, structured reporting</li><li><strong>Best Fit</strong> — Traditional Database</li></ul><h3>❑ References</h3><p><a href="https://learn.microsoft.com/en-us/data-engineering/playbook/solutions/vector-database/">Understanding Vector Databases | Microsoft Learn</a></p><p><a href="https://www.pinecone.io/learn/vector-database/">What is a Vector Database &amp; How Does it Work? Use Cases + Examples | Pinecone</a></p><p><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/vector-search">Integrated vector store — Azure Cosmos DB for NoSQL | Microsoft Learn</a></p><p><a href="https://www.dtiq.com/blog/ai/whats-my-vector-victor-understanding-vectors-in-generative-ai">“What’s My Vector, Victor?” Understanding Vectors in Generative AI — DTiQ</a></p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through the basic of Vector databases. We will surely explore and continue with advance topics of AI in upcoming blogs, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfdb66d37944" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Engineering explained — From basic to advanced techniques]]></title>
            <link>https://rkniyer999.medium.com/prompt-engineering-explained-from-basic-to-advanced-techniques-2995198eebb6?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2995198eebb6</guid>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[prompt-techniques]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sun, 01 Jun 2025 16:02:07 GMT</pubDate>
            <atom:updated>2025-06-01T16:02:07.083Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unlock the full potential of AI by learning how to craft powerful prompts that get smarter, more accurate results.</h4><h3>Prompt Engineering explained — From basic to advanced techniques</h3><h4>MAKE IT SIMPLE AI SERIES — CRAFTING EFFECTIVE PROMPTS: KEY ELEMENTS AND BEST PRACTICES— PART1</h4><h3>❑ Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Ai-FHeDS0dCEl4nOEryjw.png" /><figcaption>Prompt Engineering</figcaption></figure><p>Prompt engineering has rapidly emerged as one of the most exciting frontiers in artificial intelligence. At its core, it’s the craft of designing and refining instructions — called prompts — to guide generative AI models like ChatGPT, DALL-E, and others to produce exactly the results you desire. With well-crafted prompts, you can steer AI responses, inject domain knowledge, and even ensure safer, more reliable outputs.</p><p>As AI continues to reshape industries, prompt engineering is becoming an essential skill for developers, researchers, and business professionals. In this blog series, I’ll take you on a journey from the basics of prompt design to advanced techniques — empowering you to harness the true potential of generative AI, no matter your starting point.</p><h3>❑ Real World Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P7jjZHbRBd0zzgH-AU1sug.png" /><figcaption>Coffee Shop</figcaption></figure><p><strong>But imagine</strong> if the <strong>barista misunderstood</strong> your answers or guessed what you wanted without asking — <strong>maybe handing you an iced caramel macchiato with extra syrup</strong>, even though you <strong>never mentioned caramel or cold drinks</strong>. This would be confusing and not what you wanted.</p><h4>Analogy to Prompt Engineering and Hallucination:</h4><p>If your instructions are vague or the LLM “guesses” details not provided (hallucination), you might get a response that sounds confident but is incorrect or unrelated — just like getting the wrong coffee order. That’s why asking clear, specific questions and verifying the answers is crucial when working with LLMs, to avoid “hallucinated” outputs.</p><blockquote>Prompt engineering with LLMs is like giving clear instructions to the barista so you get the coffee you want.</blockquote><h4>❑ What is a Prompt?</h4><p><em>A prompt is a </em><strong><em>piece of natural language text</em></strong><em> — like a question, instruction, or statement — that you give to a </em><strong><em>large language model (LLM)</em></strong><em> to </em><strong><em>guide it in generating a response</em></strong><em>.</em></p><p>For text-to-text language models, a prompt can be a <strong>question</strong>,<strong> a command, or a longer statement that includes context or instructions.</strong></p><pre><strong>Translate this sentence to French</strong>: The weather is nice today.</pre><p><strong>For</strong> <strong>text-to-image or text-to-audio</strong> models, prompts are typically <strong>descriptions of the desired output. e.g.</strong></p><pre><strong>Create </strong>a <strong>landscape image</strong> of <strong>rolling green hills under a bright blue sky</strong>.</pre><pre><strong>Generate </strong>the <strong>sound of gentle rain falling on leaves in a quiet forest</strong>.</pre><p>Prompts can include <strong>four parts</strong>: the goal, context, expectations, and source, as described in the following image:</p><h4>❑ What is a Prompt Engineering?</h4><p><a href="https://en.wikipedia.org/wiki/Prompt_engineering">Prompt engineering</a> is the <strong>process of structuring</strong> or <strong>crafting an instruction</strong> in order <strong>to produce the best possible output</strong> from a <strong>generative artificial intelligence. Source Wikipedia</strong>.</p><p><strong>According to Google Cloud</strong>, prompt engineering is “The art and science of designing and optimizing prompts to guide AI models, particularly LLMs, towards generating the desired responses”.</p><h4>Parts of a Prompt</h4><ul><li><strong>Instruction</strong>: Tells the AI exactly what you want it to do.</li><li><strong>Context</strong>: Gives extra background or details to help the AI give a better answer.</li><li><strong>Input Data</strong>: The main question or information you want help with.</li><li><strong>Output Indicator</strong>: Explains what kind of answer or format you expect from the AI.</li></ul><p>These <strong>parts work together</strong> to help you get the best results from your prompts.</p><h4>❑ Prompt Engineering Best Practices</h4><p>Below are some of the best practices for writing a prompt —</p><ol><li><strong>Be Clear and Specific</strong>: Ensure that your prompt clearly states what you want the model to do. Avoid vague or ambiguous language.</li></ol><h4>Vague Prompt:</h4><pre>Write about technology.</pre><p><strong><em>Why it’s unclear:</em> </strong>It doesn’t specify what <strong>aspect of technology</strong>, <strong>the style</strong>, <strong>length</strong>, or <strong>purpose</strong>.</p><h4>Clear and Specific Prompt:</h4><pre>Write a 200-word article explaining how artificial intelligence is transforming healthcare, focusing on patient diagnosis improvements.</pre><p><strong>Why it’s clear:</strong> It specifies the <strong>topic (AI in healthcare)</strong>, the <strong>focus (patient diagnosis)</strong>, and the <strong>length (200 words)</strong>.</p><p>2.<strong> Provide Context</strong>: Give the model enough background information to understand the task. This can include relevant details, examples, or constraints.</p><h4>Without Context:</h4><pre>Summarize this article.</pre><p><strong><em>Why it’s lacking</em></strong><em>:</em> The model doesn’t know what the article is about or what kind of summary is needed.</p><h4>With Context:</h4><pre>Summarize the following article about climate change, focusing on the main causes and effects discussed. The article is intended for high school students.</pre><p><strong><em>Why it’s better:</em></strong> It gives background (topic: climate change), specifies what to focus on (causes and effects), and mentions the audience (high school students), helping the model generate a more relevant summary.</p><p><strong>3. Use Structured Inputs</strong>: When possible, structure your prompt in a way that makes it easy for the model to follow. This can include using bullet points, numbered lists, or specific formatting.</p><p><strong>4. Iterate and Refine</strong>: Prompt engineering is an iterative process. Experiment with different prompts, analyze the results, and refine your prompts based on the model’s performance.</p><p><strong>5. Test for Bias and Fairness</strong>: Ensure that your prompts do not introduce or reinforce biases. Test your prompts with diverse inputs to check for fairness and inclusivity.</p><h4>Prompt Example:</h4><pre>Describe a typical software engineer.</pre><p>This prompt <strong>might lead the AI</strong> to <strong>generate stereotypical or biased descriptions (e.g., assuming a certain gender, age, or background).</strong></p><p><strong>How to Test for Bias and Fairness:</strong><br>Try running the prompt with variations or explicitly ask:</p><pre>Describe a typical software engineer. Make sure your answer is inclusive of all genders, backgrounds, and cultures.</pre><p><strong>What to Look For:<br></strong>Check if the AI’s responses avoid stereotypes and represent diversity, ensuring the prompt and output are fair and inclusive.</p><p><strong>6. Use Examples</strong>: Providing examples of the desired output can help guide the model towards producing similar results.</p><p><strong>7. Review and Adjust</strong>: After receiving the model’s response, review it to see if it meets your needs. If not, adjust your prompt and try again.</p><h3>❑ Reference</h3><p><a href="https://cloud.google.com/discover/what-is-prompt-engineering?hl=en#what-is-prompt-engineering">Prompt Engineering for AI Guide | Google Cloud</a></p><p><a href="https://www.promptingguide.ai/">Prompt Engineering Guide | Prompt Engineering Guide</a></p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through the basic of Prompt engineering. We will surely explore and continue with advance topics of prompt engineering in upcoming blogs, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2995198eebb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Agents explained]]></title>
            <link>https://rkniyer999.medium.com/ai-agents-explained-1d7e692055de?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d7e692055de</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Sun, 25 May 2025 18:05:42 GMT</pubDate>
            <atom:updated>2025-05-26T01:22:27.631Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are Agents? All you need to know about AI agents</h4><h4>MAKE IT SIMPLE AI SERIES — UNDERSTANDING AI AGENTS</h4><h3>❑ Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cq5NPNhBLMPTLJ6CjBGdJA.png" /><figcaption>Gen AI Evolution from LLM to RAG to Agents</figcaption></figure><p><em>“Explain me AI Agents like a 5-year-old kid”</em></p><p>Ever been asked that? If not, consider yourself lucky 😊. If yes, then hopefully this blog will help you answer it with confidence.</p><p>I’ve always believed that many computer science concepts are inspired by real life — just like object-oriented principles such as inheritance and abstraction. Similarly, the idea of agents in AI is also rooted in human analogies. In this blog, I’ll Walk you through a relatable example from everyday life to make the concept easier to grasp.</p><p>In this blog, we will cover</p><ul><li>What are AI Agents?</li><li>What are characteristics of AI Agents?</li><li>What are the different types of agents?</li><li>What kinds of use cases are best suited for AI Agents?</li><li>What are unresolved challenges in AI Agents?</li></ul><h3>❑ Real World Agent Example</h3><p>To make the concept of AI agents easier to understand, think of the experience of renting a house with the help of a real estate agent. Each step in that journey reflects how a Real estate agent works — interacting with you, gathering your requirements, reasoning through options, and ultimately helping you complete a task, just like finding the right home.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ZxTkw1Y1nQdRBd5lT2uBg.png" /></figure><p>Below is a step-by-step breakdown of the agent interaction flow described above:</p><ol><li>I <strong>visit a real estate agent</strong> &amp; share my house requirements.</li><li>Agent <strong>asks for my requirements</strong>.</li><li>Based on my requirement, Agent <strong>searches different options</strong> for me.</li><li>Agent <strong>suggests me 3 options.</strong></li><li>I <strong>select couple of options</strong> and <strong>reject </strong>few options.</li><li>Agent <strong>reasons why </strong>I rejected an option which makes me think of reconsidering an option since I had missed on some benefits of provided options. Now the estate agent provides me 3 refined options</li><li>I visit <strong>3 option</strong> and <strong>selected 1 option</strong></li><li>Agent helps in <strong>finalizing the negotiations</strong> with the Owner of the flat.</li><li>Estate Agent initiates the contract process and helps me sign the contract and <strong>complete the house hunting task</strong>.</li></ol><p>Now consider we <strong>replace real estate Agent</strong> with an <strong>AI agent</strong>. Below is the step-by-step analogy for the same.</p><h3>❑ Step-by-Step Analogy: Estate Agent vs. AI Agent</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wH1yoqEo5Eb5PrP7z_WePg.png" /><figcaption>Step-by-Step Analogy: Estate Agent vs. AI Agent</figcaption></figure><p>Now that we have understood how an AI agent can replace a real estate agent, let’s understand what is AI Agent?</p><h3>❑ What is an Agent?</h3><p>An AI agent is an autonomous system that observes its environment, makes decisions, and takes actions using available tools to achieve a specific goal.</p><blockquote>An AI agent isn’t just a simple chatbot. Think of it as a <strong>digital assistant with a mission</strong>: it can set goals, carry out <strong>tasks </strong>on its own, and <strong>learn from what happens</strong> along the way. It remembers what you tell it, <strong>makes smart decisions</strong>, <strong>reasons through problems</strong>, and <strong>can even adapt to new situations</strong> as they come up. In short, it’s a software helper that’s smart and independent enough to sometimes surprise you with how useful it can be.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2aHk2M4rkzH5nnEb4mRlg.png" /><figcaption>AI Agent Characteristics</figcaption></figure><p>Key characteristics of AI agents are as follows:</p><ul><li><strong>Autonomy with Flexibility</strong>: An agent can <strong>operate independently</strong>, <strong>performing tasks/action</strong> <strong>without </strong>needing constant <strong>human oversight</strong>. However, it <strong>can also include a human in the loop</strong> when oversight or control is necessary.</li><li><strong>Memory and Knowledge</strong>: Agents are equipped with memory, enabling them to store user preferences and past interactions.<br>This memory can be <strong>short-term</strong> — used within the context of an ongoing conversation — or <strong>long-term</strong>, allowing agents to retrieve knowledge beyond the immediate environment. Long-term memory can include access to external systems, services, tools, and even other agents.</li><li><strong>Cognitive Capabilities</strong>: Agents can leverage large language models (LLMs), agents can process information, reason through options, and make decisions based on context and goals.</li><li><strong>Environmental Awareness</strong>: Agents can perceive and interpret data from their environment, enabling them to respond intelligently to changing conditions or new inputs.</li><li><strong>Tool Integration</strong>: Agents can extend their capabilities by using external tools — such as browsing the internet or making API calls — to complete complex tasks.</li><li><strong>Collaboration</strong>: Agents are not isolated — they can work alongside other agents or humans, sharing information and coordinating actions to achieve shared objectives.</li></ul><blockquote>To relate it to our real estate analogy: imagine if the estate agent, after understanding your preferences, could independently search listings, schedule viewings, negotiate terms, and even prepare contracts — without checking back with you at every step. That’s autonomy.</blockquote><h3>❑ Traditional GenAI VS Agentic GenAI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*29GhzdObPFO-vu2a.png" /><figcaption>AI agents vs. traditional generative AI. Image credit: Maximilian Vogel</figcaption></figure><h3>❑ General Agent architecture and components</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*uDluU4uPBThkjh1t.png" /><figcaption>General Architecture of An AI Agent (Source: “Agents” by Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic)</figcaption></figure><ul><li><strong>Model </strong>— In the context of AI agents, the “model” refers to the language model (LM) that drives decision-making. It can be a single or multiple LMs — general-purpose, multimodal, or fine-tuned — capable of reasoning using frameworks like ReAct or Chain-of-Thought. While the model isn’t trained on the agent’s specific configuration, it can be adapted through examples that reflect tool use and reasoning. Choosing a model aligned with your application and tool ecosystem is key to effective agent performance.</li><li><strong>Tool </strong>— Foundational models are powerful but limited in interacting with the real world. Tools bridge this gap, enabling AI agents to access external data and perform actions like calling Apis. These tools can range from simple data fetches to complex updates, expanding the agent’s capabilities. With tools, agents can support advanced systems like RAG, going far beyond what models can do alone.</li><li><strong>The orchestration layer </strong>— The orchestration layer is a loop where the agent observes, reasons, and acts until it reaches a goal. Its complexity can vary — from simple rule-based decisions to advanced logic and learning techniques. This process governs how the agent adapts its behavior based on new information.</li></ul><h3>❑ What are the different types of agents?</h3><ul><li><strong>Simple Reflex Agents — </strong>Perform immediate actions based on predefined rules.</li><li><strong>Model-Based Reflex Agents </strong>— Perform actions based on a model of the world and changes to that model.</li><li><strong>Goal-Based Agents — </strong>Create plans to achieve specific goals by interpreting the goal and determining actions to reach it.</li><li><strong>Utility-Based Agents </strong>— Consider preferences and weigh tradeoffs numerically to determine how to achieve goals.</li><li><strong>Learning Agents </strong>— Improve over time by responding to feedback and adjusting actions accordingly.</li><li><strong>Hierarchical Agents</strong> — Feature multiple agents in a tiered system, with higher-level agents breaking tasks into subtasks for lower-level agents to complete.</li><li><strong>Multi-Agent Systems (MAS)</strong> — Agents complete tasks independently, either cooperatively or competitively.</li></ul><h3>❑ What kinds of use cases are best suited for AI Agents?</h3><p>AI agents are well-suited for tasks that <strong>involve automation, complex decision-making, and integration with other systems</strong>. They are especially effective in domains such as software development, customer support, process optimization, and roles that require data analysis and reporting.</p><h3>❑ What are Unresolved Challenges in AI Agents?</h3><p>AI agents hold immense promise across a wide range of applications. However, like any evolving technology, they come with their own set of limitations:</p><ul><li><strong>Context Limitations</strong>: AI agents often struggle with limited context windows, which can hinder their ability to plan tasks or reflect effectively. While techniques like vector-based memory retrieval help extend their reach, they don’t yet match the depth of full attention mechanisms.</li><li><strong>Difficulty with Long-Term Planning</strong>: Although agents can perform well on narrowly defined tasks, they still lag behind humans when it comes to breaking down complex goals into manageable steps and executing long-term strategies.</li><li><strong>Fragility of Natural Language Interfaces</strong>: Most AI agents rely on natural language to interact with external tools and systems. However, outputs from large language models can occasionally be unreliable — ranging from formatting issues to unexpected or inconsistent behavior.</li></ul><p>Despite these hurdles, the pace of innovation in AI is rapid. Many of these challenges are actively being addressed, and we can expect significant improvements in the near future.</p><h3>❑ Reference</h3><p>Agents Google paper by Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic — <a href="https://www.kaggle.com/whitepaper-agents">Agents | Kaggle</a></p><p><a href="https://medium.com/codex/what-are-ai-agents-your-step-by-step-guide-to-build-your-own-df54193e2de3">What Are AI Agents? A Short Intro And A Step-by-Step Guide to Build Your Own. | by Maximilian Vogel | CodeX | Medium</a></p><p><a href="https://www.gptbots.ai/blog/how-to-build-ai-services-with-agents">How to Build AI Services with Agents</a> by Jacob Morrow</p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through the basic understanding of AI Agents. We will surely explore many of these topics in detail, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d7e692055de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Fabric Unveiled: Key Highlights from Ignite 2024]]></title>
            <link>https://rkniyer999.medium.com/microsoft-fabric-unveiled-key-highlights-from-ignite-2024-1f92abe3dc43?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f92abe3dc43</guid>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Tue, 03 Dec 2024 00:40:16 GMT</pubDate>
            <atom:updated>2024-12-03T07:33:10.299Z</atom:updated>
            <content:encoded><![CDATA[<h4>Redefining the Data Game at Ignite 2024!</h4><h4>FABRIC SERIES 09: News— Powering Innovation: Must-Know Features of Microsoft Fabric from Ignite 2024</h4><h3>❑ Overview</h3><p>At <a href="https://ignite.microsoft.com/en-US/home">Microsoft Ignite</a> 2024 some truly game-changing features of Microsoft Fabric were announced, promising to redefine how businesses approach data and analytics. In this blog, I’ll Walk you through the key features unveiled, breaking down what they mean and why they matter for developers, data scientists, and organizations looking to stay ahead in the AI era. Let’s get started!</p><h4>Microsoft Fabric’s objectives</h4><p>Microsoft Fabric’s roadmap is centered on below 4 key core objectives, highlighting its pivotal role in empowering modern data-driven organizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQOQbb0Po2i-8Zq6oq4JHA.png" /><figcaption>Microsoft Fabric’s 4 key objectives</figcaption></figure><ol><li>An AI-powered data platform</li><li>An open and AI-ready data lake</li><li>AI-enabled business users</li><li>A mission-critical foundation</li></ol><p>I have tried to align the <strong>key announcements</strong> into these four primary objectives.</p><h3>❑ An AI-powered data platform</h3><p>A <strong>unified platform</strong> for all data needs, supporting transactional, analytical, real-time, and batch workloads. Fully integrated in experience and architecture, it is powered by AI for seamless performance.</p><h4>#1 — Autonomous Fabric Databases as a part of unified Fabric Data Platform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mu8q11YGLP_fwkDbrfxBlA.png" /><figcaption>Autonomous Fabric Databases as a part of unified Fabric Data Platform</figcaption></figure><p>As shown in the above diagram, data engineering, data science, and data warehousing workloads have been merged into analytics making way for a new workload: <strong>Fabric Databases</strong>. With Fabric Databases, Microsoft is bringing their entire database portfolio to Fabric, and starting with a flagship SQL Server product wherein you will get full world-class transactional SQL performance, all integrated into Microsoft Fabric. And just like Fabric, it’s all software-as-a-service, and all of the data is integrated into OneLake.</p><p><em>Let’s understand — What’s the thought process behind incorporating databases into Microsoft Fabric?</em> In today’s fast-evolving data landscape, the distinctions between transactional, NoSQL, document, vector, and in-memory databases are rapidly fading. Most AI projects now combine these database types to work together seamlessly. Fabric simplifies this process by unifying these capabilities, making it easier to build applications and transition smoothly into the AI era.</p><p>You can explore more at <a href="https://learn.microsoft.com/en-us/fabric/database/sql/overview">SQL database Overview (Preview) — Microsoft Fabric | Microsoft Learn</a></p><h4>#2— General availability of Fabric Real-Time Intelligence</h4><p>Real-Time Intelligence is now generally available. Real-Time Intelligence offers both pro-dev and no-code tools to handle high-volume streaming data with precision. It enables dynamic data transformation, real-time querying for immediate insights, and automated actions based on the data. The Real-Time Hub serves as a centralized platform to discover and manage all streaming data efficiently. Checkout <a href="https://blog.fabric.microsoft.com/en-us/blog/transform-your-business-with-real-time-intelligence-now-generally-available?ft=All">Transform your business with Real-Time Intelligence: now Generally Available | Microsoft Fabric Blog | Microsoft Fabric</a> for more details.</p><h4>#3- Industry Solutions &amp; ISV integration (Fabric Workload Development Kit)</h4><p>Microsoft Fabric continues to evolve as a comprehensive data analytics platform, incorporating industry solutions and fostering extensibility to address diverse business needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N4f8PKnw0A7PHXv7A7nm0A.png" /><figcaption>Industry Solutions and Partner workloads</figcaption></figure><h4>Industry Solutions for Rapid Value Creation</h4><p>Fabric now includes a range of industry-specific solutions, designed to accelerate the time to value for organizations across sectors like sustainability, healthcare, and retail. These built-in capabilities empower businesses to leverage tailored analytics and insights without starting from scratch, enabling faster decision-making and impactful outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UDOcEGEHv6hIeozYXoQ6gw.png" /><figcaption>Industry Solutions</figcaption></figure><p>Microsoft has announced the <strong>general availability of Sustainability Data Solutions </strong>within Microsoft Fabric. This innovative feature empowers businesses to effortlessly compute, analyze, and report their environmental, social, and governance (ESG) metrics with exceptional accuracy and efficiency. The Sustainability Data Solutions in Microsoft Fabric offer a robust suite of tools, including prebuilt and preconfigured resources such as data lakes, notebooks, and dashboards.</p><p>Refer <a href="https://learn.microsoft.com/en-us/industry/sustainability/sustainability-data-solutions-fabric/get-started-overview"><strong>sustainability data solutions in Microsoft Fabric</strong></a>, for more details.</p><h4>Extensibility with the Fabric Workload Development Kit (GA)</h4><p>The Fabric Workload Development Kit, announced earlier this year, is now <strong>generally available</strong>. This powerful tool makes Fabric extensible, allowing independent software vendors (ISVs) to integrate and bring their own workloads seamlessly into the platform. These integrations are deeply embedded, ensuring data resides in OneLake, artifacts are housed within the same workspace, and permissions models remain consistent.</p><p>A diverse array of ISV solutions is already available in public preview, with more innovations expected to roll out in the coming months. These solutions exemplify the collaborative ecosystem Fabric fosters, enabling businesses to tailor the platform to their unique requirements. Refer <a href="https://blog.fabric.microsoft.com/en/blog/announcing-general-availability-of-the-fabric-workload-development-kit?ft=All">Announcing general availability of the Fabric Workload Development Kit | Microsoft Fabric Blog | Microsoft Fabric</a> for more details.</p><h3>❑ An open and AI-ready data lake</h3><p>Fabric enables seamless access to your entire multi-cloud data estate through a unified data lake, allows analytics engines to operate on a single data copy, and ensures your data is optimized for AI-driven innovation.</p><h4>#4- Mirroring in Fabric</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIiF65KJF-Bye_WupHKOHg.png" /><figcaption>Currently available mirroring sources in Microsoft Fabric</figcaption></figure><p><strong>Mirroring </strong>in Fabric is a powerful feature that allows you to replicate your Database to Fabric’s OneLake. This ensures that your data is always up-to-date and readily available for advanced analytics, AI, and data science without the need for complex ETL processes.</p><p>Snowflake mirroring was GAed recently and mirroring for Azure SQL DB is now generally available (GA). Refer <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-mirroring-azure-sql-database-in-fabric-now-generally-available-ga?ft=All">Announcing the general availability (GA) of Mirroring for Azure SQL Database in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric</a> for more details.</p><h4><strong>#5 Open Mirroring in Fabric (Public Preview) —</strong></h4><p><strong>Open Mirroring in Fabric </strong>capability extends mirroring where in any applications and data ISVs can write change data directly into a Mirrored Database in Microsoft Fabric based on the <strong>open mirroring public Apis</strong> and <strong>approach</strong>. Open mirroring is <strong>designed to be extensible, customizable, and open</strong>. For more details refer <a href="https://learn.microsoft.com/en-us/fabric/database/mirrored-database/open-mirroring">Open Mirroring (Preview) — Microsoft Fabric | Microsoft Learn</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgRNhc43RiVgSqrdAxJZCQ.png" /><figcaption>Open Mirroring</figcaption></figure><p>Open mirroring meets your data replication needs if you:</p><ul><li>Use your own application to write data into the open mirroring landing zone per the <a href="https://learn.microsoft.com/en-us/fabric/database/mirrored-database/open-mirroring-landing-zone-format">open mirroring landing zone requirements and formats</a>.</li><li>Use one of Microsoft existing <a href="https://learn.microsoft.com/en-us/fabric/database/mirrored-database/open-mirroring-partners-ecosystem">open mirroring partners</a> to help you ingest data.</li></ul><h4>#6 New OneLake catalog offers unified solution for Microsoft Fabric data</h4><p>The <strong>new OneLake catalog </strong>which serves as a unified solution to explore, manage and govern the entire Microsoft Fabric data estate, including structured and unstructured data, lakehouses, warehouses, machine learning models, notebooks, reports and more is now <strong>generally available</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*zVGtRLHb0szYP1Q5dBivwA.png" /><figcaption>OneLake catalog</figcaption></figure><p>It features an items list with an in-context item details view that makes it possible to browse through and explore items without losing your list context. The catalog also features selectors and filters to narrow down and focus the list, making it easier to find what you need. Refer <a href="https://learn.microsoft.com/en-us/fabric/governance/onelake-catalog">Discover and explore Fabric items in the OneLake catalog — Microsoft Fabric | Microsoft Learn</a> for more details.</p><h3>❑ AI-enabled business users</h3><p>Fabric can empower business users to better understand your data with AI-enhanced Q&amp;A experiences and visuals embedded in the Microsoft 365 apps they use every day.</p><h4>#7 AI Skill (Preview)</h4><p>AI skill is a tool that allows you to configure a <strong>generative AI assistant</strong> that provides insights into your data by conversational Q&amp;A experience. AI Skills allows Fabric developers to create their own GenAI experience using data in the lakehouse and now in warehouse.</p><p><a href="https://support.fabric.microsoft.com/en-us/blog/data-warehouse-copilot-ai-skill?ft=Microsoft-fabric:category">Data Warehouse: Copilot &amp; AI Skill | Microsoft Fabric Blog | Microsoft Fabric</a></p><p>AI skills are now integrated with the Azure AI Agent Service, enabling developers to create AI applications that utilize Fabric data as a core knowledge source. This includes low-code chatbots and custom generative AI applications built in Azure AI Studio. Refer <a href="https://techcommunity.microsoft.com/blog/azure-ai-services-blog/ground-your-ai-agents-with-knowledge-from-bing-search-microsoft-fabric-sharepoin/4303634">Ground your AI agents with knowledge from Bing Search, Microsoft Fabric, SharePoint and more | Microsoft Community Hub</a> for more details.</p><h3>❑ A mission-critical foundation</h3><p>You can confidently deploy and manage Fabric with category-leading performance, instant scalability, shared resilience, and built-in security, governance, and compliance.</p><h4><strong>#8 Security &amp; Governance</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzGr4eg6LGR31RPYHO6WCw.png" /><figcaption>Microsoft Fabric Certifications</figcaption></figure><p>When building solutions, ensuring they are developed on a certified platform is crucial. Microsoft Fabric has now achieved its final major certification — FedRAMP certification. This certification is essential for organizations working with the U.S. federal government, demonstrating Fabric’s compliance with stringent security and regulatory standards. With this milestone, Fabric solidifies its position as a trusted platform for secure, compliant data solutions.</p><h4>#9 <strong>Surge Protection</strong></h4><p>Now you can protect your capacities from unexpected surges in background workload consumption with new controls for capacity administrators. These controls allow admins to set thresholds or limits on background activity consumption, ensuring background jobs stop running once the limit is reached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ux1QoCpVRClk8g6zoaXXKQ.png" /><figcaption>Surge Protection</figcaption></figure><p>Capacity Admins can configure separate limits for each capacity in your organization for added flexibility. For e.g. If an organization is running a Dev, Test &amp; mission critical production workloads, you can deprioritize the Dev &amp; test workloads to ensure that mission critical continues to run in major loads in your capacity.</p><h4>#10 <strong>Workspace Monitoring (Preview)</strong></h4><p>Workspace monitoring in Microsoft Fabric equips developers and administrators with detailed logs and performance metrics, enabling efficient management and optimization of their workspaces. This feature supports troubleshooting performance issues, investigating errors, and optimizing queries, ultimately reducing data downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*jrT6HBSg_k2UPfuHBmeUog.png" /><figcaption>Workspace Monitoring</figcaption></figure><p>By providing comprehensive visibility into system operations, workspace monitoring plays a vital role in observing system health, optimizing resources, and facilitating operational planning. The actionable insights gained can enhance performance tuning, improve capacity planning, and inform resource allocation and decision-making. Additionally, the detailed logs aid in compliance and auditing, ensuring robust operational governance. For more details, refer <a href="https://blog.fabric.microsoft.com/en-us/blog/announcing-public-preview-of-workspace-monitoring?ft=All">Announcing preview of Workspace Monitoring | Microsoft Fabric Blog | Microsoft Fabric</a></p><h4><strong>#11 External data sharing (GA)</strong></h4><p>The external data sharing feature announced earlier this year is now generally available. The general availability of <a href="https://aka.ms/External-data-Sharing-GA"><strong>external data sharing</strong></a> allows you to directly share OneLake tables and folders with other Fabric tenants in an easy, quick, and secure manner. In the current release, each share may include a single folder or a table from a Lakehouse. In the coming releases, support will be added for multiple folders and tables in a single share as well as sharing from Warehouses and Eventhouses.</p><h4>#12 Microsoft Purview Integration</h4><p>Purview Data Loss Prevention (DLP) policies in Microsoft Fabric automatically detect sensitive data in lakehouses and semantic models, protecting against inappropriate sharing and ensuring compliance with regulations like GDPR and HIPAA. With the new restrict access action, compliance admins can configure policies to limit sensitive data access to authorized users or data owners. These enhancements complement existing DLP features, including auditing, alerts, and custom policy tips within the Fabric platform. Fore more details, refer <a href="https://aka.ms/Purview-DLP-Ignite-24"><strong>integration with Microsoft Purview</strong></a><strong>.</strong></p><h4>#13 Tenant switcher control</h4><p>The tenant switcher is now available in the Fabric portal. Users with access to more than one Fabric tenant can easily switch between tenants directly from the account manager in the top right corner of the Fabric portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*Sdv8dYMW2_m2JRiO.png" /></figure><h3>❑ Reference</h3><ul><li>“Microsoft Fabric: what’s new and what’s next” presented by Arun Ulagaratchagan &amp; Amir Netz at Microsoft Ignite 2024</li><li><a href="https://www.microsoft.com/en-us/microsoft-fabric/blog/2024/11/19/accelerate-app-innovation-with-an-ai-powered-data-platform/">Accelerate app innovation with an AI-powered data platform | Microsoft Fabric Blog</a></li><li><a href="https://news.microsoft.com/ignite-2024-book-of-news/">Microsoft Ignite 2024 Book of News</a></li></ul><h3>❑ Thank You</h3><p>This blog mirrors the structure and key content of a session “Microsoft Fabric: what’s new and what’s next” presented by<strong> Arun Ulagaratchagan &amp; Amir Netz </strong>at Microsoft Ignite 2024, where the latest features and updates for Microsoft Fabric were discussed in detail. So a big thankyou to Arun Ulagaratchagan &amp; Amir Netz and all the folks who have contributed to this session.</p><h3>❑ Conclusion</h3><p>I hope this blog helped you to go through the major Microsoft Fabric announcements at Microsoft Ignite 2024. We will surely explore many of these in detail, Till then, Happy Learning!!!</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f92abe3dc43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your Data Pipeline Deployments Across Tenants]]></title>
            <link>https://medium.com/microsoft/streamlining-fabric-data-pipeline-transfers-automated-deployment-across-tenants-d3e64acae8d6?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3e64acae8d6</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Fri, 31 May 2024 15:43:08 GMT</pubDate>
            <atom:updated>2024-06-09T13:06:25.566Z</atom:updated>
            <content:encoded><![CDATA[<h4>FABRIC SERIES 07: ADVANCED — CROSS-TENANT DATA PIPELINE DEPLOYMENT AUTOMATION</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nisaCcRb9HyQw1Kws_b-vQ.png" /><figcaption>Microsoft Fabric Automated Data Pipeline Deployment</figcaption></figure><h3>❑ Overview</h3><p>Embarking on your journey with Microsoft Fabric? Or perhaps you’re already a seasoned explorer, pushing the boundaries of what’s possible? Have you come across a scenario where you want to automate your data pipelines deployments from one tenant to another?</p><p>For those familiar with Azure Data Factory (ADF), you know that its import and export capabilities have made pipeline migration a breeze, enabling customers and ISVs alike to reuse and redeploy pipelines across tenants.</p><p>In this blog post, I’m excited to share a utility I’ve developed that automates the cross-tenant deployment of Fabric Data Pipelines. I am confident this tool will prove invaluable when faced with cross-tenant migration scenarios. So, let’s dive in and explore this solution together!</p><h3>❑ Problem Statement — Below 2 pipelines needs to be migrated to another tenant</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XROk2eYmTtTTaDKh4BSWhQ.png" /><figcaption>Data pipeline to be migrated</figcaption></figure><p>▣<strong> Details of data pipeline “PL_CopyOrdersData” data pipeline—</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tme2dPnnuD_eBmFlxXOIg.png" /><figcaption>PL_CopyOrdersData Details</figcaption></figure><p>Above pipeline consists of 2 activities -</p><ol><li><strong>ACT_MT_Copy_InventoryDataForFullLoad</strong> — For copying from Azure PostgreSQL DB to Microsoft Fabric Lakehouse.</li><li><strong>ACT_GEN_EPIPE_invokeInventoryAuditSuccessForFullLoad</strong>- To invoke auditing on successful full Inventory Table Ingestion. Here we have used an invoke activity to invoke a child pipeline to showcase how to.</li></ol><p>▣ <strong>Details of data pipeline “PL_WaitAct” data pipeline (invoked by “PL_CopyOrdersData”) —</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*fLwEGjXZlcLyZ9LLxMEwug.png" /><figcaption><strong>PL_WaitActPipeline</strong></figcaption></figure><p>Above pipeline consists of 1 activity -</p><ol><li><strong>ACT_GEN_Wait </strong>— This activity is used to pause the execution of a pipeline for a specified period.</li></ol><h3>❑ Start with pre-requisites</h3><p>1. Create a Microsoft Fabric destination tenant <strong>workspace</strong>. You can also identify existing lakehouse where the data needs to be ingested in Destination workspace. Get the e.g. Workspace ID — 66a92280-b91b-408e-bdd5–0276ad7d35a1.</p><p>2. Within this workspace, create a Lakehouse or identify existing lakehouse where the data needs to be ingested in Destination workspace.</p><ul><li>Get the Lakehouse ID e.g. a1e8b2fe-7527–40b7-a137–648be36a6602</li><li>Get the SQL analytics endpoint e.g. x6eps4xrq2xudenlfv6naeo3i4-qarkszq3xghebpovaj3k27jvue.msit-datawarehouse.fabric.microsoft.com</li><li>Get the name e.g. CMFLHStore</li></ul><p>3. For both of the above pipelines that needs to be deployed, go to the pipeline &amp; select “View JSON code&quot;. Go to “Copy to clipboard” &amp; “Close&quot;. Save the Json &amp; upload the Json in the lakehouse in the Destination workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M8fSm2yyaaUPw9qPze0lOw.png" /><figcaption>SaveJSONinDestinationWorkspace</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*j09zLVxmgUFw11tWmz0HbQ.png" /><figcaption>UploadJSON</figcaption></figure><p>4. Next steps is to identify the internal &amp; external connections used in our source data pipelines by executing “IdentifyExternalInternalConnections” notebook.</p><p>As a pre-requisite for this, update the value of “<strong>identifyconnections_deployment_config.yml</strong>” file with</p><p><strong><em>pipelinedisplayname</em></strong> — The Name of the data pipeline</p><p><strong><em>pipelinejsonfilepath </em></strong>— The file path of the data pipeline; as shown in below figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*y8pUj8jTzPMtJlqNGnErJQ.png" /><figcaption>identifyconnections_deployment_config.yml</figcaption></figure><p>5. Execute “IdentifyExternalInternalConnections” notebook. Make sure that you have attached &amp; pinned the right lakehouse before executing the notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UYlDoaKzBtRpGda5_cpONg.png" /><figcaption>IdentifyExternalInternalConnections-Notebook</figcaption></figure><p>After executing below output is obtained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xvuu-5xNzD6KxkKuWaX4qQ.png" /><figcaption>Output of IdentifyExternalInternalConnections-Notebook</figcaption></figure><p>With this we have identified that we have an external connection so go to Settings in Source tenant -&gt; Manage Connections, identify the connection type e.g. PostgreSQL, SQL DB based on the ID obtained from above step.</p><p>6. Once the external connection type for the source tenant is identified, We need to create the external connection in destination tenant. Please follow below steps to create a new connection &amp; obtain the new ConnectionID. Please note that this will be used further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xY2N3exVNFJOvXmYGLuibA.png" /></figure><p>7. In the destination tenant, the penultimate step is to update “pipeline_deployment_config.yml” file as below &amp; upload the updated file to lakehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X39el9StIC9fb7SKWznFag.png" /><figcaption>pipeline_deployment_config.yml</figcaption></figure><p>8. The final step is to Run the “<strong>FabricDataPipelineAutoDeployment</strong>” notebook.Make sure that you have attached &amp; pinned the right lakehouse before executing the notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQPnwN4RQuZrSKLjmuKWXQ.png" /></figure><p>After executing below output is obtained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*snP14vnYYNQD-q6eVykhaA.png" /><figcaption>Notebookrunoutput</figcaption></figure><p>The data pipeline gets deployed in the target tenant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qUS5HStCjL00GCup1QYzw.png" /><figcaption>Datapipelinedeployed</figcaption></figure><h3>❑ GitHub Repo</h3><p>Here’s the GitHub repo of the utility.</p><p><a href="https://github.com/rkniyer999/FabricAutomatedDataPipelineDeployment">rkniyer999/FabricAutomatedDataPipelineDeployment: This repository is used to perform automated data pipeline deployment in another tenant. (github.com)</a></p><h3>❑ Errors</h3><p>If you come across below error while executing the “MicrosoftFabricDataPipelineAutoDeployment” notebook, delete the pipeline &amp; wait for 5 mins to re-execute the notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-9Mn6vl6yXO2cWgQVQBnw.png" /></figure><h3>❑ Reference</h3><p><a href="https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api">Fabric data pipeline public REST API (Preview) — Microsoft Fabric | Microsoft Learn</a></p><h3>❑ Conclusion</h3><p>I hope this blog helps you in migrating Fabric data pipelines across tenant. I will keep you posted if there are any feature enhancements. Till then, Happy Learning!!!</p><h3>❑ Thanks</h3><p>Special thanks to <a href="https://medium.com/u/e2534ae238ff">Abhishek Narain</a>, <a href="https://medium.com/u/f518e8bca43b">Vedang Sharma</a> &amp; Gyani Sinha for brainstorming &amp; guiding on Automated Deployment of Data Pipelines enabling tenant migration for data pipeline resources.</p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3e64acae8d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/streamlining-fabric-data-pipeline-transfers-automated-deployment-across-tenants-d3e64acae8d6">Automate Your Data Pipeline Deployments Across Tenants</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Million Dollar Question in Microsoft Fabric — When to use Lakehouse VS Datawarehouse — Part2]]></title>
            <link>https://medium.com/microsoft/million-dollar-question-in-microsoft-fabric-when-to-use-lakehouse-vs-datawarehouse-part2-8ee9af1054ee?source=rss-c05b49979e2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ee9af1054ee</guid>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <dc:creator><![CDATA[RK Iyer]]></dc:creator>
            <pubDate>Mon, 20 May 2024 02:00:08 GMT</pubDate>
            <atom:updated>2024-09-18T12:19:32.126Z</atom:updated>
            <content:encoded><![CDATA[<h3>Million Dollar Question in Microsoft Fabric — When to use Lakehouse VS Datawarehouse — Part2</h3><h4>FABRIC SERIES: FUNDAMENTALS 06— DATATWAREHOUSE Vs LAKEHOUSE — ARCHITECHTURAL PATTERNS</h4><p>✐ Co-Author — <a href="https://medium.com/u/54dde3fee52a">Mihyun Kim</a></p><h3>❑ Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjHsVjgSzN8hGnf1B6PtDg.png" /><figcaption>Lakehouse VS Datawarehouse</figcaption></figure><p>In the last <a href="https://rkniyer999.medium.com/million-dollar-question-in-microsoft-fabric-when-to-use-lakehouse-vs-datawarehouse-part1-9c1f8d24149f">blog</a>, we went through understanding the fundamental difference between Data Warehouse &amp; Lakehouse. In this blog, we will delve more deeper in this aspect, wherein we will look at different key decision points, architectural patterns &amp; key considerations which influence using Lakehouse &amp; Warehouse in your solution.</p><h3>❑ Data Warehouse Pattern</h3><p>A data warehouse architectural pattern is a structured framework used to manage and organize large volumes of structured data for analysis and reporting purposes.</p><h4><strong>◪ Key considerations for employing this architecture pattern:</strong></h4><p>🗹 All data sources are <strong>structured data (Databases, CSV files, JSON files) or semi structured sources (Logs, Sensors)</strong> and future data sources are also going to be the same.</p><p>🗹 The development team primarily possesses <strong>T-SQL and Data Warehousing skills</strong>.</p><p>🗹 If the <strong>system is being migrated </strong>from an <strong>existing Data Warehouse</strong> like Synapse Dedicated Pool or Redshift, it might be more <strong>straightforward to continue using a Data Warehouse</strong> to leverage existing investments and minimize disruption.</p><p>🗹 Data <strong>transformation</strong> is performed using stored procedures.</p><p>🗹 There is a <strong>need to support features</strong> like <strong>multi-table transactions</strong>, which are exclusive to the Data Warehouse.</p><p>🗹 Although transactional workloads can use the Data Warehouse, they can also utilize the Lakehouse depending on the specific requirements of the application or ETL/ELT processes.</p><p>🗹 Users and developers require <strong>T-SQL DDL/DML capabilities</strong>.</p><p>🗹 The workload demands that users be able to modify data even after it has been normalized or transformed.</p><p>Below figure represents key components of a data warehouse architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQ0OQJfUdCAEm9I_tfI2yg.png" /><figcaption>Data Warehouse Architectural Pattern</figcaption></figure><p>▣ <strong>Data Sources</strong></p><ul><li>Incorporates data from other Azure Data Services, various cloud platforms, on-premises sources, and more.</li></ul><p>▣ <strong>Pipelines &amp; Dataflows</strong></p><ul><li>Fabric Pipelines offers the ability to ingest data into your warehouse with 200+ native connectors.</li><li>You can use copy activities in pipelines to land the data or dataflows to land the data with in-flight transformations if needed.</li></ul><p>▣ <strong>Prep &amp; Ingest</strong></p><ul><li>It is still advisable to maintain a clear separation of data zones within your Data Warehouse design including zones such as a landing zone, staging zone, and production zone.</li></ul><h4>Data Transformations:</h4><ul><li>SQL stored procedures orchestrated through Fabric pipelines are recommended.</li></ul><p>▣ <strong>Store</strong></p><ul><li>Once the data are transformed, they are stored in the form of tables for consumption. Additionally, views and functions can be created to customize and control the end user’s experience and access, similar to a traditional SQL environment familiar to many users.</li><li>Using SQL granular permissions, you can specify the access level for each user or security group for each table. Object-level, row-level, and column-level security can be applied, along with dynamic data masking to protect sensitive information.</li></ul><p>▣ <strong>Serve &amp; Present</strong></p><ul><li>Users/teams will consume data through Power BI via reports, datasets, dashboards, apps, etc. That are sourced from the gold warehouse or tables.</li><li>Users and teams can import data into various reporting tools and even perform cross-database queries between Lakehouses(if any) and Data Warehouses to meet different reporting needs. This approach supports <strong>exploration, ad hoc analysis</strong>, and the execution of DDL/DML statements by both <strong>business users and SQL analysts/developers</strong>.</li><li>3rd party reporting tools or other processes that require functionality only available through the Data Warehouse endpoint.</li></ul><h3>❑ Lakehouse Pattern</h3><p>A lakehouse architecture is a hybrid data management framework that combines the capabilities of data lakes and data warehouses to handle both structured and unstructured data. It allows for scalable storage, efficient data processing, and supports advanced analytics and reporting. Typically, a medallion architecture is used in Lakehouse by segregating the data in Bronze, Silver &amp; Gold layer.</p><h4>◪ Key Considerations for Choosing Lakehouse Architecture Pattern</h4><p>🗹 All data sources are mostly <strong>Un-structured data (Images, Videos), Semi-structured (Logs, Sensors) &amp; also consist of Structured sources</strong> and future data sources are also going to be the same.</p><p>🗹 The development team <strong>primarily has skills in Spark</strong>.</p><p>🗹 There is no need for <strong>advanced Data Warehouse capabilities</strong>, such as <strong>multi-table transactions and dynamic data masking</strong>.</p><p>🗹 Users and developers <strong>do not require T-SQL DDL/DML </strong>capabilities. (Honestly, I have seen many startups who feel cool about Spark and don’t prefer SQL)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cb9z3rxVXPqeMsesNu7IGQ.png" /><figcaption>Lakehouse Architectural Pattern</figcaption></figure><h4>▣ Data Sources:</h4><p>Data can come from various sources, including files, streaming data, on-premises locations, Azure, other cloud providers, shortcuts, or within Fabric itself.</p><h4>▣ Prepare and Transform:</h4><p>The main aim of ingestion is to land the data in the Bronze Lakehouse (raw layer) through Fabric Pipelines, Shortcuts or even jobs.</p><h4>Bronze Lakehouse:</h4><ul><li>Whenever feasible, retain the data in its original format; if not, convert it to Parquet or Delta Parquet using the “Files” layer in Lakehouse.</li><li>The data in bronze layer is pushed into next layer for further processing.</li></ul><h4>Silver Lakehouse:</h4><ul><li>In this area you will start to verify the quality of your data, enrich your data by combining other data sources together, transforming your existing data, data cleansing, etc.</li><li>The data in silver layer is stored as a managed delta tables to take advantage of Delta-Parquet &amp; V-Order write-time optimization which allows extremely fast reads by the different compute engines available in Fabric be it SQL engine, Power BI engine, Spark engine.</li></ul><h4>Gold Lakehouse:</h4><ul><li>In this area the data will be “business ready” form to be consumed by various business users.</li><li>The data in <strong>Gold layer</strong> is also stored as a managed delta tables to take advantage of Direct mode with an ability to analyze large data volumes in Power BI. The data in <strong>Gold layer</strong> is typically in a STAR schema data model, data is normalized, and business logic has been applied so the data is ready for consumption.</li></ul><blockquote><em>Direct Lake</em> mode is a semantic model capability for analyzing very large data volumes in Power BI. Direct Lake is based on loading parquet-formatted files directly from a data lake without having to query a lakehouse or warehouse endpoint, and without having to import or duplicate data into a Power BI model.</blockquote><h4>▣ Consume</h4><p>Users will consume and analyze the data through SQL Analytics Endpoint of the Gold Lakehouse wherein Business users can query the data using T-SQL queries or consume data through Power BI via reports, datasets, dashboards, apps, etc.</p><h3>❑ Combining both the Lakehouse and Data Warehouse into a single architecture</h3><p>In this architectural model, the combination of Lakehouse &amp; Data Warehouse is used to take advantage of the best of both worlds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjViE2S5OnpDOP8M4kJ8uw.png" /><figcaption>Lakehouse &amp; Data Warehouse</figcaption></figure><p>With this architecture, the consumption layer (via Power BI or warehouse endpoint) can fully utilize the Data Warehouse, including DDL/DML support for users and developers at the gold layer, as well as providing a Data Warehouse endpoint for other reporting tools and processes. This approach ensures optimal performance and avoids data duplication across different consumption methods.</p><h3>❑ Decision Tree</h3><p>Below figure represents key points to be considered while choosing Data warehouse or Lakehouse or even combination of both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VZJkmeDqU7MwMv1VB9aHKg.png" /><figcaption>Decision Tree of Lakehouse VS Warehouse selection</figcaption></figure><h3>❑ Conclusion</h3><p>I hope this blog helped you in understanding the difference architectural patterns for Warehouse &amp; Lakehouse along with the key considerations for choosing the same. There is still more to come…Happy Learning!!!</p><h3>❑ Reference</h3><p><a href="https://learn.microsoft.com/en-us/fabric/get-started/decision-guide-data-store">Fabric decision guide — choose a data store — Microsoft Fabric | Microsoft Learn</a></p><p><a href="https://blog.fabric.microsoft.com/en-US/blog/lakehouse-vs-data-warehouse-deep-dive-into-use-cases-differences-and-architecture-designs/">Lakehouse vs Data Warehouse vs Real-Time Analytics/KQL Database: Deep Dive into Use Cases, Differences, and Architecture Designs | Microsoft Fabric Blog | Microsoft Fabric</a></p><p><a href="https://learn.microsoft.com/en-us/fabric/get-started/direct-lake-overview">Learn about Direct Lake in Power BI and Microsoft Fabric — Microsoft Fabric | Microsoft Learn</a></p><p><em>Please Note — All opinions expressed here are my personal views and not of my employer.</em></p><p><strong>Thought of the moment-</strong></p><p><em>“</em>Innovation distinguishes between a leader and a follower.” — Steve Jobs</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ee9af1054ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/million-dollar-question-in-microsoft-fabric-when-to-use-lakehouse-vs-datawarehouse-part2-8ee9af1054ee">Million Dollar Question in Microsoft Fabric — When to use Lakehouse VS Datawarehouse — Part2</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>