<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anirban Som on Medium]]></title>
        <description><![CDATA[Stories by Anirban Som on Medium]]></description>
        <link>https://medium.com/@anirsom2012?source=rss-f84177806fc8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6tEcNee2p5KIaZe7YZl8bQ.png</url>
            <title>Stories by Anirban Som on Medium</title>
            <link>https://medium.com/@anirsom2012?source=rss-f84177806fc8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:27:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anirsom2012/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Databricks Lakeflow Pipelines: Understanding Materialized View Refresh & Refresh Policies with…]]></title>
            <link>https://medium.com/@anirsom2012/databricks-lakeflow-pipelines-understanding-materialized-view-refresh-refresh-policies-with-d4acb4f5e1ce?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4acb4f5e1ce</guid>
            <category><![CDATA[databricks-lakeflow]]></category>
            <category><![CDATA[declarative-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 08:07:58 GMT</pubDate>
            <atom:updated>2026-08-29T08:07:58.814Z</atom:updated>
            <content:encoded><![CDATA[<h3>Databricks Lakeflow Pipelines: Understanding Materialized View Refresh &amp; Refresh Policies with Real-World Examples</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFwrgZea9tPD24Tmf0i4Hw.png" /></figure><p>When I started working with <strong>Databricks Lakeflow Pipeline</strong>, one thing that caught my attention was how materialized views are refreshed. At first, it sounds pretty straightforward — the source data changes, and the materialized view gets updated. But there’s a little more to it than that.</p><p>Depending on the query, Databricks Lakeflow Pipeline can potentially refresh a materialized view incrementally instead of recalculating everything from scratch. This can make a big difference when you’re dealing with large datasets.</p><blockquote>But then comes the obvious question:</blockquote><blockquote><strong>How does Databricks Lakeflow Pipeline decide which refresh approach to use? And when should we use a refresh policy like </strong><strong>incremental_strict?</strong></blockquote><p>In this article, I’ll try to break this down with a few simple examples — starting with what a refresh policy actually means, then looking at incremental refresh, incremental_strict, and cases where a full refresh may be needed.</p><p>The goal isn’t to go deep into every internal detail, but rather to build a practical understanding that you can use when designing <strong>Databricks Lakeflow Pipelines</strong>.</p><h3>What is “Refresh” for a Materialized View?</h3><p>When we define a materialized view in a Databricks Lakeflow Pipeline, we are essentially defining a query whose result is persisted as a table.</p><p>For example, suppose we have a transactions table:</p><pre>SELECT<br>    account_id,<br>    COUNT(txn_id) AS txn_count,<br>    SUM(txn_amount) AS account_revenue<br>FROM transactions<br>GROUP BY account_id</pre><p>We can define this as a materialized view:</p><pre>from pyspark import pipelines as dp<br>from pyspark.sql.functions import count, sum<br><br>@dp.materialized_view()<br>def account_summary():<br>    return (<br>        spark.read.table(&quot;transactions&quot;)<br>        .groupBy(&quot;account_id&quot;)<br>        .agg(<br>            count(&quot;txn_id&quot;).alias(&quot;txn_count&quot;),<br>            sum(&quot;txn_amount&quot;).alias(&quot;account_revenue&quot;)<br>        )<br>    )</pre><p>The materialized view stores the <strong>result</strong> of this query rather than calculating it from scratch every time someone queries the view.</p><p>But what happens when the underlying transactions table changes?</p><p>For example, imagine the materialized view currently contains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bNTkdcHXksX30BzP5OrF8w.png" /></figure><p>Now 10 new transactions arrive for A101. The materialized view is no longer reflecting the current state of transactions. This is where <strong>refresh</strong> comes in.</p><h3>Refresh means updating the materialized view</h3><p>A refresh brings the materialized view’s stored results up to date with the current results of its defining query.</p><p>In other words:</p><blockquote><strong><em>Refresh = make the materialized view reflect the current state of its source data.</em></strong></blockquote><p>Databricks guarantees that the final result of the materialized view is equivalent to what you would get by executing the defining query as a batch query over the source data.</p><p>The interesting part is <strong>how Databricks gets to that result</strong>.</p><p>It doesn’t necessarily have to scan the entire source table every time.</p><h3>Refresh types</h3><p>When a materialized view is refreshed, Databricks can use either:</p><ol><li><strong>Incremental refresh (Default)</strong></li><li><strong>Full refresh</strong></li></ol><p>With an <strong>incremental refresh</strong>, Databricks identifies changes in the source data and processes only the data that affects the materialized view’s result instead of recomputing the entire query.</p><p>Conceptually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rboYjV3ZmszZs_ry_n_Hiw.png" /></figure><p>Whereas a full refresh looks more like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*3BorBFqZcXyZAPUGDbD0kg.png" /></figure><h3>What happens with the default refresh?</h3><p>This is where things get slightly more interesting.</p><p>In a <strong>serverless</strong> Lakeflow Pipeline, a normal/default refresh doesn’t simply mean “always do incremental refresh.”</p><p>Instead, Databricks evaluates whether incremental refresh is possible and whether it is more cost-effective.</p><p>Conceptually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XR9kD1undSGOqyMkM_ZfCw.png" /></figure><h3>Understanding Refresh Policy in a Materialized View</h3><p>Once we understand what a refresh means, the next question naturally comes up:</p><blockquote><strong>Who decides how the materialized view should be refreshed?</strong></blockquote><p>This is where <strong>refresh policy</strong> comes into the picture.</p><p>A refresh policy tells Databricks how it should maintain the result of a materialized view when the underlying data changes.</p><p>Let’s take a simple example.</p><p>Imagine we have a transactions table containing millions or even billions of transactions, and we want to maintain an account-level summary:</p><pre>from pyspark import pipelines as dp <br>from pyspark.sql.functions import count, sum <br>@dp.materialized_view() <br>def account_summary(): <br>  return <br>    ( <br>      spark.read.table(&quot;transactions&quot;)<br>      .groupBy(&quot;account_id&quot;) <br>      .agg( <br>          count(&quot;txn_id&quot;).alias(&quot;txn_count&quot;), <br>          sum(&quot;txn_amount&quot;).alias(&quot;account_revenue&quot;) <br>  ) <br>)</pre><p>Now suppose yesterday the transactions table had 500 million rows.</p><p>Today, only 50,000 new transactions have arrived.</p><p>We have two possible approaches.</p><p>The first approach is to say:</p><blockquote><em>“Forget what was calculated yesterday. Read all 500 million rows again and calculate the result from scratch.”</em></blockquote><p>That’s a <strong>full refresh</strong>.</p><p>The second approach is:</p><blockquote><em>“Only process the changes since the last refresh and update the affected results.”</em></blockquote><p>That’s an <strong>incremental refresh</strong>.</p><p>Both approaches should ultimately produce the same result. The difference is in <strong>how much data needs to be processed to get there</strong>.</p><h3>The four refresh policies</h3><p>Databricks provides four policies for materialized views:</p><ul><li><strong>AUTO</strong></li><li><strong>INCREMENTAL</strong></li><li><strong>INCREMENTAL STRICT</strong></li><li><strong>FULL</strong></li></ul><p>You can think about them as different levels of control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XSZoJHea9p3ZqPM-G7GDg.png" /></figure><pre>from pyspark import pipelines as dp<br><br>@dp.materialized_view(<br>  refresh_policy = &#39;incremental_strict&#39;<br>)<br>def my_mv():<br>  return spark.read(&quot;main.default.source_table&quot;)</pre><p>Using REFRESH POLICY, you can configure the system to:</p><ul><li>AUTO (default) - Use automatic, cost-based selection. Databricks chooses incremental or full refresh based on efficiency and query capabilities. Recommended for most users.</li></ul><blockquote><em>“Keep this materialized view up to date and choose the appropriate refresh strategy.”</em></blockquote><ul><li>INCREMENTAL - Prefer incremental refresh. Databricks performs incremental refresh whenever possible. It falls back to a full refresh if the query plan no longer supports incremental refresh.</li></ul><blockquote>“Prefer incremental refresh, but fall back to Full Refresh”</blockquote><ul><li>INCREMENTAL STRICT - Strictly require incremental refresh. Incremental refresh is required during normal operation. If incrementalization is not possible, the refresh or create operation fails.</li></ul><blockquote>“I want incremental processing, and I don’t want the pipeline to silently switch to a full refresh if incremental processing isn’t possible.”</blockquote><ul><li>FULL - Always perform full refreshes. Databricks never performs incremental refresh, even when the query is incrementalizable.</li></ul><blockquote>“Always! Full Refresh”</blockquote><h3>Wrapping Up: Choosing the Right Refresh Policy</h3><p>At the end of the day, refresh policy is less about choosing a setting and more about understanding how your data behaves. If only a small amount of data changes between pipeline updates, incremental refresh can save a lot of unnecessary processing. But when the data or transformation makes incremental processing impractical, a full refresh may be the better choice. The important thing is to understand what Databricks is doing behind the scenes and choose a policy that matches your data, query, and performance requirements. Once you look at refresh this way, the different refresh policies in Databricks Lakeflow Pipeline become much easier to reason about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4acb4f5e1ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Declarative Automation Bundle (DAB) + GitHub Actions: Deployment Patterns and Best Practices]]></title>
            <link>https://medium.com/@anirsom2012/declarative-automation-bundle-dab-github-actions-deployment-patterns-and-best-practices-ea6a3667892b?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea6a3667892b</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sun, 19 Jul 2026 18:21:58 GMT</pubDate>
            <atom:updated>2026-07-19T18:21:58.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*mziAexukdGsJRLKajxK-Gw.jpeg" /></figure><p>As organizations adopt Declarative Automation Bundle<strong> (formerly known as Databricks Assets Bundle — DAB)</strong>, one of the first questions that comes up is:</p><blockquote>“<em>How should multiple developers deploy to the same Databricks workspace without interfering with each other?”</em></blockquote><p>This article explores practical deployment patterns for <strong>DAB</strong> using <strong>GitHub Actions</strong>, discusses common pitfalls, and recommends best practices for development, testing, and production environments.</p><h3>Why Declarative Automation Bundles?</h3><p>It brings Infrastructure as Code (IaC) principles to the Databricks platform. Instead of manually creating jobs, pipelines, clusters, and workflows, developers define them declaratively in a bundle and deploy them consistently across environments.</p><p>This enables:</p><ul><li>Version-controlled deployments</li><li>Repeatable infrastructure</li><li>Automated CI/CD</li><li>Environment consistency</li></ul><p>However, once multiple developers start deploying simultaneously, deployment isolation becomes an important consideration.</p><h3>Deployment Pattern 1: Local Development</h3><p>Developers first create a git folder in their Users directory using the git clone URL of the github Repo. Based on the feature they are working they will create a feature branch and start adding code and bundle configurations.</p><p>A common <strong><em>databricks.yml</em></strong> configuration is:</p><pre>targets: <br>  dev: <br>    mode: development <br>    workspace: <br>      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}</pre><p>If Alice deploys:</p><pre>/Workspace/Users/alice@company.com/.bundle/sales/dev</pre><p>If Bob deploys:</p><pre>/Workspace/Users/bob@company.com/.bundle/sales/dev</pre><p>Each developer receives an isolated deployment.</p><p>This is the recommended approach for local development because no one overwrites another developer’s assets.</p><h3>Deployment Pattern 2: GitHub Actions with a Shared Service Principal</h3><p>Many organizations automate deployments using GitHub Actions authenticated with a single service principal.</p><p>At first glance, it appears that the same configuration should work as above:</p><pre>root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}</pre><p>Unfortunately, it does not.</p><p>Since every deployment authenticates as the same service principal,</p><pre>workspace.current_user.userName</pre><p>always resolves to the service principal.</p><p>For example:</p><pre>/Workspace/Users/databricks-app-deploy-spn/.bundle/sales/dev</pre><p>Every deployment targets the exact same location.</p><p>If two developers deploy simultaneously:</p><ul><li>Developer A deploys feature/customer-api</li><li>Developer B deploys feature/reporting</li></ul><p>the second deployment overwrites the first.</p><p>This is one of the most common mistakes when adopting DAB’s with CI/CD.</p><h3>The Solution: Make Deployments Unique</h3><p>Instead of relying on the authenticated user, create a deployment path that includes information from GitHub.</p><p>GitHub already knows:</p><ul><li>Repository</li><li>Branch</li><li>Pull Request</li><li>Commit</li><li>Workflow</li><li>Actor</li></ul><p>These values can be passed into the bundle as variables.</p><p>For example:</p><pre>workspace:<br>  root_path: /Workspace/.bundle/${bundle.name}/${bundle.target}/${var.github_actor}/${var.branch_name}</pre><p>GitHub Actions:</p><pre>env:<br>  BUNDLE_VAR_github_actor: ${{ github.actor }}<br>  BUNDLE_VAR_branch_name: ${{ github.ref_name }}</pre><p>Deployments become</p><pre>/Workspace/.bundle/sales/dev/alice/feature-orders</pre><pre>/Workspace/.bundle/sales/dev/bob/feature-dashboard</pre><p>Now every deployment is isolated.</p><h3>Deployment Pattern 3: Branch Isolation</h3><p>Another useful strategy is branch isolation.</p><pre>root_path:<br>    /Workspace/.bundle/app/dev/${var.branch_name}</pre><p><strong>Examples:</strong></p><ul><li>feature-login</li><li>feature-payments</li><li>feature-reporting</li></ul><p><strong>Advantages</strong></p><ul><li>Easy to identify deployments</li><li>Multiple feature branches can coexist</li><li>Simple cleanup</li></ul><p><strong>Disadvantage</strong></p><p>Two developers working on the same branch could still overwrite each other.</p><h3>Deployment Pattern 4: User + Branch Isolation (Recommended)</h3><p>The most flexible pattern combines both the GitHub actor and branch name.</p><pre>workspace:<br>  root_path: /Workspace/.bundle/${bundle.name}/${bundle.target}/${var.github_actor}/${var.branch_name}</pre><p>Example deployments:</p><pre>alice<br>   feature-dashboard</pre><pre>bob<br>   feature-dashboard</pre><pre>charlie<br>   feature-orders</pre><p>Every deployment is completely isolated.</p><h3>Environment-Specific Deployment Strategy</h3><p>Development and production environments have different goals.</p><p>For production deployments, there should be only one authoritative deployment.</p><p>Example:</p><pre>workspace:<br>    root_path: /Workspace/app/prod</pre><p>Production should be deterministic and not depend on who triggered the deployment.</p><h3>Final Thoughts</h3><p>Declarative Automation Bundles provide a powerful foundation for managing Databricks resources as code, and GitHub Actions makes it straightforward to automate deployments.</p><p>The most important design consideration is deployment isolation.</p><p>For local development, using workspace.current_user.userName works well because each developer authenticates with their own identity.</p><p>For GitHub Actions using a shared service principal, however, every deployment appears to come from the same identity. In that case, deployment paths should be parameterized using GitHub metadata such as the actor and branch name to prevent collisions.</p><p>By combining isolated development deployments, environment-specific bundle targets, validation, and promotion through CI/CD, teams can build a scalable, reliable, and maintainable deployment process for Databricks that supports both rapid development and production stability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea6a3667892b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Google Dremel: Why It Still Matters for Data Engineers]]></title>
            <link>https://medium.com/@anirsom2012/demystifying-google-dremel-why-it-still-matters-for-data-engineers-9d1d8894530c?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d1d8894530c</guid>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sun, 08 Feb 2026 12:04:56 GMT</pubDate>
            <atom:updated>2026-02-08T12:04:56.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzXA7PT9EP29iSUH0qWUHg.png" /></figure><h3>Introduction</h3><p>In 2008, Google faced a massive challenge: its services — Search, Gmail, YouTube, Ads — were generating <strong>trillions of log entries daily</strong>, often stored in <strong>nested, semi-structured formats</strong>. Analysts needed to query this data quickly to detect spam, monitor ad click fraud, and understand user engagement.</p><p>The existing tools, <strong>Hadoop and Hive</strong>, were powerful but slow. Hadoop’s batch jobs could take hours, and Hive’s SQL-like queries were still translated into MapReduce jobs, inheriting the same latency. For Google’s scale, this was unacceptable.</p><p>This pain point inspired the creation of <strong>Dremel</strong>, first published in a 2008 white paper. As explained in Gaurav Sen’s excellent breakdown of the paper, Dremel was designed to run <strong>interactive queries on web-scale datasets</strong> by combining columnar storage, hierarchical aggregation, and clever algorithms like finite automata and segment trees. It became the foundation for systems like <strong>BigQuery</strong>, <strong>Apache Drill</strong>, and <strong>Impala</strong>, and even won the <strong>VLDB Test of Time Award in 2020</strong></p><h3>Why Hadoop and Hive Fell Short</h3><p>Back in 2008, Hadoop and Hive were the backbone of big data, but they had limitations:</p><ul><li><strong>Batch-oriented</strong>: Hadoop was designed for long-running jobs, not interactive queries.</li><li><strong>Row-based storage</strong>: Queries had to scan entire rows, even if only a few columns were needed.</li><li><strong>Hive’s latency</strong>: Hive added SQL-like syntax but still relied on MapReduce, making queries slow.</li><li><strong>Nested data pain</strong>: JSON-like logs were inefficient to query.</li><li><strong>No interactivity</strong>: Analysts couldn’t “slice and dice” data quickly; they had to wait for batch jobs to finish.</li></ul><p>Google needed something faster, more interactive, and optimized for nested data. Dremel was the answer.</p><h3>Key Innovations in Google Dremel</h3><h4>1. Columnar Storage for Nested Data</h4><p>Dremel flattened nested records into a <strong>columnar representation</strong>, reducing I/O and enabling efficient aggregate queries.</p><p>➡️ <strong>Modern parallel</strong>: Apache Parquet and ORC use similar techniques, powering Spark, Hive, and Presto.</p><h4>2. Multi-Level Execution Trees</h4><p>Dremel introduced a <strong>tree architecture</strong> for query execution:</p><ul><li>Leaf nodes scan data shards.</li><li>Intermediate nodes aggregate partial results.</li><li>Root node merges everything.</li></ul><p>This hierarchical design enabled <strong>massive parallelism</strong> across thousands of CPUs.</p><p>➡️ <strong>Modern parallel</strong>: Presto/Trino and BigQuery use similar aggregation strategies.</p><h4>3. Algorithms for Scale</h4><p>As highlighted in the video, Dremel leveraged <strong>finite automata</strong> and <strong>segment trees</strong> to efficiently process queries across billions of records. These algorithms helped maintain speed and reliability at scale.</p><p>➡️ <strong>Modern parallel</strong>: Today’s query optimizers in Snowflake and BigQuery inherit similar principles.</p><h3>Why It Matters for Data Engineers Today</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8tyeS1yiz1z-kNnU9XnrA.png" /></figure><p>Dremel’s design principles are embedded in today’s <strong>cloud data warehouses</strong> and <strong>lakehouse architectures</strong>. Understanding them helps engineers design pipelines that balance <strong>batch ETL</strong> with <strong>interactive analytics</strong>.</p><h3>Practical Takeaways for Data Engineers</h3><ul><li><strong>Choose columnar formats</strong> (Parquet/ORC) for analytical workloads.</li><li><strong>Separate batch and interactive layers</strong> in your architecture.</li><li><strong>Leverage distributed query engines</strong> (BigQuery, Trino, Snowflake) for scalability.</li><li><strong>Think in terms of execution trees</strong> when designing aggregations — hierarchical aggregation reduces bottlenecks.</li></ul><h3>Conclusion</h3><p>Google Dremel was more than a research project — it was the blueprint for modern data warehouses. For today’s data engineers, its lessons remain highly relevant: <strong>optimize storage, embrace parallelism, and design for interactivity</strong>.</p><p>By connecting Dremel’s innovations to current technologies, and learning from resources like Gaurav Sen’s system design video, we gain a deeper appreciation of why tools like BigQuery and Snowflake feel so fast — and how to build pipelines that scale with confidence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d1d8894530c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakehouse update strategies: Copy-on-Write Vs. Merge-on-Read]]></title>
            <link>https://medium.com/@anirsom2012/lakehouse-update-strategies-copy-on-write-vs-merge-on-read-7ddc9e6f2755?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ddc9e6f2755</guid>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sun, 25 Jan 2026 13:01:32 GMT</pubDate>
            <atom:updated>2026-01-25T13:01:32.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern data lakes are no longer just append-only parquet file dumps in cloud object storage. With the rise of <strong>open table formats</strong> — <strong>Delta Lake</strong>, <strong>Apache Iceberg</strong>, and <strong>Apache Hudi</strong> — data lakes now support <strong>updates, deletes, time travel, and streaming</strong> at scale.</p><p>At the heart of this evolution are two strategies: <strong>Copy-on-Write (CoW) </strong>and <strong>Merge-on-Read (MoR)</strong></p><p>Understanding these concepts is critical when designing analytical and near–real-time data platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVU89SFhaciECpVJbINy-g.png" /></figure><h3>Why Copy-on-Write and Merge-on-Read exist</h3><p>Traditional data lakes store data as immutable files (Parquet, ORC, Avro). But business use cases demand:</p><ul><li>Upserts from CDC pipelines</li><li>GDPR deletes</li><li>Late-arriving corrections</li><li>Near-real-time analytics</li></ul><p>Since files are immutable, open table formats solve this by <strong>tracking versions and metadata</strong> while applying one of two update strategies.</p><h3>Copy-on-Write (CoW)</h3><h4>How it works</h4><p>When a record is updated or deleted:</p><ol><li>The entire affected file (or partition) is rewritten</li><li>A new version of the data file is created</li><li>Old files are retained for time travel and rollback</li></ol><h4>Characteristics</h4><p>✅ Simple read path (no merge required)<br>✅ Excellent read performance<br>❌ Higher write amplification<br>❌ Slower for high-frequency updates</p><h4>Best for</h4><ul><li>BI &amp; reporting</li><li>Read-heavy workloads</li><li>Batch updates</li></ul><h3>Merge-on-Read (MoR)</h3><h4>How it works</h4><p>When updates arrive:</p><ol><li>Changes are written to <strong>delta/log files</strong></li><li>Base files remain untouched</li><li>At read time, base files + delta files are merged<br><em>(or compacted later asynchronously)</em></li></ol><h4>Characteristics</h4><p>✅ Fast writes<br>✅ Ideal for streaming and CDC<br>❌ Slightly slower reads (merge cost)<br>❌ Operationally more complex</p><h4>Best for</h4><ul><li>Streaming ingestion</li><li>CDC pipelines</li><li>Near–real-time analytics</li></ul><h3>How Delta Lake, Apache Iceberg and Apache Hudi provide options to choose CoW or MoR</h3><h4>Delta Lake:</h4><p>Delta lake by default uses Copy-on-Write. When a write, update or delete operation modifies a record of a data file, the entire data file gets rewritten. Delta lake introduced <strong>Deletion Vector </strong>to support Merge-on-Read. Deletion vector can be enabled for delta tables using table properties -</p><pre>ALTER TABLE table_name <br>SET TBLPROPERTIES (&#39;delta.enableDeletionVectors&#39; = &#39;true&#39;);</pre><h4>Apache Iceberg</h4><p>Apache Iceberg provides options to choose between <strong>Copy-on-Write (CoW)</strong> and <strong>Merge-on-Read (MoR)</strong> through table-level configuration properties. Iceberg allows configuring the update/delete mode when creating a table or by altering an existing table.</p><pre>-- Create table with Merge-on-Read<br>CREATE TABLE prod.db.sample (<br>    id bigint,<br>    data string,<br>    category string<br>) USING iceberg<br>TBLPROPERTIES (<br>    &#39;write.delete.mode&#39; = &#39;merge-on-read&#39;,<br>    &#39;write.update.mode&#39; = &#39;merge-on-read&#39;,<br>    &#39;write.merge.mode&#39;  = &#39;merge-on-read&#39;<br>);</pre><pre>-- Create table with Copy-on-Write<br>CREATE TABLE prod.db.sample (<br>    id bigint,<br>    data string,<br>    category string<br>) USING iceberg<br>TBLPROPERTIES (<br>    &#39;write.delete.mode&#39; = &#39;copy-on-write&#39;,<br>    &#39;write.update.mode&#39; = &#39;copy-on-write&#39;,<br>    &#39;write.merge.mode&#39;  = &#39;copy-on-write&#39;<br>);</pre><h4>Apache Hudi</h4><p>Apache Hudi provides option to choose CoW or MoR by supporting two table types. Hudi <em>table types</em> define how data is stored and how write operations are implemented on top of the table (i.e how data is written)</p><pre>df.write.format(&quot;hudi&quot;)<br>  .option(&quot;hoodie.datasource.write.table.type&quot;, &quot;MERGE_ON_READ&quot;) // Or COPY_ON_WRITE<br>  .mode(SaveMode.Overwrite)<br>  .save(basePath)</pre><h3>Final thoughts</h3><p>Copy-on-Write and Merge-on-Read aren’t competing ideas — they’re <strong>design levers</strong>.</p><p>Open table formats let you decide:</p><ul><li><strong>When to pay the cost</strong> (write vs read)</li><li><strong>How fresh the data must be</strong></li><li><strong>Which engines need access</strong></li></ul><p>As data lakes evolve into <strong>lakehouses</strong>, mastering these concepts is key to building platforms that scale — not just in size, but in capability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ddc9e6f2755" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing Effective Microsoft Fabric Deployment Patterns: A Practical Guide]]></title>
            <link>https://medium.com/@anirsom2012/designing-effective-microsoft-fabric-deployment-patterns-a-practical-guide-0c194a9ea0d0?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c194a9ea0d0</guid>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sat, 23 Aug 2025 11:26:10 GMT</pubDate>
            <atom:updated>2025-08-23T11:26:10.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8uDPfS01eltfhOjJcq_mJw.png" /></figure><p>Microsoft Fabric is emerging as a game-changer in enterprise analytics, bringing together storage, processing, and AI-driven insights in one unified SaaS platform. With OneLake at its core, Fabric simplifies how organizations manage and analyze data. But adopting Fabric effectively is not just about the technology — it’s about choosing the right deployment pattern. A thoughtful approach to deployment can determine how well you manage governance, security, cost, and scalability.</p><p>In this article, we’ll explore Fabric’s deployment hierarchy, common deployment patterns, layered vs. modular design strategies. By the end, you’ll have a practical framework and architecture to guide your Fabric deployment strategy.</p><h3>The Deployment Hierarchy in Microsoft Fabric</h3><p>Before diving into patterns, it’s important to understand Fabric’s hierarchical structure:</p><ul><li><strong>Tenant</strong>: The top-level boundary for your organization’s Fabric environment.</li><li><strong>Capacity</strong>: Defines the compute resources available within a tenant.</li><li><strong>Workspace</strong>: A logical grouping of assets (datasets, pipelines, reports) that aligns with teams or projects.</li><li><strong>Item</strong>: The atomic components — such as Dataflows, Lakehouses, notebooks, or reports — that deliver analytics capabilities.</li></ul><p>Getting this hierarchy right helps align deployment choices with organizational structure, governance, and scalability goals.</p><h3>Pattern #1 — Single Capacity</h3><p>This architecture is common for very small organizations or for those just starting out with microsoft fabric. This pattern is in a single capacity and has non-production and production workspace. These workspaces will be sharing the same compute resources</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*0jSq9FFqDH_J1kuqKK6Fjw.png" /></figure><h3>Pattern #2 — At least two capacities</h3><p>One capacity serves all your non production workspaces and the other capacity serves all your production workspaces. This is the typical architecture where resources are split by environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hv8IKCyV-CUNvwrxiNsFlQ.png" /></figure><h3>Pattern #3— Multi Region deployment</h3><p>This architecture is similar to <strong>Pattern #2, </strong>but in addition it has two capacities deployed in Region B. Each region has its own non-production and production resources. This architecture is used when you have Data Residency requirement due to regulations or you have users across different geographic region and you need lowest latency possible for reporting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*dbYRMCHtJaYKMmAd74tptA.png" /></figure><h3>Pattern #4 — Data Mesh type</h3><p>In this architecture organization might have a central IT team that curates data for other departments. All of this work would be done in a capacity that the central IT team manage. You might also have different business units who want to do their own self-service work or have their own data they want to work with. They can use the data that the central IT team has prepared or utilize their own data and do their own work. This is the typical Data Mesh architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zUmKTD7l72sCDb8QKmEUZw.png" /></figure><p>You can combine above 4 patterns and create your own Microsoft Fabric deployment architecture as per your business needs.</p><h3>Putting It Together: A Practical Example</h3><p>Imagine a retail enterprise deploying Microsoft Fabric:</p><ul><li>A <strong>central data team</strong> manages ingestion pipelines and curated datasets in a Bronze/Silver/Gold layered model.</li><li><strong>Business units</strong> (e.g., Sales, Marketing, Finance) operate modular workspaces to build reports and dashboards.</li><li><strong>CI/CD pipelines</strong> automate deployments from dev → test → prod, with Git ensuring version control.</li><li><strong>Governance policies</strong> enforce data security, cost monitoring, and workspace lifecycle management.</li></ul><p>This hybrid model combines the best of layered and modular strategies while ensuring governance and scalability.</p><h3>Conclusion</h3><p>Choosing the right deployment pattern in Microsoft Fabric is about more than just technical design — it’s about aligning with your organization’s governance model, team structures, and business priorities. Whether centralized, decentralized, or hybrid, successful deployments embrace layered architectures, modular flexibility, and strong DevOps practices.</p><p>With a thoughtful approach, Microsoft Fabric becomes more than just a platform — it becomes the backbone of a scalable, governed, and future-ready analytics ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c194a9ea0d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Great Security features in Snowflake]]></title>
            <link>https://medium.com/@anirsom2012/great-security-features-in-snowflake-166857cb2466?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/166857cb2466</guid>
            <category><![CDATA[data-warehouse-solutions]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[cloud-data-warehouse]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Sun, 30 Jan 2022 09:08:15 GMT</pubDate>
            <atom:updated>2022-01-30T09:08:15.622Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*Q03_hn4X7ThheNHHFCm8vQ.png" /><figcaption>Security features of Snowflake</figcaption></figure><p>Snowflake is a <strong>cloud data platform</strong>, in fact, it is the only data platform built IN and FOR the cloud. Snowflake is also the only cloud data platform that can be used as a data warehouse and a data lake. This allows both functionalities which means that you no longer need to have a separate data lake and data warehouse/data marts. With Snowflake, you can build your data architecture within a single platform.</p><p>For the many that use Snowflake to store <strong>sensitive information</strong>, it’s paramount to engage with it according to defined <strong>security guidelines</strong> in order to detect risky events, prevent as many threats as possible and react to security incidents in the best way possible.</p><p>Snowflake provides industry-leading features that ensure the highest levels of security for your account and users, as well as all the data you store in Snowflake. The following are a high-level summary of the features :</p><ul><li><strong>Network/site access</strong></li></ul><p>Network policies provide options for managing network configurations to the Snowflake service.Network policies allow restricting access to your account based on user IP address. Effectively, a network policy enables you to create an IP <em>allowed list</em>, as well as an IP <em>blocked list</em>, if desired.</p><ul><li><strong>User &amp; Group Administration</strong></li></ul><p>SCIM integration to manage user identities and groups (i.e. roles). SCIM(System for Cross-domain Identity Management) is an open specification to help facilitate the automated management of user identities and groups (i.e. roles) in cloud applications using RESTful APIs. In my organisation we use Microsoft Azure Active Directory as a SCIM identity provider. One can configure Snowflake to Provision Azure AD users and groups in Snowflake.</p><ul><li><strong>Account/user authentication</strong></li></ul><p>Snowflake supports using key pair authentication for enhanced authentication security as an alternative to basic authentication (i.e. username and password).</p><p>Snowflake also supports OAuth for authorised account access without sharing or storing user login credentials.</p><p>Snowflake also supports SAML Authentication for SSO.</p><ul><li><strong>Object security</strong></li></ul><p>Controlled access to all objects in the account (e.g. users, warehouses, databases, tables) through a hybrid model of DAC (discretionary access control) and RBAC (role-based access control).</p><ul><li><strong>Data security</strong></li></ul><p>All ingested data stored in Snowflake tables is encrypted using AES-256 strong encryption. All files stored in internal stages for data loading and unloading are also automatically encrypted using AES-256 strong encryption. Snowflake supports encrypting data using Customer Managed Key.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=166857cb2466" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Send email notification on AWS S3 events]]></title>
            <link>https://medium.com/@anirsom2012/send-email-notification-on-aws-s3-events-186acd25a401?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/186acd25a401</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Wed, 29 Jul 2020 06:54:03 GMT</pubDate>
            <atom:updated>2020-07-29T06:56:24.568Z</atom:updated>
            <content:encoded><![CDATA[<p>This article aims to help AWS developers to setup email notification on any AWS S3 bucket events (Any S3 bucket). Let’s see the below flow diagram to understand the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*nGtje6DMB1kJ_XSkKCW_dQ.png" /></figure><p><strong>List of services we are going to use -</strong></p><ol><li>IAM ( To create IAM Role for Lambda to access S3 and SES)</li><li>S3 ( For storage as well as to trigger the event)</li><li>Lambda ( To receive S3 event and send email using SES)</li><li>SES ( Configure Email address)</li></ol><p>We need to have a S3 bucket for which we will setup the event. In this article I will explain the process for PUT event ( Whenever you will receive any new object in your S3 bucket).</p><p>Lets start by creating an IAM Role for Lambda which will allow Lambda function to access SES and S3.</p><ol><li>Go to IAM</li><li>Click on Roles from the left pane under Access Management and click on “Create Role”</li><li>From AWS Services select Lambda and click on Next to add permissions</li><li>Add these following policies and Save the Role with a name—</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*wiIWv_L1dYs-U01XWClVTg.png" /></figure><p>Once the role is created, go to your lambda console and create a function which will handle the event and send email using SES. While creating the lambda function don’t forget to attach the above role, otherwise you wont be able to send emails from lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vGletzWexzFIvuky7MDuVQ.png" /></figure><pre>import json<br>import boto3</pre><pre>def lambda_handler(event, context):<br> # TODO implement<br> print(“S3 event : {0}”.format(event[‘Records’][0][‘eventName’]))<br> print(“S3 Bucket Name : {0}”.format(event[‘Records’][0][‘s3’][‘bucket’][‘name’]))<br> print(“S3 Object Name : {0}”.format(event[‘Records’][0][‘s3’][‘object’][‘key’]))<br> <br> client = boto3.client(‘ses’)<br> subject = “New object received on {0} bucket”.format(str(event[‘Records’][0][‘s3’][‘bucket’][‘name’]))<br> body = “””<br> &lt;br&gt;<br> Bucket Name : {0} &lt;br&gt;<br> Action : {1} &lt;br&gt;<br> Object Name : {2}<br> “””.format(event[‘Records’][0][‘s3’][‘bucket’][‘name’],event[‘Records’][0][‘eventName’],event[‘Records’][0][‘s3’][‘object’][‘key’])<br> <br> message = {“Subject” : {“Data” : subject}, “Body” : {“Html” : {“Data” : body}}}<br> <br> response = client.send_email(Source = “&lt;SOURCE_EMAIL&gt;”,Destination = {“ToAddresses” : [“&lt;TO_EMAIL&gt;”]}, Message = message)<br> <br> return {<br> ‘statusCode’: 200,<br> #’body’: json.dumps(‘Hello from Lambda!’)<br> }</pre><p>AWS provides the event structure which S3 sends when a new event occurs. Follow the below link to understand the format. I have followed the same format while writing the above code.</p><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html">Event message structure</a></p><p>Also to understand format to Send email using SES and boto3, follow the boto3 SES doc below.</p><p><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ses.html">SES - Boto3 Docs 1.23.10 documentation</a></p><p>Once you are done with the lambda function and Saved the function you can go to S3 bucket for which you want to setup a trigger and follow the steps below -</p><ol><li>Go to S3 bucket</li><li>Go to <strong>Properties</strong></li><li>Scroll down to <strong>Advanced Settings </strong>and Click on <strong>Events</strong></li><li>Click on <strong>Add Notification</strong></li><li>Give it a Name</li><li>Select Events for which you want to trigger the lambda</li><li>Use <strong>Prefix </strong>if you want it for a specific folder inside the bucket</li><li>Use <strong>Suffix</strong> if you want it for a specific type of files</li><li>Under <strong>Send to </strong>Select <strong>Lambda Function </strong>and select the function name from the dropdown</li><li>Save</li></ol><p>Once you are done with the above steps there is one final thing which you need to configure is validating your source email address with AWS SES. For that -</p><ol><li>Go to <strong>Simple Email Service</strong></li><li>Click on Email Addressed (On the left)</li><li>Click on <strong>Verify a New Email Address</strong></li><li>Enter the email address and click on <strong>Verify This Email Address</strong></li><li>You will receive an email with an unique link, you just need to click on that and your email address will be verified with SES</li></ol><p>Now, go to you S3 bucket and upload some file there. Below is the email which I received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*PEnDvCukE5Z1Bk_Idf3maA.png" /></figure><p>To monitor Log you can go to CloudWatch and then Log Groups to monitor logs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=186acd25a401" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup Jupyter Notebook on AWS EC2 Ubuntu Server]]></title>
            <link>https://medium.com/@anirsom2012/setup-jupyter-notebook-on-aws-ec2-ubuntu-server-7f92a5b5b3d?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f92a5b5b3d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Mon, 11 Nov 2019 18:20:28 GMT</pubDate>
            <atom:updated>2019-11-11T18:26:51.639Z</atom:updated>
            <content:encoded><![CDATA[<p>Jupyter Notebook is an open-source web application that allows you to create and share documents called notebooks that contains live code and result.</p><p>In Big Data domain, developers and analysts are using Jupyter Notebook to test their codes and to share analysis.</p><p>We can leverage big data tool such as Spark to write code in Python to fetch data into Jupyter Notebook and use other Python analytical modules such as pandas, matplotlib, scikit-learn to explore the data.</p><p>In this article we will learn how to install and launch Jupyter Notebook on a AWS EC2 instance running Ubuntu server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*ngJunac1NVRLMA1IuBQj9A.png" /></figure><p>Below are the steps we are going to perform -</p><ol><li>Launch AWS EC2 instance and connect through SSH</li></ol><p>2. Create new user and assign to sudo</p><p>3. Install Python 3 and pip</p><p>4. Install Jupyter module for Python (<strong>pip install jupyter</strong>) and Configure Jupyter to allow access from other IP addresses</p><p>5. Launch Jupyter Notebook by using jupyter notebook command</p><p>6. Access Jupyter Notebook from Web Browser using URL and create notebooks.</p><h4><strong>Step 1 : Launch AWS EC2 instance and connect through SSH</strong></h4><p>Go to your AWS Management console and select EC2. Click on Launch Instance button and select <strong>Ubuntu Server 18.04 LTS. </strong>Keep the other options as default and launch the instance using your already existing Key pair or create a new Key pair. After your instance is ready to be connect via SSH, connect to your instance using Putty. You should be successfully connect to your instance as default ubuntu user.</p><h4><strong>Step 2 : Create new user and assign to sudo</strong></h4><p>Before creating user make sure your newly created instance is updated by running the following command -</p><pre>ubuntu@&lt;your-private-ip&gt;:~$sudo apt-get update</pre><p>After your system is updated run the following commands to create a new user called jupyteruser and grand sudo permission -</p><pre>sudo addgroup jupyter <br><em># Add a new group</em></pre><pre>sudo adduser --ingroup jupyter jupyteruser <br><em>#Creating jupyteruser and adding to the jupyter group</em></pre><pre>sudo adduser jupyteruser sudo<br>#Adding jupyteruser to sudo group to get administrator access</pre><p>Add the time of creating new user you will be asked to set password and enter some user details. Once the user is created, switch to the user using the following command -</p><pre>sudo su jupyteruser<br>#Switching to the new user</pre><p>We also need to create a secured key and add that to the authorized_keys file get ssh permission for localhost. To create a ssh key and add that to authorized_keys file, use the below commands -</p><pre>ssh-keygen -t rsa -P &quot;&quot;<br>#Creating a new key pair</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*fnia6ZQzPa3SToAuLN0P-A.png" /></figure><pre>cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys<br>#Append the newly created key to the authorized_keys file</pre><p>After that check if your are able to ssh to the localhost by the following command. You should be able to ssh to localhost</p><pre>ssh localhost</pre><h4><strong>Step 3 : Install Python 3 and pip</strong></h4><pre>sudo apt-get install python3</pre><pre>sudo apt-get -y install python3-pip</pre><h4>Step 4 : Install Jupyter and configure</h4><pre>sudo pip3 install jupyter<br>jupyter notebook --generate-config</pre><p>This command will create a configuration file called jupyter_notebook_config.py. Edit this file in vi editor and add the following two lines anywhere in the file and save the file.</p><pre>c.NotebookApp.allow_origin = &#39;*&#39; #allow all origins<br>c.NotebookApp.ip = &#39;0.0.0.0&#39; # listen on all IPs</pre><p>If you want to set a pasword for your jupyter notebook you can use the following command -</p><pre>jupyter notebook password</pre><h4>Step 5 : Launch Jupyter Notebook</h4><pre>jupyter notebook</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*fdTC5uDm6h95drjvyk1J6g.png" /></figure><h4>Step 6 : Access Jupyter Notebook and create notebook</h4><p>Jupyter Notebook starts at port number 8888 by default. Note your public DNS of your AWS EC2 instance and type the following URL in your web browser -</p><pre>&lt;Your-aws-ec2-instance-public-dns&gt;:8888</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*VyWnb5TNfLgNh2d4OdH1GQ.png" /></figure><p>Enter the password you have set while launching the jupyter notebook and you are done. You can create a new notebook by clicking on <strong>New — &gt; Python 3 (under notebook)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1LdnMzctOpXMpyajFqU9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f92a5b5b3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interact with AWS S3 using Python 3]]></title>
            <link>https://medium.com/@anirsom2012/interact-with-aws-s3-using-python-3-4d55089d3889?source=rss-f84177806fc8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d55089d3889</guid>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[simple-storage-service]]></category>
            <dc:creator><![CDATA[Anirban Som]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 15:24:13 GMT</pubDate>
            <atom:updated>2019-11-04T15:24:13.556Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS S3 or Simple Storage Service is a service offered by AWS that provides object storage through web interface.</p><p>Below is the architecture how Python interacts with AWS S3 –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*C88steszfvleGs07dVdYKw.png" /><figcaption>Architecture</figcaption></figure><p>AWS SDK for Python which is boto3 interacts with AWS S3. It can read data from S3 and can write data into S3.</p><p>There are many ways through which we can interact with S3. In this article we will discuss how we can interact with AWS S3 using python. But before that let us understand some of the fundamental concepts of AWS S3.</p><p>AWS S3 works with a concept called Buckets &amp; Objects. Bucket you can think of as a container and Objects is file that you store inside the Bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*prenSZ3bi75b9MPXKHo5oA.png" /><figcaption>AWS S3 Architecture</figcaption></figure><p>Steps to create Bucket in AWS S3 –</p><p>1. Go to AWS Management console.</p><p>2.Open Simple Storage Service (S3)</p><p>3. Click on Create Bucket</p><p>You need to select Region while creating a Bucket. Your Bucket name should be globally unique in the selected Region.</p><p>4. Once your bucket is created you are ready to store files (Objects) into the bucket. You can also create folders inside a bucket to arrange your data.</p><p>We will perform below activities with S3 using Python –</p><p>1. List down all the buckets present in S3</p><p>2. Upload a file from local to S3 bucket</p><p>3. Download a file from S3</p><p>4. Copy a file from one Bucket to another.</p><p>Prerequisites</p><p>1. Install Python 3.x</p><p>2. Install aws package (pip install aws) to configure Access Key and Secret Access Key</p><p>3.Install AWS SDK for Python (pip install boto3)</p><p>4. Install Python IDE (PyCharm)</p><p><strong>List down all Buckets present in S3</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*qm-yy1AzoS2JGfosFJoGVg.png" /><figcaption>Code</figcaption></figure><p>In this case I have 2 buckets in my S3. Let’s see the response dictionary returned by AWS –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*g6LOAXRPvhp4pQUA1BIMkA.png" /><figcaption>AWS Response Object</figcaption></figure><p>And, in the final output you will get a list of all the S3 buckets –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*inXY4NkFfrjnnqS0nAh2WQ.png" /><figcaption>Output</figcaption></figure><p><strong>Upload a file from local to S3 bucket</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*hTN1-nRhYVapZcnPNjHUzg.png" /><figcaption>Code</figcaption></figure><p>In this script we are copying sales_data_201903.csv’ file from our local to S3 bucket</p><p>Bucket before running the script –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*peycuhH2RUbT5PlRQnoyQw.png" /></figure><p>Bucket after running the script –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*Ih50QAPiNgjqcJA5CrREgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*gEG4Cx3gtXqJP4HySDjF-A.png" /></figure><p><strong>Download file into local driver from S3</strong></p><p>We will download the file ‘sales_data_201903.csv’ from anirban-som-main-bucket bucket into our local drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*eCxJNV2qVDJXgy1i_oysJA.png" /></figure><p>Folder before running script –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*hMhKUq8QdmwEH1q34Yt8tA.png" /></figure><p>Folder after running the script –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*iVEDo5piDhlKPMg5-I9n5A.png" /></figure><p><strong>Copy file from one Bucket to another Bucket</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*2Wj-x1bQbZtfS6RdBb5z0Q.png" /></figure><p>Here we are copying the file ‘sales_data_201903.csv’ which is present in anirban-som-main-bucket Bucket to anirban-som-sub-main-bucket Bucket.</p><p>Before running –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*ttUjO8w58GnWvSGjmD-25Q.png" /></figure><p>After running the script –</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*Qz-jiF5vE8x5_N1cRytFNg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d55089d3889" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>