<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by JayaAnanth on Medium]]></title>
        <description><![CDATA[Stories by JayaAnanth on Medium]]></description>
        <link>https://medium.com/@JayaAnanthDataConsultant?source=rss-1c1bd9619032------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4XrJK0UXXc1vvrvm</url>
            <title>Stories by JayaAnanth on Medium</title>
            <link>https://medium.com/@JayaAnanthDataConsultant?source=rss-1c1bd9619032------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:40:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@JayaAnanthDataConsultant/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nested XML Processing — AWS Glue with DynamicFrames, Relationalize, and Databricks Spark-XML]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@JayaAnanthDataConsultant/nested-xml-processing-aws-glue-with-dynamicframes-relationalize-and-databricks-spark-xml-f57494be3a63?source=rss-1c1bd9619032------2"><img src="https://cdn-images-1.medium.com/max/1200/0*P641M6-ft67MxqGN.png" width="1200"></a></p><p class="medium-feed-snippet">Introduction:</p><p class="medium-feed-link"><a href="https://medium.com/@JayaAnanthDataConsultant/nested-xml-processing-aws-glue-with-dynamicframes-relationalize-and-databricks-spark-xml-f57494be3a63?source=rss-1c1bd9619032------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@JayaAnanthDataConsultant/nested-xml-processing-aws-glue-with-dynamicframes-relationalize-and-databricks-spark-xml-f57494be3a63?source=rss-1c1bd9619032------2</link>
            <guid isPermaLink="false">https://medium.com/p/f57494be3a63</guid>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[notebook]]></category>
            <category><![CDATA[xml-schema]]></category>
            <dc:creator><![CDATA[JayaAnanth]]></dc:creator>
            <pubDate>Sun, 28 Apr 2024 02:00:41 GMT</pubDate>
            <atom:updated>2024-05-12T07:32:28.122Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg Architecture: Create Iceberg Tables with DDL & CloudFormation]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@JayaAnanthDataConsultant/apache-iceberg-architecture-create-iceberg-tables-with-ddl-cloudformation-8a4e52152562?source=rss-1c1bd9619032------2"><img src="https://cdn-images-1.medium.com/max/1230/0*zlBF6aegaUf4T5QH.png" width="1230"></a></p><p class="medium-feed-snippet">Apache Iceberg has gained significant traction in the big data world due to its robust features, including ACID transactions, schema&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@JayaAnanthDataConsultant/apache-iceberg-architecture-create-iceberg-tables-with-ddl-cloudformation-8a4e52152562?source=rss-1c1bd9619032------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@JayaAnanthDataConsultant/apache-iceberg-architecture-create-iceberg-tables-with-ddl-cloudformation-8a4e52152562?source=rss-1c1bd9619032------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a4e52152562</guid>
            <category><![CDATA[cloudformation-template]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[JayaAnanth]]></dc:creator>
            <pubDate>Fri, 19 Apr 2024 02:00:51 GMT</pubDate>
            <atom:updated>2024-04-22T13:00:45.907Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Choosing the Right Tool for the Job — AWS Glue, EMR, and EMR Serverless Explained]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@JayaAnanthDataConsultant/choosing-the-right-tool-for-the-job-aws-glue-emr-and-emr-serverless-explained-e1a8ef6a2975?source=rss-1c1bd9619032------2"><img src="https://cdn-images-1.medium.com/max/1200/0*3hk4PO3XJbrOSSVm.png" width="1200"></a></p><p class="medium-feed-snippet">Definition:</p><p class="medium-feed-link"><a href="https://medium.com/@JayaAnanthDataConsultant/choosing-the-right-tool-for-the-job-aws-glue-emr-and-emr-serverless-explained-e1a8ef6a2975?source=rss-1c1bd9619032------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@JayaAnanthDataConsultant/choosing-the-right-tool-for-the-job-aws-glue-emr-and-emr-serverless-explained-e1a8ef6a2975?source=rss-1c1bd9619032------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1a8ef6a2975</guid>
            <category><![CDATA[emr-serverless]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-emr]]></category>
            <dc:creator><![CDATA[JayaAnanth]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 01:00:14 GMT</pubDate>
            <atom:updated>2024-03-17T22:21:36.625Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Efficient XML Data Processing and Querying with AWS Glue and Athena — A Comprehensive Guide]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@JayaAnanthDataConsultant/in-todays-data-driven-landscape-the-analysis-of-xml-files-holds-significant-importance-across-3fff86096892?source=rss-1c1bd9619032------2"><img src="https://cdn-images-1.medium.com/max/1200/0*7Gk6imDYlW5NQqHX.png" width="1200"></a></p><p class="medium-feed-snippet">In today&#x2019;s data-driven landscape, the analysis of XML files holds significant importance across diverse industries such as finance&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@JayaAnanthDataConsultant/in-todays-data-driven-landscape-the-analysis-of-xml-files-holds-significant-importance-across-3fff86096892?source=rss-1c1bd9619032------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@JayaAnanthDataConsultant/in-todays-data-driven-landscape-the-analysis-of-xml-files-holds-significant-importance-across-3fff86096892?source=rss-1c1bd9619032------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fff86096892</guid>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[xml]]></category>
            <category><![CDATA[glue-catalog]]></category>
            <category><![CDATA[athena]]></category>
            <dc:creator><![CDATA[JayaAnanth]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 01:00:16 GMT</pubDate>
            <atom:updated>2024-03-17T23:02:53.219Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[An Introduction to the DBT Tool: Installation and Execution on an EC2 Instance]]></title>
            <link>https://medium.com/@JayaAnanthDataConsultant/an-introduction-to-the-dbt-tool-installation-and-execution-on-an-ec2-instance-c9831fa7ab08?source=rss-1c1bd9619032------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9831fa7ab08</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[data-build-tool]]></category>
            <dc:creator><![CDATA[JayaAnanth]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 01:00:28 GMT</pubDate>
            <atom:updated>2023-10-26T05:59:04.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E4h-xA5YnofcpGcdMnMHJQ.png" /></figure><p>DBT (data build tool) enables analytics engineers to transform data in their warehouses by simply writing select statements. DBT handles turning these select statements into tables and views. DBT does the T in ELT (Extract, Load, Transform) processes — it doesn’t extract or load data, but it’s extremely good at transforming data that’s already loaded into your warehouse.</p><p><strong>DBT Core</strong> is the software that takes a DBT project (.sql and .yml files) and a command and then creates tables/views in your warehouse. DBT Core includes a command line interface (CLI) so that users can execute DBT commands using a terminal program. DBT Core is open source and free to use.</p><p><strong>DBT Cloud</strong> is an application that helps teams use DBT. DBT Cloud provides a web-based IDE to develop DBT projects, a purpose-built scheduler, and a way to share DBT documentation with your team. DBT Cloud offers a number of features for free, as well as additional features in paid tiers (check out the pricing here).</p><p>DBT connects to your data warehouse to run data transformation queries. As such, you’ll need a data warehouse with source data loaded in it to use DBT. DBT natively supports connections to Snowflake, BigQuery, Redshift and Postgres data warehouses</p><p>A DBT project is a directory of .sql and .yml files, which DBT uses to transform your data. At a minimum, a DBT project must contain:</p><ul><li><strong>A project file:</strong> A DBT_project.yml file tells DBT that a particular directory is a DBT project, and also contains configurations for your project.</li><li>A model is a single .sql file. Each model contains a single select statement that either transforms raw data into a dataset that is ready for analytics, or, more often, is an intermediate step in such a transformation. A project may also contain a number of other resources, such as <a href="https://docs.getdbt.com/docs/build/snapshots">snapshots</a>, <a href="https://docs.getdbt.com/docs/build/seeds">seeds</a>, <a href="https://docs.getdbt.com/docs/build/tests">tests</a>, <a href="https://docs.getdbt.com/docs/build/jinja-macros#macros">macros</a>, <a href="https://docs.getdbt.com/docs/collaborate/documentation">documentation</a>, and <a href="https://medium.com/r?url=https%3A%2F%2Fdocs.getdbt.com%2Fdocs%2Fbuilding-a-dbt-project%2Fusing-sources">sources</a>.</li></ul><p><strong>To use the CLI:</strong></p><p>Prerequisite — Create the EC2 Machine</p><p><strong>Steps to Set up DBT core</strong> (i) Install DBT in Amazon EC2 Machine. For this demo installing DBT-snowflake Inorder to install DBT we need pip so first install pip</p><p>Verify the DBT version</p><p><strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/0*48atr84H_VqRXQOC.png" /></figure><p>(ii) Initialise the DBT init (DBT init [project-name])and provide the connection details or it can be manually updated too <strong>(/root/.DBT/profiles.yml)</strong></p><p><strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*116M6Z7zSVGu5AeF.png" /></figure><p>(iii) After the DBT init command the list of module folder will be created ls -ltr</p><p><strong>Result:</strong></p><p>(iv) To verify the credential details go to profile.yml file</p><p><strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*5BPa5huz8OyVWitN.png" /></figure><p>(v) run DBT debug to verify the connectivity</p><p><strong>Command:</strong> <strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0d8Jy0AzisUM7eJR.png" /></figure><p>(vi) Verify the dbt_profile .yml file contains all the details of the project</p><p><strong>Command:</strong> <strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2L8qGa18IHAVFWH3.png" /></figure><p>A model is a select statement. Models are defined in .sql files (typically in your models directory): Each .sql file contains one model / select statement The name of the file is used as the model name Models can be nested in subdirectories within the models directory When you execute the <a href="https://docs.getdbt.com/reference/commands/run">DBT run </a>command, DBT will build this model in your data warehouse by wrapping it in a create view as or create table as statement.</p><p>Materializations are strategies for persisting DBT models in a warehouse. There are four types of materializations built into DBT. They are:</p><p>(vii) Under the model please write the Select statement and declare the materialised as view or table</p><p>(vii) Run DBT run command to execute the</p><p>Command:</p><p><strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7NmHumWiVLwDw5rm.png" /></figure><p>(viii) Verify the first and second model created in the snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KO7TyvY5JcVtLxH4.png" /></figure><p>In this post, we discussed how to set up a DBT on an EC2 instance and connect to the snowflake and implemented a simple transformation logic by creating a new table in the Database.</p><p>DBT tool is used to Create the table or view based on the select query. With DBT, data teams work directly within the warehouse to produce trusted datasets for reporting, ML modeling, and operational workflows.</p><p><strong>If you enjoy the article, Please Follow me.</strong></p><p><em>Originally published at </em><a href="https://datainevitable.com/posts/dbtec2/"><em>https://datainevitable.com</em></a><em> on October 6, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9831fa7ab08" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>