<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arun Kumar Nagaraj on Medium]]></title>
        <description><![CDATA[Stories by Arun Kumar Nagaraj on Medium]]></description>
        <link>https://medium.com/@aruun1995?source=rss-4cbee88ce9e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Y-mZrohzl5wVlgsc.jpg</url>
            <title>Stories by Arun Kumar Nagaraj on Medium</title>
            <link>https://medium.com/@aruun1995?source=rss-4cbee88ce9e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:26:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aruun1995/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Beginner-Friendly Walkthrough of EDA: Exploring the Titanic Dataset]]></title>
            <link>https://medium.com/@aruun1995/a-beginner-friendly-walkthrough-of-eda-exploring-the-titanic-dataset-a0f633deb178?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0f633deb178</guid>
            <category><![CDATA[eda]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 01:53:19 GMT</pubDate>
            <atom:updated>2026-08-18T01:53:19.403Z</atom:updated>
            <content:encoded><![CDATA[<p>Exploratory Data Analysis (EDA) is the first real conversation you have with your data — before modeling, before conclusions, before assumptions. In this post, I’ll walk through how I explored the classic Titanic dataset using <strong>Pandas</strong>, <strong>NumPy</strong>, <strong>Matplotlib</strong>, and <strong>Seaborn</strong>, covering the full journey from raw CSV to visual insight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hl_-BtG7lKk5Xrw3" /></figure><h3>1. Setting Up the Environment</h3><p>Every EDA project starts the same way: getting the right tools in place.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><ul><li><strong>Pandas</strong> for loading and manipulating tabular data</li><li><strong>NumPy</strong> for numerical operations</li><li><strong>Matplotlib</strong> and <strong>Seaborn</strong> for visualization</li></ul><h3>2. Loading and Getting to Know the Data</h3><pre>df = pd.read_csv(&quot;titanic.csv&quot;)<br>df.head()<br>df.tail()</pre><p>The Titanic dataset has <strong>891 rows and 12 columns</strong>, covering passenger details like age, sex, ticket class, fare, and whether they survived.</p><p>A few commands go a long way in understanding a new dataset:</p><pre>df.describe()   # statistical summary of numeric columns<br>df.columns      # list of all column names<br>df.dtypes       # data type of each column<br>df.info()       # combined view of non-null counts + dtypes<br>df.nunique()    # number of unique values per column</pre><p>df.describe() immediately tells you things like the average passenger age (~29.7 years) and the wide spread in ticket fares — useful hints about where to dig deeper later.</p><h3>3. Checking for Duplicates and Missing Values</h3><p>Before drawing any conclusions, you need to know how “clean” your data actually is:</p><pre>df.duplicated()<br>df.isnull()<br>print(df.isnull().sum())</pre><p>This revealed three columns with missing data:</p><p>Column Missing Values Age 177 Cabin 687 Embarked 2</p><p>Cabin was missing in over 77% of rows — a strong signal that this column needed special handling rather than a simple fill.</p><h3>4. Data Cleaning</h3><p>With the gaps identified, the next step is deciding how to handle each one:</p><p><strong>Cabin</strong> — too sparse to be reliably imputed, so after testing a placeholder fill (&quot;bbbb&quot;), the column was dropped entirely:</p><pre>df.drop([&quot;Cabin&quot;], axis=1, inplace=True)</pre><p><strong>Embarked</strong> — only 2 missing values, dropped for simplicity:</p><pre>df.drop([&quot;Embarked&quot;], axis=1, inplace=True)</pre><p><strong>Age</strong> — missing values filled with the column mean, a common strategy for numeric fields with a manageable number of gaps:</p><pre>df[&#39;Age&#39;].fillna(df[&#39;Age&#39;].mean(), inplace=True)</pre><p><strong>Sex</strong> — standardized the text values for readability:</p><pre>df[&#39;Sex&#39;] = df[&#39;Sex&#39;].replace(&#39;male&#39;, &#39;Male&#39;)<br>df[&#39;Sex&#39;] = df[&#39;Sex&#39;].replace(&#39;female&#39;, &#39;Female&#39;)</pre><blockquote><strong><em>Takeaway:</em></strong><em> Cleaning decisions aren’t one-size-fits-all. A column missing 2 values behaves very differently than one missing 77% — treat them accordingly.</em></blockquote><h3>5. Data Transformation &amp; Feature Engineering</h3><p>Once the data is clean, small transformations can reveal new angles:</p><pre>df[&#39;Age&#39;] = df[&#39;Age&#39;].astype(&#39;int&#39;)          # convert to integer for readability<br>df[&#39;Family_count&#39;] = df[&#39;SibSp&#39;] + df[&#39;Parch&#39;]  # engineer a &quot;family size&quot; feature</pre><p>Grouping the data also surfaces patterns before any chart is drawn:</p><pre>grouped_by = df.groupby(&#39;Pclass&#39;)[&#39;Pclass&#39;].count()</pre><pre>sur_not = df.groupby([&#39;Survived&#39;, &#39;Sex&#39;])[&#39;PassengerId&#39;].count()</pre><p>This last line quietly tells a big part of the Titanic story:</p><p>Survived Sex Count No Female 81 No Male 468 Yes Female 233 Yes Male 109</p><p>Female passengers survived at roughly <strong>74%</strong>, compared to about <strong>19%</strong> for male passengers — a striking gap that sets up the visualizations perfectly.</p><h3>6. Data Visualization</h3><p>This is where the story becomes visible. A few well-chosen charts do more than any table of numbers.</p><p><strong>Gender vs. Survival Rate</strong> — a bar plot makes the survival gap immediate and intuitive:</p><pre>plt.figure(figsize=(8,6))<br>sns.barplot(x=&#39;Sex&#39;, y=&#39;Survived&#39;, data=df)<br>plt.title(&quot;Gender vs Survival Rate&quot;)<br>plt.xlabel(&quot;Gender&quot;)<br>plt.ylabel(&quot;Survived&quot;)<br>plt.show()</pre><p><strong>Age Distribution</strong> — a histogram (with an optional KDE curve) shows the shape of the passenger age range:</p><pre>plt.figure(figsize=(8,6))<br>sns.histplot(df[&#39;Age&#39;], bins=20, kde=True)</pre><p><strong>Age vs. Fare</strong> — a scatter plot to check for any relationship between a passenger’s age and what they paid:</p><pre>sns.scatterplot(x=&quot;Age&quot;, y=&quot;Fare&quot;, data=df)</pre><p><strong>Fare Trend Across Passengers</strong> — a line plot to eyeball fare variation across passenger IDs:</p><pre>sns.lineplot(x=&quot;PassengerId&quot;, y=&quot;Fare&quot;, data=df)</pre><p>Each chart type served a different purpose:</p><ul><li><strong>Bar plots</strong> → comparing a metric across categories (Sex, Pclass)</li><li><strong>Histograms</strong> → understanding the distribution/spread of a numeric variable</li><li><strong>Scatter plots</strong> → spotting relationships between two numeric variables</li><li><strong>Line plots</strong> → observing trends across an ordered sequence</li></ul><h3>Key Takeaways</h3><ol><li><strong>Always start with the shape and structure</strong> of your data (.info(), .describe(), .dtypes) before touching anything.</li><li><strong>Quantify missingness first</strong> — the right cleaning strategy depends on <em>how much</em> is missing, not just <em>that</em> something is missing.</li><li><strong>Group-bys are a cheap way to preview a story</strong> before you ever plot it — the Sex/Survived breakdown told us most of what the bar chart later confirmed.</li><li><strong>Match the chart to the question</strong>: category comparisons, distributions, relationships, and trends each call for a different plot type.</li></ol><p>EDA isn’t just a preprocessing step — it’s where you build intuition about your data, catch quality issues early, and find the questions worth answering with a model later.</p><p><em>Dataset: Titanic passenger data (891 rows, 12 columns). Tools: Python, Pandas, NumPy, Matplotlib, Seaborn.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0f633deb178" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Storage to Strategy: Integrating S3 Tables with AWS Glue for Business Success]]></title>
            <link>https://medium.com/@aruun1995/from-storage-to-strategy-integrating-s3-tables-with-aws-glue-for-business-success-ec49efb1d749?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec49efb1d749</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[s3-tables]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Sun, 04 May 2025 17:23:08 GMT</pubDate>
            <atom:updated>2025-05-04T17:24:18.693Z</atom:updated>
            <content:encoded><![CDATA[<p>As businesses grow increasingly data-driven, traditional storage models are no longer enough. Amazon S3 has long been the cornerstone of cloud-based object storage, but as analytics demands rise, organizations need more than just a data lake — they need structure, performance, and automation. That’s where <strong>Amazon S3 Tables</strong> come in.</p><p>S3 Tables represent a strategic evolution from storage to insight. Built on <strong>Apache Iceberg</strong>, S3 Tables offer a fully managed way to store tabular data on Amazon S3, with schema and metadata management, transactional consistency, and integration with the broader AWS analytics ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*vaCp80nMhVyIKk1l--sPbw.png" /></figure><p><strong>S3 vs. S3 Tables: Why the Difference Matters</strong></p><p>Amazon S3 is a general-purpose object storage solution, ideal for storing everything from images to logs to large datasets. However, when it comes to querying structured data efficiently or managing data versioning, traditional S3 has its limitations.</p><p><strong>S3 Tables</strong>, on the other hand, are purpose-built for tabular data. They support <strong>Apache Iceberg format</strong> and are fully managed by AWS. Each S3 Table has a dedicated S3 location used to store both data and metadata files. What’s important is that these tables disallow direct deletes and overwrites. This design ensures <strong>data consistency and integrity</strong>, a crucial factor in high-stakes analytical workflows.</p><p>Instead of manual cleanup, <strong>policy-driven maintenance</strong> handles data aging, and <strong>AWS CloudTrail</strong> provides a detailed audit trail of all table-level actions.</p><p><strong>Performance That Scales with Your Business</strong></p><p>One of the most significant advantages of S3 Tables is <strong>performance</strong>. They are capable of handling up to <strong>10x more transactions per second</strong> compared to standard S3 prefixes. Traditional S3 buckets support around 5,500 reads and 3,500 writes per second, whereas S3 Tables scale up to <strong>55,000 reads and 35,000 writes per second</strong>.</p><p>This boost is further enhanced by <strong>automated compaction</strong>, which periodically consolidates smaller Parquet files into larger ones. This process reduces the number of read operations needed during a query, resulting in <strong>up to 3x faster query performance</strong>. Unlike older setups where teams had to spin up compute clusters for compaction, S3 Tables handle it all in the background, completely managed by AWS.</p><p><strong>Security and Logical Data Organization</strong></p><p>Security and data governance are front and center with S3 Tables. You can organize data using <strong>namespaces</strong>, which are logical groupings of tables within a table bucket. This allows for cleaner data organization, better access control, and easier table management, especially in enterprise environments where multiple teams interact with the same storage layer.</p><p><strong>Cost Optimization Through Automated Maintenance</strong></p><p>Apache Iceberg enables powerful features like <strong>time travel</strong>, where you can view historical versions of a table. Every update results in a new snapshot, which is great for auditability — but over time, these snapshots accumulate and increase storage costs.</p><p>In standard S3, you’d have to manually track and delete old snapshots. With S3 Tables, <strong>automated maintenance runs</strong> take care of:</p><ul><li><strong>Snapshot expiration</strong>: Removing outdated snapshots.</li><li><strong>Unreferenced file removal</strong>: Cleaning up files not used in any current snapshot.</li></ul><p>These maintenance tasks are <strong>policy-driven</strong> and run in the background, keeping your storage lean and your operations efficient without manual effort.</p><p><strong>Seamless Integration with the AWS Analytics Ecosystem</strong></p><p>At the core of AWS analytics lies the <strong>AWS Glue Data Catalog</strong>, a centralized index of an organization’s datasets. With general-purpose S3, you often need to manually register tables in Glue. But with S3 Tables, registration is <strong>automatic</strong>.</p><p>Once available in the Glue catalog, S3 Tables can be:</p><ul><li>Queried instantly in <strong>Amazon Athena</strong></li><li>Analyzed in <strong>Amazon Redshift</strong></li><li>Ingested from <strong>Amazon Kinesis Data Firehose</strong></li><li>Visualized through <strong>Amazon QuickSight</strong></li><li>Integrated with logs, IoT, and other streaming data sources</li></ul><p>This frictionless integration means your data becomes immediately <strong>discoverable and actionable</strong> across your analytics stack.</p><p><strong>Demo:</strong></p><ol><li>You can create S3 tables, by navigating to AWS Console, S3 → <strong>Table Buckets</strong>. Click on <strong>Enable Integration </strong>and hit on <strong>Create table bucket.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iDEQJzYibwnBXwE1jfvKOA.png" /></figure><p>2. Once hitting <strong>Create table bucket, </strong>you need to enter the<strong> table bucket name </strong>and hit on<strong> create table bucket.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wR1Vc9dJXTWaU-kguhrc-w.png" /></figure><p>3. You may have a similar screenshot like the following after creating the table bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lo93biJC-jCH7mKTc_1i_A.png" /></figure><p>4. Navigate to AWS Glue → ETL Jobs → Create Job (Spark Job), copy the <strong>Amazon Resource Name(ARN) </strong>of the table bucket.</p><p>5. Download the required jar dependency from the following link:</p><p><a href="https://mvnrepository.com/artifact/software.amazon.s3tables/s3-tables-catalog-for-iceberg-runtime">https://mvnrepository.com/artifact/software.amazon.s3tables/s3-tables-catalog-for-iceberg-runtime</a></p><p>6. Navigate to the AWS Glue job, and add it under the Dependent JAR path in the job details section of AWS Glue.</p><p>7. In the code editor, create a Spark session like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHt48l15MQ_3u6hD6CUhHQ.png" /></figure><p>8. You can run similar queries like SQL using spark session. The basic table creation in S3 tables and inserting into the S3 tables have been added here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*HrK9puJubaUjnNc8Beqg2w.png" /></figure><p>9. In PROD scenario’s, we may encounter with multiple files(may be CSV/Parquet), to handle with file, you can read the file in similar way like how you do in spark and write it to s3 tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKbyMD2NBMW-7XY2JWaXgw.png" /></figure><p>10. Once the data have been written into S3 tables, you can get into S3 tables, and you can see the table got registered in S3 tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4r4f-ufmbTxVjKyg5C9zg.png" /></figure><p>11. You can also query the data using <strong>Query table with Athena.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uv6q9Dg6Awx4QK6wZubGDQ.png" /></figure><p><strong>Conclusion:</strong></p><p>The shift from storage to strategy starts with tools like Amazon S3 Tables. With built-in performance optimizations, automated maintenance, tight integration with AWS analytics services, and full management of schema and transactions, S3 Tables offer a powerful way to store, manage, and analyze your data at scale.</p><p>No more worrying about metadata management, compaction jobs, or manual table registration. With S3 Tables and AWS Glue, you can focus on what matters most: delivering insights that drive business success.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec49efb1d749" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLFlow on AWS EC2 Server vs MLFlow on AWS Sagemaker AI]]></title>
            <link>https://medium.com/@aruun1995/mlflow-on-aws-ec2-server-vs-mlflow-on-aws-sagemaker-ai-3b0ccf1d23d9?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b0ccf1d23d9</guid>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[mlflow]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 09:37:30 GMT</pubDate>
            <atom:updated>2025-03-14T09:37:30.049Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Machine learning model lifecycle management is crucial for any ML-driven organization. MLflow, an open-source platform for managing the end-to-end ML lifecycle, is widely used for tracking experiments, packaging models, and deploying them to production. When deploying MLflow on AWS, two common choices arise, setting it up on an EC2 server or leveraging the managed ML capabilities of AWS SageMaker.</p><p>This blog compares MLflow on EC2 and MLflow on SageMaker AI in terms of setup, cost, scalability, security, integration, and ease of use to help you decide which best suits your use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*gei7qtVUKhrfJWUj" /></figure><p>Setting Up MLflow</p><h3>MLflow on EC2</h3><p>Setting up MLflow on an EC2 instance involves the following steps:</p><ol><li><strong>Provision an EC2 instance:</strong> Select an appropriate instance type (e.g., t3.medium for small workloads or p3.2xlarge for GPU-based training).</li><li><strong>Install MLflow:</strong> Use pip install mlflow to install MLflow.</li><li><strong>Configure storage:</strong> Use Amazon S3 for artifact storage or set up an EBS volume.</li><li><strong>Set up a database:</strong> Use an RDS instance (PostgreSQL, MySQL) for MLflow’s tracking database.</li><li><strong>Expose MLflow UI:</strong> Configure security groups and use a reverse proxy like Nginx to serve MLflow’s UI.</li></ol><h3>MLflow on SageMaker</h3><p>AWS SageMaker provides a managed experience for MLflow with built-in experiment tracking. To use MLflow on SageMaker:</p><ol><li><strong>Enable SageMaker Experiments:</strong> Use the AWS Console or SDK to create experiments.</li><li><strong>Use MLflow in a SageMaker Notebook:</strong> Install MLflow and integrate it with SageMaker tracking.</li><li><strong>Leverage built-in model registry:</strong> SageMaker handles model lineage and tracking natively.</li></ol><h3>Cost Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*Qrf3wgGfki63cKrx8TFqTQ.png" /></figure><p><strong>MLflow on EC2</strong></p><ul><li><strong>Scalability:</strong> Requires manual scaling using Auto Scaling Groups.</li><li><strong>Performance Optimization:</strong> Customizable instance selection but requires tuning.</li><li><strong>Parallel Execution:</strong> This can be achieved with Kubernetes (EKS) or EMR for distributed training.</li></ul><p><strong>MLflow on SageMaker</strong></p><ul><li><strong>Auto Scaling:</strong> Built-in auto-scaling for training and inference.</li><li><strong>Optimized Compute:</strong> Supports GPU instances for deep learning workloads.</li><li><strong>Distributed Training:</strong> Native support for distributed training with frameworks like TensorFlow, PyTorch, and Hugging Face.</li></ul><h3>Security &amp; Compliance</h3><p><strong>MLflow on EC2</strong></p><ul><li>Requires IAM role setup and security group configurations.</li><li>Data encryption must be manually enabled.</li><li>Compliance with industry standards depends on custom configuration.</li></ul><p><strong>MLflow on SageMaker</strong></p><ul><li>IAM roles are managed with fine-grained permissions.</li><li>Data encryption is enabled by default.</li><li>Compliance with industry standards like HIPAA, SOC, and GDPR is built-in.</li></ul><h3>Integration with AWS Services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*leqSityDi1I-7GJmanIXFQ.png" /></figure><h3>Ease of Use &amp; Maintenance</h3><p><strong>MLflow on EC2</strong></p><ul><li>Requires <strong>manual setup</strong> and maintenance.</li><li>Needs <strong>patching, monitoring, and scaling</strong>.</li><li>More control over the environment.</li></ul><p><strong>MLflow on SageMaker</strong></p><ul><li><strong>Fully managed</strong> with minimal setup.</li><li><strong>Automatic scaling and patching.</strong></li><li><strong>Tighter integration</strong> with SageMaker workflows.</li></ul><h3>Use Cases &amp; Recommendations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*bUpjqqTMoUI3wfV6Uz1dew.png" /></figure><h3>Conclusion</h3><p>Both MLflow on EC2 and MLflow on SageMaker AI have their strengths. If you require full control over your infrastructure, flexible cost management, and custom integrations, EC2 is a better choice. However, if you prefer a fully managed, scalable, and secure environment with minimal maintenance, SageMaker is the way forward.</p><p>Ultimately, the choice depends on the organization’s ML needs, budget, and preference for hands-on vs. managed solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b0ccf1d23d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enabling AWS Glue Data Quality]]></title>
            <link>https://medium.com/@aruun1995/aws-glue-data-quality-e0a460ce321b?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0a460ce321b</guid>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Sat, 30 Nov 2024 14:28:40 GMT</pubDate>
            <atom:updated>2025-05-25T14:09:27.788Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s data-driven world, organizations depend heavily on accurate, consistent, and reliable data to make informed decisions. <strong>Data quality</strong> refers to the degree to which data meets the standards required for its intended use. High-quality data is characterized by attributes such as <strong>accuracy, completeness, consistency, timeliness, and relevance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tfBtH0eD3COack3s.jpg" /></figure><p>When data quality is compromised, the consequences can be significant: from poor decision-making to operational inefficiencies, regulatory non-compliance, and financial losses. For example, a retail company using outdated inventory data might overstock certain products while understocking others, leading to lost sales opportunities and increased costs.</p><p>Achieving high data quality isn’t just about correcting errors — it’s about embedding quality checks throughout the data lifecycle. This is where <strong>AWS Glue Data Quality</strong> steps in, enabling organizations to define, monitor, and enforce data quality rules seamlessly within their ETL workflows.</p><p>In this blog, we’ll explore the key aspects of data quality, discuss the challenges it addresses, and demonstrate how AWS Glue Data Quality can help you ensure that your ETL pipelines deliver trustworthy data.</p><p><strong>Why Data Quality is Important?</strong></p><p>In an era where decisions are driven by data, the importance of <strong>data quality</strong> cannot be overstated. Poor-quality data can have far-reaching consequences, impacting not only day-to-day operations but also long-term strategic goals. Here are some key reasons why ensuring high data quality is critical:</p><ol><li><strong>Better Decision-Making</strong>: Decisions are only as good as the data they’re based on. High-quality data provides accurate insights, enabling organizations to make informed choices that lead to success. Conversely, inaccurate or incomplete data can result in costly mistakes.</li><li><strong>Improved Operational Efficiency</strong>: Clean and reliable data reduces the need for constant rework and troubleshooting. When teams can trust their data, processes run smoothly, saving both time and resources.</li><li><strong>Enhanced Customer Experience</strong>: Inaccurate customer data can lead to poor experiences — such as sending promotions to the wrong audience or failing to meet customer expectations. Reliable data ensures better targeting, personalization, and service delivery.</li><li><strong>Regulatory Compliance</strong>: Industries like finance, healthcare, and e-commerce face stringent data regulations. Poor data quality can lead to compliance violations, resulting in hefty fines and reputational damage.</li><li><strong>Cost Reduction</strong>: Bad data can be expensive. According to research, poor data quality costs organizations an average of 15–20% of their revenue. This stems from inefficiencies, errors, and missed opportunities caused by unreliable information.</li><li><strong>Building Trust</strong>: Stakeholders — whether internal teams, customers, or partners — need to trust the data they interact with. High-quality data strengthens this trust, fostering better collaboration and credibility.</li></ol><p><strong>Example</strong>: A logistics company relying on outdated address data might deliver packages to incorrect locations, incurring higher costs and frustrating customers. By ensuring data quality, such scenarios can be avoided, ensuring efficient delivery and customer satisfaction.</p><p>By prioritizing data quality, organizations can unlock the true potential of their data, ensuring it serves as a valuable asset rather than a liability.</p><p><strong>Introduction to AWS Glue Data Quality:</strong></p><p><strong>AWS Glue Data Quality</strong>, launched in December 2022 reInvent as a preview version, later AWS Glue Data Quality globally available on June 2023, is a powerful feature that empowers organizations to ensure their data meets critical quality standards throughout its lifecycle. Integrated seamlessly with AWS Glue, this capability allows users to define, monitor, and validate data quality rules directly within their <strong>ETL pipelines</strong> and for <strong>data at rest</strong>.</p><p>AWS Glue Data Quality is built on the <strong>DeeQu framework</strong>, an open-source library developed by AWS for data quality validation on large datasets. This integration allows Glue Data Quality to leverage DeeQu’s declarative approach to specifying data quality constraints, supporting comprehensive validation metrics like completeness, uniqueness, and validity.</p><p>AWS Glue Data Quality can work at two levels:<br>- ETL Pipelines<br>- At Rest</p><h4>Significance in ETL Pipelines</h4><p>Data quality issues often arise during extraction, transformation, and loading (ETL) processes. By incorporating data quality checks at this stage, AWS Glue Data Quality ensures that only accurate and consistent data moves through the pipeline. This proactive approach prevents poor-quality data from propagating to downstream systems, reducing the risk of errors and inefficiencies.</p><p>Key benefits include:</p><ul><li><strong>Automated Quality Rules</strong>: Define rules to assess data completeness, uniqueness, consistency, and more.</li><li><strong>Streamlined Monitoring</strong>: Continuously monitor data quality metrics during ETL workflows.</li><li><strong>Immediate Feedback</strong>: Detect and address quality issues in real-time, minimizing downstream impact.</li></ul><h4>Ensuring Quality for Data at Rest</h4><p>Data quality isn’t just about the ETL process — it’s equally important for <strong>data stored in data lakes and warehouses</strong>. AWS Glue Data Quality enables users to run checks on stored data, ensuring it remains reliable over time. This is crucial for building trust in analytics, AI/ML models, and decision-making systems that depend on historical data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BSL6k_KHe_BUoxO9NvtZbQ.png" /></figure><p>With AWS Glue Data Quality, organizations gain a comprehensive framework for:</p><ul><li><strong>Validating Historical Data</strong>: Run rules against stored datasets to ensure ongoing accuracy and relevance.</li><li><strong>Simplifying Compliance</strong>: Maintain high-quality data to meet regulatory requirements and audits.</li><li><strong>Building Confidence in Insights</strong>: Ensure analytical outputs are based on trustworthy data.</li></ul><p>By integrating AWS Glue Data Quality into ETL workflows and data-at-rest processes, organizations can transform their data from a liability into a reliable asset that drives business success.</p><p><strong>Key Terminologies:</strong></p><ul><li>DQDL</li><li>Rule</li><li>Ruleset</li><li>Data Quality</li><li>Data Quality Score</li><li>Analyzer</li><li>Observation</li></ul><ol><li><strong>DQDL</strong><br>- DQDL stands for Data Quality Definition Language<br>- Domain-specific language that you use to define rules for AWS Glue Data Quality.</li><li><strong>Rule<br></strong>A DQDL expression that checks data for a specific characteristic and returns a Boolean value.<br>Currently, there are 27 rules available in AWS Glue Data quality. Will see all the rules in the later section.</li><li><strong>Ruleset<br></strong>AWS Glue resource that comprises a set of data quality rules.<br>More than one rule and defining within a rules array is a ruleset.</li><li><strong>Data Quality<br></strong>AWS Glue Data Quality evaluates rules against a dataset to measure data quality. Each rule checks for particular characteristics like data freshness or integrity.</li><li><strong>Data Quality Score<br></strong>The percentage of data quality rules that pass (result in true) when you evaluate a ruleset with AWS Glue Data Quality.</li><li><strong>Analyzer<br></strong>Analyzer gathers data statistics that can be used by ML algorithms to detect anomalies and hard-to-detect data quality issues over time.</li><li><strong>Observation<br></strong>An unconfirmed insight generated by AWS Glue by analyzing data statistics gathered from rules and analyzers over time.</li></ol><p><strong>Data Quality in Transit vs At-Rest:</strong></p><p>The below table illustrates the common differences between Data Quality at Rest and Data Quality in transit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gnc1JF1yrRftLzbIVsIt1g.png" /></figure><p><strong>How to enable Data Quality at Rest in AWS Glue?</strong></p><ol><li>Navigate to AWS Glue, run a crawler on the dataset, and tables should be available on the data catalog.</li><li>Get into the table within the AWS Glue data catalog, you can see a similar screen.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q9n86k_lYgPJQfseHa2g1g.png" /></figure><p>3. Get into Data Quality and click on <strong>Create data quality rules</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6IIyRgmV08Ah89gWf_L8pw.png" /></figure><p>4. Choose IAM role and click on <strong>Recommend Rules</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*-bztMliCHcXXl1TtVPgx9A.png" /></figure><p>5. Based on the dataset size, it will scan the data and generate rule recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JB4aWEuhXrjaYIZR3FiQDQ.png" /></figure><p>6. Once the scan has been completed, you can see the following screen and you can select the rules that need to be applied/edited and click on insert rule recommendation, which will be added to your ruleset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8rDcvjYUfeU1_lQUZtXDQ.png" /></figure><p>7. After adding the rule, the rules array will become like the following and click on<strong> Save ruleset</strong>.</p><p>8. Run the ruleset, which you saved, and click on Run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0y2c4PRNxpl7BGj8gdIKtA.png" /></figure><p>9. Data quality will be performed on the dataset and you can see the results of data quality like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6uoaiymXeKwkCF6u2kcRQ.png" /></figure><p><strong>How to enable Data Quality in Transit at AWS Glue?</strong></p><p>Get into AWS Glue ETL jobs, Data Quality in transit can be enabled by writing own script by importing libraries.</p><p>Another way, it can be enabled by using Visual ETL. In this blog, we will see how we can enable via script.</p><ol><li>Import the following important statement into your script.<br>from awsgluedq.transform import EvaluateDataQuality</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iYVwzGhKzP8GbR6qUt4Tw.png" /></figure><p>2. AWS Glue Data Quality at Rest recommended rules for the dataset, but when it comes to in-transit, we may need to very well aware about the data and it needs to be made manually. No rule recommendation is allowed on in-transit.</p><p>3. Evaluate the dataset against the list of rules(ruleset) defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*beRziye9i4pb1umbfxlEKQ.png" /></figure><p>4. You can get into data quality tab of your glue ETL jobs and see how the data quality on the dataset have been occured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kbt68CeSpr5ZrY1q2HQv_Q.png" /></figure><p>5. The more details of the job run and list of the rules passed/failed, can be seen on the above tab.</p><p><strong>Limitations of AWS Glue Data Quality:</strong></p><p>●Nested struct or list type is not supported</p><p>●Ruleset size shouldn’t exceed 65 kb.</p><h3>Conclusion</h3><p>Ensuring data quality is no longer an optional task — it’s a fundamental requirement for organizations aiming to derive meaningful insights from their data. AWS Glue Data Quality offers a powerful, automated, and scalable solution to maintain high-quality datasets, whether during ETL processes or for data at rest. Built on the robust <strong>DeeQu framework</strong>, it brings advanced validation, anomaly detection, and monitoring capabilities to your data pipelines, ensuring your business decisions are backed by reliable and trustworthy data.</p><p>By integrating AWS Glue Data Quality into your workflows, you can transform your data into a strategic asset, reduce operational risks, and unlock the true potential of analytics and AI. Whether you’re a seasoned data engineer or just beginning your cloud journey, AWS Glue Data Quality provides the tools you need to uphold the integrity of your data.</p><p>Start leveraging AWS Glue Data Quality today and experience the confidence of making data-driven decisions powered by clean, accurate, and consistent data.</p><p>Thanks for reading out ❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0a460ce321b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Lake on AWS Glue]]></title>
            <link>https://medium.com/@aruun1995/delta-lake-on-aws-glue-c6a32ee99050?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6a32ee99050</guid>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 11:35:16 GMT</pubDate>
            <atom:updated>2024-10-15T11:38:49.022Z</atom:updated>
            <content:encoded><![CDATA[<p>Delta Lake is an open-source data lake <strong>storage layer</strong> that provides ACID(Atomicity, Consistency, Integrity, and Durability) transactions, versioning, schema enforcement, and other advanced features on top of the cloud and big data storage systems like <strong>Amazon S3, Azure Data Lake Storage, and Apache Hadoop</strong>. Databricks originally developed it and is now an open-source project under the Linux Foundation.</p><p>Delta Lake enables organizations to store and manage large-scale data sets, including structured, semi-structured, and unstructured data, with high reliability, performance, and scalability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*pL1H5rBjEt285AldYpN6Aw.png" /></figure><p><strong>To enable Delta Lake for AWS Glue:</strong></p><p>Add the following configuration int the job parameters to use the delta lake features inside the aws glue</p><p>datalake-formats — delta<br> — conf — spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension</p><p>— conf — spark.sql.catalog.spark_catalog = org.apache.spark.sql.delta.catalog.DeltaCatalog</p><p><strong>Basic Operations in Delta:</strong></p><p>The available modes to handle big data in the Delta table:</p><ul><li>overwrite: Overwrite all the contents of the Delta table with the new data.</li><li>append: Append the new data to the existing data in the Delta table.</li><li>ignore: Do nothing if the Delta table already exists.</li><li>error: Raise an error if the Delta table already exists.</li></ul><ol><li><strong>Creating Delta Tables:</strong></li></ol><p>Table creation can be done in two ways</p><ul><li>without inserting data</li><li>with inserting data</li></ul><p><strong>a. Without inserting data:</strong></p><pre>schema = “id INT, name STRING, age INT”<br>spark.sql(f”CREATE TABLE delta.`s3://&lt;bucket_name&gt;/deltaLake/delta_table1` ({schema}) USING DELTA “)</pre><p><strong>b. With Inserting data:</strong></p><p>Load CSV file from S3 bucket. To load the file to the delta table first, we have to convert the CSV file to a data frame and then we load the data frame to the delta table.</p><pre>dataFrame = spark.read.format(&quot;csv&quot;).load(&quot;s3://&lt;bucket_name&gt;/deltaLake/delta_table/addresses.csv&quot;)<br>dataFrame.write.format(&quot;delta&quot;).mode(&quot;overwrite&quot;).save(&quot;s3://&lt;bucket_name&gt;s/deltaLake/delta_table&quot;)</pre><p><strong>2. Insert data into Delta Tables</strong></p><p>You can insert data into a Delta table using the DataFrame.write API in PySpark.Here&#39;s an example code snippet that shows how to do this</p><ul><li><strong>Using python:</strong></li></ul><pre>from pyspark.sql import SparkSession <br><br>spark = SparkSession.builder.appName(&quot;MyApp&quot;).getOrCreate() <br><br><br># create a DataFrame to insert <br>data = [(&quot;Alice&quot;, 25), (&quot;Bob&quot;, 30), (&quot;Charlie&quot;, 35)] <br>df = spark.createDataFrame(data, [&quot;name&quot;, &quot;age&quot;]) <br><br><br># write the DataFrame to a Delta table <br>df.write.format(&quot;delta&quot;).mode(&quot;append&quot;).save(&quot;/path/to/delta-table&quot;)</pre><ul><li>If your data doesn’t have column names, If you want to specify the column names you print the schema of the table it gives the default column name for the table you can use that</li></ul><pre>from pyspark.sql import SparkSession <br>from pyspark.sql.types import StructType, StructField, StringType, IntegerType <br>spark = SparkSession.builder.appName(&quot;MyApp&quot;).getOrCreate() <br><br># create a DataFrame to insert <br>data = [(&quot;Alice&quot;, 25), (&quot;Bob&quot;, 30), (&quot;Charlie&quot;, 35)] <br>df = spark.createDataFrame(data, [&quot;_c0&quot;,&quot;_c1&quot;]) <br><br># write the DataFrame to a Delta table <br>df.write.format(&quot;delta&quot;).mode(&quot;append&quot;).save(&quot;/path/to/delta-table&quot;)</pre><ul><li><strong>Using spark.sql:</strong></li></ul><pre>spark.sql(&quot;INSERT INTO delta.`s3://&lt;bucket_name&gt;/deltaLake/delta_table` VALUES (&#39;value1&#39;, &#39;value2&#39;, &#39;value3&#39;,&#39;value4&#39;,&#39;value5&#39;,&#39;value6&#39;)&quot;)</pre><p><strong>3. Delete Delta Tables:</strong></p><ul><li><strong>Using spark.sql</strong></li></ul><pre>spark.sql(f&quot;DELETE FROM delta.`{table_path}` where _c0=&#39;danial&#39;&quot;)</pre><ul><li><strong>Using Python</strong></li></ul><pre>deltaTable.delete(&quot;_c0=&#39;danial&#39;&quot;) </pre><ul><li><strong>History</strong></li></ul><pre># get the full history of the table(shows all operations performed in the table)<br>fullHistoryDF = deltaTable.history()    <br># get the last operation <br>lastOperationDF = deltaTable.history(1) </pre><p><strong>4. Update Delta Tables:</strong></p><p>It is divided into two modes</p><ul><li>Update all columns</li><li>Update specific columns</li></ul><p>Update all Columns:</p><pre>data = [(&quot;danial&quot;, &quot;simes&quot;,&quot;456 north&quot;,&quot;Coimbatore&quot;,&quot;CBE&quot;,&quot;789&quot;)]<br>updates = spark.createDataFrame(data,[&quot;_c0&quot;, &quot;_c1&quot;,&quot;_c2&quot;,&quot;_c3&quot;,&quot;_c4&quot;,&quot;_c5&quot;])<br>deltaTable.alias(&quot;et&quot;).merge(<br>    updates.alias(&quot;new&quot;),<br>    &#39;et._c0 = new._c0&#39;<br>    ).whenMatchedUpdateAll().execute()</pre><p>Update Specific Columns:</p><pre>deltaTable.alias(&quot;et&quot;).merge(<br>  updates.alias(&quot;new&quot;),<br>  &#39;et._c0 = new._c0&#39;<br>).whenMatchedUpdate(<br>  set = {<br>    &quot;et._c1&quot;: &quot;new._c1&quot;,<br>    &quot;et._c2&quot;: &quot;new._c2&quot;,<br>    &quot;et._c3&quot;: &quot;new._c3&quot;,<br>    &quot;et._c4&quot;: &quot;new._c4&quot;,<br>    &quot;et._c5&quot;: &quot;new._c5&quot;<br>  }<br>).execute()</pre><p><strong>5. Upsert Delta Table:</strong></p><p>It searches for the condition to update if the condition is satisfied it will update the row or insert a new row.</p><pre>data = [(&quot;lionel&quot;,&quot;messi&quot;,&quot;434 east&quot;,&quot;Chennai&quot;,&quot;MAS&quot;,&quot;456&quot;)]<br>new_df = spark.createDataFrame(data,[&quot;_c0&quot;,&quot;_c1&quot;,&quot;_c2&quot;,&quot;_c3&quot;,&quot;_c4&quot;,&quot;_c5&quot;])<br>deltaTable.alias(&quot;et&quot;).merge(<br>        new_df.alias(&quot;nd&quot;),<br>        condition = &#39;et._c1=&quot;messi&quot;&#39;<br>        ).whenMatchedUpdate(<br>            set={<br>                    &quot;et._c0&quot;:&quot;nd._c0&quot;,<br>                    &quot;et._c1&quot;:&quot;nd._c1&quot;<br>            }<br>        ).whenNotMatchedInsert(<br>            values={<br>                &quot;et._c0&quot;:&quot;nd._c0&quot;,<br>                &quot;et._c1&quot;:&quot;nd._c1&quot;,<br>                &quot;et._c2&quot;:&quot;nd._c2&quot;,<br>                &quot;et._c3&quot;:&quot;nd._c3&quot;,<br>            } #Here we omit the column _c4 and _c5 while insert so null value is added<br>).execute()</pre><p><strong>6. Vacuum Delta Table:</strong></p><ul><li>The Vacuum () function in Delta Lake is used to clean up old versions of data files that are no longer needed for a Delta Lake table. The function takes a retention period in hours as its argument, and any data files older than the retention period will be deleted.</li><li>It only data files that are more than retention hours should be removed during the vacuum operation.</li></ul><pre>spark.conf.set(&quot;spark.databricks.delta.retentionDurationCheck.enabled&quot;, &quot;false&quot;) <br>table_path = &#39;s3://&lt;bucket_name&gt;/deltaLake/delta_table&#39; <br>deltaTable = DeltaTable.forPath(spark, table_path) <br>deltaTable.vacuum()</pre><p><strong>Note:</strong> To give vacuum(0) we should config the spark with spark.conf.set(“spark.databricks.delta.retentionDurationCheck.enabled”, “false”)</p><p>Delta Lake has revolutionized how we handle big data by combining the reliability of traditional data warehouses with the scalability of modern data lakes. Whether you’re managing simple data ingestion, handling complex updates, or performing large-scale analytics, Delta Lake ensures ACID transactions, scalability, and data consistency.</p><p>By leveraging Delta Lake, you can efficiently manage the full lifecycle of your data — ingesting, processing, and analyzing it seamlessly. As more organizations adopt Delta Lake for its power and flexibility, now is the perfect time to explore its potential for building robust, data-driven applications.</p><p>With Delta Lake, you’re not just storing data; you’re empowering your workflows with a future-proof solution designed to handle today’s data challenges and beyond. Thanks for reading it :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6a32ee99050" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Bedrock — End-to-End Image Generation using Stable Diffusion]]></title>
            <link>https://medium.com/@aruun1995/aws-bedrock-end-to-end-image-generation-using-stable-diffusion-79ab1574f01f?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/79ab1574f01f</guid>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[aws-bedrock]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 15:05:06 GMT</pubDate>
            <atom:updated>2024-09-10T10:38:32.844Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Bedrock — End-to-End Image Generation using Stable Diffusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*2KZV8lngDkSb7S_KHqemIQ.png" /></figure><p><strong>Generating a random image:</strong></p><p><strong>Prompt</strong>: Funny carrot sitting on Duck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*9nP_CQc1AUk9wJpFFwoI7Q.png" /></figure><p>When we provided the prompt to the stable diffusion model, it generated the above image, what is occurring behind the scenes? Let’s see in more detail.</p><p><strong>Introduction to Image Generation</strong></p><p><strong>What is Image?</strong></p><ul><li>Images are spatial data structured in Pixel grids.</li><li>Images are represented as an array of Pixel values.</li></ul><p><strong>What is Pixel?</strong></p><ul><li>Pixels define the resolution of the image.</li></ul><p>There are two types of Image generation:</p><ul><li>Conditional Image Generation ( Eg., the one which we displayed on top, where prompt is considered as a condition )</li><li>Unconditional Image Generation ( Eg., thepersondoesnotexist.com where it generates random images )</li></ul><p><strong>What is Diffusion?</strong></p><ul><li>Diffusion from 9th standard chemistry perspective:</li></ul><p><em>“When a solute is added to a substance, a solute will spread out and completely dissolve into the substance.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*9DR3wgWUgZqZQJBW1vnyfg.png" /></figure><ul><li>Definition from an AI Perspective</li></ul><p><em>“When a noise is added to an image, a noise will spread out and completely dissolve into the image”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*O5GUDXyquEZvJ-LrfGabQA.png" /></figure><p>There are a few notable diffusion models available in the market, they are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4un4KEnuSlPB_7mvVGTNw.png" /></figure><p><strong>Key Terminologies in Stable Diffusion</strong></p><ol><li><strong>Noise:</strong></li></ol><p>Imagine you have a blurry, messy picture with lots of random dots and squiggles, like a fuzzy TV screen. The “noise” is just that random mess</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l63NuCDWF3URS3Mh7SrSXw.png" /></figure><p>Why Noise is added to an image?</p><ul><li>Noise is added to teach the AI how to <strong>generate images from scratch</strong>.</li><li>Here’s a simple way to think about it: Imagine you want to train someone to clean a messy room. First, you show them how the room looks when it’s clean. Then, you make the room messy bit by bit, and their job is to learn how to clean it up step by step until it looks like the original.</li><li>In Stable Diffusion, the AI sees clear images, then the noise is added to make them messy. By learning how to clean up that mess (remove noise), the AI becomes good at starting from noise and building a new, clear image. This is how it learns to generate realistic images from random noise!</li><li>So, the noise is added as a way to help the AI learn how to <strong>reverse the mess</strong> and create images</li></ul><p><strong>2. Latent Diffusion Model</strong></p><p>The <strong>Latent Diffusion Model</strong> (LDM) is an AI model used to create images from text. Instead of working directly with full, detailed images (which can be slow and complex), it works in a “latent” or simplified space.</p><p><strong>3. Forward Diffusion ( Noising )</strong></p><p>In forward diffusion, the process starts with a clear image and gradually adds noise, making it messier and messier over time. By the end, the image becomes almost pure noise or static.</p><p><strong>4. Reverse Diffusion ( De-Noising )</strong></p><p>Reverse diffusion<strong> </strong>is the opposite of forward diffusion. It starts with a noisy, messy image (like static or random dots) and gradually removes the noise, step by step, to reveal a clear and recognizable image.</p><p><strong>5. Latent Space</strong></p><p>Latent space is like a simplified, hidden version of data. Imagine you’re looking at a very detailed picture but don’t need every detail to understand what it shows. Latent space is like taking that picture and boiling it down to just the most important features or patterns.</p><p>In the latent space of models like Stable Diffusion, the image dimensions are typically much smaller than the original image. The latent space represents a compressed version of the image.</p><p>In <strong>latent space</strong>, the image is typically represented in <strong>3 dimensions</strong>.</p><p>These three dimensions are:</p><ol><li><strong>Height</strong> (e.g., 64 pixels)</li><li><strong>Width</strong> (e.g., 64 pixels)</li><li><strong>Channels</strong> (e.g., 4 channels in Stable Diffusion)</li></ol><p><strong>6. Variational Autoencoder</strong></p><p>The Variational Autoencoder (VAE) neural network has two parts: (1) an encoder and (2) a decoder. The encoder compresses an image to a lower dimensional representation in the latent space. The decoder restores the image from the latent space.</p><p>Variational autoencoder transforms the image to and from the latent space.</p><p>The encoder compresses the 512x512 pixel image into a smaller 64x64 model in latent space that’s easier to manipulate. The decoder restores the model from latent space into a full-size 512x512 pixel image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*HkxgnrWyrcYkcVxYXn6f5Q.png" /></figure><p><strong>7. U-Net</strong></p><p><strong>U-Net</strong> acts as a <strong>noise predictor. </strong>The U-Net’s job is to determine how much noise should be removed at each step to reveal the final, clear image gradually.</p><p><strong>8. CLIP</strong></p><p>CLIP (Contrastive Language–Image Pretraining) is a model designed to understand both images and text together. Its goal is to match images with their correct descriptions and vice versa.</p><p><strong>Stable Diffusion in AWS Bedrock:</strong></p><p><strong>What is Bedrock?</strong></p><ul><li>Amazon Bedrock is a fully managed service that allows users to build generative AI applications on Amazon’s cloud computing platform</li><li>Bedrock provides access to foundational models (FMs) from leading AI companies, such as Anthropic, AI21 Labs, Cohere, Meta, Mistral AI, Stability.ai, and Amazon.</li><li>Stable Diffusion comes under part of StabilityAI.</li></ul><ol><li>Ensure whether Access has been granted for the stable diffusion model available in AWS Bedrock</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uy-c_CMg8pLeZa2v6iPNLw.png" /></figure><p>2. Models available in StabilityAI in AWS Bedrock</p><ul><li>SDXL 1.0</li><li>Stable Image Ultra</li><li>Stable Diffusion 3 large</li><li>Stable Image Core</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*LyxmX7wfmJrcIyQgtvObQA.png" /></figure><p><strong>Guardrails for sending safe prompts and generating safe images:</strong></p><p>Guardrails are sending safe prompts and generating safe images are guidelines or safety measures that prevent the AI model from producing harmful, inappropriate, or unsafe content.</p><p><strong>Configuring Guardrails in AWS Bedrock:</strong></p><p>Navigate to Guardrails in AWS Bedrock, where we can set the Moderations in the prompt and the LLM response.</p><p>List of sections moderated in Prompt and LLM response:</p><ul><li>Hate</li><li>Insults</li><li>Sexual</li><li>Violence</li><li>Misconduct</li></ul><p>As an additional guardrail, we can bring AWS Comprehend for validating the prompt from the user and also AWS Rekognition for validating the image generated by LLM.</p><p><strong>Invoking AWS Bedrock from Lambda:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIy5HQVQxFHURU7rUckXbQ.png" /></figure><p><strong>Passing Prompts to Stable Diffusion Model:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGAs6rO9rGXXD9qNBEuQ2g.png" /></figure><p><strong>Parameters:</strong></p><ol><li>prompt: This is the text input that is used to generate a new image</li><li>negative prompt: This is the text input that is used to suppress parts of the output image</li><li>Cfg_scale: CFG stands for Classifier- Free Guidance scale, this value controls how closely generated images follow the input prompts</li><li>Seed: The parameter used to generate a random vector from which to produce an image</li><li>steps: This is the number of inference steps that SD executes before returning an output image</li><li>style_preset: This will guide the image model towards a particular style. Possible values are 3d-model, analog film, anime, cinematic, comic-book, digital-art, enhance, fantasy-art, isometric, line-art, low-poly, modeling-compound, origami, photographic, pixel-art and tile-texture.</li><li>clip_guidance_preset: This parameter helps to control how closely the AI sticks to the description (text prompt) you give.</li><li>sampler<strong>:</strong> This helps the model create the image step by step.</li><li>height/width: Define how much height/width the image should be.</li><li>num_images_per_prompt: Defines the number of images needs to be displayed per prompt.</li></ol><p><strong>Stable Diffusion Working in Single Image:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*7LUMmRcPVSTNYbYS4FLpaQ.png" /></figure><p>This is how the stable diffusion works, and generates images for the prompts we are providing. It’s great news that AWS Bedrock has access to stabilityAI models, which makes our work easier.</p><p>Thank you for reading out :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79ab1574f01f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark UI on AWS Glue]]></title>
            <link>https://medium.com/@aruun1995/spark-ui-on-aws-glue-8c4eb6b87d68?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c4eb6b87d68</guid>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 10:24:46 GMT</pubDate>
            <atom:updated>2024-08-16T10:24:46.705Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark provides a suite of Web UI/User Interfaces (<a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#spark-jobs">Jobs</a>, <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#spark-stages">Stages</a>, <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#tasks">Tasks</a>, <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#storage">Storage</a>, <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#environment">Environment</a>, <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#executors">Executors</a>, and <a href="https://sparkbyexamples.com/spark/spark-web-ui-understanding/#sql">SQL</a>) to monitor the status of your Spark/PySpark application, resource consumption of Spark cluster, and Spark configurations.</p><p>On the other hand, if any job on AWS Glue fails, we will analyze the AWS Cloudwatch logs, and that will help us to understand the failure better.</p><p>To better understand the failures within AWS Glue spark jobs, enabling the Spark UI, will significantly help us to understand the failures better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*m_LcMObA2lTQmhS9CeUvHA.png" /></figure><p>Let’s discuss the steps of Enabling SparkUI on AWS Glue:</p><p><strong>Steps to create SparkUI:</strong></p><ol><li>Navigate to AWS Glue and click on Create a Glue job.</li><li>Create an IAM Role that provides access to CloudWatch Logs.</li><li>Select the Glue version.</li><li>Enable Spark UI. If it’s not enabled by default, enable it. You can enable Spark UI on the job details tab of AWS Glue.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QO67IP14cimAgRjRvK_3g.png" /></figure><p>5. Set the Spark UI logs path under Advanced Properties.</p><p>6. Run the job. Once the job is completed, it will create a log under the specified Spark UI logs path.</p><p>7. To create Spark UI, use the CloudFormation template.</p><p>8. click on the following link:</p><p><a href="https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html">https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html</a> and choose your region and Glue version, ensuring that the Glue version matches your job’s version</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KuOKeoI_xu8YPVj599i3tg.png" /></figure><p>9. In the Stack Detail section, provide the following details:</p><ul><li>IPAddressRange</li><li>Spark UI path for a specific job or the common path to take all Spark job histories.</li><li>Key store password</li><li>VPC (recommended: Default VPC)</li><li>Subnet with an Internet gateway</li></ul><p>10. Click Next after filling in all the stack details.</p><p>11. In the Configure Stack option, keep the default option as it is and click Next.</p><p>12. Review your changes and acknowledge them. Click Create Stack.</p><p>13. Once the stack is successfully created, switch to the Output tab. There will be two URLs listed: one public and another similar to the public URL. Click the public URL, and it will redirect to the Spark UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PutaSIAdFQSPmhZUP0UNyA.png" /></figure><p>Thanks for reading out :). Happy troubleshooting using Spark UI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c4eb6b87d68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrate LLM with Real-time Data — Part 2]]></title>
            <link>https://blog.gopenai.com/integrate-llm-with-real-time-data-part-2-e690d0ba10f3?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e690d0ba10f3</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Tue, 23 Apr 2024 17:16:47 GMT</pubDate>
            <atom:updated>2024-04-26T10:51:16.963Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integrate LLM with Real-time Data — Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Ya-l4iuCA50lRuP.png" /></figure><p>This is in continuation of the previous blog — Integrating LLM with Real-time Data — Part 1. Please find the following link:</p><p><a href="https://medium.com/@aruun1995/integrating-llm-with-real-time-data-part-1-c93c38bd32e5">Integrating LLM with Real-time Data — Part 1</a></p><p>In this blog, we will see with real-time example by integrating it with real-time data. The use case chosen here is integrating it with stock data.</p><p>Getting stock prices in real-time using LLM is not possible right now, what if we used a Function calling API to achieve it? We will go step by step as we have seen in the last post, in a similar way we will create 3 steps in achieving it. Before diving into it, let’s ask ChatGPT and see the response from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kB_Zw0fjl9wci7TE6lUgw.png" /></figure><p>As we discussed, chatGPT was not able to provide real-time information. Let’s try to fix this using a Function calling API.</p><p><strong>Step 1</strong>: Create a function description</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FPJwQqe3NU5PaezKN0I6xA.png" /></figure><p><strong>Step 2</strong>: Define Function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*QvyRBtiB82nf90daWl620A.png" /></figure><p>In the above screenshot, we can see external API yfinance(Yahoo Finance) has been used. Also, you could notice that we are returning a static response from the function so that we can always extract a structured response from the LLM.</p><p><strong>Step 3</strong>: Calling defined function via OpenAI API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*xLpwrpxqcYU81eZc0VC8jQ.png" /></figure><p>Since we mentioned tool_choice to “auto”, it will automatically identify the function to be called based on the list of requirement arguments extracted from the user prompt or LLM response.</p><p>The final output will look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kAheBKkY0jWPRe81hapm2w.png" /></figure><p>From the above results, we can conclude that we can successfully connect with real-time data using LLM.</p><p>If you have any queries/setting up similar environment and facing issues while setting up, feel free to reach out to me via LinkedIn (<a href="https://www.linkedin.com/in/arun-kumar-nagaraj/">https://www.linkedin.com/in/arun-kumar-nagaraj/</a>)</p><p>Thanks for reading out :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e690d0ba10f3" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/integrate-llm-with-real-time-data-part-2-e690d0ba10f3">Integrate LLM with Real-time Data — Part 2</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating LLM with Real-time Data — Part 1]]></title>
            <link>https://blog.gopenai.com/integrating-llm-with-real-time-data-part-1-c93c38bd32e5?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c93c38bd32e5</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Mon, 15 Apr 2024 17:12:09 GMT</pubDate>
            <atom:updated>2024-04-26T10:51:20.383Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integrating LLM with Real-time Data — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ob7dX0OSLAcbdq6ZNh3M5g.png" /></figure><p>In today’s data-driven world, the demand for intelligent systems that can not only process vast amounts of information but also adapt and respond to real-time data is ever-growing. In this blog, we’ll explore how we can use language models(Function Calling API) to analyze and respond to information as it comes in real time.</p><p><strong>Introduction to Function Calling API:</strong></p><p>Function Calling API connects large language models to external tools. While invoking an API call, can describe functions, and the model intelligently chooses to output a JSON object containing arguments to call the appropriate function.</p><p><strong>When to use Function calling API?</strong></p><p>Function calling API can be used in the following scenarios:</p><ol><li>Real-time queries</li><li>Calling Internal API’s</li><li>Extract structured data from the LLM response</li></ol><p><strong>Note</strong>: Function calling API is supported only on the GPT-3.5-turbo-0125 and GPT-4-turbo-preview models.</p><p><strong>Design:</strong></p><p>To implement a function calling APIs within our LLM application, we may need to develop the application in the following ways:</p><p><strong>Step 1</strong>: Create a function description</p><p><strong>Syntax:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iqbc9Y4fMWVf9DO59aHkEA.png" /></figure><p><strong>Step 2</strong>: Define Function</p><p>A function can be defined by writing logic to connect with external API’s/Internal API’s. This function can also be used to extract structured data from the provided responses.</p><p>So that, we can get LLM responses in a structured way.</p><p><strong>Template:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNVqxch335ILm_KulwkdUA.png" /></figure><p><strong>Example:</strong> Function to return structured responses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bUx7nsdu0_snG09MRwq7RQ.png" /></figure><p><strong>Step 3</strong>: Calling defined function via OpenAI API</p><p>Open AI API will call the implemented function based on the parameters provided below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x9OHhXUSiVKIEgNscV1hEg.png" /></figure><p>Here tools should be an array, which will take the function description as a value, and setting</p><p>tool_choice to auto to automatically pick the matching function. Also, we have the flexibility to provide a specific function name here.</p><p>In the next blog, we will see with real-time example, how we can use it along with the demonstration. Hope, this blog will give a basic idea of how to integrate real-time data with LLM.</p><p>Thank you for reading out :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c93c38bd32e5" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/integrating-llm-with-real-time-data-part-1-c93c38bd32e5">Integrating LLM with Real-time Data — Part 1</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hyperledger Fabric — Part 3]]></title>
            <link>https://blog.blockmagnates.com/hyperledger-fabric-part-3-87f062fc5c65?source=rss-4cbee88ce9e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/87f062fc5c65</guid>
            <category><![CDATA[hyperledger-fabric]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Arun Kumar Nagaraj]]></dc:creator>
            <pubDate>Sun, 25 Feb 2024 18:20:42 GMT</pubDate>
            <atom:updated>2024-03-01T14:13:26.924Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hyperledger Fabric — Part 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hJd1n8dnut86BlVDT2ppA.png" /></figure><p>This is in continuation of the previous blog — Introduction to Hyperledger Fabric — Part 2. Please find the following link:</p><p><a href="https://blog.blockmagnates.com/hyperledger-fabric-part-2-3306e5977870">Hyperledger Fabric — Part 2</a></p><p>In this blog, we will see the basic things required to set up a fabric network and also on setting up a fabric network.</p><p><strong>Basic things required to set up fabric network:</strong></p><p><strong>Docker:</strong></p><ul><li>To be a part of the fabric network, we need to use docker.</li></ul><p><strong>Docker Images:</strong></p><ul><li>These are pre-defined, preset docker containers, that can be downloaded from the docker cloud.</li><li>Hyperledger fabric provides docker images for the various roles that docker containers play inside of the fabric network. So a docker image for fabric peer, a docker image for fabric CA, and a docker image for fabric orderer, we have separate images for different roles.</li></ul><p><strong>Docker Compose:</strong></p><ul><li>Docker compose is an automation tool where you can define all of the various docker services, which means various docker containers that you want to start and the configuration that you want to start these containers within a single file and spread it across multiple files using the concept of hierarchy.</li></ul><p><strong>CLI container:</strong></p><ul><li>CLI container provided by fabric as docker image, it has been downloaded from docker cloud and will start as (n+1)th docker container inside the network.</li></ul><p>The certificate that the fabric uses is X509. It can be created using third-party applications, certificate issuers, or even on local machines.</p><p><strong>Cryptogen:</strong></p><ul><li>Cryptogen is a tool that is used for bulk certificate generation. This is a binary file that comes along with fabric installation.</li><li>Cryptogen generate results in:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JO_K_Cx0HhY5UuAoNHo2fw.png" /></figure><ul><li>By default, an admin user will be created.</li></ul><p><strong>Setting up Fabric- Network :</strong></p><p>Setting up a fabric network can be sub-divided into two steps, they are:</p><ul><li>Pre-Setup Phase</li><li>Network Setup Phase</li></ul><p><strong>Pre-setup phase</strong></p><p>The pre-setup phase will set the following things:</p><ul><li>Crypto-config generation</li><li>Channel artifacts creation</li></ul><p>To generate a crypto config and channel artifacts, cryptogen plays a key role in creating it.</p><p><strong>Network Setup Phase:</strong></p><p>The network Setup phase will setup the following things:</p><ul><li>Docker Network creation</li><li>Channel creation</li><li>Peer joining</li><li>Anchor peer updation for each organization</li></ul><p>In the next blog, will dive deeper into the depth of hyper ledger fabric in the topics of setting up a fabric network, starting the network, and playing with the network. Please feel free to interact and share your thoughts about it. Thanks for reading out :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87f062fc5c65" width="1" height="1" alt=""><hr><p><a href="https://blog.blockmagnates.com/hyperledger-fabric-part-3-87f062fc5c65">Hyperledger Fabric — Part 3</a> was originally published in <a href="https://blog.blockmagnates.com/">Block Magnates</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>