<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Binaya Puri on Medium]]></title>
        <description><![CDATA[Stories by Binaya Puri on Medium]]></description>
        <link>https://medium.com/@binaya.puri?source=rss-ae7a71b172a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZsN09_pvcEMlaUt211xO-w.jpeg</url>
            <title>Stories by Binaya Puri on Medium</title>
            <link>https://medium.com/@binaya.puri?source=rss-ae7a71b172a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:11:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@binaya.puri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Passed the Microsoft Azure AI-900 Certification in Just One Week]]></title>
            <link>https://medium.com/@binaya.puri/how-i-passed-the-microsoft-azure-ai-900-certification-in-just-one-week-f750d92ef8ad?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f750d92ef8ad</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[ai-certification]]></category>
            <category><![CDATA[azure-ai-900]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Fri, 16 May 2025 16:52:18 GMT</pubDate>
            <atom:updated>2025-05-16T16:52:18.382Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MFxvaX6LjDD7uVrf.png" /></figure><h3>Introduction</h3><p>I recently challenged myself to prepare for and pass the Microsoft Azure AI-900 certification exam in just one week. On May 1, 2025, I successfully achieved this goal with a score of 931, well above the passing threshold! The AI-900 exam, officially known as “Microsoft Azure AI Fundamentals,” tests your understanding of basic AI concepts and their implementation in Microsoft Azure services. In this comprehensive guide, I’ll share my intense one-week preparation strategy, the resources that proved most effective, and specific tactics that helped me score 100% in three exam domains despite the limited preparation time.</p><p>If you’re facing a tight deadline or simply want to efficiently earn this certification without months of study, my experience will show you exactly how to accomplish this ambitious goal.</p><h3>Why I Chose the One-Week Challenge</h3><p>If you’re facing a tight deadline or simply want to efficiently earn this certification without months of study, my experience will show you exactly how to accomplish this ambitious goal.</p><p>My decision to compress the preparation into one week was driven by several factors:</p><ul><li><strong>Upcoming Project Deadline</strong>: I needed to demonstrate AI proficiency quickly for a new Azure-based project at work.</li><li><strong>Prior Experience</strong>: I already had some familiarity with cloud concepts and basic AI terminology.</li><li><strong>Test the Limits</strong>: I wanted to discover if Microsoft’s “fundamentals” certification could genuinely be mastered in a short timeframe.</li><li><strong>Pre-booked Exam Date</strong>: I had already booked my exam date but started preparing much later than intended, leaving me with just one week before the scheduled test.</li><li><strong>Time Constraints</strong>: With other work and personal commitments, I simply didn’t have the luxury of an extended study period but still needed to pass the certification.</li></ul><p>This accelerated timeline isn’t ideal for everyone, but if you have some technical background and can dedicate full days to studying, it’s definitely achievable.</p><h3>Understanding My Exam Results</h3><p>Looking at my score breakdown:</p><ul><li><strong>Describe Artificial Intelligence Workloads and Considerations</strong>: 100%</li><li><strong>Describe Fundamental Principles of Machine Learning on Azure</strong>: 80%</li><li><strong>Describe Features of Computer Vision Workloads on Azure</strong>: 100%</li><li><strong>Describe Features of Natural Language Processing (NLP) Workloads on Azure</strong>: 100%</li><li><strong>Describe Features of Generative AI Workloads on Azure</strong>: 92%</li></ul><p>I strategically focused on areas with the highest domain weights while ensuring basic proficiency across all topics. My lower scores came in the machine learning principles section, which is the most technically complex and has the highest weighting (20–25%).</p><h3>My 7-Day Intensive Study Plan</h3><p>Here’s the day-by-day breakdown of my one-week preparation strategy:</p><h3>Day 1: Assessment and Foundational Concepts</h3><ul><li>Completed a practice test to identify knowledge gaps</li><li>Studied basic AI concepts, terminology, and Azure fundamentals</li><li>Focused on the “Describe Artificial Intelligence Workloads and Considerations” domain</li><li>Set up my Azure free account for hands-on practice</li></ul><h3>Day 2: Machine Learning Fundamentals</h3><ul><li>Dedicated the entire day to the highest-weighted domain: “Describe Fundamental Principles of Machine Learning on Azure”</li><li>Completed Microsoft Learn modules on Azure Machine Learning</li><li>Practiced creating a simple model in Azure ML Studio</li><li>Took detailed notes on supervised, unsupervised, and reinforcement learning</li></ul><h3>Day 3: Computer Vision</h3><ul><li>Focused exclusively on “Describe Features of Computer Vision Workloads on Azure”</li><li>Studied Computer Vision API, Face API, and Form Recognizer</li><li>Completed hands-on labs with Custom Vision</li><li>Created flashcards for service capabilities and use cases</li></ul><h3>Day 4: Natural Language Processing</h3><ul><li>Concentrated on “Describe Features of Natural Language Processing (NLP) Workloads on Azure”</li><li>Studied Text Analytics, Translator, and LUIS</li><li>Conducted practical exercises with QnA Maker and Azure Bot Service</li><li>Reviewed speech recognition and speech synthesis services</li></ul><h3>Day 5: Generative AI</h3><ul><li>Focused on “Describe Features of Generative AI Workloads on Azure”</li><li>Studied Azure OpenAI Service capabilities and limitations</li><li>Reviewed responsible AI principles for generative models</li><li>Created comparison charts of different generative capabilities</li></ul><h3>Day 6: Practice Exams and Gap Filling</h3><ul><li>Took multiple full-length practice exams</li><li>Identified and addressed remaining knowledge gaps</li><li>Focused extra attention on machine learning principles (my weakest area)</li><li>Created concise review sheets for each domain</li></ul><h3>Day 7: Final Review and ExamTopics Practice</h3><ul><li>Morning: Focused review of ExamTopics question bank, which proved to be my most valuable resource — more than 85% of my actual exam questions were similar to those on ExamTopics</li><li>Afternoon: Completed final passes through the 165 practice questions I had collected from ExamTopics, focusing especially on questions I had previously missed</li><li>Evening: Additional practice on other exam platforms including:</li><li>TutorialsDojo AI-900 practice tests</li><li>Udemy Scott Duffy practice exam set</li><li>DumpsArena sample questions</li><li>CertLibrary AI-900 question bank</li><li>ExamCollection free practice questions</li></ul><p>This daily structure ensured I covered all exam domains thoroughly while allowing sufficient time for hands-on practice, which proved crucial for cementing my understanding.</p><h3>Essential Resources for Quick Mastery</h3><p>For a one-week preparation window, resource selection becomes critical. Here are the resources that proved most valuable:</p><h3>Official Microsoft Resources</h3><ol><li><strong>Microsoft Learn’s AI-900 Learning Path</strong>: I completed the <a href="https://learn.microsoft.com/en-us/training/paths/get-started-with-artificial-intelligence-on-azure/">official modules</a> with a focus on the knowledge check questions and lab exercises. This provided a structured overview of all exam topics directly from the source.</li><li><strong>Microsoft AI-900 Study Guide</strong>: The <a href="https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-900">official exam guide</a> breaks down exactly what’s covered, which allowed me to create targeted study notes.</li><li><strong>Azure AI Services Documentation</strong>: For quick reference on specific services and their capabilities, the <a href="https://learn.microsoft.com/en-us/azure/ai-services/">official documentation</a> was invaluable.</li></ol><h3>Third-Party Resources</h3><ol><li><strong>Whizlabs Practice Tests</strong>: These closely simulated the actual exam format and difficulty level, helping me identify weak areas quickly.</li><li><strong>YouTube Cram Videos</strong>: For rapid review, I found John Savill’s <a href="https://www.youtube.com/watch?v=OwZHNH8EfSU">AI-900 Cram</a> exceptionally helpful — it condenses key concepts into a digestible format.</li><li><strong>MeasureUp Practice Test</strong>: Though expensive, this official practice test partner provided the most exam-like questions and detailed explanations.</li><li><strong>GitHub AI-900 Labs</strong>: The <a href="https://github.com/MicrosoftLearning/AI-900-AIFundamentals">Microsoft Learning repository</a> provided structured lab exercises I could complete quickly.</li></ol><h3>My Secret Weapon: Service Comparison Matrices</h3><p>I created simple comparison matrices for each category of Azure AI services, showing:</p><ul><li>Key features and capabilities</li><li>Typical use cases</li><li>Limitations</li><li>Related/alternative services</li></ul><p>This approach helped me quickly identify the most appropriate service for different scenarios — a common theme in exam questions.</p><h3>Hands-On Practice: Quality Over Quantity</h3><p>With only one week, I couldn’t afford to spend days in extensive hands-on practice. Instead, I focused on high-impact, targeted exercises:</p><ol><li><strong>Azure Free Account</strong>: I maximized the $200 credit from my <a href="https://azure.microsoft.com/en-us/free/">free Azure account</a> by focusing on the most exam-relevant services.</li><li><strong>Guided Labs</strong>: Rather than building projects from scratch, I followed the <a href="https://microsoftlearning.github.io/AI-900-AIFundamentals/">official Microsoft labs</a>, which efficiently demonstrated key concepts.</li><li><strong>Service Exploration</strong>: For each major service, I:</li></ol><ul><li>Created a resource in the Azure portal</li><li>Explored the configuration options</li><li>Tested basic functionality with sample data</li><li>Noted integration points with other services</li></ul><ol><li><strong>Focused Scenarios</strong>: I practiced with scenarios directly related to exam objectives:</li></ol><ul><li>Analyzing text with Text Analytics</li><li>Classifying images with Custom Vision</li><li>Creating a simple Q&amp;A system with QnA Maker</li><li>Testing Azure OpenAI Service capabilities</li></ul><p>This focused approach ensured I gained practical familiarity with key services without getting lost in complex projects.</p><h3>Exam Day Strategy</h3><p>My strategy for the exam day itself was crucial to achieving a high score with limited preparation:</p><ol><li><strong>Question Prioritization</strong>: I first answered all questions I was confident about, flagging uncertain ones for review.</li><li><strong>Elimination Technique</strong>: For difficult questions, I systematically eliminated obviously wrong answers first.</li><li><strong>Scenario Analysis</strong>: For scenario-based questions, I identified the key requirements before considering the services.</li><li><strong>Time Management</strong>: I allocated my time proportionally to the domain weights, spending more time on machine learning principles questions.</li><li><strong>Final Review</strong>: I reserved 15 minutes at the end to review flagged questions, making educated guesses where necessary.</li></ol><h3>Critical Concepts That Secured My High Score</h3><p>Based on my exam experience, these key areas were vital to my success:</p><h3>1. Service Selection Logic</h3><p>Understanding which Azure AI service to apply in different scenarios was perhaps the most tested concept. I memorized this decision matrix:</p><ul><li><strong>Text analysis needs</strong>: Text Analytics API for sentiment, key phrase extraction, language detection, and entity recognition</li><li><strong>Image analysis needs</strong>: Computer Vision for general analysis, Custom Vision for specialized classification or object detection</li><li><strong>Form processing needs</strong>: Form Recognizer for structured documents</li><li><strong>Conversational needs</strong>: QnA Maker for FAQs, LUIS for intent recognition, Bot Service for deployment</li><li><strong>Translation needs</strong>: Translator for text, Speech Service for speech</li><li><strong>Generative content needs</strong>: Azure OpenAI Service for text generation, image generation, and embeddings</li></ul><h3>2. Machine Learning Fundamentals</h3><p>The most technical domain requires understanding:</p><ul><li><strong>ML Process</strong>: Data collection → Preparation → Training → Evaluation → Deployment</li><li><strong>Model Types</strong>: Regression (numeric prediction) vs. Classification (categorization) vs. Clustering (grouping)</li><li><strong>Key Metrics</strong>: Accuracy, precision, recall, and F1 score (and when each matters most)</li><li><strong>Azure ML Studio</strong>: Components and basic workflow for model creation</li><li><strong>Automated ML</strong>: Capabilities and limitations compared to custom model development</li></ul><h3>3. Responsible AI Principles</h3><p>Microsoft emphasizes ethical AI across all domains. I focused on memorizing their six principles:</p><ul><li>Fairness</li><li>Reliability &amp; Safety</li><li>Privacy &amp; Security</li><li>Inclusiveness</li><li>Transparency</li><li>Accountability</li></ul><h3>4. Integration Points</h3><p>Understanding how different services connect was crucial for scenario questions:</p><ul><li>How NLP services connect with Bot Service</li><li>How Computer Vision outputs can feed into other applications</li><li>How Azure OpenAI integrates with other Azure services</li></ul><h3>Domain-Specific Strategies That Led to Perfect Scores</h3><p>I achieved 100% in three domains. Here’s how I approached each:</p><h3>AI Workloads and Considerations (100%)</h3><ul><li>Created flashcards for AI terminology and types (ANI vs. AGI)</li><li>Memorized responsible AI principles and their practical applications</li><li>Studied common AI workload patterns in Azure</li></ul><h3>Computer Vision Workloads (100%)</h3><ul><li>Created a service capability comparison table (Computer Vision vs. Custom Vision vs. Face API vs. Form Recognizer)</li><li>Practiced with the actual services to understand their outputs</li><li>Focused on recognizing appropriate use cases for each service</li></ul><h3>NLP Workloads (100%)</h3><ul><li>Memorized which NLP tasks each Azure service handles</li><li>Created a workflow diagram showing how different NLP services can work together</li><li>Practiced building a simple QnA bot to understand the integration points</li></ul><h3>Challenges I Faced and How I Overcame Them</h3><p>The one-week timeline created specific challenges:</p><h3>Challenge 1: Limited Time for Machine Learning Concepts</h3><p><strong>Solution:</strong> I focused on understanding the core principles rather than implementation details. I created simple analogies to remember complex concepts (e.g., comparing supervised learning to learning with a teacher).</p><h3>Challenge 2: Information Overload</h3><p><strong>Solution:</strong> I created a one-page summary for each domain, distilling only the most exam-relevant information. This prevented me from getting lost in technical details beyond the exam scope.</p><h3>Challenge 3: Service Capabilities Confusion</h3><p><strong>Solution:</strong> I created comparison tables that clearly showed which capabilities belonged to which services, focusing on the distinctions most likely to appear in exam questions.</p><h3>Challenge 4: Fatigue from Intensive Study</h3><p><strong>Solution:</strong> I implemented the Pomodoro technique (25 minutes of focused study, 5-minute break) and ensured I took longer breaks every 2–3 hours. I also varied my study methods between reading, watching videos, and hands-on practice.</p><h3>Post-Certification Reflections</h3><p>After passing the exam with a 931 score, I identified what worked best in my one-week approach:</p><ol><li><strong>Structured daily focus</strong> on specific domains worked better than trying to cover everything each day.</li><li><strong>Practice tests</strong> early in the process helped me target my studies more effectively than starting with learning materials.</li><li><strong>Service comparison tables</strong> were more valuable than detailed notes on individual services.</li><li><strong>Official Microsoft Learn modules</strong> provided sufficient depth for this fundamental-level exam.</li><li><strong>Understanding the “why”</strong> behind service recommendations was more important than memorizing features.</li></ol><h3>Advice for Other One-Week Challengers</h3><p>If you’re attempting to pass AI-900 in one week, here’s my targeted advice:</p><ol><li><strong>Take a practice test on day one</strong> to identify your starting knowledge level and focus areas.</li><li><strong>Create a strict hour-by-hour schedule</strong> for the week and commit to following it.</li><li><strong>Focus on understanding service selection</strong> rather than trying to memorize all features of each service.</li><li><strong>Limit your resources</strong> to prevent information overload — stick to official Microsoft materials plus one practice test provider.</li><li><strong>Prioritize hands-on experience</strong> with the most commonly tested services (Azure ML Studio, Custom Vision, Text Analytics, QnA Maker).</li><li><strong>Don’t neglect responsible AI</strong> — Microsoft emphasizes these principles throughout the exam.</li><li><strong>Create concise review materials</strong> during your studies that you can quickly review on the final day.</li></ol><h3>Comprehensive Resource List for One-Week Preparation</h3><h3>Day 1: Assessment and Foundations</h3><ul><li><a href="https://www.whizlabs.com/microsoft-azure-certification-ai-900/">Whizlabs AI-900 Practice Test</a> (for initial assessment)</li><li><a href="https://learn.microsoft.com/en-us/credentials/certifications/azure-ai-fundamentals/">Microsoft AI Fundamentals Certification Overview</a></li><li><a href="https://azure.microsoft.com/en-us/products/ai-services/">Azure AI Services Overview</a></li></ul><h3>Day 2: Machine Learning</h3><ul><li><a href="https://learn.microsoft.com/en-us/training/paths/create-no-code-predictive-models-azure-machine-learning/">Microsoft Learn: Create no-code predictive models with Azure Machine Learning</a></li><li><a href="https://learn.microsoft.com/en-us/azure/machine-learning/">Azure Machine Learning documentation</a></li><li><a href="https://microsoftlearning.github.io/AI-900-AIFundamentals/instructions/02-module-02.html">Machine Learning Lab Exercise</a></li></ul><h3>Day 3: Computer Vision</h3><ul><li><a href="https://learn.microsoft.com/en-us/training/paths/explore-computer-vision-microsoft-azure/">Microsoft Learn: Explore Computer Vision</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/">Custom Vision Service documentation</a></li><li><a href="https://microsoftlearning.github.io/AI-900-AIFundamentals/instructions/03-module-03.html">Computer Vision Lab Exercise</a></li></ul><h3>Day 4: Natural Language Processing</h3><ul><li><a href="https://learn.microsoft.com/en-us/training/paths/explore-natural-language-processing/">Microsoft Learn: Explore natural language processing</a></li><li><a href="https://learn.microsoft.com/en-us/azure/cognitive-services/language-service/">Text Analytics documentation</a></li><li><a href="https://microsoftlearning.github.io/AI-900-AIFundamentals/instructions/04-module-04.html">NLP Lab Exercise</a></li></ul><h3>Day 5: Generative AI</h3><ul><li><a href="https://learn.microsoft.com/en-us/training/modules/explore-azure-openai/">Microsoft Learn: Explore generative AI with Azure OpenAI Service</a></li><li><a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/">Azure OpenAI Service documentation</a></li><li><a href="https://microsoftlearning.github.io/AI-900-AIFundamentals/instructions/05-module-05.html">Generative AI Lab Exercise</a></li></ul><h3>Day 6–7: Review and Practice</h3><ul><li><a href="https://www.youtube.com/watch?v=OwZHNH8EfSU">John Savill’s AI-900 Cram Video</a></li><li><a href="https://www.measureup.com/microsoft-azure-ai-fundamentals-ai-900.html">MeasureUp AI-900 Practice Test</a></li><li><a href="https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-900">Microsoft Official AI-900 Sample Questions</a></li><li><a href="https://www.examtopics.com/exams/microsoft/ai-900/">Exam Topics Questions</a></li></ul><h3>Conclusion</h3><p>Passing the Azure AI-900 certification with a score of 931 in just one week demonstrates that with focused effort, strategic resource selection, and targeted practice, you can efficiently prepare for this foundational exam. While this accelerated approach requires discipline and prior technical familiarity, it proves that Microsoft’s fundamental-level certifications can be successfully achieved in a compressed timeframe.</p><p>The key to my success was prioritizing breadth over depth, focusing on service selection logic, and creating structured comparison tools that helped me quickly identify the right Azure AI service for different scenarios. By understanding the exam’s focus on conceptual knowledge rather than deep implementation details, I was able to efficiently prepare despite the limited time available.</p><p>If you’re facing a similar challenge with limited preparation time, I hope my experience provides a practical roadmap to your own success with the Azure AI-900 certification.</p><p><em>Have you attempted a certification in a compressed timeframe? What strategies worked for you? Share your experiences in the comments below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f750d92ef8ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging AWS Lake Formation for Scalable Data Governance: Beyond the Basics]]></title>
            <link>https://medium.com/@binaya.puri/leveraging-aws-lake-formation-for-scalable-data-governance-beyond-the-basics-d6e4a971c230?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6e4a971c230</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[aws-lake-formation]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Fri, 16 May 2025 16:22:06 GMT</pubDate>
            <atom:updated>2025-05-16T16:22:06.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*M6eJ6uJEJdZmh5g4.jpg" /></figure><h3>Introduction</h3><p>In today’s data-driven landscape, organizations face increasingly complex challenges in managing, securing, and deriving value from their data assets. AWS Lake Formation has emerged as a powerful service for building, securing, and managing data lakes, yet many of its advanced capabilities remain underutilized. This comprehensive blog explores the nuanced aspects of AWS Lake Formation that go beyond typical implementations, focusing on strategies that enable sophisticated data governance at scale while maintaining flexibility for diverse analytical workloads.</p><p>As data volumes continue to grow exponentially and regulatory requirements become more stringent, traditional approaches to data management are proving insufficient. This exploration will demonstrate how Lake Formation can serve as the foundation for a modern data governance framework that balances security, compliance, and analytical agility.</p><h3>The Evolution of Data Lakes and the Lake Formation Paradigm</h3><p>The concept of data lakes has evolved significantly since James Dixon first coined the term in 2010. Initially conceptualized as centralized repositories for storing structured and unstructured data, data lakes have transformed into sophisticated, multi-layered architectures that support diverse analytical workloads (Miloslavskaya &amp; Tolstoy, 2016).</p><p>AWS Lake Formation, introduced in 2019, represented a paradigm shift by abstracting much of the complexity associated with building and managing data lakes. However, its true potential goes far beyond the basic implementation patterns commonly discussed in AWS documentation and community resources.</p><h3>Key Components in the Lake Formation Architecture</h3><p>At its core, Lake Formation functions as a permissions management and data catalog service that interfaces with multiple AWS services. Understanding the interactions between these components is crucial for implementing advanced governance patterns:</p><ol><li><strong>AWS Glue Data Catalog</strong>: Serves as the metadata repository, containing table definitions, schema information, and partition details</li><li><strong>Lake Formation Permissions Model</strong>: Provides fine-grained access control at the database, table, column, and row levels</li><li><strong>Integration with Analytics Services</strong>: Extends permissions to services like Athena, Redshift, EMR, and QuickSight</li></ol><p>The power of Lake Formation lies in how these components work together to enable centralized governance while supporting decentralized data consumption patterns.</p><h3>Advanced Permission Management Strategies</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/0*VoS6sRjzeRbi94xy.png" /></figure><h3>Implementing Data Perimeters with Resource-Based Policies</h3><p>One of the less explored aspects of Lake Formation is the ability to implement data perimeters using resource-based policies. Unlike traditional role-based access control (RBAC) approaches, resource-based policies allow for context-aware access decisions that consider factors beyond identity:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;AWS&quot;: &quot;arn:aws:iam::123456789012:role/AnalyticsRole&quot;<br>      },<br>      &quot;Action&quot;: [<br>        &quot;lakeformation:GetDataAccess&quot;,<br>        &quot;glue:GetTable&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;aws:SourceVpc&quot;: &quot;vpc-01234567890abcdef&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>This approach enables organizations to enforce network-level controls on data access, ensuring that sensitive data can only be queried from approved network environments (AWS Security Blog, 2023).</p><h3>Cross-Account Data Sharing with Granular Controls</h3><p>While cross-account data sharing is a documented feature of Lake Formation, implementing it with complex permission hierarchies requires careful planning. Advanced cross-account patterns include:</p><ol><li><strong>Hub-and-Spoke Distribution Models</strong>: Centralize data management while distributing read access across multiple accounts</li><li><strong>Federated Governance Models</strong>: Maintain distributed data ownership while implementing centralized access controls</li><li><strong>Hybrid Permission Models</strong>: Combine Lake Formation permissions with service-specific controls</li></ol><p>Research by Gartner (2022) indicates that organizations with mature data governance frameworks are increasingly adopting federated models that balance centralized control with distributed ownership.</p><h3>Integrating Lake Formation with Data Quality Frameworks</h3><p>Data governance extends beyond access control to include data quality management. Lake Formation can be integrated with data quality frameworks to enforce quality standards as part of the governance process.</p><h3>Implementing Quality-Based Access Controls</h3><p>A novel approach is to integrate data quality metrics into the access control framework:</p><ol><li><strong>Quality Tagging Pipeline</strong>: Automatically assess and tag datasets with quality metrics using AWS Glue jobs</li><li><strong>LF-Tag Based Access</strong>: Configure Lake Formation to use LF-Tags that incorporate quality metrics</li><li><strong>Quality-Aware Query Processing</strong>: Implement query modification patterns that filter low-quality data</li></ol><p>This pattern enables a new paradigm where access to data is determined not only by who is requesting it, but also by the quality characteristics of the data itself (Redman &amp; Blanton, 2020).</p><pre># Example Glue job snippet for quality tagging<br>from awsglue.context import GlueContext<br>from pyspark.context import SparkContext<br>from awsglue.job import Job<br>import boto3<br>sc = SparkContext()<br>glueContext = GlueContext(sc)<br>spark = glueContext.spark_session<br>job = Job(glueContext)<br># Calculate completeness score for a dataset<br>def calculate_completeness(df, critical_columns):<br>    total_rows = df.count()<br>    completeness_scores = {}<br>    <br>    for col in critical_columns:<br>        non_null_count = df.filter(df[col].isNotNull()).count()<br>        completeness_scores[col] = (non_null_count / total_rows) * 100<br>    <br>    overall_score = sum(completeness_scores.values()) / len(completeness_scores)<br>    return overall_score<br># Tag the table with quality metrics in Lake Formation<br>def apply_quality_tags(database_name, table_name, quality_score):<br>    lf_client = boto3.client(&#39;lakeformation&#39;)<br>    <br>    quality_category = &quot;HIGH&quot; if quality_score &gt;= 90 else &quot;MEDIUM&quot; if quality_score &gt;= 70 else &quot;LOW&quot;<br>    <br>    response = lf_client.add_lf_tags_to_resource(<br>        Resource={<br>            &#39;Table&#39;: {<br>                &#39;DatabaseName&#39;: database_name,<br>                &#39;Name&#39;: table_name<br>            }<br>        },<br>        LFTags=[<br>            {<br>                &#39;TagKey&#39;: &#39;DATA_QUALITY&#39;,<br>                &#39;TagValues&#39;: [quality_category]<br>            }<br>        ]<br>    )<br>    return response</pre><h3>Advanced Data Catalog Management</h3><h3>Automating Catalog Evolution with Schema Inference</h3><p>As data schemas evolve, maintaining an accurate data catalog becomes challenging. Advanced Lake Formation implementations leverage automated schema inference and catalog updates:</p><ol><li><strong>Continuous Schema Monitoring</strong>: Deploy Glue crawlers with event-driven triggers to detect schema changes</li><li><strong>Change Management Workflows</strong>: Implement approval workflows for schema changes using Step Functions</li><li><strong>Impact Analysis Automation</strong>: Automatically identify downstream dependencies affected by schema changes</li></ol><p>Research by the Data Management Association (DAMA) highlights automated catalog management as a critical capability for organizations with mature data governance practices (DAMA-DMBOK, 2022).</p><h3>Enriching Metadata with Business Context</h3><p>The Glue Data Catalog provides basic metadata capabilities, but advanced implementations extend this with rich business context:</p><ol><li><strong>Custom Metadata Extensions</strong>: Add business definitions, data lineage, and quality metrics</li><li><strong>Metadata Federation</strong>: Integrate with external business glossaries and metadata repositories</li><li><strong>Contextual Search Capabilities</strong>: Implement semantic search across enriched metadata</li></ol><p>Organizations like Netflix and Airbnb have published case studies demonstrating how enriched metadata significantly improves data discovery and understanding (Netflix Technology Blog, 2022).</p><h3>Optimizing Lake Formation for Analytical Workloads</h3><h3>Query Performance Optimization Patterns</h3><p>Lake Formation’s security model introduces additional authorization layers that can impact query performance. Advanced implementations employ several strategies to mitigate this:</p><ol><li><strong>Permission Caching Patterns</strong>: Implement application-level caching of permissions decisions</li><li><strong>Optimized Data Layout Strategies</strong>: Design partition structures that align with permission boundaries</li><li><strong>Hybrid Processing Models</strong>: Selectively use different processing engines based on query patterns</li></ol><p>Benchmarks conducted by Databricks (2023) demonstrate that properly optimized Lake Formation implementations can maintain performance comparable to direct S3 access while providing enhanced security.</p><h3>Real-time Analytics Integration</h3><p>While Lake Formation is often associated with batch analytics, it can be extended to support real-time analytical workloads:</p><ol><li><strong>Streaming Data Catalogs</strong>: Automatically register streaming sources in the Glue Data Catalog</li><li><strong>Permission-Aware Stream Processing</strong>: Extend Lake Formation permissions to Kinesis Data Analytics</li><li><strong>Hybrid Batch-Streaming Architectures</strong>: Unify governance across batch and streaming workloads</li></ol><p>The ability to maintain consistent governance across batch and streaming workflows addresses a significant gap in many data lake implementations (Kleppmann et al., 2023).</p><h3>Implementing Advanced Compliance Controls</h3><h3>Supporting Data Residency Requirements</h3><p>For organizations operating in multiple jurisdictions, data residency requirements present significant challenges. Lake Formation can be configured to support these requirements:</p><ol><li><strong>Location-Aware Access Controls</strong>: Restrict access based on the physical location of data</li><li><strong>Cross-Region Catalog Synchronization</strong>: Maintain consistent metadata across regions while keeping data local</li><li><strong>Residency-Aware Processing</strong>: Route analytical workloads to appropriate regions based on data location</li></ol><pre># Terraform example for regional data lake configuration<br>resource &quot;aws_lakeformation_resource&quot; &quot;regional_data_location&quot; {<br>  arn = &quot;arn:aws:s3:::${var.regional_bucket_name}&quot;<br>}<br>resource &quot;aws_lakeformation_permissions&quot; &quot;regional_access&quot; {<br>  principal   = aws_iam_role.regional_analytics_role.arn<br>  permissions = [&quot;DATA_LOCATION_ACCESS&quot;]<br>  data_location {<br>    arn = aws_lakeformation_resource.regional_data_location.arn<br>  }<br>  condition {<br>    string_equals {<br>      &quot;aws:RequestedRegion&quot; = var.allowed_region<br>    }<br>  }<br>}</pre><h3>Automating Regulatory Compliance</h3><p>Lake Formation can serve as a foundation for automated compliance controls:</p><ol><li><strong>Compliance Tagging Framework</strong>: Tag datasets with relevant regulatory requirements</li><li><strong>Rule-Based Permission Adjustments</strong>: Automatically adjust permissions based on regulatory changes</li><li><strong>Compliance Verification Workflows</strong>: Implement automated audit trails and verification processes</li></ol><p>Research by the International Association of Privacy Professionals (IAPP) indicates that organizations with automated compliance controls demonstrate significantly higher compliance rates and lower costs (IAPP, 2023).</p><h3>Architectural Patterns for Enterprise Scale</h3><h3>Multi-Team Lake Formation Implementation</h3><p>Scaling Lake Formation across multiple teams requires careful architectural planning:</p><ol><li><strong>Federated Administration Model</strong>: Delegate specific administrative functions to team-level administrators</li><li><strong>Service Control Policies</strong>: Implement guardrails to prevent circumvention of Lake Formation controls</li><li><strong>Centralized Policy Management</strong>: Maintain organization-wide policies with team-specific extensions</li></ol><p>Organizations like Capital One have published reference architectures demonstrating how these patterns enable enterprise-scale governance (AWS re:Invent, 2022).</p><h3>Integration with CI/CD Pipelines</h3><p>Advanced Lake Formation implementations treat data governance as code, integrating permission management into CI/CD pipelines:</p><ol><li><strong>Permission-as-Code</strong>: Define Lake Formation permissions using infrastructure as code tools</li><li><strong>Automated Testing</strong>: Test permission changes against compliance requirements</li><li><strong>Controlled Deployment</strong>: Implement staged rollouts of permission changes</li></ol><pre># Example GitHub Actions workflow for Lake Formation permission management<br>name: Deploy Lake Formation Permissions<br>on:<br>  push:<br>    paths:<br>      - &#39;permissions/*.json&#39;<br>    branches:<br>      - main<br>jobs:<br>  validate:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v2<br>      <br>      - name: Setup Python<br>        uses: actions/setup-python@v2<br>        with:<br>          python-version: &#39;3.9&#39;<br>      <br>      - name: Install dependencies<br>        run: |<br>          pip install aws-cdk-lib<br>      - name: Validate permission structures<br>        run: |<br>          python ./scripts/validate_permissions.py<br>  <br>  deploy:<br>    needs: validate<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v1<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: us-east-1<br>      <br>      - name: Deploy permissions<br>        run: |<br>          python ./scripts/deploy_permissions.py</pre><h3>Future Directions and Research Opportunities</h3><h3>Machine Learning for Automated Governance</h3><p>Emerging research points to the potential of using machine learning to enhance data governance:</p><ol><li><strong>Anomalous Access Detection</strong>: Identify unusual access patterns that may indicate misuse</li><li><strong>Intelligent Permission Recommendations</strong>: Automatically suggest appropriate permission levels</li><li><strong>Predictive Impact Analysis</strong>: Forecast the impact of permission changes on analytical workflows</li></ol><p>Early implementations by organizations like JP Morgan Chase demonstrate promising results, with up to 40% reduction in governance overhead (Financial Data Science Association, 2023).</p><h3>Quantum-Resistant Encryption Integration</h3><p>As quantum computing advances, securing sensitive data in data lakes becomes increasingly important:</p><ol><li><strong>Post-Quantum Cryptography Integration</strong>: Prepare Lake Formation for post-quantum cryptography standards</li><li><strong>Hybrid Encryption Models</strong>: Implement layered encryption strategies that combine traditional and quantum-resistant approaches</li><li><strong>Cryptographic Agility Frameworks</strong>: Build systems that can rapidly adapt to new cryptographic standards</li></ol><p>The National Institute of Standards and Technology (NIST) has published guidelines that can inform this integration (NIST, 2023).</p><h3>Conclusion</h3><p>AWS Lake Formation offers far more than a simplified approach to building data lakes. When implemented with advanced patterns and strategies, it becomes a powerful foundation for sophisticated data governance that balances security, compliance, and analytical agility.</p><p>Organizations that move beyond basic Lake Formation implementations can achieve several significant benefits:</p><ol><li><strong>Reduced Governance Overhead</strong>: Automate routine governance tasks while maintaining robust controls</li><li><strong>Enhanced Analytical Agility</strong>: Enable self-service analytics with appropriate guardrails</li><li><strong>Scalable Compliance</strong>: Adapt to evolving regulatory requirements without disrupting analytical workflows</li><li><strong>Future-Proof Architecture</strong>: Build data foundations that can evolve with changing business needs</li></ol><p>As data continues to grow in both volume and strategic importance, the ability to implement sophisticated governance at scale will become a critical differentiator for organizations across industries.</p><h3>References</h3><p>AWS Security Blog. (2023). Implementing data perimeters using AWS Lake Formation. Retrieved from <a href="https://aws.amazon.com/blogs/security/implementing-data-perimeters-using-aws-lake-formation/">https://aws.amazon.com/blogs/security/implementing-data-perimeters-using-aws-lake-formation/</a></p><p>DAMA International. (2022). DAMA-DMBOK: Data Management Body of Knowledge (3rd ed.). Technics Publications.</p><p>Databricks. (2023). Performance benchmarks for AWS Lake Formation integration. Databricks Engineering Blog.</p><p>Financial Data Science Association. (2023). Machine learning applications in financial data governance. Journal of Financial Data Science, 5(2), 78–92.</p><p>Gartner. (2022). Market Guide for Data Governance Solutions. Gartner Research.</p><p>IAPP. (2023). Annual Privacy Governance Report. International Association of Privacy Professionals.</p><p>Kleppmann, M., Beresford, A. R., &amp; Vinoski, S. (2023). Data-Intensive Applications: Principles and Emerging Architectures. O’Reilly Media.</p><p>Miloslavskaya, N., &amp; Tolstoy, A. (2016). Big Data, Fast Data and Data Lake Concepts. Procedia Computer Science, 88, 300–305.</p><p>NIST. (2023). Post-Quantum Cryptography Standardization. National Institute of Standards and Technology.</p><p>Netflix Technology Blog. (2022). Metacat: Making Big Data Discoverable and Meaningful at Netflix. Medium.</p><p>Redman, T. C., &amp; Blanton, A. (2020). Data Quality: The Field Guide. Harvard Business Review Press.</p><p>AWS re:Invent. (2022). Enterprise-scale data governance using AWS Lake Formation. Session ARC328. Las Vegas, NV.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6e4a971c230" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing AWS Secrets Manager Rotation for MongoDB with Auto-deploy to ECS]]></title>
            <link>https://medium.com/@binaya.puri/implementing-aws-secrets-manager-rotation-for-mongodb-with-auto-deploy-to-ecs-15e1b7af03c7?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/15e1b7af03c7</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[secretmanager]]></category>
            <category><![CDATA[autorotation]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 04:04:20 GMT</pubDate>
            <atom:updated>2024-09-24T04:04:20.485Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2n0uZ5w-EegnNrH9" /></figure><p>This article provides a comprehensive guide on setting up an automated system to rotate MongoDB credentials using AWS Secrets Manager, update the credentials in MongoDB Atlas, and automatically redeploy an ECS service with the new credentials. This setup enhances security by regularly rotating database credentials and ensures that application always has the most up-to-date access information.</p><h3>Why to Rotate Secrets?</h3><p><strong>Enhanced Security:<br></strong>Regularly rotating secrets minimizes the risk of unauthorized access. If credentials are compromised, the impact is limited to the time until the next rotation.</p><p><strong>Compliance</strong>:<br>Many regulatory standards and best practices require periodic rotation of credentials to ensure security and compliance.</p><p><strong>Reduce Attack Surface:<br></strong>By frequently changing secrets,reduce the window of opportunity for attackers to use compromised credentials.</p><p><strong>Automated Management:<br></strong>Automating secret rotation reduces the manual effort required to manage credentials, lowering the chances of human error and ensuring consistency.</p><p><strong>Improved Incident Response:<br></strong>In the event of a security incident, having an automated rotation policy in place can quickly mitigate risks by replacing compromised credentials.</p><h3>Prerequisites</h3><p>Before beginning, ensure we should have :</p><ol><li>An AWS account with appropriate permissions</li><li>A MongoDB Atlas account</li><li>An existing ECS cluster and service</li><li>Terraform installed on local machine</li><li>AWS CLI configured with credentials</li><li>A Slack workspace with permissions to create webhooks</li><li>Python 3.8 or later installed</li></ol><h3>Step-by-Step Implementation</h3><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*szfEw2hf8AiKlGrAPURWkA.png" /></figure><h3>1. Set Up MongoDB Atlas</h3><p>1.1. Log in to MongoDB Atlas account.</p><p>1.2. Create a new project (if not already created):</p><ul><li>Click on “New Project”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mXu9KTkLPvKJ6UA5mmU6GA.png" /></figure><ul><li>Enter a project name</li><li>Click “Create Project”</li></ul><p>1.3. Create a new cluster (if not already created):</p><ul><li>Click “Build a Cluster”</li><li>Choose preferred cloud provider and region</li><li>Select cluster tier</li><li>Click “Create Cluster”</li></ul><p>1.4. Create an API key:</p><ul><li>Go to “Project Settings” &gt; “Access Manager” &gt; “API Keys”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*ZTj5YHu28ffkU-TKog6xGA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*R4RRxhkMeAB8Donvqm5TUQ.png" /></figure><ul><li>Click “Create API Key”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8BlfyZecfa4pFBaQZxlcLQ.png" /></figure><ul><li>Set the appropriate permissions (Project Owner recommended for this use case)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*Hbwwnfls2HTh7Bp6jccl-Q.png" /></figure><ul><li>Note down the Public and Private keys</li></ul><p>1.5. Create a database user:</p><ul><li>Go to “Database Access”</li><li>Click “Add New Database User”</li><li>Choose authentication method (Password recommended)</li><li>Enter username and password</li><li>Assign appropriate database permissions</li><li>Click “Add User”</li></ul><p>1.6. Configure network access:</p><ul><li>Go to “Network Access”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*lnkWkNJcfhN95c6BHRGQuw.png" /></figure><ul><li>Click “Add IP Address”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Og02SPsLkEuuq68q-H-bQ.png" /></figure><ul><li>Add the NAT Gateway IP from VPC (we’ll get this later)</li><li>Temporarily add current IP address for initial setup</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*jNrWWZCATaNQWfT8w_O47A.png" /></figure><p>1.7. Get connection string:</p><ul><li>Go to “Clusters”</li><li>Click “Connect” on cluster</li><li>Choose “Connect application”</li><li>Copy the connection string</li></ul><h3>2. Set Up AWS Secrets Manager</h3><p>2.1. Log in to the AWS Management Console</p><p>2.2. Navigate to AWS Secrets Manager</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*kDyopPZDov23TVC0ywkJ7Q.png" /></figure><p>2.3. Create a new secret:</p><ul><li>Click “Store a new secret”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cm2JBVjZcyzmbpDwLC_3sA.png" /></figure><ul><li>Choose “Other type of secret”</li><li>Add a key-value pair with key “MONGO_URL” and value as our MongoDB connection string</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*UDt3OsgpOo-Zkk-eTE8qKQ.png" /></figure><ul><li>Click “Next”</li><li>Name secret (e.g., “test/rotation/aws”)</li><li>Add description (optional)</li><li>Click “Next” and then “Store”</li><li>* Note down the ARN of secret **</li></ul><h3>3. Create a Slack Webhook</h3><p>3.1. Go to Slack workspace</p><p>3.2. Create a new Slack app or use an existing one:</p><ul><li>Go to <a href="https://api.slack.com/apps">https://api.slack.com/apps</a></li><li>Click “Create New App”</li><li>Choose “From scratch”</li><li>Name the app and select workspace</li></ul><p>3.3. Enable Incoming Webhooks:</p><ul><li>In app’s settings, go to “Incoming Webhooks”</li><li>Activate Incoming Webhooks</li></ul><p>3.4. Create a new webhook:</p><ul><li>Scroll down and click “Add New Webhook to Workspace”</li><li>Choose the channel where notifications will be posted</li><li>Authorize the app</li></ul><p>3.5. Copy the Webhook URL for later use</p><h3>4. Prepare Local Environment</h3><p>4.1. Create a new directory for project:</p><p>mkdir secrets-manager-rotation cd secrets-manager-rotation</p><p>4.2. Initialize a new Terraform working directory:</p><p>terraform init</p><p>4.3. Create the following Terraform configuration files:</p><ul><li>variables.tf: For declaring variables</li><li>network.tf: For VPC and networking resources</li><li>lambda.tf: For Lambda function configuration</li><li>secrets.tf: For Secrets Manager configuration</li><li>cw_event.tf: For CloudWatch Event configuration</li></ul><p>4.4. Create a Python virtual environment:</p><p>python3 -m venv venv source venv/bin/activate</p><p>4.5. Install required Python packages:</p><p>pip install boto3 requests pymongo</p><p>4.6. Create lambda_function.py and mongodb.py files for Lambda function code.</p><h3>5. Configure Terraform Files</h3><h4>5.1 Project Structure</h4><p>secrets-manager-rotation/<br>│<br>├── my_lambda_package/<br>│ ├── bin/<br>│ ├── bson/<br>│ ├── certifi/<br>│ ├── certifi-2024.7.4.dist-info/<br>│ ├── chardet/<br>│ ├── chardet-4.0.0.dist-info/<br>│ ├── dns/<br>│ ├── dnspython-2.6.1.dist-info/<br>│ ├── gridfs/<br>│ ├── idna/<br>│ ├── idna-2.10.dist-info/<br>│ ├── pymongo/<br>│ ├── pymongo-4.8.0.dist-info/<br>│ ├── requests/<br>│ ├── requests-2.25.1.dist-info/<br>│ ├── urllib3/<br>│ ├── urllib3–1.26.5.dist-info/<br>│ └── urllib3–1.26.19.dist-info/<br>│ └── lambda_function.py<br>├── .gitignore<br>├── cw_event.tf<br>├── lambda.tf<br>├── lambda.zip<br>├── network.tf<br>├── secrets.tf<br>├── test.tfvars<br>└── variables.tf</p><h4>5.2. Define variables for project</h4><ul><li>project_name</li><li>environment</li><li>region</li><li>slack_webhook_url</li><li>secret_arn</li><li>atlas_project_id</li><li>atlas_public_key</li><li>atlas_private_key</li><li>service_name</li><li>cluster_name</li></ul><h4>5.3. Configure VPC, subnets, internet gateway, NAT gateway, and route tables</h4><pre>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}<br><br>resource &quot;aws_vpc&quot; &quot;main&quot; {<br>  cidr_block = &quot;10.0.0.0/16&quot;<br>}<br><br>resource &quot;aws_subnet&quot; &quot;public&quot; {<br>  vpc_id                  = aws_vpc.main.id<br>  cidr_block              = &quot;10.0.1.0/24&quot;<br>  map_public_ip_on_launch = true<br>}<br><br>resource &quot;aws_subnet&quot; &quot;private&quot; {<br>  vpc_id     = aws_vpc.main.id<br>  cidr_block = &quot;10.0.2.0/24&quot;<br>}<br><br>resource &quot;aws_internet_gateway&quot; &quot;igw&quot; {<br>  vpc_id = aws_vpc.main.id<br>}<br><br>resource &quot;aws_route_table&quot; &quot;public&quot; {<br>  vpc_id = aws_vpc.main.id<br><br>  route {<br>    cidr_block = &quot;0.0.0.0/0&quot;<br>    gateway_id = aws_internet_gateway.igw.id<br>  }<br>}<br><br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  subnet_id      = aws_subnet.public.id<br>  route_table_id = aws_route_table.public.id<br>}<br><br>resource &quot;aws_nat_gateway&quot; &quot;nat&quot; {<br>  allocation_id = aws_eip.nat.id<br>  subnet_id     = aws_subnet.public.id<br>}<br><br>resource &quot;aws_eip&quot; &quot;nat&quot; {<br>  domain = &quot;vpc&quot;<br>}<br><br>resource &quot;aws_route_table&quot; &quot;private&quot; {<br>  vpc_id = aws_vpc.main.id<br><br>  route {<br>    cidr_block     = &quot;0.0.0.0/0&quot;<br>    nat_gateway_id = aws_nat_gateway.nat.id<br>  }<br>}<br><br>resource &quot;aws_route_table_association&quot; &quot;private&quot; {<br>  subnet_id      = aws_subnet.private.id<br>  route_table_id = aws_route_table.private.id<br>}<br><br>resource &quot;aws_security_group&quot; &quot;lambda_sg&quot; {<br>  vpc_id = aws_vpc.main.id<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>}<br><br>output &quot;nat_gateway_ip&quot; {<br>  value       = aws_eip.nat.public_ip<br>  description = &quot;The Elastic IP address associated with the NAT Gateway.&quot;<br>}<br><br></pre><h4>5.4. Set up the IAM role, policies, and Lambda function configuration.</h4><pre>resource &quot;aws_iam_role&quot; &quot;lambda_role&quot; {<br>  name = &quot;${var.project_name}_${var.environment}_lambda_role&quot;<br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Action = &quot;sts:AssumeRole&quot;,<br>        Effect = &quot;Allow&quot;,<br>        Principal = {<br>          Service = &quot;lambda.amazonaws.com&quot;<br>        },<br>      },<br>    ],<br>  })<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;lambda_policy&quot; {<br>  name        = &quot;${var.project_name}_${var.environment}_lambda_policy&quot;<br>  description = &quot;Policy for Lambda to access Secrets Manager, ECS, and CloudWatch&quot;<br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Action = [<br>          &quot;secretsmanager:GetSecretValue&quot;,<br>          &quot;secretsmanager:PutSecretValue&quot;<br>        ],<br>        Effect   = &quot;Allow&quot;,<br>        Resource = &quot;*&quot;<br>      },<br>      {<br>        Action = [<br>          &quot;logs:CreateLogGroup&quot;,<br>          &quot;logs:CreateLogStream&quot;,<br>          &quot;logs:PutLogEvents&quot;<br>        ],<br>        Effect   = &quot;Allow&quot;,<br>        Resource = &quot;*&quot;<br>      },<br>      {<br>        Action = [<br>          &quot;ec2:*&quot;,<br>        ],<br>        Effect   = &quot;Allow&quot;,<br>        Resource = &quot;*&quot;<br>      },<br>      {<br>        Action = [<br>          &quot;ecs:UpdateService&quot;<br>        ],<br>        Effect   = &quot;Allow&quot;,<br>        Resource = &quot;*&quot;<br>      },<br>    ],<br>  })<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;lambda_role_policy_attachment&quot; {<br>  role       = aws_iam_role.lambda_role.name<br>  policy_arn = aws_iam_policy.lambda_policy.arn<br>}<br><br>resource &quot;aws_lambda_function&quot; &quot;mongo_rotate_function&quot; {<br>  filename         = &quot;lambda.zip&quot;<br>  function_name    = &quot;${var.project_name}_${var.environment}_mongo_rotate_function&quot;<br>  role             = aws_iam_role.lambda_role.arn<br>  handler          = &quot;lambda_function.lambda_handler&quot;<br>  runtime          = &quot;python3.8&quot;<br>  timeout          = 30<br>  memory_size      = 256<br>  source_code_hash = filebase64sha256(&quot;lambda.zip&quot;)<br>  vpc_config {<br>    subnet_ids         = [aws_subnet.private.id]<br>    security_group_ids = [aws_security_group.lambda_sg.id]<br>  }<br>  environment {<br>    variables = {<br>      SLACK_WEBHOOK_URL = var.slack_webhook_url<br>      SECRET_ARN        = var.secret_arn<br>      PROJECT_ID        = var.project_id<br>      PUBLIC_KEY        = var.public_key<br>      PRIVATE_KEY       = var.private_key<br>    }<br>  }<br>}<br><br>resource &quot;aws_lambda_permission&quot; &quot;allow_secretsmanager&quot; {<br>  statement_id  = &quot;AllowExecutionFromSecretsManager&quot;<br>  action        = &quot;lambda:InvokeFunction&quot;<br>  function_name = aws_lambda_function.mongo_rotate_function.function_name<br>  principal     = &quot;secretsmanager.amazonaws.com&quot;<br>}<br></pre><h4>5.5. Configure the Secrets in the secrets manager</h4><p>Configure the Secrets Manager secret version.</p><pre>resource &quot;aws_secretsmanager_secret_version&quot; &quot;mongo_url_secret_version&quot; {<br>  secret_id = &quot;arn*********************************************************&quot;<br>  secret_string = jsonencode({<br>    MONGO_URL = &quot;mongodb+srv://BINAYA:BINAYA@*******.******.mongodb.net/binaya_todo?retryWrites=true&amp;w=majority&amp;appName=*****&quot;<br>  })<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUMznkrotP5vErPS7i5FVA.png" /></figure><h4>5.6. Configure CloudWatch Event Rule for Scheduled Rotation</h4><p>Set up the CloudWatch Event rule to trigger Lambda function.</p><pre>resource &quot;aws_cloudwatch_event_rule&quot; &quot;every_5_minutes&quot; {<br>  name        = &quot;${var.project_name}_${var.environment}_every_1_minutes&quot;<br>  description = &quot;CloudWatch Event rule to trigger Lambda function every 5 minutes&quot;<br>#   schedule_expression = &quot;rate(5 minutes)&quot;<br>  schedule_expression = &quot;cron(0 0 ? * 7 *)&quot;<br>}<br><br>resource &quot;aws_cloudwatch_event_target&quot; &quot;lambda_target&quot; {<br>  rule      = aws_cloudwatch_event_rule.every_5_minutes.name<br>  target_id = &quot;lambda_target&quot;<br>  arn       = aws_lambda_function.mongo_rotate_function.arn<br>}<br><br>resource &quot;aws_lambda_permission&quot; &quot;allow_cloudwatch_to_invoke&quot; {<br>  statement_id  = &quot;AllowExecutionFromCloudWatch&quot;<br>  action        = &quot;lambda:InvokeFunction&quot;<br>  function_name = aws_lambda_function.mongo_rotate_function.function_name<br>  principal     = &quot;events.amazonaws.com&quot;<br>  source_arn    = aws_cloudwatch_event_rule.every_5_minutes.arn<br>}</pre><h3>6. Implement Lambda Function</h3><p>→ In my_lambda_package/lambda_function.py, implement the main handler and helper functions as provided in code.</p><pre>import json<br>import boto3<br>import requests<br>import random<br>import string<br>import os<br>import logging<br>from urllib.parse import urlparse, quote_plus, unquote_plus<br>from requests.auth import HTTPDigestAuth<br><br># Set up logging<br>logger = logging.getLogger()<br>logger.setLevel(logging.INFO)<br><br>def random_password(length=16):<br>    letters_and_digits = string.ascii_letters + string.digits<br>    return &#39;&#39;.join(random.choice(letters_and_digits) for i in range(length))<br><br>def parse_mongo_uri(uri):<br>    parsed = urlparse(uri)<br>    username = unquote_plus(parsed.username) if parsed.username else &#39;&#39;<br>    password = unquote_plus(parsed.password) if parsed.password else &#39;&#39;<br>    hosts = parsed.netloc.split(&#39;@&#39;)[-1] if &#39;@&#39; in parsed.netloc else parsed.netloc<br>    hosts = hosts.split(&#39;,&#39;)<br>    database = parsed.path[1:] if parsed.path else &#39;&#39;<br>    query = parsed.query<br>    return {<br>        &#39;username&#39;: username,<br>        &#39;password&#39;: password,<br>        &#39;hosts&#39;: hosts,<br>        &#39;database&#39;: database,<br>        &#39;query&#39;: query<br>    }<br><br>def get_aws_ip_ranges():<br>    response = requests.get(&quot;https://ip-ranges.amazonaws.com/ip-ranges.json&quot;)<br>    ip_ranges = response.json()[&quot;prefixes&quot;]<br>    region = &quot;us-east-1&quot;  # Ensure only us-east-1 region IPs are used<br>    service = &quot;AMAZON&quot;<br>    return [item[&quot;ip_prefix&quot;] for item in ip_ranges if item[&quot;region&quot;] == region and item[&quot;service&quot;] == service]<br><br>def update_ip_whitelist(atlas_project_id, atlas_api_public_key, atlas_api_private_key, ip_ranges):<br>    api_url = f&quot;https://cloud.mongodb.com/api/atlas/v1.0/groups/{atlas_project_id}/accessList&quot;<br>    headers = {<br>        &quot;Content-Type&quot;: &quot;application/json&quot;<br>    }<br>    auth = HTTPDigestAuth(atlas_api_public_key, atlas_api_private_key)<br>    <br>    # Get current whitelist<br>    response = requests.get(api_url, headers=headers, auth=auth)<br>    if response.status_code != 200:<br>        logger.error(f&quot;Failed to get current IP whitelist: {response.status_code} - {response.text}&quot;)<br>        raise ValueError(f&quot;Failed to get current IP whitelist: {response.status_code} - {response.text}&quot;)<br><br>    try:<br>        current_whitelist = response.json()[&#39;results&#39;]<br>        logger.info(f&quot;Current whitelist: {current_whitelist}&quot;)<br>        current_whitelist = [entry[&quot;cidrBlock&quot;] for entry in current_whitelist if &quot;cidrBlock&quot; in entry]<br>    except (ValueError, TypeError, KeyError) as e:<br>        logger.error(f&quot;Error parsing current whitelist response: {e}&quot;)<br>        raise ValueError(f&quot;Error parsing current whitelist response: {e}&quot;)<br>    <br>    # Check the number of current entries<br>    current_count = len(current_whitelist)<br>    logger.info(f&quot;Current number of entries in whitelist: {current_count}&quot;)<br><br>    # Determine how many new entries we can add<br>    max_entries = 200<br>    new_entries = []<br>    for ip_range in ip_ranges:<br>        if ip_range not in current_whitelist:<br>            new_entries.append({&quot;ipAddress&quot;: ip_range, &quot;comment&quot;: &quot;Added by Lambda function&quot;})<br>    <br>    new_count = len(new_entries)<br>    logger.info(f&quot;Number of new entries to add: {new_count}&quot;)<br><br>    # If adding new entries exceeds the limit, remove the oldest entries first<br>    if current_count + new_count &gt; max_entries:<br>        excess_count = (current_count + new_count) - max_entries<br>        logger.info(f&quot;Removing {excess_count} old entries to make space for new ones&quot;)<br>        <br>        # Remove the oldest entries (or implement logic for which entries to remove)<br>        for i in range(excess_count):<br>            if i &gt;= len(current_whitelist):<br>                logger.error(f&quot;Index {i} out of range for current whitelist of length {len(current_whitelist)}&quot;)<br>                break<br>            entry_to_remove = current_whitelist[i]<br>            remove_url = f&quot;{api_url}/{quote_plus(entry_to_remove)}&quot;<br>            remove_response = requests.delete(remove_url, headers=headers, auth=auth)<br>            if remove_response.status_code != 204:<br>                logger.error(f&quot;Failed to remove IP range {entry_to_remove}: {remove_response.status_code} - {remove_response.text}&quot;)<br>                raise ValueError(f&quot;Failed to remove IP range {entry_to_remove}: {remove_response.status_code} - {remove_response.text}&quot;)<br>            logger.info(f&quot;Removed IP range from whitelist: {entry_to_remove}&quot;)<br>    <br>    if new_entries:<br>        data = json.dumps(new_entries)<br>        response = requests.post(api_url, headers=headers, auth=auth, data=data)<br>        if response.status_code != 201:<br>            logger.error(f&quot;Failed to add IP ranges: {response.status_code} - {response.text}&quot;)<br>            raise ValueError(f&quot;Failed to add IP ranges: {response.status_code} - {response.text}&quot;)<br>        logger.info(f&quot;Added IP ranges to whitelist: {new_entries}&quot;)<br>    else:<br>        logger.info(&quot;No new IP ranges to add to whitelist&quot;)<br><br>def send_slack_notification(webhook_url, message):<br>    slack_message = {&#39;text&#39;: message}<br>    response = requests.post(webhook_url, data=json.dumps(slack_message), headers={&#39;Content-Type&#39;: &#39;application/json&#39;})<br>    if response.status_code != 200:<br>        logger.error(f&quot;Request to Slack returned an error {response.status_code}: {response.text}&quot;)<br>        raise ValueError(f&quot;Request to Slack returned an error {response.status_code}: {response.text}&quot;)<br>    logger.info(&quot;Slack notification sent successfully&quot;)<br><br>def lambda_handler(event, context):<br>    try:<br>        # Check for required environment variables<br>        required_env_vars = [&#39;SECRET_ARN&#39;, &#39;SLACK_WEBHOOK_URL&#39;, &#39;PROJECT_ID&#39;, &#39;PUBLIC_KEY&#39;, &#39;PRIVATE_KEY&#39;, &#39;CLUSTER_NAME&#39;, &#39;SERVICE_NAME&#39;]<br>         for var in required_env_vars:<br>            if var not in os.environ:<br>                raise ValueError(f&quot;Missing required environment variable: {var}&quot;)<br>        <br>        secret_arn = os.environ[&#39;SECRET_ARN&#39;]<br>        slack_webhook_url = os.environ[&#39;SLACK_WEBHOOK_URL&#39;]<br>        atlas_project_id = os.environ[&#39;PROJECT_ID&#39;]<br>        atlas_api_public_key = os.environ[&#39;PUBLIC_KEY&#39;]<br>        atlas_api_private_key = os.environ[&#39;PRIVATE_KEY&#39;]<br>        cluster_name = os.environ[&#39;CLUSTER_NAME&#39;]        <br>        service_name = os.environ[&#39;SERVICE_NAME&#39;] <br>        <br>        client = boto3.client(&#39;secretsmanager&#39;)<br>        <br>        # Retrieve the current secret value<br>        get_secret_value_response = client.get_secret_value(SecretId=secret_arn)<br>        secret = json.loads(get_secret_value_response[&#39;SecretString&#39;])<br>        <br>        # Generate a new password<br>        new_password = random_password()<br>        <br>        # Parse MongoDB connection string<br>        old_url = secret[&#39;MONGO_URL&#39;]<br>        parsed_url = parse_mongo_uri(old_url)<br>        username = parsed_url[&#39;username&#39;]<br>        <br>        # Construct the new MongoDB URL with the new password<br>        hosts_str = &#39;,&#39;.join(parsed_url[&#39;hosts&#39;])<br>        new_url = f&quot;mongodb+srv://{quote_plus(username)}:{quote_plus(new_password)}@{hosts_str}/{parsed_url[&#39;database&#39;]}?{parsed_url[&#39;query&#39;]}&quot;<br>        <br>        # Update the user password in MongoDB Atlas<br>        update_user_url = f&quot;https://cloud.mongodb.com/api/atlas/v1.0/groups/{atlas_project_id}/databaseUsers/admin/{username}&quot;<br>        auth = HTTPDigestAuth(atlas_api_public_key, atlas_api_private_key)<br>        headers = {&#39;Content-Type&#39;: &#39;application/json&#39;}<br>        data = {<br>            &#39;password&#39;: new_password<br>        }<br>        <br>        logger.info(f&quot;Attempting to update password for user: {username}&quot;)<br>        response = requests.patch(update_user_url, auth=auth, headers=headers, json=data)<br>        <br>        if response.status_code != 200:<br>            logger.error(f&quot;Failed to update user password in MongoDB Atlas: {response.status_code}&quot;)<br>            logger.error(f&quot;Response content: {response.text}&quot;)<br>            raise ValueError(f&quot;Failed to update user password in MongoDB Atlas: {response.status_code} - {response.text}&quot;)<br>        <br>        logger.info(&quot;Password updated successfully in MongoDB Atlas&quot;)<br>        <br>        # Update the secret in Secrets Manager<br>        client.put_secret_value(<br>            SecretId=secret_arn,<br>            SecretString=json.dumps({<br>                &#39;MONGO_URL&#39;: new_url<br>            })<br>        )<br>        <br>        logger.info(&quot;Secret updated successfully in AWS Secrets Manager&quot;)<br>        <br>        # Send notification to Slack about secret rotation<br>        send_slack_notification(slack_webhook_url, f&quot;MongoDB URL rotated successfully for user: {username}&quot;)<br>        <br>        # Send notification to Slack about ECS deployment attempt<br>        send_slack_notification(slack_webhook_url, &quot;Attempting to deploy ECS service with updated secrets&quot;)<br>        <br>        # Update the ECS service to force new deployment<br>        ecs_client = boto3.client(&#39;ecs&#39;)<br>        ecs_response = ecs_client.update_service(<br>            cluster=cluster_name,<br>            service=service_name,<br>            forceNewDeployment=True<br>        )<br>        logger.info(f&quot;ECS service update response: {ecs_response}&quot;)<br><br>        # Send notification to Slack about successful ECS deployment<br>        send_slack_notification(slack_webhook_url, &quot;ECS service deployed successfully with updated secrets&quot;)<br><br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: json.dumps(&#39;MongoDB URL rotated and ECS service updated successfully&#39;)<br>        }<br><br>    except Exception as e:<br>        logger.error(f&quot;An error occurred: {str(e)}&quot;)<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;body&#39;: json.dumps(f&quot;Error: {str(e)}&quot;)<br>        }</pre><h3>7. Package Lambda Function</h3><p>Create a deployment package:</p><pre>pip install --target ./my_lambda_package boto3 requests pymongo<br><br>cd my_lambda_package<br><br># Create a zip file of the Lambda package<br>zip -r ../lambda.zip .<br><br># Go back to the previous directory<br>cd ..</pre><h3>8. Apply Terraform Configuration</h3><p>→ Initialize Terraform (if not done already):</p><p>terraform init</p><p>→ Plan Terraform changes:</p><p>terraform plan -var-file=&quot;test.tfvars&quot;</p><p>→ Apply the Terraform configuration:</p><p>terraform apply -var-file=&quot;test.tfvars&quot;</p><h3>9. Configure MongoDB Atlas Whitelist</h3><p>→ After applying Terraform, get the NAT Gateway IP:</p><p>terraform output nat_gateway_ip</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*sJBKyDHZ8IbBy3mYoxW7og.png" /></figure><p>→ Add this IP to MongoDB Atlas network access list:</p><ul><li>Go to MongoDB Atlas &gt; Network Access</li><li>Add the NAT Gateway IP to the IP Whitelist</li></ul><h3>10. Test the Setup</h3><p>→ Manually trigger the Lambda function through the AWS Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUMznkrotP5vErPS7i5FVA.png" /></figure><p>→ Check Secrets Manager to verify that the secret has been updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PE-luhUTG9Qws8S4drH5xg.png" /></figure><p>→ Verify that the ECS service has been redeployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qSufCFjChuT0u9v5VaLqkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMiShi5826E8ztl-rRb4Cw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SeWUpFSLSFeCq0rineIzw.png" /></figure><p>→ Check Slack channel for notifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*JiG_NzAUfvDBke8fsgRdZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mzFr0tstEtz1A66dqobbiQ.png" /></figure><h3>Conclusion</h3><p>This setup provides an automated way to rotate MongoDB credentials, update them in AWS Secrets Manager, and redeploy ECS service. It enhances security by regularly changing database credentials and ensures that application always uses the most up-to-date access information.</p><p>Remember to monitor the system regularly and adjust the rotation schedule as needed based on security requirements.</p><p>This detailed guide covers all the necessary steps, including those not explicitly mentioned in original code. It provides a comprehensive walkthrough of setting up the entire system, from MongoDB Atlas configuration to Terraform setup and Lambda function implementation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15e1b7af03c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build, Train, and Deploy a Machine Learning Model using Sagemaker]]></title>
            <link>https://medium.com/@binaya.puri/build-train-and-deploy-a-machine-learning-model-using-sagemaker-c98c6ed098d2?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c98c6ed098d2</guid>
            <category><![CDATA[training]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[build]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sagemaker]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Fri, 26 Apr 2024 16:55:22 GMT</pubDate>
            <atom:updated>2024-04-26T16:55:22.767Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hz-dLFrD1usJ1skdpVXW8A.jpeg" /></figure><p>Amazon SageMaker is a fully managed service that enables developers and data scientists to quickly build, train, and deploy machine learning (ML) models. It offers a wide range of features including data labeling, model training, tuning and deployment, as well as pre-built algorithms and the ability to import your own algorithms. SageMaker is designed to make the process of developing, training and deploying ML models easier and more efficient, allowing users to focus on their core machine learning tasks rather than on the underlying infrastructure.</p><h3>Why Use SageMaker?</h3><p>SageMaker’s versatility, scalability, and integration with other AWS services make it an excellent choice. For every phase of the machine learning lifecycle, from data preparation to model deployment, it offers a single platform. With SageMaker’s managed environment, users can concentrate on their models and applications instead of worrying about the hassles of setting up and maintaining the infrastructure needed for ML projects. SageMaker’s capabilities are further enhanced by its integration with AWS services, such as S3 for data storage and EC2 for computing resources, which makes it an effective tool for creating and implementing ML models at scale.</p><h3>To Build, Train, and Deploy a Machine Learning Model using Sagemaker follow these steps:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*j8SPxkC3UzvbMeEdKmwo_Q.png" /></figure><h4>Step 1: Create an Amazon SageMaker Notebook Instance</h4><p>First, create an instance of an Amazon SageMaker Notebook. To use SageMaker for machine learning projects, you must first create a notebook instance. This instance provides a pre-configured environment for creating and managing Jupyter notebooks. It is a fully managed Amazon EC2 compute instance that runs the Jupyter Notebook App. You choose the preferred instance type, platform identifier, and IAM role when creating a notebook instance. This assigns the required permissions to access S3 buckets and other AWS resources. The notebook instance is prepared for machine learning development out of the box with a pre-configured Jupyter notebook server, SageMaker and AWS SDK libraries, and a collection of Anaconda libraries.</p><p>→ Choose <strong>Notebook instances</strong>, and then choose <strong>Create notebook instance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-bbSPb7nGqWOB83lQ41HQ.png" /></figure><p>→ Click on create notebook instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tq03kZlPNDXThiFTmZaHA.png" /></figure><p>→ On the <strong>Create notebook instance</strong> page, provide the following information (if a field is not mentioned, leave the default values):</p><ol><li>For <strong>Notebook instance name</strong>, type a name for your notebook instance.</li><li>For <strong>Notebook Instance type</strong>, choose ml.t2.medium. This is the least expensive instance type that notebook instances support, and it suffices for this exercise. If a ml.t2.medium instance type isn&#39;t available in your current AWS Region, choose ml.t3.medium.</li><li>For <strong>Platform Identifier</strong>, choose a platform type to create the notebook instance on. This platform type dictates the Operating System and the JupyterLab version that your notebook instance is created with. For information about platform identifier type, see <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-al2.html">Amazon Linux 2 notebook instances</a>. For information about JupyterLab versions, see <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-jl.html">JupyterLab versioning</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*gmL91zoZ1ebIvlqk5DRJfA.png" /></figure><p>→ For <strong>IAM role</strong>, choose <strong>Create a new role</strong>, and then choose <strong>Create role</strong>. This IAM role automatically gets permissions to access any S3 bucket that has sagemaker in the name. It gets these permissions through the AmazonSageMakerFullAccess policy, which SageMaker attaches to the role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*2hNxH5KUMTa0DnARIlynJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*W9Frtvol9PPmrM9xXOuKlg.png" /></figure><p>→ Choose <strong>Create notebook instance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*YcX9O8T6dw0dUHhMtJrbcw.png" /></figure><p>In a few minutes, SageMaker launches an ML compute instance — in this case, a notebook instance — and attaches a 5 GB of Amazon EBS storage volume to it. The notebook instance has a preconfigured Jupyter notebook server, SageMaker and AWS SDK libraries, and a set of Anaconda libraries</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ql1TRV24Prd5FbTuZVdsrQ.png" /></figure><p>→ It’s status has changed to inservice now and we got actions as open jupyter and jupyter lab</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rBq55l6XRdz92RyZI9k6A.png" /></figure><h3>Step 2: Create a Jupyter Notebook</h3><p>After setting up the notebook instance, the next step is to create a Jupyter notebook within the instance. This notebook serves as the primary interface for scripting, training, and deploying ML models. You can choose between the JupyterLab interface for a more modern, web-based UI, or the classic Jupyter view. The notebook environment is pre-configured with the necessary libraries and tools for ML development, facilitating the process of data exploration, model training, and deployment.</p><p>→ On the <strong>Notebook instances</strong> page, open your notebook instance by choosing either <strong>Open JupyterLab</strong> for the JupyterLab interface or <strong>Open Jupyter</strong> for the classic Jupyter view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*oH3Ue8GdJcccHFk5Tc2WaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slA2BHiOp8D5_pDwf-Xl1Q.png" /></figure><p>→ This UI appears after using Jypyter lab view</p><p>→ If you opened the notebook in the JupyterLab view, on the <strong>File</strong> menu, choose <strong>New</strong>, and then choose <strong>Notebook</strong>. For <strong>Select Kernel</strong>, choose <strong>conda_python3</strong>. This preinstalled environment includes the default Anaconda installation and Python 3. or Directly can select conda_python3 in the Notebook as showing below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bnZYM_Ip6z8UrQ3_pYw_lQ.png" /></figure><p>→ Save the notebooks : choose <strong>File</strong>, choose <strong>Save Notebook As…</strong>, and then rename the notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fw8DcL2p9-06wED-8Ao78A.png" /></figure><h3>Step 3: Download, Explore, and Transform a Dataset</h3><p>The third step involves loading and preparing the dataset for analysis. In this example, the Adult Census dataset is used, which is loaded using the SHAP library. The dataset is then explored and transformed as needed, with the transformed data being split into training, validation, and test sets. These sets are then converted to CSV files and uploaded to an Amazon S3 bucket, preparing the data for model training.</p><h3>Load Adult Census Dataset Using SHAP</h3><p>Using the SHAP library, import the Adult Census dataset as shown following:</p><pre>import shap<br>X, y = shap.datasets.adult()<br>X_display, y_display = shap.datasets.adult(display=True)<br>feature_names = list(X.columns)<br>feature_names</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*V4jzlKxwKqmm9r2Y5LIB3g.png" /></figure><p>→ If the current Jupyter kernel does not have the SHAP library, install it by running the following conda command:<br>***required***:</p><pre>%conda install -c conda-forge shap</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*47xLqSKATWT3w1gOg4nvJA.png" /></figure><p>If you’re using JupyterLab, you must manually refresh the kernel after the installation and updates have completed. Run the following IPython script to shut down the kernel (the kernel will restart automatically):</p><pre>import IPython<br>IPython.Application.instance().kernel.do_shutdown(True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*VnoGFC2zT5YX3AKmWcKf_A.png" /></figure><p>The feature_names list object should return the following list of features:</p><h3>Overview the Dataset</h3><p>Run the following script to display the statistical overview of the dataset and histograms of the numeric features.</p><pre>display(X.describe())<br>hist = X.hist(bins=30, sharey=True, figsize=(20, 10))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RivmilYcHmK_5jiLpSWyIQ.png" /></figure><p>If you want to use a dataset that needs to be cleaned and transformed, you can simplify and streamline data preprocessing and feature engineering using Amazon SageMaker Data Wrangler. To learn more, see Prepare ML Data with Amazon SageMaker Data Wrangler.</p><h3>Split the Dataset into Train, Validation, and Test Datasets</h3><p>Using Sklearn, split the dataset into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate the performance of the final trained model. The dataset is randomly sorted with the fixed random seed: 80 percent of the dataset for training set and 20 percent of it for a test set.</p><pre>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)<br>X_train_display = X_display.loc[X_train.index]</pre><p>Split the training set to separate out a validation set. The validation set is used to evaluate the performance of the trained model while tuning the model’s hyperparameters. 75 percent of the training set becomes the final training set, and the rest is the validation set.</p><pre>X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)<br>X_train_display = X_display.loc[X_train.index]<br>X_val_display = X_display.loc[X_val.index]</pre><p>Using the pandas package, explicitly align each dataset by concatenating the numeric features with the true labels.</p><pre>import pandas as pd<br>train = pd.concat([pd.Series(y_train, index=X_train.index,<br>                             name=&#39;Income&gt;50K&#39;, dtype=int), X_train], axis=1)<br>validation = pd.concat([pd.Series(y_val, index=X_val.index,<br>                            name=&#39;Income&gt;50K&#39;, dtype=int), X_val], axis=1)<br>test = pd.concat([pd.Series(y_test, index=X_test.index,<br>                            name=&#39;Income&gt;50K&#39;, dtype=int), X_test], axis=1)</pre><p>Check if the dataset is split and structured as expected:</p><pre>train</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iduoJTDoKytCQnrDZrRSWg.png" /></figure><pre>validation</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeRYSBySEXkPyXCPwaT9ug.png" /></figure><pre>test</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxarS9OHxsdbyYn32dxN7w.png" /></figure><h3>Convert the Train and Validation Datasets to CSV Files</h3><p>Convert the train and validation dataframe objects to CSV files to match the input file format for the XGBoost algorithm.</p><pre># Use &#39;csv&#39; format to store the data<br># The first column is expected to be the output column<br>train.to_csv(&#39;train.csv&#39;, index=False, header=False)<br>validation.to_csv(&#39;validation.csv&#39;, index=False, header=False)</pre><h3>Upload the Datasets to Amazon S3</h3><p>Using the SageMaker and Boto3, upload the training and validation datasets to the default Amazon S3 bucket. The datasets in the S3 bucket will be used by a compute-optimized SageMaker instance on Amazon EC2 for training.</p><p>The following code sets up the default S3 bucket URI for your current SageMaker session, creates a new demo-sagemaker-xgboost-adult-income-prediction folder, and uploads the training and validation datasets to the data subfolder.</p><pre>import sagemaker, boto3, os<br>bucket = sagemaker.Session().default_bucket()<br>prefix = &quot;demo-sagemaker-xgboost-adult-income-prediction&quot;<br><br>boto3.Session().resource(&#39;s3&#39;).Bucket(bucket).Object(<br>    os.path.join(prefix, &#39;data/train.csv&#39;)).upload_file(&#39;train.csv&#39;)<br>boto3.Session().resource(&#39;s3&#39;).Bucket(bucket).Object(<br>    os.path.join(prefix, &#39;data/validation.csv&#39;)).upload_file(&#39;validation.csv&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gAQvw_Wws2qFKQknfBGWHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsNPlQkdzUtD1K2VjTRmrw.png" /></figure><pre>! aws s3 ls {bucket}/{prefix}/data --recursive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*2WFYMobDgj4u3mrWINzI7g.png" /></figure><h3>Step 4: Train a Model</h3><p>With the dataset prepared, the next step is to train a model using SageMaker. This involves selecting an appropriate algorithm, such as XGBoost, and configuring an estimator with the necessary parameters. The training data is specified, and the model training is initiated. SageMaker provides tools for monitoring the training process and evaluating the model’s performance.</p><p>To train a model using Amazon SageMaker, follow these steps:</p><p><strong>Choose the Training Algorithm</strong>: Begin by selecting an appropriate algorithm for your dataset. Amazon SageMaker offers various built-in algorithms, such as the XGBoost algorithm for simplicity in this example. For those needing automation in model selection, especially for tabular datasets, Amazon SageMaker Autopilot can be utilized to automate the machine learning process.</p><p><strong>Set Up Your SageMaker Environment</strong>: Import the SageMaker Python SDK to access AWS services and features necessary for training. Retrieve essential details like the AWS Region and IAM role associated with your SageMaker session using the following code snippets:</p><pre>import sagemaker<br><br>region = sagemaker.Session().boto_region_name<br>print(&quot;AWS Region: {}&quot;.format(region))<br><br>role = sagemaker.get_execution_role()<br>print(&quot;RoleArn: {}&quot;.format(role))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*F-0qzUUMqbQiyJxRcOKwhA.png" /></figure><p><strong>Install or Upgrade SageMaker Python SDK</strong>: Ensure you have the latest version of the SageMaker Python SDK. If not, update it using the command</p><pre>! pip install -qU sagemaker</pre><p>Remember to restart the kernel if you’re working within SageMaker Studio or notebook instances to apply the update.</p><p><strong>Create an XGBoost Estimator</strong>: Configure an estimator for the XGBoost algorithm by specifying parameters such as the container image URI, IAM role, instance type, and output path for storing model artifacts. For instance:</p><pre>from sagemaker.debugger import Rule, ProfilerRule, rule_configs<br>from sagemaker.session import TrainingInput<br><br>s3_output_location=&#39;s3://{}/{}/{}&#39;.format(bucket, prefix, &#39;xgboost_model&#39;)<br><br>container=sagemaker.image_uris.retrieve(&quot;xgboost&quot;, region, &quot;1.2-1&quot;)<br>print(container)<br><br>xgb_model=sagemaker.estimator.Estimator(<br>    image_uri=container,<br>    role=role,<br>    instance_count=1,<br>    instance_type=&#39;ml.m4.xlarge&#39;,<br>    volume_size=5,<br>    output_path=s3_output_location,<br>    sagemaker_session=sagemaker.Session(),<br>    rules=[<br>        Rule.sagemaker(rule_configs.create_xgboost_report()),<br>        ProfilerRule.sagemaker(rule_configs.ProfilerReport())<br>    ]<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*ZkKfV6YgMSnvF0wkydOcmw.png" /></figure><p><strong>Set Hyperparameters</strong>: Define hyperparameters for the XGBoost algorithm, like max_depth, eta, and num_round.</p><pre>xgb_model.set_hyperparameters(<br>    max_depth = 5,<br>    eta = 0.2,<br>    gamma = 4,<br>    min_child_weight = 6,<br>    subsample = 0.7,<br>    objective = &quot;binary:logistic&quot;,<br>    num_round = 1000<br>)</pre><p><strong>Prepare Training Data</strong>: Configure TrainingInput objects for your training and validation datasets stored in S3, specifying their content type (e.g., CSV).</p><pre>from sagemaker.session import TrainingInput<br><br>train_input = TrainingInput(<br>    &quot;s3://{}/{}/{}&quot;.format(bucket, prefix, &quot;data/train.csv&quot;), content_type=&quot;csv&quot;<br>)<br>validation_input = TrainingInput(<br>    &quot;s3://{}/{}/{}&quot;.format(bucket, prefix, &quot;data/validation.csv&quot;), content_type=&quot;csv&quot;<br>)</pre><p><strong>Start Training</strong>: Launch the training job with the fit method, providing the training and validation data. You can monitor the training progress by setting wait=True.</p><pre>xgb_model.fit({&quot;train&quot;: train_input, &quot;validation&quot;: validation_input}, wait=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*uBLkuUu0BQyd6CjSJBediQ.png" /></figure><p><strong>Specify the S3 Bucket URI for Debugger Training Reports :</strong>First, determine the location of the Debugger training reports by specifying the S3 bucket URI. This URI points to where the reports are stored, based on the output path of your XGBoost model and its latest training job name.</p><pre>rule_output_path = xgb_model.output_path + &quot;/&quot; + xgb_model.latest_training_job.job_name + &quot;/rule-output&quot;<br>! aws s3 ls {rule_output_path} --recursive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u6byLNYTT5TvQ4Z-93WZtQ.png" /></figure><p><strong>Download the Debugger Training and Profiling Reports : </strong>Next, download the Debugger training reports, including both the XGBoost training report and the profiling report, to your current workspace for further examination.</p><pre>! aws s3 cp {rule_output_path} ./ --recursive</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eY-t3nSMXmPWs2zVLGfitw.png" /></figure><p><strong>Access the XGBoost Training Report : </strong>To view the XGBoost training report, use the IPython FileLink function to create a clickable link to the report file, allowing easy access right from your notebook.</p><pre>from IPython.display import FileLink, FileLinks<br>display(&quot;Click link below to view the XGBoost Training report&quot;, FileLink(&quot;CreateXgboostReport/xgboost_report.html&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*-Ls0omeiKVRuvxXrEKYP3Q.png" /></figure><p><strong>Access the Debugger Profiling Report</strong> : Identify the name of the profiling report from the latest training job’s rule job summary. This report offers insights into the EC2 instance’s resource utilization, system bottleneck detection, and Python operation profiling results. Generate a clickable link to access this report.</p><pre>profiler_report_name = [rule[&quot;RuleConfigurationName&quot;] <br>                        for rule in xgb_model.latest_training_job.rule_job_summary() <br>                        if &quot;Profiler&quot; in rule[&quot;RuleConfigurationName&quot;]][0]<br>profiler_report_name<br>display(&quot;Click link below to view the profiler report&quot;, FileLink(profiler_report_name+&quot;/profiler-output/profiler-report.html&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*vNTbms5yg6vyPEV-3rf7lQ.png" /></figure><p>Now you have a trained XGBoost model. SageMaker stores the model artifact in your S3 bucket. To find the location of the model artifact, run the following code to print the model_data attribute of the xgb_model estimator:</p><pre>xgb_model.model_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*92_lDFEGoWRqJaJPVSEU-A.png" /></figure><h3>Step 5: Deploy the Model to Amazon EC2</h3><p>Once the model is trained, it can be deployed to Amazon EC2 instances for real-time predictions. SageMaker simplifies this process by providing a method to deploy the model, specifying the number and type of EC2 instances to use. The deployed model can then be used to make predictions on new data.</p><p>Here’s a Python code snippet that demonstrates how to deploy your model:</p><pre>import sagemaker<br>from sagemaker.serializers import CSVSerializer<br><br># Assuming xgb_model is your trained model<br>xgb_predictor = xgb_model.deploy(<br>    initial_instance_count=1,<br>    instance_type=&#39;ml.t2.medium&#39;,<br>    serializer=CSVSerializer()<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*sgzZx-R-nBGNUH7FRUSlGA.png" /></figure><p>In this code:</p><ul><li>initial_instance_count specifies the number of instances to deploy the model on.</li><li>instance_type defines the type of instances you want to use for hosting your model.</li><li>serializer is used to convert input data into a CSV-formatted string, as the XGBoost algorithm requires input in this format.</li></ul><p>The deploy method sets up a deployable model, configures the SageMaker hosting services endpoint, and launches the endpoint to host your model.</p><h4>Retrieving the Endpoint Name</h4><p>After deploying your model, you can retrieve the name of the generated endpoint by running:</p><pre>xgb_predictor.endpoint_name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*74FlSmWnTIG3LzvDZGe8KQ.png" /></figure><p>This will return the endpoint name in the format “sagemaker-xgboost-YYYY-MM-DD-HH-MM-SS-SSS”. This endpoint remains active on the ML instance, allowing you to make predictions at any time unless you decide to shut it down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VZjy8KLUDjHWXkuy0MOXTg.png" /></figure><h3>Step 6: Evaluating and Using Your Model in Production</h3><p>After deploying your model to Amazon EC2 using Amazon SageMaker, it’s essential to evaluate its performance and optimize it for production use. This process involves several steps, including making predictions with a test dataset, determining the optimal cutoff value for predictions, and potentially adjusting the model based on these evaluations.</p><h4>Step 1: Making Predictions with the Test Dataset</h4><p>First, you’ll need to set up a function to predict each line of the test set. This function allows for batch inference, fully utilizing the instance’s hardware resources. Here’s how you can do it:</p><pre>import numpy as np<br><br>def predict(data, rows=1000):<br>    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))<br>    predictions = &#39;&#39;<br>    for array in split_array:<br>        predictions = &#39;,&#39;.join([predictions, xgb_predictor.predict(array).decode(&#39;utf-8&#39;)])<br>    return np.fromstring(predictions[1:], sep=&#39;,&#39;)</pre><p>Next, use this function to make predictions on your test dataset and visualize the results with a histogram. Remember to exclude the 0th column, which contains the actual values, from the test dataset:</p><p>make predictions of the test dataset and plot a histogram. You need to take only the feature columns of the test dataset, excluding the 0th column for the actual values.</p><pre>import matplotlib.pyplot as plt<br><br>predictions=predict(test.to_numpy()[:,1:])<br>plt.hist(predictions)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*lrUcTyP3QYJ3UBqA3nvO3Q.png" /></figure><h3>Determining True or False Based on Predictions</h3><p>Since the predicted values are of float type, you’ll need to set a cutoff value to determine whether the prediction is true or false. The following code uses the Scikit-learn library to return the confusion matrix and classification report with a cutoff of 0.5:</p><pre><br>import sklearn<br><br>cutoff = 0.5<br>print(sklearn.metrics.confusion_matrix(test.iloc[:, 0], np.where(predictions &gt; cutoff, 1, 0)))<br>print(sklearn.metrics.classification_report(test.iloc[:, 0], np.where(predictions &gt; cutoff, 1, 0)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*C7A_LuiUuA0eB6UmPJoY9w.png" /></figure><h3>Finding the Best Cutoff Value</h3><p>To find the optimal cutoff value, you can compute the log loss function. This function measures the performance of a classification model where the prediction input is a probability value between 0 and 1. The following code calculates the log loss values for different cutoff values and plots the log loss versus cutoff graph:</p><pre><br>import matplotlib.pyplot as plt<br><br>cutoffs = np.arange(0.01, 1, 0.01)<br>log_loss = []<br>for c in cutoffs:<br>    log_loss.append(<br>        sklearn.metrics.log_loss(test.iloc[:, 0], np.where(predictions &gt; c, 1, 0))<br>    )<br><br>plt.figure(figsize=(15,10))<br>plt.plot(cutoffs, log_loss)<br>plt.xlabel(&quot;Cutoff&quot;)<br>plt.ylabel(&quot;Log loss&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*F9SVdrC7RDiWR4rZQSJoaA.png" /></figure><p>To find the minimum points of the error curve, use the NumPy argmin and min functions:</p><pre>print(<br>    &#39;Log loss is minimized at a cutoff of &#39;, cutoffs[np.argmin(log_loss)], <br>    &#39;, and the log loss value at the minimum is &#39;, np.min(log_loss)<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*XeMs6TXnjNG7vhhzri80pA.png" /></figure><p>By following these steps, you can evaluate your model’s performance and optimize it for production use. This process is crucial for ensuring that your model is accurate and ready for deployment in real-world scenarios. Whether you’re working on a customer churn prediction problem or any other business problem, understanding how to evaluate and optimize your model is key to achieving the best results.</p><p><strong>Finally,</strong> Amazon SageMaker <strong>provides</strong> a comprehensive platform for developing, training, and deploying machine learning models, making it an excellent choice for data scientists and developers <strong>who want</strong> to <strong>take</strong> their ML projects <strong>into</strong> production. Whether you’re working on a simple classification task or a complex predictive model, SageMaker provides the tools and infrastructure <strong>you need</strong> to <strong>efficiently</strong> develop, train, and deploy your <strong>models.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c98c6ed098d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Obtain a Discord Webhook URL: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@binaya.puri/how-to-obtain-a-discord-webhook-url-a-step-by-step-guide-719fddcd592b?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/719fddcd592b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[discord]]></category>
            <category><![CDATA[discord-webhooks]]></category>
            <category><![CDATA[discord-server]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Fri, 16 Feb 2024 15:13:34 GMT</pubDate>
            <atom:updated>2024-02-16T15:13:34.196Z</atom:updated>
            <content:encoded><![CDATA[<p>Discord is a platform that facilitates voice, video, and text communication, primarily designed for creating communities around gaming. With over 150 million monthly active users, Discord has become a popular choice for gamers and other communities to connect and collaborate .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*kYBAsOMcWrBlI3eG.png" /></figure><p>Discord offers features such as voice channels, video calls, text channels, and direct messages, which enable users to communicate in real-time. It also supports file sharing, allowing users to send images, documents, and other files within their conversations. Discord’s text channels can be organized into categories and roles, making it easy to manage and navigate through different topics and user groups</p><h3>Introduction</h3><p>Discord webhooks are a powerful feature that allows external applications to send messages to your Discord channels. They are useful for integrating Discord with other services, automating notifications, and creating custom bots. This article will guide you through the process of obtaining a Discord webhook URL.</p><h3>Use Case</h3><p>Imagine you have a project management tool that sends updates to your team. You want these updates to be sent directly to a Discord channel where your team can see them instantly. Or perhaps you’re developing a custom bot that needs to post messages to a specific channel. With Discord webhooks, you can achieve this seamlessly.</p><h3>Step 1: Accessing Server Settings</h3><ul><li>Open Discord and navigate to the server where you wish to create the webhook.</li><li>Click on the server name located at the top left corner to reveal the drop-down menu.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*YUQJP5KtMg_uOjPCxgthQQ.png" /></figure><ul><li>From the drop-down menu, select ‘Server Settings’ and this interface appears</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*iD0ItjgcedpkTn56duUV4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0IEMV_9pcGa4OEttAa1K_Q.png" /></figure><h3>Step 2: Navigating to Integrations</h3><ul><li>Within the Server Settings page, find the ‘Integrations’ option in the left sidebar.</li><li>Click on ‘Integrations’ to proceed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZtoq5jYSwD_LijerIBmLw.png" /></figure><h3>Step 3: Managing Webhooks</h3><ul><li>Scroll down to the ‘Webhooks’ section on the Integrations page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOVZgMusnDJE9eHv-mKYEg.png" /></figure><ul><li>Click on ‘View Webhooks’ to see the list of existing webhooks or add a new one</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*WbovGeoqLBRX0t7WqOMuWw.png" /></figure><h3>Step 5: Customizing Your Webhook (Optional)</h3><p>Optionally, you can customize the avatar for your webhook to make it more recognizable.</p><h3>Step 6: Copying the Webhook URL</h3><ul><li>Once you’ve configured your webhook, click on ‘Copy Webhook URL.’</li><li>This URL is what you’ll use to send messages to your Discord channel from your application or bot.</li></ul><pre>https://discord.com/api/webhooks/*****************************l</pre><h3>Step 7: Saving Your Webhook</h3><ul><li>After setting up your webhook and copying the URL, click ‘Save’ to</li><li>finalize the creation.</li></ul><p>After completing these steps, you will have a webhook URL that you can use to send messages to the selected Discord channel programmatically</p><h3>Conclusion</h3><p>Now that you have your Discord webhook URL, you can integrate it with your applications, bots, or any other service that supports webhook notifications. This URL is your bridge to programmatically sending messages to your Discord server, enhancing communication and automation across various platforms.</p><p>Remember to keep your webhook URL secure, as anyone with the URL can send messages to your Discord channel. If you ever need to revoke the webhook or change its settings, you can do so from the ‘View Webhooks’ section in your server’s Integrations settings.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=719fddcd592b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging CloudWatch Insights for Real-Time Analysis of Model Invocation Logs in Generative AI…]]></title>
            <link>https://medium.com/@binaya.puri/leveraging-cloudwatch-insights-for-real-time-analysis-of-model-invocation-logs-in-generative-ai-841cf7296ef3?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/841cf7296ef3</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[bedrock]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Thu, 15 Feb 2024 13:51:19 GMT</pubDate>
            <atom:updated>2024-02-15T13:51:19.003Z</atom:updated>
            <content:encoded><![CDATA[<h3>Leveraging CloudWatch Insights for Real-Time Analysis of Model Invocation Logs in Generative AI: AWS Bedrock</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2YDNFaL95N2x5QAHcS_gg.png" /></figure><p>Amazon Bedrock simplifies the creation and scaling of generative AI applications by leveraging foundation models (FMs) from prominent AI providers like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon itself. As a fully managed platform, Bedrock offers a vast array of features necessary for developing AI applications efficiently while prioritizing privacy and security.</p><p>Integrating with Amazon CloudWatch, Bedrock enables users to meticulously track and analyze usage metrics, facilitating the creation of custom dashboards for auditing purposes. This integration allows for detailed insights into model invocations and token usage, ranging from individual model usage in a single account to comprehensive tracking across multiple accounts. Additionally, Bedrock supports model invocation logging, enabling the capture of metadata along with request and response data for every model invocation. This feature, which is initially turned off, can be activated by users to begin log publication.</p><p>This article delves into leveraging CloudWatch for real-time monitoring of Bedrock, employing metrics and logs to set up alerts and actions based on predetermined thresholds. CloudWatch’s extensive features further enhance monitoring capabilities, including cross-account observability, correlation between logs and metrics, composite alarms, log analytics, and application performance monitoring.</p><h4>Step-by-Step Guide to Monitoring Bedrock with CloudWatch</h4><h4>Step 1: Enabling Model Invocation Logging</h4><p>To enable logging, access the Bedrock console’s Settings page via the left navigation bar. Here, you’ll find the option to toggle Model Invocation Logging, prompting you to specify the types of data to be logged, such as text, images, and embeddings.</p><p>To configure logging, navigate to the <a href="https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/settings">Settings page</a> in the Bedrock console, from the left navigation bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ia2aYnLI6BiozzRndPGA6w.png" /></figure><p>Then toggle the <strong>Model invocation logging</strong> button which will present you with several fields that will need to be filled out before logging can be enabled.</p><p>First, select the data types to include with logs. You can choose text, image and embedding.</p><h4>Step 2: Selecting a Logging Destination</h4><p>Next, select your logging destination,</p><p>Select your logging destination, where you have three options. The first option is S3 Only which configures Bedrock to only send logs to a S3 bucket of your choice. The second option is CloudWatch Logs only, which sends the logs to CloudWatch and when model input or output data is larger than 100kb or in binary format it can be optionally delivered to S3. The last option is Both S3 &amp; CloudWatch Logs where logs are sent to both S3 &amp; CloudWatch and when model input or output data is larger than 100kb or in binary format data it will only be sent to S3. Whatever option you choose you remain in control of model inputs and outputs, including encryption with KMS and retention duration. In my case I have chosen the CloudWatch Logs only option.</p><ul><li><strong>S3 Only:</strong> Directs logs exclusively to an Amazon S3 bucket.</li><li><strong>CloudWatch Logs Only:</strong> Sends logs to CloudWatch, with the option to offload larger or binary data to S3.</li><li><strong>Both S3 &amp; CloudWatch Logs:</strong> Logs are sent to both destinations, with large or binary data being S3-exclusive.</li></ul><p>Each option offers full control over data management, including encryption via KMS and setting data retention durations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKOwtOAoPUFJRvZZ1udKpw.png" /></figure><h4>Step 3: Configuring CloudWatch Log group and S3 Settings</h4><p>After choosing your logging destination, create a log group in CloudWatch and an S3 bucket for large data files. Within the Bedrock Settings, link your newly created S3 bucket for large data delivery and finalize the setup by saving your changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cQfFcVZsHtdwH5adBItow.png" /></figure><h4>Step 4 : Specifying a CloudWatch Log Group for Model Invocation Logging</h4><p>After enabling model invocation logging in the Bedrock console, you’ll need to specify where these logs should be stored within CloudWatch. This involves:</p><ul><li>Navigating back to the Model Invocation Console within Bedrock.</li><li>Locating the CloudWatch Logs configuration section.</li><li>Specifying a unique log group name.</li></ul><p>A log group in CloudWatch is a container that holds a collection of log streams, which in turn contain the actual log events. By specifying a log group name, you are instructing Bedrock to send its logs to a specific place within CloudWatch, where they can be organized, monitored, and accessed easily. This step is crucial for managing logs effectively, as it allows for better organization and retrieval of log data related to model invocations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uxue6jckz7ienfOOshhTtg.png" /></figure><h4>Step 5 : Creating an S3 Bucket for Log Storage</h4><p>Amazon S3 (Simple Storage Service) offers scalable object storage. When dealing with large amounts of log data or logs that include large files (such as images or embeddings), it’s often necessary to store them in an S3 bucket. This step involves:</p><ul><li>Going to the Amazon S3 console.</li><li>Creating a new bucket, which involves specifying a unique name for the bucket and configuring settings such as region, access permissions, and other options depending on your requirements.</li></ul><p>S3 buckets are used in conjunction with CloudWatch Logs when logs are too large or in a format that is better suited for S3 storage. It ensures that all data, regardless of size, is stored securely and efficiently, with the flexibility to access and analyze this data as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*9AoWuX05VSWGnHf5tT3BKQ.png" /></figure><h4>Step 6: Integrating the S3 Bucket with Bedrock Logging Configuration</h4><p>After creating an S3 bucket, the next step is to link this bucket with the Bedrock logging configuration. This involves:</p><ul><li>Returning to the Bedrock Settings page.</li><li>Navigate to the section where you previously selected your logging destination.</li><li>Inputting the URL or ARN (Amazon Resource Name) of your newly created S3 bucket into the relevant field, specifically for “S3 bucket for large data delivery.”</li></ul><p>By adding the S3 bucket URL to the Bedrock configuration, you ensure that any log data, especially those exceeding size limits for CloudWatch or in binary format, are automatically sent to the specified S3 bucket. This setup provides a comprehensive logging solution that leverages both CloudWatch for real-time monitoring and analytics, and S3 for durable, scalable storage of large or binary log files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoeSw50ApUUbk_0N_hhwEQ.png" /></figure><p>Once the model invocation logs are being delivered, you can use two features in CloudWatch to inspect your logs. The first is Live Tail and the second is Log Insights.</p><h4>Generating log data from Bedrock</h4><p>CloudWatch Logs Insights enables you to interactively search and analyze your log data in CloudWatch Logs. You can perform queries to help you more efficiently and effectively respond to operational issues.</p><p>In the case of Bedrock we can use Log Insights to search and analyze the model invocation logs and search for specific keywords or simply the latest invocation logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7admtOJDljiTk2lhmP64bg.png" /><figcaption>Streaming logs using Live Tail</figcaption></figure><h4>Streaming logs using Live Tail</h4><p>Live Tail in CloudWatch Logs is a feature that provides an interactive log analytics experience that helps you view your logs interactively in near real-time as they’re ingested. Live Tail provides customers a rich out-of-the-box experience to view and detect issues in their incoming logs. Additionally, it provides fine-grained controls to filter, highlight attributes of interest, and pause/replay logs while troubleshooting issues</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AS5et2h4b1p3jL3qMqfHQ.png" /></figure><h4>Conclusion</h4><p>Monitoring Amazon Bedrock with CloudWatch offers a comprehensive, real-time overview of your generative AI applications’ performance and usage. By following the outlined steps to configure and utilize logging and monitoring features, developers can ensure their applications run efficiently and securely, with the ability to quickly identify and address potential issues. This approach not only simplifies the management of generative AI applications but also enhances their reliability and effectiveness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=841cf7296ef3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Data from Confluence to Amazon S3 using Boto3]]></title>
            <link>https://medium.com/@binaya.puri/step-by-step-guide-to-exporting-data-from-confluence-to-amazon-s3-0a5ead3bc844?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a5ead3bc844</guid>
            <category><![CDATA[data-manipulation]]></category>
            <category><![CDATA[data-extraction]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[confluence]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Sat, 03 Feb 2024 07:58:34 GMT</pubDate>
            <atom:updated>2024-02-03T14:34:29.129Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ZAiGTVJNHIINN2xHhlI2OQ.png" /><figcaption>Exporting data from confluence page to s3</figcaption></figure><p>Exporting data from Confluence to Amazon S3 involves a step-by-step process to securely transfer Confluence data to an S3 bucket. First, you would typically access your Confluence instance and identify the data you want to export, which may include pages or attachments. Then, you might use Confluence’s built-in export functionality to create a backup of the selected data. Next, you would configure AWS credentials and create an S3 bucket if one doesn’t exist already. Finally, you’d utilize AWS CLI or an SDK like Boto3 to upload the exported backup file to your S3 bucket. This process ensures that your Confluence data is safely stored in Amazon S3, making it accessible, durable, and scalable for future needs or as part of a backup strategy.</p><p>To copy all data from Confluence to Amazon S3, you can follow these steps:</p><h3>Step 1: Generate an API token:</h3><p>This is a security feature in Confluence that allows you to authenticate with the API without using your password. To generate an API token we have to follow the following steps</p><ul><li>Need to go to your profile settings in Confluence</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXf3avGdkeJL8SdMuzEoQg.png" /></figure><ul><li>Navigate to the “Security” section at the top of the page</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*gFSVRUkedbUHcVxcFwIqWg.png" /></figure><ul><li>Click on “Create and manage API tokens”. Here, you can create a new token and copy it for use in your API requests</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-OvW0S_u3Naw6AuF4jGBg.png" /></figure><ul><li>Lastly, Create a API Token by writing a label for it and Save the token somewhere</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*jybdtEKpItO7CrVSCa0WXg.png" /></figure><h3>Step 2: Find the content ID:</h3><p>The content ID in Confluence is a unique identifier assigned to each piece of content in a Confluence page or space. It is used to reference the content within the Confluence system.<br>You need to know the ID of the content you want to export. This is a unique identifier for each piece of content in Confluence. You can find the content ID by navigating to the content in Confluence and looking at the URL. The content ID is the number in the URL.</p><p>let&#39;s look into the URL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GA3d-tvox6gKfMbUJbi2w.png" /></figure><p>In the URL of a Confluence page, the content ID is the numerical part after the last slash (/). For example, in the URL</p><p><strong>Note</strong> that the content ID is not the same as the page ID, which is also used in Confluence URLs and is the number after the second slash in the URL.</p><ul><li>Find the content ID by navigating to the page in Confluence and looking at the page URL.</li></ul><p><strong>For example</strong>, in this URL</p><ul><li>Space Key</li></ul><blockquote><a href="https://binayapuri.atlassian.net/wiki/spaces/~712020c47cbdcacc5d49d6a4b4a153bd510c33/pages/327681/Step-by-Step+Guide+to+Exporting+Data+from+Notion+to+Amazon+S3+using+Python+and+Boto3,">https://binayapuri.atlassian.net/wiki/spaces/~712020c47cbdcacc5d49d6a4b4a153bd510c33/pages/327681/Step-by-Step+Guide+to+Exporting+Data+from+Notion+to+Amazon+S3+using+Python+and+Boto3,</a> 712020c47cbdcacc5d49d6a4b4a153bd510c33</blockquote><ul><li>Content Key</li></ul><blockquote>327681</blockquote><ul><li>Title of the page</li></ul><blockquote>Step-by-Step+Guide+to+Exporting+Data+from+Notion+to+Amazon+S3+using+Python+and+Boto3</blockquote><h3>Step 3: Export content from Confluence &amp; save it to the local file</h3><p>Confluence does not provide a built-in way to export all data to Amazon S3. However, you can export content from Confluence Cloud using the REST API or the XML/RPC API. The content can be exported in XML format or as a ZIP file.</p><p>Here is an example of how to export content using the REST API:</p><p><strong>curl -u username: password -X GET ‘</strong><a href="https://your-domain.atlassian.net/wiki/rest/api/content/%7BcontentId%7D?expand=body.storage"><strong>https://your-domain.atlassian.net/wiki/rest/api/content/{contentId}?expand=body.storage</strong></a><strong>&#39;</strong></p><pre>curl -u username:password -X GET &#39;https://your-domain.atlassian.net/wiki/rest/api/content/{contentId}?expand=body.storage&#39;</pre><p>curl -u testa**x0@gmail.com:AT------------**********--------------bt-********************-Yeo7cy*****0qPE=ACA1BA9D -X GET ‘https://binayapuri.atlassian.net/wiki/rest/api/content/3***81?expand=body.storage&#39; -o text.txt</p><ul><li>This part of the command uses the -u option of curl to pass in the username and password for the Confluence API. The username is testa**x0@gmail.com and the password is</li><li>AT******CA***9D. The -u option is a shortcut for the user, which is used to pass the username and password to the server for authentication.</li><li>-X GET ‘https://binayapuri.atlassian.net/wiki/rest/api/content/3****1?expand=body.storage&#39; This part of the command specifies the HTTP method to use for the request (GET) and the URL to send the request to. The URL is the REST API endpoint for a specific piece of content in Confluence, identified by its content ID (327681). The expand=body.storage query parameter is used to include the storage format of the page content in the response.</li><li>-o text.txt<br>This part of the command uses the -o option of curl to write the output of the command to a file named text.txt. This is useful for saving the response from the server for later use.</li><li>So, in summary, this command uses curl to send a GET request to the Confluence REST API, asking for the content of a specific page (identified by its content ID) in storage format. The server’s response is then written to a file named text.txt.</li></ul><p>curl -u username:password -X GET ‘<a href="https://your-domain.atlassian.net/wiki/rest/api/content/%7BcontentId%7D?expand=body.storage&#39;">https://your-domain.atlassian.net/wiki/rest/api/content/{contentId}?expand=body.storage&#39;</a></p><p>Replace {contentId} with the ID of the content you want to export and username:password with your Confluence username and password.</p><h3>Step: 4 Upload the local file to Amazon S3</h3><p>Once you have the exported content in a local file, you can upload it to Amazon S3 using the AWS CLI. Here is an example of how to do this:</p><p><strong>Option 1: Use Python and Boto 3 Library</strong></p><p>This Python script uses the Boto3 library to interact with Amazon Web Services (AWS) S3 (Simple Storage Service). It begins by defining AWS access credentials (access key and secret key) and specifying the target S3 bucket and file details. Then, it creates an S3 client using these credentials. The script attempts to upload a local file (‘text.txt’) to the specified S3 bucket (‘mlopstrain’) with the specified file name (‘PAGE_ID.txt’). It includes error handling for cases where AWS credentials are missing, there are validation errors or any other unexpected exceptions during the upload process. If successful, it prints a confirmation message; otherwise, it prints an error message with details about the issue. It’s important to note that using IAM roles or environment variables for AWS credentials is a more secure practice than hardcoding them in the script.</p><ul><li>Make a python file “<strong>conf.py”</strong></li><li>Add the following commands</li></ul><pre><br>import boto3<br>import botocore.exceptions<br><br># AWS credentials (replace these with your own or use a better method like IAM roles or environment variables)<br>aws_access_key_id = &#39;***********************&#39;<br>aws_secret_access_key = &#39;****************************&#39;<br><br># S3 bucket and file details<br>bucket_name = &#39;your_bucket_name&#39;<br>file_name = &#39;PAGE_ID.txt&#39;<br>local_file_path = &#39;text.txt&#39;<br><br># Create an S3 client<br>s3 = boto3.client(&#39;s3&#39;, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)<br><br>try:<br>    # Upload the local file to the specified S3 bucket and key<br>    s3.upload_file(local_file_path, bucket_name, file_name)<br>    print(f&#39;Successfully uploaded {local_file_path} to {bucket_name}/{file_name}&#39;)<br>except botocore.exceptions.NoCredentialsError:<br>    print(&#39;AWS credentials not found. Please configure your AWS credentials.&#39;)<br>except botocore.exceptions.ParamValidationError as e:<br>    print(f&#39;Invalid AWS parameters: {e}&#39;)<br>except Exception as e:<br>    print(f&#39;An error occurred: {e}&#39;)</pre><p>Run this Python file :</p><pre>Python conf.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*tCPzxmQP613wj1DyC9MaEg.png" /></figure><p><strong>Option 2: From the Command line</strong></p><p>This AWS CLI command, ‘aws s3 cp’, is used to copy a local file (‘text.txt’) to an Amazon S3 bucket (‘mlopstrain’). It simplifies the process of uploading a file to S3 by specifying the source file and the destination S3 bucket location in the command. AWS CLI handles authentication and file transfer, making it a convenient way to interact with S3 storage from the command line.</p><pre>aws s3 cp text.txt s3://mlopstrain/</pre><p>Please note that you need to have the AWS CLI and Confluence REST API installed and configured on your machine to perform these steps. Also, you need to have the necessary permissions to export content from Confluence and upload files to your S3 bucket.</p><h4>Output:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJ7Z7gUtngFah3u1K3j3Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*Aq1xPsLH88ZbLbtZX3ngmA.png" /></figure><h3>Extract data with a particular space:</h3><p>The space key in Confluence is a short, unique identifier that forms part of the URL for that space. Here are steps on how to find the space key in Confluence:</p><ol><li>Navigate to the space for which you want to find the key.</li><li>Look at the URL of the space in your web browser’s address bar. The space key is usually found in the URL after /spaces/ and before /overview. For example, in http://&lt;yoursite&gt;/wiki/spaces/MARKETING/overview, the space key is</li></ol><p>If you can’t find the space key in the URL, you can also find it through the space details:</p><ol><li>Go to the “Space Directory” in Confluence. You can access this by clicking on “Spaces” in the header and then selecting “Space Directory”.</li><li>Look for the space for which you want to find the key and click on the “Space Details” icon beside the space.</li><li>The space key will be listed in the details for the space</li></ol><pre>curapuri.atlassian.net/wiki/rest/api/space/~7120**d4***********/content?limit=100&#39; -o tezt8.txt</pre><p>curl -u testa**x0@gmail.com:ATATT3xF****A1BA9D -X GET ‘https://binayapuri.atlassian.net/wiki/rest/api/space/~712020c47cbdcac***************/content?limit=100&#39; -o tezt8.txt</p><p>If it does not work apply the space name as well</p><h3>Extract the depth of the confluence</h3><p>To extract the depth of content in a Confluence space, you need to understand how Confluence structures its content. In Confluence, content is organized hierarchically with spaces at the top level, then pages, and child pages under pages. The depth of a page is its level in this hierarchy.</p><p>Confluence does not directly provide a way to extract the depth of a page through its REST API. However, you can indirectly calculate the depth by using the ancestors field in the page’s metadata. The ancestor&#39;s field contains an array of the page’s ancestor pages, with the closest ancestor first and the space’s root page last. The depth of the page is the number of ancestors it has.</p><p>Here’s an example of how you can retrieve a page’s metadata, including its ancestors, using the Confluence REST API:</p><pre>curl -u email:api_token -X GET &#39;https://your-domain.atlassian.net/wiki/rest/api/content/{page_id}?expand=ancestors&#39;</pre><p>Replace email:api_token with your email and API token, your domain with your Confluence domain, and {page_id} with the ID of the page.</p><p>In the JSON response, look for the ancestor&#39;s field. The depth of the page is the number of items in the ancestors array.</p><p>Please note that this approach requires you to make a separate API request for each page. If you need to get the depth for many pages, you might need to write a script that iterates over the pages and makes the necessary API requests</p><h3>Limitation :</h3><p>If you want to transfer an entire space, Confluence does not provide a built-in method for this. You would need to export each page or content within the space individually using the described method. To transfer the entire space more efficiently, you may need to explore third-party tools or services designed for Confluence data migrations.</p><h3>Conclusion :</h3><p>Exporting data from Confluence to Amazon S3 involves several steps. First, you need to generate an API token in Confluence for authentication. Next, identify the content you want to export and obtain its content ID. Then, use Confluence’s REST API to export the content in XML format, saving it to a local file.</p><p>To transfer the data to Amazon S3, you can use either Python with the Boto3 library or the AWS CLI. The Python script provided defines AWS credentials, specifies the S3 bucket and file details, and uploads the local file to S3. Alternatively, the AWS CLI command ‘aws s3 cp’ simplifies the process by copying the local file to the S3 bucket specified in the command.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a5ead3bc844" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mnist Dataset for Machine Learning]]></title>
            <link>https://medium.com/@binaya.puri/mnist-dataset-for-machine-learning-8987e0b20bb3?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8987e0b20bb3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mnist-dataset]]></category>
            <category><![CDATA[big-data-training]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Thu, 21 Dec 2023 12:27:20 GMT</pubDate>
            <atom:updated>2023-12-21T12:27:20.053Z</atom:updated>
            <content:encoded><![CDATA[<p>Mnist consists of a collection of 70,000 grayscale images of handwritten digits from 0 to 9. Each image is a 28x28 pixel square. The dataset is divided into training and testing sets, making it suitable for training and evaluating machine learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-CynJj2lrUv8K_zx8fzo9w.png" /></figure><p>The MNIST dataset is a popular and widely used dataset in the field of machine learning and computer vision. It stands for “Modified National Institute of Standards and Technology” and consists of a large collection of handwritten digits. The dataset is often used as a benchmark for testing and developing machine learning algorithms, particularly for digit recognition tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*my2iycmx0GcRBb209QhlkQ.png" /></figure><ol><li><strong>Data</strong>: The dataset contains 70,000 grayscale images of handwritten digits (0 through 9). Each image is a 28x28 pixel square, making the images relatively small compared to many other datasets. These images have been preprocessed and centered to ensure consistency.</li><li><strong>Training and Testing Sets</strong>: The MNIST dataset is typically divided into two subsets: a training set and a test set. The training set contains 60,000 images, and the test set contains 10,000 images. This separation allows researchers and practitioners to train machine learning models on one subset and evaluate their performance on the other.</li><li><strong>Labeling</strong>: Each image in the dataset is associated with a corresponding label, indicating which digit (0–9) is written in the image. These labels are used to train and evaluate the accuracy of machine learning models.</li><li><strong>Use Cases</strong>: MNIST is often used as a benchmark for various machine learning algorithms, especially for image classification tasks. It’s a relatively simple dataset compared to more complex image datasets like CIFAR-10 or ImageNet, making it a good starting point for learning and experimentation.</li><li><strong>Challenges</strong>: While MNIST is considered a standard dataset, it’s not without its challenges. Achieving high accuracy on MNIST is a relatively straightforward task for modern machine learning models. As a result, researchers often use more challenging datasets to evaluate the robustness and generalization capabilities of models.</li></ol><p>The MNIST dataset has been widely used for educational purposes, as a starting point for exploring deep learning, and for benchmarking various machine learning algorithms. However, it’s important to note that it has become somewhat outdated in recent years due to its simplicity. Researchers often seek more complex and diverse datasets to better reflect the challenges encountered in real-world applications.</p><p>→ Set of 70,000 small images of digits handwritten by high school Students and employees of the US Census Bureau</p><p>→ All images are labelled with the respective digit they represent</p><p>→ MNIST is the hello world of machine learning</p><p>→ There are 70,000 images, and each images has 784 (28*28) features.</p><p>→ Each image is 28*28 pixels , and each feature simply represents one pixel’s intensity from 0 (white) to 255 (Black)</p><h3>Requisites :</h3><p><strong>Jupyter Notebook :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*gh8eJJuhWho0Bjk6K_NimQ.png" /></figure><p>The Jupyter Notebook is an open-source web application that allows to creation and share documents that contain live code, equations, visualizations, and narrative tests. this will make it easy for. this will make it easy for us to document and use the code as a notebook.</p><p><strong>2. Python Libraries</strong></p><ul><li>NumPy<strong>:</strong><br> NumPy is a fundamental package for scientific computing in Python. It provides support for arrays and matrices, essential for data manipulation.</li><li>scikit-learn: <br>Scikit-learn (sklearn) is a popular machine-learning library in Python. It includes various tools for data analysis and modeling, including algorithms for classification and regression.</li><li>Matplotlib<strong>:</strong> <br>Matplotlib is a plotting library for creating visualizations in Python. It’s useful for visualizing data and results.</li></ul><p>You can install these libraries using the following command:</p><pre>pip install numpy scikit-learn matplotlib</pre><p><strong>3. Basic Python Programming Knowledge:</strong> Familiarity with basic Python programming concepts such as variables, arrays/lists, functions, and control flow is essential for working with the MNIST dataset and implementing machine learning algorithms.</p><h3>Set up jupyter Notebook</h3><p>To install and set up Jupyter Notebook :</p><ol><li><strong>Install Python:</strong></li></ol><p>If you don’t have Python installed on your Windows system, you’ll need to install it first. You can download the latest Python installer for Windows from the official Python website (</p><p><a href="https://www.python.org/downloads/windows/">Python Releases for Windows</a> ). Be sure to check the box that says “Add Python X.X to PATH” during the installation process, where “X.X” represents the version number of Python.</p><p><strong>2. Open Command Prompt or PowerShell:</strong></p><p>After installing Python, open either Command Prompt or PowerShell on your Windows system. You can do this by searching for “cmd” or “PowerShell” in the Start menu.</p><p><strong>3. Install Jupyter using pip:</strong></p><p>To install Jupyter, you can use the pip package manager that comes with Python. Run the following command to install Jupyter:</p><pre><br>pip install jupyter</pre><p>This command will download and install Jupyter and its dependencies.</p><p><strong>4. Start Jupyter Notebook:</strong></p><p>Once Jupyter is installed, you can start it by running the following command in your Command Prompt or PowerShell:</p><pre>jupyter notebook</pre><p>This command will launch a Jupyter Notebook server, and it will open a web browser window displaying the Jupyter Notebook interface</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5x6MqiUmSyeCLaxM5qrEQ.png" /></figure><p>web browser window displaying the Jupyter Notebook interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*r4r0doDKJiGPc-Yu1PHQKw.png" /></figure><p><strong>5. Use Jupyter Notebook:</strong></p><p>You can create new Jupyter notebooks, open existing ones, and start working with Python code, text, and visualizations in the Jupyter Notebook interface. It’s an interactive environment for data analysis, machine learning, and more.</p><p><strong>6. click on “New” located in the top of the right corner</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIBgeHwLhI_NcX5Z4fuKQg.png" /></figure><p><strong>7. Click on “Notebook” that redirects to a page</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bHoduMwVSffJ4LgGFv_K6g.png" /></figure><p><strong>8. Select Kernal Python 3(ipykernal)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*DiZBL6A0dxLBnWSSWKShBA.png" /></figure><h3>Mnist: Fetch the data and then split it into train and test sets and apply a few ML algorithms to detect a given digit</h3><ol><li><strong>Fetching Dataset</strong></li></ol><pre>from sklearn.datasets import fetch_openml<br>mnist = fetch_openml(&quot;mnist_784&quot;)</pre><p>This code imports the fetch_openml function from scikit-learn and uses it to fetch the MNIST dataset, which consists of 28x28 pixel images of handwritten digits (0-9). The dataset is stored in the mnist variable for further use in machine learning tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1P5eWzJpw6guKJKmvugSw.png" /></figure><p><strong>3. we can see ‘data’ , ‘pixels (1–784)’ , ‘targets’ ,’description’ in the minist when we write command :</strong></p><p>mnist</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAp4ozbTglMYShkUU8SAQg.png" /></figure><p><strong>4. Loading the MNIST Dataset</strong></p><pre>x, y = mnist[&#39;data&#39;], mnist[&#39;target&#39;]</pre><p>here, we’ve loaded the MNIST dataset into x (which contains the input data) and y (which contains the target labels).</p><p>5. <strong>Checking Data Shapes:</strong></p><p>x.shape</p><p>y.shape</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*xrjHeiCV8pRptpdu2MUDbw.png" /></figure><p>These lines are used to check the dimensions (shape) of the x and y arrays, which represent the dataset&#39;s input data and target labels, respectively.</p><p><strong>6. Using </strong>%matplotlib inline <strong>for Inline Plotting in Jupyter Notebook</strong></p><p>%matplotlib inline</p><pre>import matplotlib <br>import matplotlib.pyplot as plt</pre><p>In Jupyter Notebook, %matplotlib inline is a magic command that enables the rendering of Matplotlib plots directly within the notebook interface. This command is particularly useful for data visualization and generating plots, as it allows you to see the graphical output of Matplotlib commands within your Jupyter Notebook cells.</p><p>Overall, %matplotlib inline is a valuable tool for anyone working with data analysis and visualization in Jupyter Notebook, as it simplifies the process of creating and examining plots as part of your data analysis workflow.</p><p><strong>7. Extracting and Reshaping a Handwritten Digit Image</strong></p><pre>some_digit = x.to_numpy()[36002] <br>some_digit_image = some_digit.reshape(28,28)</pre><ul><li>some_digit is a variable that stores a flattened representation of a handwritten digit image from the MNIST dataset.</li><li>x represents the MNIST dataset, where each row corresponds to a flattened 28x28 pixel image of a handwritten digit.</li><li>.to_numpy() is used to convert the dataset x into a NumPy array.</li><li>[36002] selects a specific row (sample) from the dataset, in this case, the 36,002nd row, which corresponds to one of the handwritten digit images.</li><li>some_digit_image is another variable that stores the same handwritten digit image, but in its original 2D format (28 rows by 28 columns).</li><li>.reshape(28, 28) is applied to some_digit to transform the flattened representation into a 2D array. This reshaping is done to prepare the image for visualization or further analysis.</li></ul><p>this code segment extracts a specific handwritten digit from the MNIST dataset, first in its flattened form and then reshapes it into a 2D image format, making it ready for display or further processing.</p><p><strong>8. Visualizing a Handwritten Digit Image</strong></p><p>To visualizes a handwritten digit image using Matplotlib. It displays the digit image in grayscale, removes axis labels, and shows the image within the Jupyter Notebook or Python environment. This is a common step in data exploration and analysis when working with image datasets like MNIST.</p><pre>plt.imshow(some_digit_image, cmap=matplotlib.cm.binary, interpolation=&quot;nearest&quot;)<br>plt.axis(&quot;off&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*-thfpMCiTzjkbzn9n6YKtw.png" /></figure><ul><li>plt.imshow(some_digit_image, cmap=matplotlib.cm.binary, interpolation=&quot;nearest&quot;) is used to display the handwritten digit image stored in the variable some_digit_image using Matplotlib. Here&#39;s what each part of this line does:</li><li>plt.imshow: This function from Matplotlib is used to display an image.</li><li>some_digit_image: It&#39;s the 2D NumPy array representing the handwritten digit image.</li><li>cmap=matplotlib.cm.binary: The cmap parameter specifies the colormap to be used for displaying the image. In this case, matplotlib.cm.binary is used to display the image in grayscale, where black represents the digit&#39;s ink and white represents the background.</li><li>interpolation=&quot;nearest&quot;: The interpolation parameter determines how the image should be interpolated (scaled) if its dimensions don&#39;t match the display size. &quot;nearest&quot; interpolation is used to maintain the pixel&#39;s original values without interpolation.</li><li>plt.axis(&quot;off&quot;): This line is used to turn off the axis labels and ticks in the Matplotlib plot. Since this is an image display, you typically don&#39;t need axis labels.</li><li>plt.show(): Finally, this command is used to display the image plot on your screen.</li></ul><p><strong>9. Splitting the MNIST Dataset into Training and Testing Sets</strong></p><p>x_train, x_test = x[0:60000], x[60000:70000]</p><p>y_train, y_test = y[0:60000], y[60000:70000]</p><p>this code snippet performs a critical step in the machine learning workflow by splitting the MNIST dataset into separate training and testing sets. This separation is crucial for training a machine learning model on one portion of the data and evaluating its performance on another unseen portion. It helps assess the model’s generalization ability to make predictions on new, unseen data. Typically, the training set is used to train the model, and the testing set is used to evaluate its accuracy and performance metrics.</p><ul><li>x_train and x_test are created to store subsets of the input data x from the MNIST dataset. Specifically:</li><li>x_train contains the first 60,000 samples (images) from x, which are typically used for training machine learning models.</li><li>x_test contains the next 10,000 samples (images) from x, which are reserved for testing the trained model&#39;s performance.</li><li>y_train and y_test are created to store subsets of the target labels y corresponding to the input data. These subsets correspond to the same 60,000 training samples and 10,000 testing samples, respectively.</li></ul><p><strong>10. Shuffling the Training Data for Improved Model Training</strong> (Optional)</p><p>By performing this shuffling operation, the training data is presented to the machine learning model in a random order. This randomness can be advantageous, particularly in situations where the data might exhibit inherent patterns or biases related to its order. Shuffling ensures that the model isn’t biased by any specific order in the training data, ultimately leading to more robust and unbiased model training.</p><pre>import numpy as np<br><br><br>shuffle_index = np.random.permutation(60000)<br>shuffle_index<br><br><br>x_train, y_train = x_train.[shuffle_index], y_train.[shuffle_index]</pre><p><strong>11. Creating a 2 detector || if the digit is 2 or not , Preprocessing Target Labels for Binary Classification</strong></p><pre>y_train = y_train.astype(np.int8)    <br>#  y train ko sabai string number ma convert hunxhha<br>y_test = y_test.astype(np.int8)<br>y_train_2 = (y_train==2)<br>y_test_2 = (y_test==2)</pre><pre>y_train</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*JpzPU3CL9UtWVjEAMJeHOQ.png" /></figure><p>The purpose of these transformations is to prepare the target labels for binary classification, where you’re interested in distinguishing one specific class (digit 2) from the rest. By converting the labels to numerical format and creating binary labels (True or False), you set up the data for training and evaluating a binary classification model that can predict whether a given image represents the digit 2 or not.</p><p><strong>12. Importing the Logistic Regression Model</strong></p><p>from sklearn.linear_model import LogisticRegression</p><ul><li>from sklearn.linear_model is a part of the scikit-learn (sklearn) library, which is a popular machine-learning library in Python.</li><li>LogisticRegression is the specific machine learning model being imported from sci-kit-learn. Logistic Regression is a widely used algorithm for binary and multi-class classification tasks.</li></ul><p>By importing LogisticRegression from sklearn.linear_model, you make the Logistic Regression algorithm available for use in your Python code. This model can be used to train and make predictions for classification tasks, including binary classification tasks like the one you&#39;ve been working on, where you&#39;re trying to classify whether an image represents the digit 2 or not. Once imported, you can create an instance of the Logistic Regression model and train it using your training data.</p><ol><li><strong>Creating and Training a Logistic Regression Classifier</strong></li></ol><p>clf = LogisticRegression(tol=0.1)</p><p>clf.fit(x_train, y_train_2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*Q-5sMuia67-XovGBK8mXuQ.png" /></figure><ul><li>clf is a variable representing a Logistic Regression classifier.</li><li>LogisticRegression(tol=0.1) creates a Logistic Regression classifier with a specified tolerance (tol) of 0.1. The tolerance parameter controls the stopping criteria for the optimization algorithm used by the Logistic Regression model during training. It determines when the optimization process should converge to find the best parameters.</li><li>clf.fit(x_train, y_train_2) trains the Logistic Regression classifier (clf) using the training data x_train and the binary target labels y_train_2. This is where the model learns to make predictions based on the input data and the specified binary classification task, which, in this case, is to predict whether a digit image represents the digit 2 (True) or not (False).</li></ul><p>After training, the clf classifier will have learned a decision boundary that separates the digit 2 from other digits based on the features (pixel values) of the images. This trained classifier can then be used to make predictions on new, unseen data and evaluate its performance on the test dataset.</p><p><strong>14. Making Predictions with a Trained Logistic Regression Classifier</strong></p><p>clf.predict([some_digit])</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*XLVxeEj-MyFUlPn1pyCYcw.png" /></figure><p><strong>15. Performing Cross-Validation to Evaluate Model Accuracy</strong></p><p>from sklearn.model_selection import cross_val_score a = cross_val_score(clf, x_train, y_train_2, cv=3, scoring=&quot;accuracy&quot;)</p><p>a</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*jjGnKHHN0w7_Ii9HJGdYkg.png" /></figure><ul><li>from sklearn.model_selection import cross_val_score imports the cross_val_score function from sci-kit-learn&#39;s model_selection module. This function is used for cross-validation, which is a technique for assessing the performance of machine-learning models.</li><li>clf is the previously trained Logistic Regression classifier.</li><li>x_train is the training data, and y_train_2 is the corresponding binary target variable indicating whether each training sample represents the digit 2 or not.</li><li>cv=3 specifies the number of cross-validation folds. In this case, cv=3 this means that the dataset will be split into three parts, and the model will be trained and evaluated three times, each time using a different fold for testing and the remaining folds for training.</li><li>scoring=&quot;accuracy&quot; specifies that the evaluation metric for the cross-validation should be accuracy. Accuracy measures the proportion of correctly classified samples, which is a common metric for classification tasks.</li><li>a is a variable that stores the results of the cross-validation. After running this code, a will contain an array of accuracy scores, with each score representing the accuracy achieved on one of the cross-validation folds.</li></ul><p>By performing cross-validation in this way, you obtain multiple accuracy scores for the model, which can help assess its overall performance and robustness. You can then analyze these scores to understand how well the model generalizes to different subsets of the training data and make informed decisions about model tuning and hyperparameter selection.</p><p>16. <strong>Calculating the Mean Accuracy from Cross-Validation Scores</strong></p><pre>a.mean()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*zM7RLFfsXDheL1O6MKz7Zg.png" /></figure><p>By calculating the mean accuracy, you can assess the overall performance of the Logistic Regression classifier and gain insights into how well it is likely to perform on new, unseen data. It helps you make informed decisions about the model’s suitability for the given task and whether further model refinement or parameter tuning is necessary.</p><p>In <strong>conclusion,</strong> this analysis and machine learning pipeline showcase the fundamental steps involved in handling and understanding the MNIST dataset, preparing data for model training, training a classifier, making predictions, and evaluating model performance. While MNIST is a classic dataset used for educational purposes, it’s important to recognize its simplicity and consider more challenging datasets for real-world applications.</p><p>Here is the full code that you can try out in your Jupyter notebook</p><pre>from sklearn.datasets import fetch_openml<br>import matplotlib<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.model_selection import cross_val_score<br><br>mnist = fetch_openml(&#39;mnist_784&#39;)<br>x, y = mnist[&#39;data&#39;], mnist[&#39;target&#39;]<br><br>some_digit = x.to_numpy()[36001]<br>some_digit_image = some_digit.reshape(28, 28)  # let&#39;s reshape to plot it<br><br>plt.imshow(some_digit_image, cmap=matplotlib.cm.binary,<br>           interpolation=&#39;nearest&#39;)<br>plt.axis(&quot;off&quot;)<br>plt.show()<br><br>x_train, x_test = x[:60000], x[6000:70000]<br>y_train, y_test = y[:60000], y[6000:70000]<br><br>shuffle_index = np.random.permutation(60000)<br>x_train, y_train = x_train.[shuffle_index], y_train.[shuffle_index]<br><br># Creating a 2-detector<br>y_train = y_train.astype(np.int8)<br>y_test = y_test.astype(np.int8)<br>y_train_2 = (y_train == &#39;2&#39;)<br>y_test_2 = (y_test == &#39;2&#39;)<br><br># Train a logistic regression classifier<br>clf = LogisticRegression(tol=0.1)<br>clf.fit(x_train, y_train_2)<br>example = clf.predict([some_digit])<br>print(example)<br><br># Cross Validation<br>a = cross_val_score(clf, x_train, y_train_2, cv=3, scoring=&quot;accuracy&quot;)<br>print(a.mean())</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8987e0b20bb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon Bedrock for Generative AI Applications]]></title>
            <link>https://medium.com/@binaya.puri/amazon-bedrock-for-generative-ai-applications-e7aefdf4b98e?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7aefdf4b98e</guid>
            <category><![CDATA[aws-ai-services]]></category>
            <category><![CDATA[aws-bedrock]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 09:43:36 GMT</pubDate>
            <atom:updated>2023-12-20T09:43:36.938Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*MfFdqBntdMoxvzFF" /></figure><p>Amazon Bedrock is a fully managed service that offers leading foundation models (FMs) and a set of tools to quickly build and scale generative AI applications. The service also helps ensure privacy and security.</p><p>You can choose from various FMs to find the model that is best suited for your use case. With the Amazon Bedrock serverless experience, you can quickly get started and experiment with FMs. You can also privately customize FMs with your own data, and seamlessly integrate and deploy them into your applications using Amazon Web Services (AWS) tools and capabilities.</p><p>With agents for Amazon Bedrock (preview), you can automate complex business tasks and orchestrations using fully managed agents. With knowledge bases for Amazon Bedrock (preview), you can integrate company data with the query so the responses from the FMs are specific to your domain and organization.</p><h3>What are the three key benefits of Amazon Bedrock?</h3><h4>1. Build awesome AI apps fast:</h4><ul><li>Pick from a bunch of super powerful AI models (like Amazon Titan or Stability Diffusion) to do things like write funny stories or generate realistic images.</li><li>Use a single, simple tool to control these models and tell them what to do.</li><li>Don’t worry about setting up all the technical stuff — Amazon Bedrock takes care of it.</li></ul><h4>2. Keep your AI safe and private:</h4><ul><li>Your data (like prompts and model responses) stays hidden and secure Amazon Bedrock keeps it locked up in your region.</li><li>Connect your AI models directly to your private systems without sending data over the internet.</li><li>Control who can use your AI models and keep everything accountable with detailed logs and reports.</li></ul><h4>3. Make your AI smart about your business:</h4><ul><li>Build AI assistants that automatically handle complex tasks using your company’s data.</li><li>Tell your AI models about your specific industry and customers to make their responses more accurate and relevant.</li><li>Train your AI models on your own data to make them even better at doing what you need them to do.</li></ul><p>Basically, Amazon Bedrock makes it easy and safe to build powerful AI applications that are tailored to your needs. Think of it as a friendly superpower for your business!</p><h3>Architecture and Use Cases</h3><h4>How can you use Amazon Bedrock to architect a generative AI application?</h4><p>The following architecture illustrates a cloud-based chatbot solution powered by generative AI and customer data. This solution has the capability to converse with you in natural language while maintaining accuracy by including customer data.</p><ol><li><strong>User Query</strong><br>Users or consumers query the chatbot through the user interface, which is capable of understanding natural processing</li><li><strong>Compute and Business Logic</strong><br>The query is processed and sent to the compute layer, which routes and orchestrates the request and response between relevant services based on the business logic and user query</li><li><strong>Relevant data<br></strong>The query is then augmented with relevant data that has been filtered. This data remains in sync with enterprise data stored and indigested from databases, websites, and file repositories.</li><li><strong>Query Augmentation</strong><br>The Query, now augmented with the relevant enterprise data, is sent to the generative AI service Amazon Bedrock</li><li><strong>Generative AI response</strong><br>Amazon Bedrock uses FMs to get the query response and routed that back to the compute layer.</li><li><strong>Response to user</strong><br>The compute layer sends the response to the chatbot user interface to provide a final natural language response to the user or consumers.</li></ol><h3>What are typical use cases for Amazon Bedrock?</h3><h4>Text generation</h4><p>Create new pieces of original content, such as short stories, essays, social media posts, and webpage copy.</p><h4>Chatbots</h4><p>Build conversational interfaces, such as chatbots and virtual assistants, to enhance the user experience for your customers.</p><h4>Search</h4><p>Search, find, and synthesize information to answer questions from a large corpus of data.</p><h4>Text summarization</h4><p>Get a summary of textual content, such as articles, blog posts, books, and documents, to gain understanding without having to read the full content.</p><h4>Image generation</h4><p>Create realistic and artistic images of various subjects, environments, and scenes from language prompts.</p><p><strong>Note</strong>: With Amazon Bedrock, you can quickly get started, privately customize FMs with your data, and seamlessly integrate and deploy them into applications using AWS tools and capabilities.</p><h3>What to keep in mind about Amazon Bedrock?</h3><p>For more information about using Amazon Bedrock, expand each of the following nine sections.</p><h4>Model access</h4><p>As an Amazon Bedrock user, you will need to request access to foundation models before they are available to use. If you want to add models for text, chat, and image generation, you need to use the model access link in the left-side navigation panel of the Amazon Bedrock console. Model access is subject to IAM permissions for the account and the users. Your use of Amazon Bedrock and its models is subject to the seller’s pricing terms, End User License Agreement (EULA), and Amazon Bedrock service terms.</p><h4>Shared responsibility model</h4><p>AWS is responsible for protecting the infrastructure that runs all of the services Amazon Bedrock offers. You are responsible for managing and encrypting your data and applying correct user access controls to data and API calls made to Amazon Bedrock. You are also responsible for the accuracy of the results returned by the customized FMs and third-party FMs.</p><h4>Quotas</h4><p>Service quotas also referred to as limits, are the maximum number of service resources allowed for your AWS account. Amazon Bedrock implements a number of account-level limits across the following APIs:</p><ul><li>The number of concurrent fine-tuning jobs allowed</li><li>The number of calls to the APIs per minute</li><li>The total number of tokens consumed and emitted by the model API per minute</li></ul><h4>Integrating AWS and the corporate network</h4><p>Amazon Bedrock is a fully managed service that is not configured in your virtual private cloud (VPC). You can access the service endpoint from a VPC over AWS PrivateLink. You can also connect your corporate networks to Amazon Bedrock through one of your VPCs.</p><h4>Region availability</h4><p>Currently, Amazon Bedrock is available in a few select Regions, with plans to expand.</p><h4>Data does not cross AWS Regional Boundaries</h4><p>With Amazon Bedrock, all of your content is processed inside the same AWS Region where the relevant API call was made. Data does not cross any AWS Regional boundaries.</p><h4>Use of customer data: handling PII data</h4><p>You may supply personally identifiable information (PII) data in input prompts to Amazon Titan or to third-party models. The third-party models will have their own ways of handling such data, but Amazon Titan will always accept it as input.</p><p>When Amazon Titan provides output from the prompt, any PII data present that was also in the input prompt will remain in plain text in the output. Any PII data in the output that was not present in the input prompt will be masked. For more information about the handling of PII data, see the third-party model provider’s End User License Agreement (EULA).</p><h3>Use of customer data for service enhancement</h3><p>None of your data is used to improve or enhance the base FMs. Your data is not shared with any model providers.</p><h3>Use of customer data for model customization</h3><p>When you fine-tune a base FM, your data is used to fine-tune a copy of the base FM. This FM copy is private to you. Neither this fine-tuned FM nor the data used to fine-tune it is shared with any other customers or model providers.</p><p>Identification of these fine-tuned models uses standard Amazon Resource Names (ARNs), and only the AWS account that created the model can access it. Amazon Bedrock does not expose any of the model-specific tuning details, such as the weights, nor does it allow the export of any of the custom model artifacts.</p><h3>Bedrock Pricing</h3><p>With Amazon Bedrock, you have a choice of two consumption plans for inference:</p><h4>On-Demand:</h4><p>With the On-Demand mode, you only pay for what you use, with no time-based term commitments. For text generation models, you are charged for every input token processed and every output token generated. For embeddings models, you are charged for every input token processed. A token is comprised of a few characters and refers to the basic unit that a model uses to understand user input and prompts to generate results. For image generation models, you are charged for every image generated.</p><h4>Provisioned Throughput:</h4><p>With this mode, you can purchase model units for a specific base or custom model. The Provisioned Throughput mode is primarily designed for large, consistent inference workloads that need guaranteed throughput. Custom models can only be accessed using Provisioned Throughput. A model unit provides a certain throughput, which is measured by the maximum number of input or output tokens processed per minute. With this Provisioned Throughput pricing, charged by the hour, you have the flexibility to choose between 1-month or 6-month commitment terms.</p><h3>How Do You Use Amazon Bedrock?</h3><h3>What are the basic technical concepts of Amazon Bedrock?</h3><p>Amazon Bedrock can be used from the AWS Management Console or with an API. Choose from the following two tabs to learn about common usage patterns for Amazon Bedrock.</p><h4>AWS MANAGEMENT CONSOLE</h4><p>You can use the Amazon Bedrock playgrounds to interact with FMs to generate text or an image 56retw4r or to have a conversation using chat. Amazon Bedrock supports the selection of an FM from a set of model providers.</p><p>Using the playgrounds in Amazon Bedrock, you can submit a natural language command (prompt) to the FM and get a response or an answer. You can influence the response from the model by adjusting model parameters, such as the temperature so that the answer can vary from being more factual to being more creative. You can provide prompts to generate text, generate images, summarize text, receive answers to questions, or have a conversation using chat.</p><h4>Amazon Bedrock API</h4><p>You can use a single Amazon Bedrock API to securely access FMs. Using the same API, you can privately and more easily pass prompts and responses between the user and the FM. The Amazon Bedrock API can be used through the AWS SDK to build a generative AI application and to integrate with other AWS services.</p><h4>How do you interact with the Amazon Bedrock playground?</h4><p>You can access Amazon Bedrock with the AWS Management Console to use the text playground, chat, or image playground. In the text or image playground, you can choose an FM, enter a prompt in the text field, and choose Run to generate a response.</p><p>In the chat playground, you can interact with the FM of your choice using a conversational interface. In the image playground, you can use Stable Diffusion FM for text-to-image query and response. The following architecture diagram is used for the demonstration in this training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2aBEIUPVyFGtLsBUG7aaw.png" /></figure><p>Consumers interact with Amazon Bedrock in the console for chat, text, and image playground activity with the FMs.</p><p>You can optionally set inference parameters to influence the response generated by the model. FMs support the following types of inference parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*xnko7qXsKf6C3EOZIj2J-g.png" /></figure><p><strong>Randomness and diversity</strong></p><ul><li><strong>Temperature: </strong>Large language models (LLMs) use probability to construct the words in a sequence. For any given sequence, there is a probability distribution of options for the next word in the sequence. When you set the temperature closer to zero, the model tends to select the higher-probability words. When you set the temperature farther away from zero, the model might select a lower-probability word.</li><li><strong>Top P: </strong>Top P defines a cutoff based on the sum of probabilities of the potential choices. If you set Top P below 1.0, the model considers the most probable options and ignores the less probable ones.</li></ul><p><strong>Length</strong></p><ul><li><strong>Response length: </strong>The response length configures the maximum number of tokens to use in the generated response.</li><li><strong>Stop sequences: </strong>A stop sequence is a sequence of characters. If the model encounters a stop sequence, it stops generating further tokens. Different models support different types of characters in a stop sequence, different maximum sequence lengths, and may support the definition of multiple stop sequences.</li></ul><h3>How to use the playground in Amazon Bedrock</h3><p><strong>Step 1:</strong> Start by conducting a search for “Bedrock” within the AWS console’s search panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bn8CaZLACI3rNPEM6KFPfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fVY5Fnw6qLt45v76e9yHLg.png" /></figure><p><strong>Step 2:</strong> Click on the “Get Started” page, which will lead you to the Overview page displaying various models and playground options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rIFXBTzoJXCpc-ikbIDhw.png" /></figure><p><strong>Step 3</strong>: Navigate to the Chat option within the available playgrounds (Chat, Text, and Image).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9kQm11fj0D1ztEVvwJT5A.png" /></figure><p><strong>Step 4:</strong> In the Chat Playground, click on the “Select Model” button and choose the desired model provider from the categorized menu. For instance, select Amazon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFTQD8JC1XHwUgGfJFi7ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CAZkLlBM4FsXwLxw9VlCsw.png" /></figure><p><strong>Step 5: </strong>Use the Chat Playground of the selected Amazon model provider to pose questions and related inquiries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cv9emfyNXzpxXlSKM2aGaA.png" /></figure><p><strong>Step 6:</strong> Use the “Add Instructions” Section to change the playground responses</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ywyAnDMlCEuFzba2947Z1A.png" /></figure><p><strong>Step 7:</strong> A window pop up after clicking on Add instructions and then can add instruction as below to change the response format</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*ysx7b8imVzgOZaj1eN6FNA.png" /></figure><p><strong>Step 8: </strong>Here are the Instructions Updated and can ask new questions. We can click on the “Start new chat” option for the new chat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjkB8s8BLYkFeq6-lqO1yA.png" /></figure><p><strong>Step 9:</strong> Next select “Text Playground”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aN9wsUPUdzU9RTef9vMZww.png" /></figure><p><strong>Step 10:</strong> In text playground also you have to select one of the Model providers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAp5-ZH4OkVImBBtIhpfzA.png" /></figure><p><strong>Step 11:</strong> Here you can use Text Playground by questioning in this way</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHn4BqbAuXc0hUVDO8qs6g.png" /></figure><p>Step 12: Choose “Image Playground” for using it . The image playground allows you to experiment with the generation of images from text input. Select Open Image Playground.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v4OkvCC8OlGjx6RSnZmOog.png" /></figure><p><strong>Step 13</strong>: Choose one of the model providers for the Image Playground</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhkNHWaqZCNoT9IVt9YHDQ.png" /></figure><p><strong>Step 14</strong>: You can adjust the settings of the model to control the quality and detail of the images through the use of the Prompt strength, Generation step, and Seed sliders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2yj44baJAfeINF5UPxHdTw.png" /></figure><p><strong>Step 15:</strong> Now, enter a text input such as “cat sitting on a fence.” Choose Run. The image produced from the prompt is displayed in the response window. You can then choose Download to download the image to your local system or Save to add the current prompt to your history items as a favorite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8kT3ARBfF9F6lRXSzP98Ug.png" /></figure><p>In summary, Amazon Bedrock stands out as a fully managed service providing powerful foundation models (FMs) along with tools for building and scaling generative AI applications. Amazon Bedrock supports various use cases, including text generation, chatbots, search, text summarization, and image generation. Its user-friendly features, like playgrounds for text, chat, and image interactions, enable seamless experimentation and customization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7aefdf4b98e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing AWS CloudFront for Dynamic Content Delivery: A Deep Dive]]></title>
            <link>https://medium.com/@binaya.puri/optimizing-aws-cloudfront-for-dynamic-content-delivery-a-deep-dive-b28a5f46312c?source=rss-ae7a71b172a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b28a5f46312c</guid>
            <category><![CDATA[cdn]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[cloudfront]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Binaya Puri]]></dc:creator>
            <pubDate>Mon, 11 Dec 2023 15:36:37 GMT</pubDate>
            <atom:updated>2023-12-11T15:39:47.508Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon CloudFront is a web service that gives business and web application developers an easy and cost-effective way to distribute content with low latency and high data transfer speed.</p><p>Amazon CloudFront is a content delivery network (CDN) service provided by AWS. It is designed to accelerate the distribution of static and dynamic web content, such as HTML files, CSS files, JavaScript files, and images, to end-users around the world. CloudFront achieves this by caching content at edge locations, which are a network of data centers distributed globally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kTxuORXnD1Id_p0BxXL1TQ.png" /></figure><h3>Key Terminologies for AWS CloudFront:</h3><ol><li><strong>Edge locations:</strong> These are the data centers located in various geographical regions around the world where CloudFront caches copies of the content. When a user requests content, CloudFront routes the request to the edge location that provides the lowest latency, ensuring fast delivery of content.</li><li><strong>Origin server:</strong> An origin server is the source of the content that CloudFront delivers to users. It can be an Amazon S3 bucket, a MediaPackage channel, or an HTTP server. CloudFront retrieves the content from the origin server and caches it in the edge locations for future requests.</li><li><strong>Distribution</strong>: A CloudFront distribution is a configuration that specifies the origin server(s) and other settings for content delivery. It tells CloudFront which origin server(s) to get the files from and how to handle requests for the content. CloudFront assigns a domain name to each distribution, which can be used to access the content.</li><li><strong>TTL (Time-to-Live):</strong> TTL is the duration for which CloudFront caches the content in the edge locations. It determines how long CloudFront serves the content from the cache without checking the origin server. A shorter TTL ensures that users receive fresh content but may increase the load on the origin server, while a longer TTL reduces the load on the origin server but may serve stale content.<br>default time is for 24 hours.</li><li><strong>Signed URLs and Signed Cookies:</strong> CloudFront supports access control mechanisms using signed URLs and signed cookies. Signed URLs are URLs that include authentication information, allowing only authorized users to access the content. Signed cookies are HTTP cookies that contain authentication information, providing a way to restrict access to specific resources.</li><li><strong>SSL/TLS Encryption:</strong> CloudFront supports SSL/TLS encryption to secure the communication between viewers and CloudFront. It uses the latest version of Transport Layer Security (TLSv1.3) for secure connections. AWS Certificate Manager (ACM) can be used to create and manage SSL certificates for CloudFront distributions.</li><li><strong>DDoS Protection:</strong> CloudFront protects network and application layer DDoS (Distributed Denial of Service) attacks. It works seamlessly with AWS Shield, AWS Web Application Firewall (WAF), and Amazon Route 53 to create a layered security perimeter for applications and content.</li><li><strong>Edge Computing: </strong>CloudFront offers programmable and secure edge computing capabilities through CloudFront Functions and AWS Lambda@Edge. CloudFront Functions allow for high-scale and latency-sensitive operations like HTTP header manipulations, URL rewrites/redirects, and cache-key normalizations. Lambda@Edge is a serverless computing feature that supports a wide range of computing needs and customizations at the edge locations.</li><li><strong>Origin Access Identity (OAI)</strong>: OAI is a feature in CloudFront that restricts access to an Amazon S3 bucket, making it only accessible from CloudFront. It adds a layer of security by allowing access to the content only through CloudFront, preventing direct access to the S3 bucket.</li></ol><h3>Advantages of CloudFront</h3><ul><li><strong>Improved Performance</strong>: CloudFront caches content at edge locations, reducing latency and enabling faster response times for end-users. By serving content from the nearest edge location, CloudFront minimizes the distance and network hops required to deliver content, resulting in quicker content loading.</li><li><strong>Global Reach</strong>: With its extensive network of edge locations, CloudFront ensures content can be delivered to users almost anywhere in the world. This global reach provides a consistent and reliable experience for users, regardless of their geographic location.</li><li><strong>Scalability</strong>: CloudFront is designed to handle high traffic volumes and can automatically scale to accommodate spikes in demand. This ensures that your application remains available and responsive, even during periods of high traffic.</li><li><strong>Security</strong>: CloudFront offers robust security measures to protect your content against various threats. It supports HTTPS encryption, ensuring secure data transfer between the edge locations and end-users. Additionally, CloudFront integrates with AWS Shield and Web Application Firewall (WAF) to mitigate DDoS attacks and protect against common web exploits.</li><li><strong>Cost-Efficiency</strong>: CloudFront follows a pay-as-you-go pricing model, with no upfront costs. By leveraging the integration with other AWS services and optimization features, CloudFront can result in significant cost savings compared to managing a CDN in-house.</li><li><strong>Integration with AWS Ecosystem</strong>: CloudFront seamlessly integrates with other AWS services, such as Amazon S3, Amazon EC2, and Elastic Load Balancing. This integration allows you to leverage the full AWS ecosystem to enhance and customize your content delivery.</li><li><strong>Customization and Flexibility</strong>: CloudFront provides various customization options, such as cache behaviors and Lambda@Edge, which allows you to execute serverless code at edge locations. These features enable fine-grained control over how your content is served to different users or regions.</li></ul><h3>Use Cases for CloudFront</h3><p>CloudFront is commonly used in the following scenarios:</p><ul><li><strong>Static Website Hosting</strong>: CloudFront can be used to host static websites stored in Amazon S3. By configuring CloudFront to serve the website content, you can take advantage of caching and global distribution, resulting in faster and more reliable website access for users worldwide.</li><li><strong>Dynamic Content Delivery</strong>: CloudFront can accelerate the delivery of dynamic content generated by applications running on Amazon EC2 or other origin servers. By caching frequently accessed content at edge locations, CloudFront reduces the load on the origin servers and improves the overall performance of dynamic applications.</li><li><strong>Video Streaming</strong>: CloudFront’s support for streaming media delivery makes it an ideal choice for video streaming applications. It offers low-latency streaming and adaptive bitrate streaming, ensuring a smooth playback experience for end-users.</li><li><strong>API Acceleration</strong>: CloudFront can be used to accelerate the delivery of APIs, improving the responsiveness of API calls and reducing latency for end-users. This is particularly beneficial for applications with a global user base or those that rely heavily on API interactions.</li></ul><h3>Hosting a Static Website with S3 and CloudFront</h3><p>To host a static website using CloudFront and Amazon S3, follow these steps:</p><ol><li>Create an S3 bucket and upload your static website files to the bucket. Make sure to enable public access to the bucket and configure the bucket for static website hosting.</li><li>Create a CloudFront distribution and configure it to use your S3 bucket as the origin. Set up additional settings such as SSL certificate, custom domain name, and caching behavior.</li><li>Once the CloudFront distribution is deployed, you can access your website using the CloudFront domain name or your custom domain name. CloudFront will serve the content from the nearest edge location, resulting in faster website access for users.</li></ol><p>By utilizing CloudFront’s caching and global distribution capabilities, you can significantly enhance the performance and reliability of your static website. Additionally, CloudFront provides features like HTTPS encryption and integration with other AWS services, ensuring a secure and seamless website hosting experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-hYx_EPswXe-F4vppRikg.png" /></figure><p>In this blog, I’ll be sharing my experience of configuring AWS CloudFront with the data origin in S3.</p><h4>Step 1: Create an S3 bucket and upload your static website files to the bucket. Make sure to enable public access to the bucket and configure the bucket for static website hosting</h4><ul><li>Navigate to the S3 dashboard of the AWS</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEK86pigtKD0GT0qjsKOgA.png" /></figure><ul><li>Enter the Region and bucket name for the S3 Bucket which should be globally unique.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*cckrFQeFCs5hu73ekwPU-w.png" /></figure><ul><li>Grant Public access to this bucket by unselecting “Block all public access” which is enabled by default.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnIWqrTvd3VuL74jOwCR8A.png" /></figure><ul><li>Set encryption type as SSE-S3</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQ6EgqKuwxFGoKzUnhb_Ng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*evpgpbQLgA6oASlKOIc-Ww.png" /></figure><ul><li>Here we can see S3 bucket has been successfully created</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*KgJM27azRoQvfLrErsuygQ.png" /></figure><ul><li>Upload Static HTML files by opening a recently created S3 bucket</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NWdTTA7SiAw7hfbU6B4jZg.png" /></figure><ul><li>Navigate to the “Permission” section and add attach this policy to the “Bucket Policy”</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::your-bucket-name/*&quot;<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3FAwqL1OnU8ERpaFK9bKDg.png" /></figure><ul><li>Navigate to the “Properties” Section</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5SyXBDtdrRMfQ3YfVCu8Dw.png" /></figure><ul><li>Scroll down to the bottom of the page Enable “Static website hosting” which is Disabled in default</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*Hg-aOHg6J7VQWTnRVR-qeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OBtxSa21hxUu_5za0U4CsA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*248TSzMqJHpnsAFQmJ6gww.png" /></figure><ul><li>Here we can see simple static site has been hosted through S3</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twBuKgkuoEzJRochb10fvQ.png" /></figure><ul><li>After testing the site from S3 Bucket Website Hosting then we have to Reconfigure to Secure CloudFront Access to S3 Read-Only Policy for a Specific Distribution . for this go to the permission section of S3 and change the policy as :</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowCloudFrontServicePrincipalReadOnly&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;cloudfront.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::cfexample/*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;AWS:SourceArn&quot;: &quot;arn:aws:*************************&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vuhbMF1duKA8RKtHLqOBNg.png" /></figure><p>This IAM policy grants read-only access to an Amazon S3 bucket (“cfexample”) exclusively for the CloudFront service. Specifically, it allows CloudFront to retrieve objects from the S3 bucket with the condition that the request originates from a specific CloudFront distribution (“E2BLG95LCCN3EG”). This policy ensures a controlled and secure configuration, limiting access to the designated CloudFront distribution for enhanced S3 content retrieval.</p><h4>Step 2: Create a CloudFront distribution and configure it to use your S3 bucket as the origin. Set up additional settings such as SSL certificate, custom domain name, and caching behavior.</h4><ul><li>Go to CloudFront service, which should give you a window just like in the below screenshot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9G4ly-qbROZwNJq8R527w.png" /></figure><ul><li>Choose Origin Domain as the S3 endpoint</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Ne3aI4blFYFPAd1HY8x7w.png" /></figure><ul><li>Use the S3 Static website hosting Endpoint here for the origin domain</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsJGjrmren54RL9eJkEKGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*PlJ79o61mViPlBUsn8Z8ig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*ox_wK9HcF8h-pw5c7t0HjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*1Q6z4xDOfUEg185PwvkspA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IwowqeHC8OMu6Atzm5aIQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q8YZjgG1mmUveVGrskaJlA.png" /></figure><h4>Step 3: Once the CloudFront distribution is deployed, you can access your website using the CloudFront domain name or your custom domain name. CloudFront will serve the content from the nearest edge location, resulting in faster website access for users.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F8OsWszpcq5ygo4r0fPFmA.png" /></figure><ul><li>Here we can see our site has been redirected by the CloudFront endpoint</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iex59y1yDSkYy3qQTGj0mw.png" /></figure><p><strong>Conclusion: </strong>In conclusion, optimizing AWS CloudFront for dynamic content delivery is crucial for enhancing the performance and responsiveness of applications. By implementing intelligent caching strategies, utilizing CloudFront Functions for edge computing, and ensuring robust security measures, businesses can create a seamless and efficient experience for users globally. The combination of AWS CloudFront and its advanced features empowers developers to deliver dynamic content with low latency, high speed, and a secure environment, ultimately improving the overall user experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b28a5f46312c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>