<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Darren Bassett on Medium]]></title>
        <description><![CDATA[Stories by Darren Bassett on Medium]]></description>
        <link>https://medium.com/@darrenbassett67?source=rss-70a03346aa87------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*92YO20kLuH2OfdTqFpb7pg.jpeg</url>
            <title>Stories by Darren Bassett on Medium</title>
            <link>https://medium.com/@darrenbassett67?source=rss-70a03346aa87------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:01:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@darrenbassett67/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond the Migration: Governing Cloud Transformation with AWS Frameworks]]></title>
            <link>https://medium.com/@darrenbassett67/beyond-the-migration-governing-cloud-transformation-with-aws-frameworks-f9884d51da83?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9884d51da83</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[solutions-architecture]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 09:01:01 GMT</pubDate>
            <atom:updated>2026-03-19T09:01:01.579Z</atom:updated>
            <content:encoded><![CDATA[<p>Architectural reflections from a strategic legacy-to-serverless simulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOS3auzd9kXKRfQ3AMt1qQ.png" /></figure><h4><strong>The Modernisation Trap</strong></h4><p>In the current cloud landscape, the push toward “Serverless First” architecture is relentless. Yet, experience suggests that jumping straight to a fully decoupled environment can be a high-risk gamble for legacy systems. The most direct path to the cloud is seldom the most stable.</p><p>Real-world success requires moving past the “Lift and Shift vs. Refactor” debate and focusing on evolutionary steps defined by governance.</p><p>This was the core focus of my recent Capstone project, where I simulated the migration of a decade-old monolith. Using the AWS Cloud Adoption Framework as my compass, I transformed this legacy system into a resilient, cloud-native architecture through a deliberate, multi-stage process.</p><p>I led the transition by applying the AWS Cloud Adoption Framework (CAF) and the Well-Architected Framework (WAF), ensuring the move was documented, secure, and aligned with business value.</p><p><strong>1. The Strategic Foundation: The AWS Cloud Adoption Framework</strong></p><p>Before I wrote a single line of CloudFormation code, I used the AWS CAF to align the technical roadmap with organisational goals. Moving a decade of legacy logic requires more than just a new hosting environment; it requires a change in how the business perceives its digital assets.</p><p><strong>The Business Perspective:</strong> From a business standpoint, the goal was to transform IT from a “cost centre” into a “value driver.” I identified two primary financial triggers for this migration:</p><ul><li><strong>Capital to Operational Expenditure:</strong> Shifting from the high upfront cost of maintaining hardware at HQ to a consumption-based model on AWS.</li><li><strong>Innovation Velocity:</strong> By decreasing the time developers spend on “undifferentiated heavy lifting”, such as OS patching, the business could pivot more quickly to customer-facing features like the new “Latest News” feed.</li></ul><p><strong>The Operations Perspective:</strong> In the existing on-premises environment, operational health was often reactive and treated as a “black box,” making issues difficult to detect or anticipate. My goal was to move toward a proactive model. By implementing Infrastructure as Code (IaC) via CloudFormation, I ensured that the entire environment was version-controlled and repeatable, effectively moving the “Source of Truth” from a technician’s manual notes to a governed, automated template.</p><p><strong>2. Engineering for Excellence: Applying the Well-Architected Framework</strong></p><p>With the strategic foundation in place, the technical execution was guided by the <strong>AWS Well-Architected Framework’s pillars</strong>. I chose to implement the migration in two distinct phases: a <strong>Replatform</strong> (Phase 1, moving to new infrastructure with minimal changes) followed by a <strong>Refactor</strong> (Phase 2, changing the app’s structure to fit the cloud).</p><p><strong>The Reliability Pillar (Phase 1: Stabilisation):</strong> In Phase 1, my priority was establishing a resilient baseline. I migrated the monolith to a highly available (HA) architecture using EC2 (AWS virtual machines) and an Application Load Balancer, which distributes traffic across servers.</p><ul><li><strong>Self-Healing Infrastructure:</strong> I configured an Auto Scaling Group to monitor instance health and automatically replace degraded nodes.</li><li><strong>Multi-AZ Distribution:</strong> By spreading instances across multiple Availability Zones, I ensured the application remained reachable even during a data centre outage.</li></ul><p><strong>The Security Pillar (Phase 2: Modernisation):</strong> Once stable, I moved to Phase 2: dividing the monolith into serverless microservices (small, independent cloud components), which allowed me to implement “Identity-Centric” security, permissions tied to users or services, which is not possible in a monolithic server:</p><p><strong>Micro-segmentation:</strong> I moved static assets to Amazon S3 and used Origin Access Control (OAC) to ensure that only my CloudFront distribution, not the public internet, could access the bucket.</p><ul><li><strong>The Principle of Least Privilege:</strong> By breaking the logic into AWS Lambda functions, I was able to assign specific IAM Roles to each task. For example, the function that reads the news feed has zero permission to write to the contact form database.</li><li><strong>Encryption in Transit:</strong> I utilised AWS Certificate Manager (ACM), which provides SSL/TLS certificates for encrypted connections, to enforce HTTPS across the entire application, ensuring user data remained encrypted from the edge to the backend.</li></ul><p><strong>The Cost Optimisation Pillar:</strong> This was perhaps the most visible win of the simulation. In Phase 1, the business paid for “idle time”; instances were running 24/7 regardless of traffic. By refactoring to Lambda and S3 in Phase 2, I moved the cost model to Pay-per-Use, aligning the AWS bill directly with end-user activity and making sure the architecture was both optimised and powerful.</p><p><strong>3. The Final ‘R’: Retire and Decommission</strong></p><p>True ROI in a migration is only realised when the legacy footprint is eliminated. In the <strong>AWS 6 ‘R’s</strong> strategy, <strong>Retire</strong> is the functional closing of the loop that ensures the business is no longer carrying the technical or financial debt of the old environment.</p><p>Once I had validated the serverless architecture and routed the production traffic via Route 53 to the new CloudFront distribution, I successfully decommissioned the Phase 1 EC2 infrastructure. From a <strong>CAF Operations Perspective</strong>, this marked the successful end of life for the legacy platform. From a <strong>Well-Architected</strong> standpoint, it was the ultimate act of cost optimisation: ensuring that zero budget was wasted on infrastructure that no longer served the business.</p><p><strong>Conclusion: Architecture as an Evolution</strong></p><p>Migrating a 10-year-old monolith to an AWS serverless stack is not only a technical change; it’s also a governance exercise. Using AWS CAF, I turned a technical requirement into a strategic business advantage.</p><p>The greatest lesson from this Capstone was learning to balance risk and reward through proven frameworks. Respect the legacy, but prioritise a secure, governed path to the cloud-native future.</p><p>Modernisation works best when it is deliberate, governed, and framework-driven.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9884d51da83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Chatbot]]></title>
            <link>https://medium.com/@darrenbassett67/beyond-the-chatbot-8cad9d0fff5b?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cad9d0fff5b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 08:51:01 GMT</pubDate>
            <atom:updated>2026-03-17T08:51:01.020Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Operational Reality of RAG on AWS</h4><h3>The Credibility Problem</h3><p>Generative AI is moving quickly from experimentation into real-world deployment. In many organisations, the conversation has shifted from “What can Large Language Models do?” to “How quickly can we deploy them?” The progress is undeniable, and the outputs are becoming increasingly sophisticated. The potential productivity gains are significant.</p><p>Yet there is a fundamental problem. An LLM can generate a beautifully written answer that is entirely incorrect. It may confidently fabricate policies, misinterpret regulations, or invent facts that sound plausible but have no grounding in reality. In some consumer-facing applications, the consequences may be limited.</p><p>In regulated or enterprise environments, however, answers must be reliable and traceable<em>.</em></p><p>The real challenge is not simply generating language. It is generating accurate, traceable, and trustworthy responses. This is where Retrieval Augmented Generation (RAG) becomes important.</p><p>As part of my AI/ML bootcamp, I built an RAG solution on AWS to understand how the pattern works in practice and what it takes to ground a model’s responses in real data. The objective was not to declare a “production-ready” system, but to understand the moving parts, trade-offs, and operational considerations that arise as soon as you move beyond a demo.</p><p>The sections that follow examine several architectural decisions that arise in practice when designing RAG systems, including vector storage, chunking strategy, embedding selection, security controls, and the operational trade-offs among cost, latency, and accuracy.</p><h3>Why Standard LLM Deployments Struggle in the Enterprise</h3><p>Large Language Models are trained on large datasets drawn from public and licensed sources. They are exceptional pattern recognisers. However, they do not inherently understand an organisation’s internal context: its policies, proprietary documentation, compliance requirements, or the latest operational updates.</p><p>When asked about these topics, the model does not decline to answer. It generates a response. This behaviour is the root of hallucinations: probabilistic confidence without contextual grounding.</p><p>Enterprise systems, by contrast, demand something very different. Answers must be traceable back to source material. They must be consistent, auditable, and aligned with organisational controls.</p><p>Deploying an LLM without a retrieval layer, therefore, creates a structural mismatch with enterprise requirements.</p><h3>The RAG Pattern: Simple in Theory, Complex in Practice</h3><p>Retrieval Augmented Generation addresses this problem by introducing a retrieval step before generation. When a user submits a question, the system first searches a curated knowledge base for relevant documents. Those documents are then incorporated into the prompt, grounding the model’s response in trusted information.</p><figure><img alt="Diagram showing the flow of a Retrieval Augmented Generation (RAG) request. An AWS Lambda function acts as the central orchestrator, retrieving context from a FAISS index stored in S3 and conversation history from DynamoDB before calling Amazon Bedrock for response generation." src="https://cdn-images-1.medium.com/max/581/1*aE92FBoZ1Es5skJ0LsZvuA.png" /><figcaption>Figure 1: The bootcamp implementation architecture using AWS Lambda as the central orchestrator.</figcaption></figure><p>At a conceptual level, the architecture appears straightforward:</p><ul><li>Capture the user’s question</li><li>Convert the query into a vector embedding</li><li>Retrieve semantically similar document fragments</li><li>Construct a structured prompt</li><li>Generate a response using the retrieved context</li><li>Store the conversation in persistent memory</li></ul><p>In my bootcamp implementation, the workflow was intentionally simple. A bespoke Python chat client captured user input. Facebook AI Similarity Search (FAISS) served as the vector store. Document chunking and embedding were handled by a Python application using the FAISS APIs. AWS Lambda provided orchestration, Amazon Bedrock generated the natural language responses, and Amazon S3 stored both the source documents and the FAISS index. To maintain state across the session, Amazon DynamoDB was used to store conversation memory, ensuring the orchestrator had context from previous interactions.</p><p>Each step introduces important design decisions around data preparation, retrieval quality, latency, security, and cost.</p><p>In production environments, RAG behaves less like a simple pattern and more like a system composed of interacting components.</p><h3>Choosing a Vector Store: Simplicity vs Sustainability</h3><p>One of the earliest architectural decisions concerns the vector store.</p><p>In the bootcamp implementation, FAISS provided a lightweight solution. Running the FAISS index within the Lambda execution environment avoided additional infrastructure and kept costs low for the prototype system. While FAISS in Lambda is cost-effective, it requires the index to be loaded into memory on every ‘cold start,’ making it better suited for small, static datasets than dynamic enterprise knowledge bases.</p><p>For tightly scoped datasets or controlled environments, this approach can be highly effective. However, FAISS does not inherently provide persistence or horizontal scalability. As datasets grow, operational complexity increases.</p><p>Managed alternatives, such as OpenSearch or Amazon Bedrock Knowledge Bases, provide persistence, scalability, and tighter integration with the broader AWS ecosystem, but they also introduce additional infrastructure and costs.</p><p>The choice is therefore not purely technical. It reflects a broader architectural philosophy: whether to prioritise minimal infrastructure and cost efficiency, or to optimise for durability and long-term scalability.</p><p>In production environments, the latter often becomes more important.</p><h3>Chunking: A Critical but Often Overlooked Design Choice</h3><p>Before documents can be embedded, they must be segmented. The size and overlap of these segments influence retrieval accuracy, token consumption, latency, and cost.</p><p>Larger chunks increase prompt size and token consumption. Smaller chunks can fragment meaning and weaken semantic coherence.</p><p>There is no universal configuration. Chunking becomes an exercise in balancing contextual integrity with operational efficiency. In practice, moderately sized overlapping chunks often perform well, although the optimal configuration depends heavily on the structure of the underlying source material.</p><p>Retrieval quality ultimately influences both user experience and operational cost.</p><h3>Embeddings: Precision Has Consequences</h3><p>Embedding models convert text into vector representations that capture semantic similarity. On AWS, services such as Amazon Bedrock provide managed embedding models that integrate naturally into an RAG workflow.</p><p>In my bootcamp implementation, embeddings were generated during the document ingestion phase using a Python application that processed each chunk and stored the resulting vectors in the FAISS index. This step forms the foundation of retrieval, because the quality of the embedding directly influences how accurately relevant documents can be located.</p><p>Selecting an embedding model is rarely trivial. Higher-dimensional embeddings can improve semantic precision, but they also increase storage requirements and retrieval latency. Larger vectors consume more memory and influence infrastructure sizing decisions.</p><p>It is often tempting to default to the most powerful model available. In production systems, however, embedding selection should be validated against real organisational data through benchmarking and evaluation.</p><p>Greater precision typically introduces additional cost.</p><h3>Security: When Experimentation Meets Reality</h3><p>During early experimentation, security configurations are often permissive. As soon as real organisational data is introduced, security becomes foundational.</p><p>Sensitive information requires encryption at rest, tightly scoped IAM policies, controlled model access, and careful logging practices. In systems serving multiple organisations, teams, or user groups, tenant isolation becomes necessary to ensure that retrieval is restricted to the data a user is authorised to access. Additionally, prompt injection risks must be addressed through validation and guardrails.</p><p>Beyond IAM, enterprise RAG requires traceability. Using tools like Amazon Bedrock Guardrails or CloudWatch logs ensures that we aren’t just getting the right answer, but we’re doing so within the bounds of corporate policy.</p><p>A RAG system is therefore more than a conversational interface. It mediates access to institutional knowledge.</p><p>At this stage, the architecture surrounding the model becomes a primary determinant of system reliability and safety.</p><h3>Cost and Latency: The Invisible Governors</h3><p>Early experiments often appear inexpensive. Production deployments reveal the cumulative effect of tokens, embeddings, and infrastructure runtime.</p><p>Increasing retrieval depth introduces more tokens. Inefficient chunking increases prompt size. Additional processing steps compound latency.</p><p>Users ultimately experience response time, not architectural elegance.</p><p>Architectural decisions must therefore balance precision, responsiveness, and cost. Systems that appear optimal in isolation may prove unsustainable at scale.</p><h3>From Retrieval to Agency</h3><p>As RAG systems mature, organisations often ask whether the assistant can do more than answer questions. Can it trigger workflows, call APIs, or participate in multi-step processes?</p><p>At this stage, retrieval architectures begin to intersect with agent-based design. The orchestration layer evolves, and the assistant transitions from answering questions to interacting with operational systems.</p><p>The model remains an important component, but architectural design increasingly determines what the system can do.</p><h3>Key Takeaways: Designing RAG on AWS</h3><p>If you are moving a RAG prototype toward production, keep these five architectural realities in mind:</p><ul><li><strong>Model is secondary; Data is primary:</strong> The quality of your response is only as good as your chunking and embedding strategy. If your retrieval layer pulls the wrong context, even the most powerful LLM will fail.</li><li><strong>Scale dictates your Vector Store:</strong> While FAISS is excellent for lightweight, cost-effective prototypes, enterprise-grade persistence and horizontal scaling typically require managed services like Amazon OpenSearch or Amazon Bedrock Knowledge Bases.</li><li><strong>Efficiency is a balancing act:</strong> Every architectural choice has a “tax.” High-dimensional embeddings improve precision but increase latency and storage costs. Overlapping chunks improve context but increase token consumption.</li><li><strong>Security is the “Day 1” requirement:</strong> In the enterprise, RAG isn’t just a chat interface; it’s a data gateway. IAM policies, encryption, and tenant isolation are foundational to system trust.</li><li><strong>Architecture is the true differentiator:</strong> While Generative AI feels like magic, its success in production depends on disciplined engineering.</li></ul><blockquote>“Trust doesn’t come from the model; it emerges from the architecture surrounding it.”</blockquote><h3>Conclusion: Architecture Is the Differentiator</h3><p>Designing a RAG architecture on AWS involves more than assembling cloud services. It requires deliberate decisions across multiple dimensions: accuracy, cost, latency, security, scalability, and governance.</p><p>Generative AI may feel revolutionary. In enterprise environments, it must operate within disciplined architectural constraints.</p><p>RAG provides a powerful mechanism for grounding probabilistic models in trusted knowledge. Its effectiveness depends less on novelty and more on careful system design.</p><p>In production systems, the model is only one component.</p><p>Trust ultimately emerges from architecture.</p><p>As organisations continue to explore generative AI, the focus may gradually shift.</p><p>The challenge is no longer whether a model can produce convincing language, but whether the surrounding architecture can ensure that those responses are reliable, secure, and sustainable in real-world systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cad9d0fff5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Chaos to Clarity]]></title>
            <link>https://medium.com/@darrenbassett67/from-chaos-to-clarity-cf799bffad78?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf799bffad78</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[job-search]]></category>
            <category><![CDATA[career-development]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Thu, 22 May 2025 08:45:05 GMT</pubDate>
            <atom:updated>2025-05-22T08:45:05.383Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>My 3-Phase AI-Powered Blueprint for Job Search Success</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAhLV_RRF-WG602om-tD4w.png" /><figcaption>Image generated by DALL-E</figcaption></figure><p>Juggling a demanding AWS cloud bootcamp, crafting portfolio projects, and navigating the modern job search felt like an overwhelming tangle of priorities. I knew there had to be a smarter way to manage it all. Here’s the 3-phase system I built, with an AI co-pilot, to bring order to the chaos and drive real progress.</p><p>My approach involved leveraging not just any AI but a sophisticated large language model (LLM) capable of maintaining context over extended dialogue and engaging in a “chain of thought” style collaboration, which was key to the iterative and detailed nature of our planning. This AI became my collaborative partner, alongside the Agile methodology for execution. I deliberately structured this process by first understanding my capacity, then building a strategy to fit that capacity, and finally using Agile to execute.</p><p><strong>Phase 1: Building the Engine — AI-Assisted Timetable Creation, Capacity Planning</strong></p><p>Before realistically committing to a strategy that involves specific hours for job applications, networking, or company research, the foundational step is to know if and when that time is available. Therefore, my first phase focused on creating a comprehensive timetable. This wasn’t just about listing tasks; it was about meticulously allocating time for my Bootcamp, existing commitments like exercise, essential portfolio work, and then carving out dedicated blocks for all job search activities. This process defined my capacity.</p><p>To achieve this, I employed a specific meta-prompting technique rather than just telling the AI what I thought I needed. I asked the AI: “I need to create a detailed weekly timetable. Ask me all the questions you need answers to to generate a realistic and comprehensive schedule that balances my bootcamp, portfolio work, job search, and personal commitments.” This approach shifted the AI from merely following instructions to acting as a Socratic partner, asking the right questions and driving deeper thinking. It systematically generated a structured list of inquiries, prompting me for specifics about fixed commitments like Bootcamp live sessions, estimated weekly hours for self-paced study, desired time for portfolio projects, target allocations for job applications and networking, and crucially, my peak focus times, preferred break durations, and even my exercise slots. This methodical questioning ensured we covered all critical variables.</p><p>Through an iterative design process, the AI’s ability to maintain context and follow that “chain of thought” allowed it to reconfigure schedules rapidly based on my feedback. We arrived at a detailed, 30-minute slot-by-slot weekly timetable. This schedule, balanced and aligned with my energy, became the bedrock, clearly showing my available capacity for each activity.</p><p><strong>Phase 2: Charting the Course — AI-Powered Job Search Strategy Built on Available Capacity</strong></p><p>With clarity on the amount of time I had available and the allocations from the timetable, I could then develop a realistic job search strategy. Knowing my available capacity directly informed how ambitious or focused the plan could be. This phase involved using a similar strategic prompting approach with the AI, asking it to prompt me for the information needed to define my ideal company and role preferences, thus serving as a strategic sounding board.</p><p>The AI first helped me establish a dual-goal definition: identifying “Good Match” companies for proactive networking and identifying roles for immediate application. To truly define a “Good Match” company, its targeted questions helped me articulate my ideal preferences across company culture, technical environment, mission/values, and growth opportunities. Similarly, for role identification, we detailed essential skills and absolute deal-breakers.</p><p>A critical output of this phase was translating these preferences into a ‘Good Fit’ evaluation checklist. This involved defining my absolute “Must-Haves,” a “Green Flag / Red Flag” system to assess positive and negative indicators, and a Tiered Prioritisation method for companies. This checklist, designed to be used during the time slots defined in my timetable, ensures consistent company evaluation. Finally, with evaluation criteria set, the AI assisted in brainstorming specific tactics for company discovery and role sourcing. This comprehensive phase resulted in a documented job search strategy, tailored to my aspirations and grounded in the reality of my available time.</p><p><strong>Phase 3: Driving Agile Execution — Executing the Strategy within the Timetable</strong></p><p>A great strategy, built upon a realistic understanding of capacity, still requires effective execution. With the detailed timetable (Phase 1) defining my available working hours and the comprehensive job search strategy (Phase 2) outlining what needed to be done, I then leveraged the AI to help structure the execution using Agile principles.</p><p>I didn’t just manually list tasks. Instead, I presented the AI with the outputs of the previous two phases: my structured timetable with allocated time blocks for ‘Job Applications,’ ‘Networking,’ etc., and the key strategic objectives I identified in my job search strategy. My prompt was along the lines of: ‘Given this timetable and this strategy outlining my key objectives and task types, help me decompose this into Agile Epics and a sample set of user stories or tasks that would fit into weekly Sprints.’</p><p>The AI, comprehending the context of both my time availability and strategic goals, was able to assist in several ways. It proposed logical “Epics”: high-level strategic pillars like Build &amp; Refine Target Company List or Apply to Roles, because these aligned with the main thrusts of the strategy document. For an objective like “Identify Good Match Companies,” the AI could break this down into concrete, sprint-able tasks such as “Research 3 potential companies using LinkedIn” or “Evaluate researched companies using ‘Good Fit’ Checklist &amp; assign Tier,” as these were logical next steps implied by the strategy and fitted within the focused work blocks my timetable allowed. While the AI didn’t assign tasks to specific days (that’s part of my Sprint Planning), its understanding of the available time blocks ensured that the suggested tasks were suitably sized. This process effectively kickstarted my Product Backlog, providing a robust set of initial Epics and tasks that I could refine, prioritise, and pull into my weekly “Sprints.”</p><p>To ensure focused action, I adopted these one-week Sprints, defining a clear sprint goal at the start of each week during “Sprint Planning.” The tasks are meticulously tracked in my spreadsheet. The heart of this Agile approach lies in continuous improvement through weekly “Sprint Reviews &amp; Retrospectives,” which allow me to assess progress, learn, and adapt my approach, ensuring my efforts remain effective.</p><p><strong>The Impact: Clarity, Control, and Consistent Progress</strong></p><p>This three-phase approach — establishing capacity via the timetable, building a strategy to fit that capacity (aided by strategic AI prompting), and using Agile for execution — has fundamentally changed how I approach my career development and job search. It has replaced overwhelm with clarity, reactivity with proactivity, and a feeling of being adrift with a sense of controlled, measurable progress.</p><p><strong>Key Learnings from This Process:</strong></p><p>This journey of structuring my job search and study schedule with AI and Agile hasn’t just been about creating plans; it’s been a profound learning experience. Here are my key takeaways:</p><ol><li>Clarity on Capacity is Non-Negotiable: Before any strategy or task list, understanding my actual available time (after accounting for non-negotiables like bootcamp and personal well-being) was the most critical first step. A strategy built on unrealistic time assumptions is doomed to fail. The AI-assisted timetable creation forced this essential realism.</li><li>Strategic Prompting Unlocks AI’s Potential: Simply telling an AI what to do yields limited results. Asking the AI to ask me the right questions (meta-prompting) transformed it into a powerful Socratic partner, ensuring comprehensive requirement gathering and a more tailored output. This significantly deepened the collaborative process.</li><li>Agile Isn’t Just for Software (It’s for Life Management): The principles of breaking down significant goals into manageable sprints, focusing on weekly achievements, and regularly reviewing are incredibly powerful for complex personal projects like a job search. It combats overwhelm and builds momentum.</li><li>Iteration is Perfection’s Best Friend: Neither the timetable, the strategy, nor Agile tasks were perfect on the first attempt. The iterative nature of working with AI, combined with the built-in feedback loops of Agile, enabled continuous refinement and adaptation. Embracing this iterative process was key.</li><li>Structure Breeds Consistency (and Sanity!): Having a defined system, from the timetable to the company evaluation checklist to the sprint planning, reduced daily decision fatigue. It allowed me to focus my energy on doing the work rather than constantly figuring out what work to do next.</li><li>AI is a Co-Pilot, not Autopilot: Always Review and Refine. While the AI was instrumental in generating drafts, frameworks, and ideas, every output required careful human review, critical thinking, and personalisation. AI accelerates, but human judgment and ownership remain paramount to ensure accuracy, relevance, and authentic voice.</li></ol><p>This structured, AI-augmented, and Agile-driven approach has not only given me a more straightforward path forward but also equipped me with a versatile framework for tackling future complex endeavours.</p><p><strong>How Do You Navigate Your Career Goals?</strong></p><p>Navigating the tech job market while upskilling is a unique challenge. If you’ve faced similar hurdles, what’s one piece of advice you’d give to someone just starting this journey? Drop your wisdom below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf799bffad78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-powered Image Processing Application Using AWS]]></title>
            <link>https://medium.com/@darrenbassett67/building-an-ai-powered-image-processing-application-using-aws-e8d843fd44ed?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8d843fd44ed</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Tue, 17 Sep 2024 11:33:12 GMT</pubDate>
            <atom:updated>2024-09-17T11:49:30.280Z</atom:updated>
            <content:encoded><![CDATA[<p>How I leveraged AWS Bedrock, Lambda, Fargate and CI/CD to create an AI-powered image-processing app</p><h3>Introduction</h3><p>In today&#39;s cloud-driven world, the ability to rapidly develop and deploy scalable, intelligent applications is more crucial than ever. As a developer, I recently embarked on an exciting journey to create a cutting-edge web application that combines the power of serverless architecture, artificial intelligence, and continuous integration/continuous deployment (CI/CD) practices.</p><p>In this article, I&#39;ll walk you through how I built an AI Image Processing Application. This sophisticated web application allows users to upload images and receive AI-generated descriptions. This project leverages a suite of AWS services, including Lambda, S3, DynamoDB, ECS Fargate, and Amazon Bedrock, to create a robust, scalable, and highly available solution.</p><p>This project&#39;s use of modern DevOps practices makes it particularly interesting. I&#39;ll explore how to set up a complete CI/CD pipeline using AWS CodePipeline, CodeBuild, and GitHub, ensuring the application can be updated and deployed quickly and confidently.</p><p>Whether you&#39;re a seasoned AWS professional or just starting your cloud journey, this article will provide valuable insights into architecting complex serverless applications, integrating AI services, and implementing DevOps best practices. So, let&#39;s dive in and discover how to bring this exciting project to life!</p><h3>Project Overview</h3><p>The application architecture leverages AWS serverless services to manage and process images efficiently. The user uploads images through a web interface hosted on AWS Fargate. Once uploaded, Lambda functions process the images, store image metadata in DynamoDB and move the images to a designated S3 bucket. The AI image descriptions are generated using Amazon Bedrock and updated in the DynamoDB table. The application also includes a CI/CD pipeline using AWS CodePipeline, CodeBuild, and CodeCommit for seamless updates and deployments.</p><p>The architecture diagram below illustrates the different AWS services used in the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdsVLX64f2ZmyW77zjkkmg.png" /><figcaption>AI Generator Web Application Architecture</figcaption></figure><h4><strong>Image Processing Pipeline</strong></h4><p>A web interface hosted on AWS Fargate is accessible by users through a custom domain name managed by Route 53. An image upload page allows users to upload images from their local computers. When a user uploads an image, the backend processing, managed by AWS Lambda functions, is invoked via a Function URL. The &#39;Upload Webpage&#39; function stores the image in an S3 upload bucket. The Lambda function, triggered by the upload, then processes the image by:</p><ul><li>Generating a unique URL for the image;</li><li>Updating a DynamoDB table with metadata for the image, including the image key and a placeholder for the description;</li><li>Moving the image to a separate S3 bucket for website assets.</li></ul><h4><strong>AI Image Description Generation</strong></h4><p>Amazon Bedrock powers the image processing pipeline, which generates AI descriptions by leveraging the &#39;Anthropic Claude 3-haiku&#39; model. The web interface allows users to create a description of their uploaded image. When the user clicks on an image, they invoke a Lambda function, which generates a description of the image by performing the following actions:</p><ul><li>The Lambda function retrieves the image from the website assets S3 bucket.</li><li>It then passes the image to Amazon Bedrock, which generates a description of the image based on a prompt defined in the code.</li><li>Finally, the DynamoDB table is updated to add the image description.</li></ul><h4><strong>CI/CD Docker Image Pipeline</strong></h4><p>The CI/CD pipeline is critical for automating the build and deployment process to ensure that it is repeatable and reliable. I configured a GitHub repository in which I stored the application assets. I then configured AWS CodeBuild and CodePipeline, using the GitHub repository as the source, to build and deploy the Docker image by:</p><ul><li>Create a Docker image containing an Apache web server and the website&#39;s static HTML and JavaScript files.</li><li>Login to the Elastic Container Registry and push the Docker image.</li><li>Deploy the new image to AWS Fargate.</li></ul><h3>Key AWS Services and Their Role</h3><p>I used the following AWS services to implement this project:</p><ul><li><strong>AWS Fargate:</strong> Hosts the web application, providing scalability and eliminating the need to manage servers.</li><li><strong>Amazon S3:</strong> Stores uploaded images and website assets.</li><li><strong>AWS Lambda:</strong> Handles backend logic for image uploads, processing, and AI description generation.</li><li><strong>Amazon DynamoDB:</strong> Stores metadata and AI-generated descriptions, ensuring fast information retrieval.</li><li><strong>Amazon Bedrock:</strong> Powers the AI image description generation using the &#39;Anthropic Claude 3-haiku&#39; model.</li><li><strong>AWS CodePipeline:</strong> Automates the build and deployment process, ensuring continuous integration and delivery.</li><li><strong>Amazon Route 53:</strong> for the custom domain name and AWS Certificate Manager (ACM) for the website certificate.</li></ul><h3>Challenges and Solutions</h3><p>Configuring permissions for this solution was challenging due to the need for fine-grained access control across multiple AWS services. Each application component, including Lambda functions, S3 buckets, DynamoDB, and Amazon Bedrock, required specific permissions to perform actions like uploading images, accessing data, and invoking AI models. To resolve this, I carefully crafted IAM policies that provided sufficient access for each service to perform its tasks while adhering to the principle of least privilege.</p><p>Configuring CORS (Cross-Origin Resource Sharing) for this solution was challenging, as it involved ensuring seamless communication between the web interface and the backend services hosted on AWS. The image upload interface, hosted on AWS Fargate, needed to interact with Lambda functions and S3 buckets, which required proper CORS settings to allow cross-origin requests. I configured the CORS rules on the S3 buckets and the Lambda function URLs to allow specific methods, headers, and origins required for the application. This process involved carefully adjusting CORS settings. By balancing functionality and security, I ensured that I only exposed the necessary resources to cross-origin requests.</p><p>A challenge here was managing Docker image configurations and ensuring seamless integration with the existing Fargate tasks. I overcame this by carefully defining and testing the pipeline&#39;s &#39;buildspec&#39; and pipeline definition files, ensuring they matched the application&#39;s requirements.</p><h3>Key Takeaways and Learnings</h3><p>This project was a comprehensive exercise in building a full-stack, serverless web application using AWS services. It deepened my understanding of serverless architecture and how various AWS components can work together to create a scalable and robust solution. One key takeaway was the importance of precise IAM configuration for ensuring security without hindering functionality. By adhering to the principle of least privilege, I learned how to provide just enough access for each service, maintaining security while allowing seamless interaction among Lambda, S3, DynamoDB, and Bedrock.</p><p>I implemented the project using the AWS CLI, which significantly enhanced my command-line tools and scripting proficiency. It provided invaluable hands-on experience in programmatically deploying and managing AWS resources, offering greater flexibility and control than using the AWS Management Console alone.</p><p>Another significant learning was the intricacy of managing CORS in a cloud environment. Configuring cross-origin requests for Fargate-hosted applications to interact smoothly with backend services like S3 and Lambda required a careful balance between security and usability.</p><p>Additionally, implementing a CI/CD pipeline with AWS CodePipeline and CodeBuild enhanced my understanding of automating the deployment process, reducing manual intervention and minimizing errors.</p><p>Overall, this project showcased the power of AWS services to quickly build, deploy, and scale intelligent applications while highlighting best practices in security, architecture, and DevOps.</p><h3>Final Thoughts and Next Steps</h3><p>For readers and fellow developers, I encourage you to experiment with AWS services and take on similar projects to deepen your cloud development skills. There&#39;s immense value in learning how different AWS components can be combined to build scalable, secure, and efficient applications.</p><p>For the future of this project, you could expand its capabilities by incorporating features like image classification using Amazon Rekognition to provide more detailed image analysis. Additionally, you could enhance the CI/CD pipeline by integrating automated testing and monitoring to ensure the application remains reliable and performant as it evolves. This journey into building cloud-based, intelligent applications is just the beginning, and I look forward to exploring even more complex and innovative solutions in the AWS ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8d843fd44ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Currency Normalisation with AWS]]></title>
            <link>https://medium.com/@darrenbassett67/automating-currency-normalisation-with-aws-eb935f93de32?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb935f93de32</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 08:01:54 GMT</pubDate>
            <atom:updated>2024-08-23T08:01:54.072Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Step-by-Step Guide Using Python and the AWS CDK</h4><h4><strong>Project Overview</strong></h4><p>Welcome to my guide on implementing a Currency Normalisation Project using Python with the AWS Cloud Development Kit (CDK).</p><p>This project automates the normalisation of financial data across different currencies using AWS cloud services. The application processes property sales data from various regions in North America, converting all sale prices to US Dollars (USD). By the end of this project, we&#39;ll have a fully functional cloud-based application that converts currencies and efficiently stores and processes transaction records using AWS serverless technologies.</p><h4><strong>Learning Objectives</strong></h4><ul><li><strong>Python Programming for Data Manipulation:</strong> Utilise Python to handle and transform financial data.</li><li><strong>AWS Cloud Development Kit: </strong>Deploy infrastructure and code repeatedly and consistently with Infrastructure as Code (IaC).</li><li><strong>AWS Lambda Functions:</strong> Implement serverless functions to automate data processing tasks.</li><li><strong>Amazon S3 for Data Management:</strong> Use S3 buckets to manage raw and processed data.</li><li><strong>DynamoDB for Storage:</strong> Store normalised transaction records efficiently in DynamoDB.</li></ul><h4><strong>The Task</strong></h4><p>We aim to create a cloud-based system that normalises property sales data by converting prices from Canadian Dollars (CAD) to US Dollars (USD). The system will store the converted data in a DynamoDB table and save the processed CSV files in a designated S3 bucket for further use or archival.</p><h4><strong>Architecture Overview</strong></h4><p>The system will consist of:</p><ul><li><strong>Two S3 buckets:</strong> One for raw data and another for processed data.</li><li><strong>Lambda Function: </strong>This automates the processing of data files.</li><li><strong>DynamoDB Table: </strong>To store processed financial records.</li><li><strong>AWS CDK: </strong>For initial setup and deployment of AWS resources.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*X08GICUsgcWySEM1GE1PxA.jpeg" /><figcaption>Currency Normaliser Architecture</figcaption></figure><h4>Implementation Details</h4><p><strong>Step 1: Setting Up the CDK Project</strong></p><p>First, we will set up our AWS environment using the AWS Cloud Development Kit (CDK). Follow these steps to configure your project:</p><ol><li><strong>Initialise the Project Environment:</strong></li></ol><p>Create a repository for your code in your Integrated Development Environment (IDE). Here is a suggested structure:</p><pre>aws-currency-normalisation/<br>├── .venv/                          # Virtual environment directory (generated)<br>├── cdk.out/                        # CDK output directory (generated by CDK)<br>├── README.md                       # Project overview, setup instructions, etc.<br>├── requirements.txt                # Python dependencies for the project<br>├── properties_app/                 # Main project directory (CDK app)<br>│   ├── __init__.py                 # Indicates this directory is a Python package<br>│   ├── properties_app_stack.py     # CDK stack definition<br>│   ├── resources/                  # Directory for resources like templates<br>│   └── tests/                      # Unit tests for CDK stack<br>│       └── test_properties_app_stack.py  # Unit test file for the CDK stack<br>├── src/                            # Source directory for Lambda functions<br>│   └── CurrencyNormaliser/         # Directory for Currency Normaliser Lambda<br>│       ├── lambda_function.py      # Main Lambda function code<br>│       ├── event.json              # Example event JSON for testing<br>│       ├── template.yaml           # Optional: SAM template for local testing<br>│       └── tests/                  # Unit tests for the Lambda function<br>│           └── test_lambda_function.py  # Unit test file for Lambda function<br>├── scripts/                        # Scripts for setup, deployment, etc.<br>│   ├── deploy.sh                   # Shell script to deploy the CDK stack<br>│   └── clean.sh                    # Shell script to clean up resources<br>└── .gitignore                      # Git ignore file</pre><p>Prepare your project environment and initialise a CDK application by executing the following commands in your IDE terminal:</p><ul><li>Create a new directory for your project:</li></ul><pre># Create a new directory for the project<br>mkdir properties_app<br><br># Navigate into the project directory<br>cd properties_app</pre><ul><li>Initialise a new CDK project with Python as the programming language, activate the Python virtual environment and install the required dependencies:</li></ul><pre># Initialize a new CDK project with Python<br>cdk init app --language python<br><br># Activate a Python virtual environment<br>source .venv/bin/activate<br><br># Install dependencies<br>pip install -r requirements.txt<br><br># Bootstrap environment in AWS<br>cdk bootstrap</pre><p><strong>2. CDK Stack Definition:</strong></p><ul><li>Create a new properties_app_stack.py file in the properties_app directory of your repository. Then, define the CDK stack by adding the following Python code to set up your infrastructure:</li></ul><pre>from aws_cdk import (<br>    Duration,<br>    Stack,<br>    aws_s3 as s3,<br>    aws_s3_notifications as s3_notif,<br>    RemovalPolicy,<br>    aws_lambda,<br>    aws_dynamodb as dynamodb<br>)<br><br>from constructs import Construct<br><br>class PropertiesAppStack(Stack):<br><br>    def __init__(self, scope: Construct, construct_id: str, **kwargs) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br><br>        # S3<br>        buckets = {&#39;raw_properties&#39;: None, &#39;processed_properties&#39;: None}<br>        for id in buckets:<br>            buckets[id] = self.create_bucket(id)<br>        <br>        <br>        # DYNAMODB<br>        table = dynamodb.Table(self, &#39;PropertiesTable&#39;,<br>            partition_key=dynamodb.Attribute(<br>                name=&#39;zpid&#39;,<br>                type=dynamodb.AttributeType.STRING<br>            ),<br>            removal_policy=RemovalPolicy.DESTROY,<br>            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST<br>        )<br>        <br>        <br>        # LAMBDA   <br>        lambda_cn = aws_lambda.Function(self, &#39;CurrencyNormalizer&#39;,<br>            runtime=aws_lambda.Runtime.PYTHON_3_10,<br>            timeout=Duration.seconds(10),<br>            handler=&#39;lambda_function.lambda_handler&#39;,<br>            code=aws_lambda.Code.from_asset(&quot;src/CurrencyNormalizer/&quot;)<br>        )<br>        <br>        <br>        # PERMISSIONS &amp; EVENTS        <br>        buckets[&#39;raw_properties&#39;].add_event_notification(s3.EventType.OBJECT_CREATED, s3_notif.LambdaDestination(lambda_cn))        <br>        buckets[&#39;raw_properties&#39;].grant_read_write(lambda_cn)<br>        buckets[&#39;processed_properties&#39;].grant_read_write(lambda_cn)<br>        <br>        table.grant_write_data(lambda_cn)<br>        <br>        <br>        # ENVIRONMENTAL VARS<br>        lambda_cn.add_environment(&#39;RAW_PROPERTIES_BUCKET&#39;, buckets[&#39;raw_properties&#39;].bucket_name)<br>        lambda_cn.add_environment(&#39;PROCESSED_PROPERTIES_BUCKET&#39;, buckets[&#39;processed_properties&#39;].bucket_name)<br>        lambda_cn.add_environment(&#39;PROPERTIES_TABLE_NAME&#39;, table.table_name)<br>            <br>    def create_bucket(self, id):<br>        bucket = s3.Bucket(self, id,<br>                removal_policy=RemovalPolicy.DESTROY,<br>                auto_delete_objects=True<br>        )<br>        <br>        return bucket</pre><p>This code sets up the AWS infrastructure, including the S3 buckets, DynamoDB table, and the Lambda function necessary for the application.</p><p><strong>Step 2: Implementing the Lambda Function</strong></p><ul><li>Create the necessary directories and files for your Lambda function:</li></ul><pre>mkdir -p src/CurrencyNormaliser<br>touch src/CurrencyNormaliser/event.json <br>touch src/CurrencyNormaliser/template.yaml <br>touch src/CurrencyNormaliser/lambda_function.py</pre><ul><li>Open the /aws-curreny-normalisation/src/CurrencyNormaliser/lambda_function.py file and insert the following code to handle CSV processing:</li></ul><pre>import json<br>import boto3<br>import csv<br>import io<br>import os<br>from decimal import Decimal<br><br>def lambda_handler(event, context):<br>    s3 = boto3.client(&#39;s3&#39;)<br>    <br>    # Get S3 bucket and DynamoDB table names from Environment variables<br>    processed_bucket = os.environ[&#39;PROCESSED_PROPERTIES_BUCKET&#39;]<br>    table_name = os.environ[&#39;PROPERTIES_TABLE_NAME&#39;]<br><br>    # Retrieving the S3 object based on event triggers<br>    raw_bucket = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;bucket&#39;][&#39;name&#39;]<br>    csv_filename = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;object&#39;][&#39;key&#39;]<br>    obj = s3.get_object(Bucket=raw_bucket, Key=csv_filename)<br>    csv_data = obj[&#39;Body&#39;].read().decode(&#39;utf-8&#39;).splitlines()<br>    <br>    processed_rows = []<br>    required_keys = [&#39;zpid&#39;, &#39;streetAddress&#39;, &#39;unit&#39;, &#39;bedrooms&#39;, <br>                     &#39;bathrooms&#39;, &#39;homeType&#39;, &#39;priceChange&#39;, &#39;zipcode&#39;,<br>                     &#39;city&#39;, &#39;state&#39;, &#39;country&#39;, &#39;livingArea&#39;, <br>                     &#39;taxAssessedValue&#39;, &#39;priceReduction&#39;, &#39;datePriceChanged&#39;,<br>                     &#39;homeStatus&#39;, &#39;price&#39;, &#39;currency&#39;]<br>    <br>    # Parsing CSV data<br>    reader = csv.DictReader(csv_data)<br>    for row in reader:<br>        filtered_row = {key: row[key] for key in required_keys if key in row}<br><br>        # Check if &#39;price&#39; and &#39;currency&#39; exist in the row before conversion<br>        if &#39;price&#39; in filtered_row and &#39;currency&#39; in filtered_row:<br>            try:<br>                price = int(filtered_row[&#39;price&#39;])<br>                converted_price, converted_currency = /<br>                  convert_currency(price, filtered_row[&#39;currency&#39;])<br>                filtered_row[&#39;price&#39;] = converted_price<br>                filtered_row[&#39;currency&#39;] = converted_currency<br>            except ValueError:<br>                print(f&quot;Invalid price value: {filtered_row[&#39;price&#39;]}&quot;)<br>    <br>        processed_rows.append(filtered_row)<br>    <br>    # Upload processed data to DynamoDB<br>    # Move processed files to the processed bucket<br>    if processed_rows:<br>        upload_to_dynamodb(table_name, processed_rows)<br>        move_file_to_processed_bucket(raw_bucket, <br>                                      processed_bucket, <br>                                      csv_filename)<br>    <br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: json.dumps(&#39;Currency normalization complete!&#39;)<br>    }<br>    <br># Currency conversion function<br>def convert_currency(price: int, currency: str) -&gt; tuple[int, str]:<br>    exchange_rate = 0.75<br><br>    if currency == &#39;CAD&#39;:<br>        usd_price = price * exchange_rate<br>        return int(usd_price), &#39;USD&#39;<br>    else:<br>        return price, currency<br><br># Function to upload data to DynamoDB<br>def upload_to_dynamodb(table_name, items):<br>    dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>    table = dynamodb.Table(table_name)<br><br>    with table.batch_writer() as batch:<br>        for item in items:<br>            batch.put_item(Item=item)<br><br># Function to move files within S3<br>def move_file_to_processed_bucket(raw_bucket_name, <br>                                  processed_bucket_name, <br>                                  file_key):<br>    s3 = boto3.client(&#39;s3&#39;)<br><br>    try:<br>        # Copy the object<br>        copy_source = {<br>            &#39;Bucket&#39;: raw_bucket_name,<br>            &#39;Key&#39;: file_key<br>         }<br>    <br>        s3.copy_object(CopySource=copy_source, <br>                        Bucket=processed_bucket_name, <br>                        Key=file_key)<br>    <br>        # Delete the original object<br>        s3.delete_object(Bucket=raw_bucket_name, Key=file_key)<br><br>    except Exception as e:<br>        # Log the error<br>        print(f&quot;Error occurred while moving {file_key} from /<br>                {raw_bucket_name} to {processed_bucket_name}: {str(e)}&quot;)</pre><p>The Lambda function handles the events triggered by new property sales files stored in the raw properties S3 bucket. It processes each file and converts currency values to USD where necessary. Finally, the function stores the updated file in the processed files S3 bucket and writes any converted rows to the DynamoDB table</p><p><strong>Step 3: Deploy the CDK App</strong></p><p>Now that we have configured the infrastructure and the Lambda function, it&#39;s time to deploy the application. We generate the CloudFormation template and then deploy the application to AWS using the following CDK commands.</p><pre># Synthesize the CloudFormation template<br>cdk synth<br><br># Deploy resources defined in the CDK app to AWS<br>cdk deploy</pre><p><strong>Step 4: Testing and Validation</strong></p><p>Finally, to ensure that everything works as expected, we can test the Lambda function by triggering it with a sample event:</p><ol><li>Create a Test Event:</li></ol><p>Open aws-currency-normalisation/src/CurrencyNormaliser/event.json. Insert the following JSON and replace the placeholders with the actual S3 bucket names:</p><pre>{<br>  &quot;Records&quot;: [<br>    {<br>      &quot;s3&quot;: {<br>        &quot;bucket&quot;: {<br>          &quot;name&quot;: &quot;YOUR-RAW-PROPERTIES-S3-BUCKET-NAME&quot;<br>        },<br>        &quot;object&quot;: {<br>          &quot;key&quot;: &quot;properties_data.csv&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>2. Upload the Test Data:</p><p>Create a sample CSV file (properties_data.csv) containing mock property sales data to the raw properties S3 bucket. I have included a sample of the property sales data below. You can use this data to test your application.</p><pre>index_id,result_from,livingArea,state,open_house_info,lotAreaValue,shouldHighlight,daysOnZillow,priceForHDP,homeStatusForHDP,unit,streetAddress,bedrooms,city,zpid,listing_sub_type,isUnmappable,openHouse,zipcode,homeStatus,longitude,bathrooms,isPremierBuilder,isPreforeclosureAuction,isZillowOwned,latitude,isShowcaseListing,lotAreaUnit,homeType,price,isNonOwnerOccupied,isFeatured,country,currency<br>1,listing_url,,ON,,,false,-1,499900,FOR_SALE,,445 Bridge Ave,7,Windsor,347741185,[object Object],false,,N9B2M3,FOR_SALE,-83.05935,2,false,false,false,42.30826,false,,SINGLE_FAMILY,499900,true,false,CAN,CAD</pre><p>3. Trigger and Verify:</p><p>Upload the `properties_data.csv` to the raw properties S3 bucket created during the CDK deployment, triggering an S3 event. This will trigger the Lambda function to process the file. Once the lambda function has finished executing, verify that the data is correctly processed, the currency is normalised, and records are stored in DynamoDB.</p><h4>Conclusion and Summary</h4><p>In this project, we successfully designed and implemented a cloud-based application that automates the normalisation of property sales data across different currencies using AWS serverless technologies. We efficiently deployed and managed the necessary resources by leveraging the AWS Cloud Development Kit (CDK), including S3 buckets, a DynamoDB table, and a Lambda function.</p><p><strong>Key Accomplishments:</strong></p><ul><li><strong>Infrastructure Setup:</strong> We used the AWS CDK to automate the creation of a scalable and efficient infrastructure. We provisioned all necessary AWS resources and ensured that they were configured correctly.</li><li><strong>Data Processing:</strong> We implemented a serverless function through Python and AWS Lambda that processes CSV files, converts property prices from CAD to USD, and stores the normalised data in DynamoDB.</li><li><strong>Data Management:</strong> We utilised Amazon S3 to manage raw and processed data, ensuring that your application handles data efficiently and securely.</li></ul><p>This project is a foundational example of how serverless architectures can solve real-world problems, such as currency normalisation, in a scalable and cost-effective manner.</p><h4>Next Steps:</h4><p>Now that we&#39;ve built this application consider expanding its capabilities:</p><ul><li><strong>Support for Additional Currencies:</strong> Enhance the Lambda function to handle multiple currencies beyond CAD and USD, possibly integrating with an external API for real-time exchange rates.</li><li><strong>Advanced-Data Analytics:</strong> Integrate AWS services like Amazon Athena or AWS Glue to perform more advanced data analytics on the normalised data.</li><li><strong>Continuous Integration/Deployment:</strong> Implement CI/CD pipelines with AWS CodePipeline and CodeBuild to automate your application&#39;s deployment and testing.</li></ul><p>By exploring these additional features, you can further enhance the functionality and scalability of your serverless application, making it a more robust solution for real-world financial data processing challenges.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb935f93de32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Power of AWS Monitoring]]></title>
            <link>https://medium.com/@darrenbassett67/unleashing-the-power-of-aws-monitoring-82fb22d69472?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/82fb22d69472</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Mon, 19 Aug 2024 08:01:42 GMT</pubDate>
            <atom:updated>2024-10-03T07:23:24.322Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Scalable Website Monitoring Solution with Serverless Technologies</h3><h3>Introduction</h3><p>In today&#39;s digital landscape, achieving optimal website performance and availability is paramount for user satisfaction and business success. A seamless user experience can significantly impact customer retention, conversion rates, and overall brand reputation. However, monitoring website performance in real-time to ensure uptime and swift resolution of issues can be complex and resource-intensive.</p><p>Organisations need a reliable, scalable, cost-effective solution to monitor their websites continuously and promptly address disruptions. Traditional monitoring solutions often involve substantial investments in infrastructure and management overhead. AWS serverless technologies offer an alternative, enabling the creation of a robust and efficient website monitoring solution without the burdens of conventional approaches.</p><p>This project aims to develop a reliable and expandable website monitoring solution using AWS services, particularly serverless technologies, to ensure the constant availability of monitored websites. The solution aims to detect website failures and promptly alert designated stakeholders via email.</p><h3>Solution Design</h3><p>The design of my website monitoring solution leverages several AWS services to ensure high availability, scalability, and cost-effectiveness. The requirements for this solution are:</p><ul><li>Monitor websites by implementing a mechanism to check their availability periodically.</li><li>Detect website failures by deploying logic to check predefined criteria, such as HTTP response codes and response time thresholds.</li><li>To alert stakeholders<strong> </strong>when a website failure is detected by integrating notifications such as email.</li><li>To be scalable and resilient to handle varying workloads and potential AWS service failures.</li><li>Optimise cost-effectiveness by leveraging serverless technologies and using AWS pricing models optimally.</li></ul><h4>Design Principles</h4><ul><li><strong>Security </strong>is a fundamental principle in any AWS architecture, and this solution adheres to AWS&#39;s best practices by implementing the principle of least privilege. Only the necessary IAM permissions are granted to the Lambda function to perform its tasks, such as logging, creating metrics, and sending notifications. This minimises the potential attack surface and ensures that each component operates with the minimal required access. Additionally, all data in transit, including logs and notifications, is securely encrypted using AWS-managed keys, further enhancing the security posture of the solution.</li><li><strong>Resiliency:</strong> AWS&#39;s highly available and fault-tolerant services, EventBridge and SNS, are designed to operate across multiple Availability Zones (AZs), ensuring the monitoring solution remains operational even if an AZ experiences an outage. The built-in retry mechanisms in EventBridge and Lambda, combined with CloudWatch alarms&#39; ability to detect and alert on failures, ensure that the solution can handle temporary disruptions and continue monitoring website availability without interruption.</li><li><strong>Performance: The solution is optimised for performance by</strong> leveraging AWS Lambda&#39;s ability to scale automatically in response to incoming events. Lambda&#39;s ephemeral nature ensures that the monitoring checks execute efficiently and concurrently, reducing latency and guaranteeing timely detection of website failures. CloudWatch&#39;s real-time logging and metric capabilities provide near real-time insights into the system&#39;s health, enabling rapid identification and resolution of performance bottlenecks.</li><li><strong>Cost Optimisation:</strong> Cost optimisation is a key design principle in this solution, which is realised by utilising AWS&#39;s serverless offerings. Lambda, CloudWatch, and SNS&#39;s pay-as-you-go pricing model ensures that costs are incurred only for actual usage, eliminating the need for upfront infrastructure investment. By estimating usage with the AWS Pricing Calculator and leveraging the AWS Free Tier, the solution is designed to be highly cost-effective, providing robust monitoring capabilities while keeping operational expenses minimal.</li></ul><p>The following diagram provides a visual representation of the solution design:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*lKD8mJf6osV4kx7Czi9I0Q.jpeg" /></figure><p>EventBridge is an AWS technology capable of scaling to millions of events per second. It uses a distributed architecture spread across multiple Availability Zones (AZ) that ensures availability in the case of an AZ failure. It guarantees at least once delivery and has built-in retry and error-handling mechanisms.</p><p>In this solution, an EventBridge scheduled rule periodically triggers the monitoring process. The rule invokes an AWS Lambda function every minute, passing a list of websites to the function in an event object.</p><p>AWS Lambda is a compute service that runs code without provisioning or managing servers. It automatically scales applications by running code responding to triggers, such as events generated by EventBridge rules. You only pay for the compute time you use; there is no charge when your code is not running.</p><p>The CheckSite Lambda function in this solution performs the following tasks:</p><ul><li>Processes the list of websites received from EventBridge by sending an HTTP GET request to each website and logging the HTTP response to a CloudWatch log group.</li><li>It creates custom CloudWatch metrics for each website, pushing any failures to the corresponding metric for the website.</li><li>This function creates a CloudWatch alarm for each website and configures the alarm to alert if three failures occur within a 5-minute evaluation period.</li><li>The function also creates a CloudWatch dashboard for near real-time monitoring of the alarms.</li><li>Finally, it creates a Simple Notification Service (SNS) topic, which publishes notifications to stakeholders about changes in the alarm state.</li></ul><p>Amazon CloudWatch is the AWS monitoring and logging service. CludWatch scales automatically and can handle high-throughput data streams. Built upon the AWS Global Infrastructure, Cloudwatch is highly available and reliable. Metrics and logs are replicated across multiple Availability Zones (AZs) to ensure durability and availability. It has a pay-as-you-go pricing model, i.e., you only pay for what you use, and there are no long-term commitments.</p><p>This solution uses CloudWatch to provide centralised logging, real-time monitoring via custom metrics and dashboards, and alerts via alarms. It is also integrated with the Simple Notification Service to publish notifications to stakeholders of website failures and recovery.</p><p>Amazon Simple Notification Service (SNS) is a fully managed messaging service. It decouples and scales serverless applications by providing a highly available, durable, and flexible publish/subscribe messaging model. This model can deliver messages to multiple subscriber endpoints such as Lambda functions, HTTP/S endpoints, email addresses, SQS queues, and mobile devices. It is designed to handle large volumes of messages and automatically scales to changes in the workload. Like CloudWatch, it utilises the AWS Global Infrastructure to provide a highly available service by replicating messages across multiple AZs and using an automatic failover to route messages to healthy AZs.</p><p>In this solution, SNS is used to notify stakeholders of changes in the state of the CloudWatch alarms. It receives notifications from the CloudWatch alarms and sends emails to the stakeholders, providing them with details about the alarm and the status of the monitored websites.</p><p>This architecture provides a comprehensive solution for monitoring website uptime and performance. It leverages the power of AWS serverless technologies to deliver a robust, scalable, and efficient monitoring system.</p><h3>Implementing Website Monitoring with EventBridge, Lambda, CloudWatch and SNS</h3><p>This section describes how I configured the solution in the AWS cloud. Early in the implementation phase, I decided to use Infrastructure as Code(IaC) using the AWS SDK. Automating the infrastructure build makes it consistent and reliable while reducing costs and manual effort. It also makes the infrastructure scaleable by dynamically updating website checks, metrics, alarms, and dashboards as the list of websites being checked changes.</p><h4>EventBridge Configuration</h4><p>An Amazon EventBridge scheduled rule initiates checks on the websites once per minute. It sends an event to the Lambda &#39;CheckSiteFunction&#39; containing a list of websites as a JSON payload in the event object. Shown below is an example of the payload:</p><pre>{<br>  &quot;site_check&quot;: {<br>    &quot;Amazon&quot;: &quot;https://www.amazon.com&quot;,<br>    &quot;eBay&quot;: &quot;https://www.ebay.com&quot;,<br>    &quot;BBC&quot;: &quot;http://www.bbc.co.uk&quot;,<br>    &quot;Demo&quot;: &quot;http://&lt;ELASTIC_IP_OF_EC2_INSTANCE&gt;&quot;<br>  }<br>}</pre><p>For this project, I accepted the default retry policy, which includes five retry attempts and a maximum event age of one hour. Additionally, there is an option to specify an Amazon SQS Dead Letter Queue (DLQ) to store any undelivered events for debugging later. However, I decided not to configure a DLQ for this particular implementation.</p><h4>Setting Up the Lambda Function</h4><p>The Lambda function monitors the availability of various wvisualiseWhen invoked by an EventBridge schedule rule, it iterates through a list of websites, making HTTP GET requests to check if the websites are online.</p><p>The Lambda function needs several IAM permissions to perform its tasks. These include creating and interacting with CloudWatch log groups, log streams, metrics, alarms, dashboards, and SNS topics. Following the principle of least privilege, I created the Lambda task execution role shown below:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;logs:CreateLogGroup&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:logs:us-east-1:&lt;ACCOUNT_ID&gt;:*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:logs:us-east-1:&lt;ACCOUNT_ID&gt;: /<br>                  log-group:/aws/lambda/CheckSiteFunction:*&quot;,<br>                &quot;arn:aws:logs:us-east-1:&lt;ACCOUNT_ID&gt;: /<br>                  log-group:/aws/lambda/SiteMonitor:*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;cloudwatch:PutMetricData&quot;,<br>                &quot;cloudwatch:PutMetricAlarm&quot;,<br>                &quot;cloudwatch:PutDashboard&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;sns:CreateTopic&quot;,<br>                &quot;sns:Publish&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><h4>Handling HTTP Requests with Timeout</h4><p>The Lambda handler iterates through the list of websites received in the event object and attempts to connect each site. If a website check is successful, i.e., it returns an HTTP 200 status code, Lambda logs the response and updates the metric for the website. If the check fails, i.e. an HTTP status code other than 200 is received or there is a socket timeout, then the error is caught. Lambda writes the error code and message to the CloudWatch log, and the custom metric for the site is updated to show a failure. The code snippet below shows the implementation of the Lambda Handler:</p><pre>def lambda_handler(event, context):<br>    # Create custom logger and SNS Topic<br>    # Retrieve the list of sites to be checked from the event<br>    logger = setup_custom_logging(context)<br>    sns_topic_arn = create_sns_topic(&#39;WebsiteFailureNotifications&#39;)<br>    sites = event.get(&#39;site_check&#39;, {})<br>    <br>    # For each site check the status and record the results<br>    for site, url in sites.items():<br>        print(f&quot;\nChecking {site} at URL {url}\n&quot;)<br>        try:<br>            request = urllib.request.Request(url, <br>                                        headers={&#39;User-Agent&#39;: &#39;AWS Lambda&#39;})<br>            socket.setdefaulttimeout(TIMEOUT_DURATION)<br>            with urllib.request.urlopen(request) as response:<br>                logger.info(f&quot;{site} - {url} - /<br>                              statusCode={response.getcode()} - /<br>                              {response.reason}&quot;)<br>                record_result(site, 0)<br>        except urllib.error.HTTPError as e:<br>            logger.error(f&quot;{site} - {url} - statusCode={e.code} - {e.reason}&quot;)<br>            record_result(site, 1)<br>        except urllib.error.URLError as e:<br>            logger.error(f&quot;{site} - {url} - Error: {e.reason}&quot;)<br>            record_result(site, 1)<br>        except socket.timeout:<br>            logger.error(f&quot;{site} - {url} - Error: Timeout&quot;)<br>            record_result(site, 1)<br>        except Exception as e:<br>            logger.error(f&quot;{site} - {url} - Unexpected Error: {str(e)}&quot;)<br>        finally:<br>            print(f&quot;End of loop run&quot;)<br>        <br>        # Create/Update alarm for each website<br>        create_cloudwatch_alarm(site, sns_topic_arn)<br>    <br>    # Update the dashboard with current websites<br>    create_update_dashboard(sites)<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: &#39;Site checks completed&#39;<br>    }</pre><h4>Using a Custom CloudWatch Log Handler</h4><p>For clarity, I separated messages generated during website checks from the default Lambda execution logs. I set up a custom CloudWatch log group to store messages related to the website checks. Below, you can see the Python code that creates a custom CloudWatch log group, log stream and a log handler, which the Lambda Handler uses to log the results.</p><pre>def create_log_group(log_group_name):<br># Create a custom CloudWatch log group for the site check results<br>    try:<br>        logs_client.create_log_group(logGroupName=log_group_name)<br>        print(f&quot;Log group {log_group_name} created successfully.&quot;)<br>    except logs_client.exceptions.ResourceAlreadyExistsException:<br>        print(f&quot;Log group {log_group_name} already exists.&quot;)<br>    except Exception as e:<br>        print(f&quot;Error creating log group: {str(e)}&quot;)<br><br>def put_log_event(log_group_name, log_stream_name, message):<br># Send a log event to CloudWatch<br>    timestamp = int(datetime.now().timestamp() * 1000)<br>    logs_client.put_log_events(<br>        logGroupName=log_group_name,<br>        logStreamName=log_stream_name,<br>        logEvents=[<br>            {<br>                &#39;timestamp&#39;: timestamp,<br>                &#39;message&#39;: message<br>            },<br>        ],<br>    )<br><br>def setup_custom_logging(context):<br># Set up logging to a custom CloudWatch log group and stream.<br><br>    # Create the custom log group if it does not exist<br>    create_log_group(CUSTOM_LOG_GROUP_NAME)<br><br>    # Define the log stream name based on the Lambda request ID<br>    log_stream_name = context.aws_request_id<br><br>    # Create the log stream<br>    try:<br>        logs_client.create_log_stream(<br>            logGroupName=CUSTOM_LOG_GROUP_NAME,<br>            logStreamName=log_stream_name<br>        )<br>        print(f&quot;Log stream {log_stream_name} created successfully.&quot;)<br>    except logs_client.exceptions.ResourceAlreadyExistsException:<br>        print(f&quot;Log stream {log_stream_name} already exists.&quot;)<br>    except Exception as e:<br>        print(f&quot;Error creating log stream: {str(e)}&quot;)<br><br>    # Set up the logger for custom logs<br>    logger = logging.getLogger(&#39;customLogger&#39;)<br>    logger.setLevel(logging.INFO)  # Set the logging level<br><br>    # Attach the custom CloudWatch handler to the logger<br>    class CloudWatchHandler(logging.Handler):<br>        def emit(self, record):<br>            log_entry = self.format(record)<br>            put_log_event(CUSTOM_LOG_GROUP_NAME, log_stream_name, log_entry)<br><br>    cloudwatch_handler = CloudWatchHandler()<br>    cloudwatch_handler.setFormatter(logging.Formatter(&#39;%(asctime)s - /<br>                                      %(name)s - %(levelname)s - /<br>                                      %(message)s&#39;))<br>    logger.addHandler(cloudwatch_handler)<br><br>    return logger</pre><h4>Recording Results and Creating Alarms</h4><p>When the check is completed, the custom metric for a website is updated. For each metric, the code creates a CloudWatch alarm. The alarm will alert if 3 data points fail within a 5-minute window, preventing false alerts from being triggered by a single failure, which can occur as the system performs checks across the internet. The alarm publishes a notification on the SNS topic whenever its state changes.</p><pre>def record_result(site, value):<br>    # Update the CloudWatch metric with the results of the website check<br>    cloudwatch.put_metric_data(<br>        Namespace=&#39;WebsiteStatusCodes&#39;,<br>        MetricData=[<br>            {<br>                &#39;MetricName&#39;: f&#39;{site}_status&#39;,<br>                &#39;Dimensions&#39;: [{&#39;Name&#39;: &#39;Website&#39;, &#39;Value&#39;: site}],<br>                &#39;Value&#39;: value,<br>                &#39;Unit&#39;: &#39;Count&#39;<br>            },<br>        ]<br>    )</pre><pre># Create CloudWatch alarm if it doesn&#39;t exist<br>def create_cloudwatch_alarm(site, topic_arn):<br>    # Remove protocol from URL<br>    alarm_name = f&quot;{site}_status_alarm&quot;<br>    metric_name = f&#39;{site}_status&#39;    <br><br>    cloudwatch.put_metric_alarm(<br>        AlarmName=alarm_name,<br>        AlarmDescription=f&quot;Alarm for {site} failing 3 times in 5 minutes&quot;,<br>        ActionsEnabled=True,<br>        # Actions to take when the alarm state is ALARM<br>        AlarmActions=[topic_arn],<br>        # Actions to take when the alarm state is OK<br>        OKActions=[topic_arn],<br>        MetricName=metric_name,<br>        Namespace=&#39;WebsiteStatusCodes&#39;,<br>        Statistic=&#39;Sum&#39;,<br>        Dimensions=[<br>            {<br>                &#39;Name&#39;: &#39;Website&#39;,<br>                &#39;Value&#39;: site<br>            },<br>        ],<br>        Period=60,  # 1 minute<br>        EvaluationPeriods=5,<br>        DatapointsToAlarm=3,<br>        Threshold=1,<br>        ComparisonOperator=&#39;GreaterThanOrEqualToThreshold&#39;,<br>        TreatMissingData=&#39;breaching&#39;<br>    )</pre><h4>Visualising Alarms in CloudWatch Dashboard</h4><p>Finally, I implemented a function to create and update a CloudWatch dashboard to visualise the status of all websites, as shown below:</p><pre># Create or update the CloudWatch dashboard<br>def create_update_dashboard(sites):<br>    widgets = []<br>    # Get the AWS region<br>    region = os.environ[&#39;AWS_REGION&#39;]  <br>    # Get the AWS account ID    <br>    account_id = boto3.client(&#39;sts&#39;).get_caller_identity().get(&#39;Account&#39;)<br>    <br>    # Iterate through the sites, construct the ARN for the site alarm<br>    # and append it to the list of alarm ARNs to be added to the dashboard<br>    alarm_arns = []<br>    for site in sites.keys():<br>        alarm_name = f&quot;{site}_status_alarm&quot;<br>        alarm_arn = f&quot;arn:aws:cloudwatch:{region}:{account_id}: /<br>                    alarm:{alarm_name}&quot;<br>        alarm_arns.append(alarm_arn)<br>        <br>    # Add a single widget with all alarms<br>    widgets.append({<br>        &#39;type&#39;: &#39;alarm&#39;,<br>        &#39;x&#39;: 0,<br>        &#39;y&#39;: 0,<br>        &#39;width&#39;: 24,<br>        &#39;height&#39;: 6,<br>        &#39;properties&#39;: {<br>            &#39;alarms&#39;: alarm_arns,  # Use the ARNs of all alarms<br>            &#39;title&#39;: &#39;Alarm Status for All Websites&#39;,<br>            &#39;sortBy&#39;: &#39;stateUpdatedTimestamp&#39;<br>        }<br>    })<br>    <br>    dashboard_body = {<br>        &#39;widgets&#39;: widgets<br>    }<br>    <br>    cloudwatch.put_dashboard(<br>        DashboardName=&#39;WebsiteMonitoringDashboard&#39;,<br>        DashboardBody=json.dumps(dashboard_body)<br>    )</pre><p>The screenshot below shows the dashboard generated through coding with the AWS SDK. Though somewhat basic, the dashboard is highly functional and provides a solid foundation for monitoring and visualising each website&#39;s status in near real-time. I could enhance it by adding more custom metrics to show how each site performs against Key Performance Indicators. CloudWatch can also be integrated with more advanced visualisation tools like Grafana to provide richer visualisation and customisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ia2jtpTk6R0zqUslgkphg.jpeg" /></figure><p>The status of each alarm can also be monitored through the AWS management console, as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uOem_5cJHXiHC0m8f4OtfA.jpeg" /></figure><p>The steps above document how I implemented a monitoring solution using AWS EventBridge, Lambda, CloudWatch and Amazon SNS to monitor website availability, handle errors and timeouts effectively, log the results, and visualise the status on a CloudWatch dashboard. This setup ensures that stakeholders are promptly alerted to any issues, helping maintain the reliability of the web services.</p><h3>Cost Analysis</h3><p>One of the key achievements of this project is its cost-effectiveness, which I realised by leveraging the AWS Pricing Calculator and the AWS Free Tier. The AWS Pricing Calculator helped me estimate the cost of various AWS services upfront, enabling me to design a monitoring solution that maximises efficiency while minimising expenses. By accurately predicting my usage and costs, I was able to make informed decisions about resource allocation and optimisation.</p><p>The AWS Free Tier also provided significant cost savings by offering free usage of specific services for the first 12 months, allowing me to experiment and build my monitoring solution without incurring immediate costs. Services such as AWS Lambda, Amazon CloudWatch, and Amazon SNS fall under usage limits, ensuring that my initial development and testing phases were virtually cost-free. Combining the AWS Pricing Calculator and the free tier offer enabled me to create a highly cost-effective monitoring solution.</p><h3>Conclusion</h3><p>This article explored harnessing AWS serverless technologies to build a scalable, reliable, cost-effective website monitoring solution. By leveraging Amazon EventBridge, AWS Lambda, Amazon CloudWatch, and Amazon SNS, I developed a system that efficiently monitors website availability, detects failures, and promptly alerts stakeholders.</p><p>I stressed the need for a robust monitoring solution in today&#39;s digital landscape and the critical role of maintaining optimal website performance in enhancing user satisfaction and achieving business success. Additionally, I provided a detailed overview of the design and architecture, outlining how each AWS service contributes to the solution&#39;s scalability, availability, and cost-effectiveness.</p><p>The implementation phase demonstrated the practical steps required to configure EventBridge for periodic checks, Lambda for executing the monitoring logic, CloudWatch for logging, metrics, and dashboards, and SNS for notifications. Integrating each component to ensure seamless operation and rapid response to detected issues.</p><p>Using Infrastructure as Code (IaC), I automated the setup, making it consistent and easily replicable across different environments. This reduced manual effort and enhanced the solution&#39;s reliability and scalability.</p><p>Overall, the solution provides a comprehensive and efficient way to monitor website performance and quickly address disruptions. This proactive approach helps maintain the reliability and availability of web services, ultimately contributing to a better user experience and improved business outcomes.</p><p>As we look to the future, there are numerous opportunities for enhancing this solution. Adding more custom metrics, integrating advanced visualisation tools like Grafana, and expanding monitoring capabilities can improve the system. By continuously leveraging AWS&#39;s evolving capabilities, we can ensure that our monitoring solution remains robust and effective in meeting the demands of modern web applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82fb22d69472" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of Web Applications]]></title>
            <link>https://medium.com/@darrenbassett67/the-evolution-of-web-applications-a7578f3d26a3?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7578f3d26a3</guid>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[aws-security]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 08:01:35 GMT</pubDate>
            <atom:updated>2024-09-10T11:33:07.803Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Evolution of a Web Application</h3><h3>A Journey from Monolithic to Serverless on AWS — Part II</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*45FxmfDyAx75e6wt" /><figcaption>Photo by <a href="https://unsplash.com/@lucabravo?utm_source=medium&amp;utm_medium=referral">Luca Bravo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Journey to a Fully Serverless Architecture on AWS</h3><p>In the second part of this article, we continue our discussion on the transition of a web application from a monolithic architecture to a fully serverless architecture utilising AWS services. The article describes the limitations of the existing monolithic architecture and the benefits of transitioning to a fully serverless architecture. Additionally, we will detail the migration process to a serverless infrastructure using AWS serverless technologies like S3, Lambda, and DynamoDB. Finally, we will discuss the necessary steps to host the website with S3, Route 53, ACM, and CloudFront for a fast, secure, and reliable web hosting experience.</p><h3>A Quick Look Back: Moving Away from Monolithic Architecture</h3><p>In the <a href="https://medium.com/@darrenbassett67/the-evolution-of-a-web-application-4131e772edbf">first part</a> of our series, we explored a start-up’s journey from operating a monolithic application on a single server to transitioning their application to a more resilient and scalable server-based architecture on AWS. The company faced significant performance bottlenecks, high operational costs, and technical debt, prompting the need for this migration. The initial phase involved organising the application’s code into microservices and using AWS technologies such as EC2, VPC, CloudFront, Lambda, S3, and DynamoDB to create a highly available, secure, and cost-optimised infrastructure. Key steps included deploying static website files on EC2 with global distribution via CloudFront, implementing various Lambda functions for dynamic content handling, and establishing robust security measures with VPC, security groups, and encrypted communications. By leveraging AWS’s auto-scaling, load balancing, and serverless services, the start-up aimed to enhance performance, scalability, and cost efficiency, setting the stage for the next phase of migrating to a fully serverless architecture.</p><h3>Kicking Off: Transitioning to a Server-Based Architecture in AWS Cloud</h3><p>The initial migration phase involved creating a secure, resilient, and scalable solution within an AWS Virtual Private Cloud (VPC). This included setting up public and private subnets, strict traffic rules using Network Access Control Lists (ACLs) and Security Groups, and directing traffic through Route 53 and CloudFront for enhanced security and performance. The architecture employed an Application Load Balancer (ALB) for traffic distribution and an Auto Scaling group to handle fluctuating demands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AzwkNDN5cDZJ4oc2-BquhA.jpeg" /><figcaption>AWS Portfolio Server-Based Website</figcaption></figure><p>The solution utilised AWS services such as Lambda for on-demand scaling, S3 for durable storage with cost-effective options, SNS for scalable messaging, DynamoDB for fast and predictable performance, and EventBridge for cost-effective event-driven architecture.</p><p>The implementation was primarily automated using AWS CloudFormation, covering multiple steps:</p><ol><li><strong>Deploying the AMI</strong>: Launching an EC2 instance with a custom Amazon Machine Image (AMI) containing a webserver and the website code as the base.</li><li><strong>Deploying Microservices</strong>: Using CloudFormation stacks to create microservices for blog posts, view counters, contact forms, and AWS news updates.</li><li><strong>Creating Microservices Endpoints and AMI</strong>: Manually configure the endpoints and update the application code on the EC2 instance; create a new custom AMI and launch template.</li><li><strong>Deploying Network Infrastructure Stacks</strong>: Automating the construction of VPC, subnets, route tables, security groups, load balancing, and CloudFront distribution.</li></ol><p>Overall, this architecture demonstrated how AWS services could create a resilient, scalable, and cost-efficient web application, setting the stage for the next phase of transitioning to a fully serverless architecture.</p><h3>Unlocking the Power of AWS: Migrating to a Fully Serverless Infrastructure</h3><p>In today’s digital era, it’s critical to guarantee that your website is fast, secure, and dependable. Amazon Web Services (AWS) provides a toolkit of robust resources to help accomplish this: Amazon S3, Amazon Route 53, AWS Certificate Manager (ACM), and Amazon CloudFront. The diagram below illustrates how these services can create a fully serverless web hosting infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nQu0Pw-w7yiSxrL4dFLxQ.jpeg" /><figcaption>AWS Portfolio Serverless Website</figcaption></figure><p>Let’s explore how these elements enhance the web hosting experience and the remarkable advantages they deliver.</p><p><strong>Amazon S3 (Simple Storage Service)</strong></p><p>Amazon S3 is the perfect solution for storing static website files, including HTML, CSS, JavaScript, and images. It provides highly scalable, durable, and accessible storage, which makes setting up and managing static website hosting a breeze. With S3, you can ensure that your website content is always available and can effortlessly handle varying traffic loads, greatly benefiting your website’s performance and reliability.</p><p><strong>Amazon Route 53</strong></p><p>Amazon Route 53 is a robust DNS routing service specifically designed to manage the DNS records of your custom domain efficiently. It provides reliable and cost-effective DNS routing, enabling you to utilise a custom domain for your website. Route 53 streamlines the management of your domain’s DNS settings, ensuring that your visitors always reach the correct destination.</p><p><strong>AWS Certificate Manager (ACM)</strong></p><p>AWS Certificate Manager streamlines SSL/TLS certificate management, guaranteeing secure website connections. ACM offers complimentary SSL/TLS certificates, automates deployment, and ensures all user-website traffic is encrypted in transit, reinforcing security and fostering user trust.</p><p><strong>Amazon CloudFront</strong></p><p>Amazon CloudFront is a Content Delivery Network (CDN) that operates globally to bring your website content closer to your users. It achieves this by caching content at edge locations worldwide, leading to improved website performance, decreased latency, and faster load times. CloudFront also provides advanced security features, including DDoS protection, which helps enhance your website’s overall security.</p><h4>Navigating the Serverless Transition</h4><p>Transitioning from a server-based infrastructure to a fully serverless architecture on AWS involves several key steps. Here’s a detailed guide to help you through the process:</p><p><strong>Create an S3 Bucket for Website Hosting:</strong> To host your website files, create an S3 bucket. Upload your HTML, CSS, and server-side JavaScript files to this bucket. We’ll use Object Access Control (OAC) to secure the bucket and prevent direct access to its content. Note that OAC isn’t supported when static website hosting is enabled, so don’t enable this option.</p><p><strong>Set Up a CloudFront Distribution:</strong> Create a new CloudFront distribution with the S3 bucket as the origin. Enable Object Access Control (OAC) to ensure that only CloudFront can access the content in your S3 bucket, thus restricting direct access to your files. Also, ensure you set the viewer protocol policy to redirect HTTP to HTTPS.</p><p><strong>Configure the Bucket Policy for OAC: </strong>After enabling Origin Access Control on the CloudFront distribution, the next step is to enforce it by creating a bucket policy for OAC. This policy will permit the CloudFront distribution to retrieve objects from the S3 bucket while blocking any other direct access. Below is an example of the policy you’ll need to implement:</p><pre>{<br>    &quot;Version&quot;: &quot;2008-10-17&quot;,<br>    &quot;Id&quot;: &quot;PolicyForCloudFrontPrivateContent&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowCloudFrontServicePrincipal&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;cloudfront.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::BUCKET_NAME/*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;AWS:SourceArn&quot;: &quot;arn:aws:cloudfront::ACCOUNT_ID:distribution/ORIGIN_ACCESS_ID&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><p><strong>Consider the use of Cross-Origin Resource Sharing (CORS): </strong>When using Route 53, CloudFront, ACM, and S3 together, it’s essential to consider whether CORS (Cross-Origin Resource Sharing) is necessary. CORS is generally unnecessary if you access the web application and CloudFront distribution via the same domain (e.g., <a href="https://yourdomain.com/">https://yourdomain.com</a>). Route 53 facilitates domain mapping, and the ACM certificate ensures secure HTTPS access.</p><p><strong>Configure Route53 for the New Distribution: </strong>Setting up an Alias record in Amazon Route 53 is crucial to ensure your custom domain correctly routes traffic to your new CloudFront distribution. To set this up, create a new alias record within your hosted zone that points to the new CloudFront distribution. For the routing policy, ‘Simple Routing’ is sufficient. After saving the configuration, propagating the DNS changes may take a few minutes. Once they do, verify that your domain correctly routes traffic through CloudFront by visiting your website. Check that CloudFront is serving the content by inspecting the response headers in your browser’s developer tools — look for headers like x-amz-cf-id, which indicates CloudFront is serving the content.</p><p><strong>Cleaning up:</strong> After setting up and testing the new architecture, we can delete the EC2 and VPC configurations created in Part I of this article. I initially developed these configurations using AWS CloudFormation; therefore, I deleted the relevant CloudFormation Stack to remove them.</p><h4><strong>Benefits of the Fully Serverless Architecture</strong></h4><p>Our transition to a fully serverless architecture has brought numerous benefits, significantly enhancing scalability, performance, security, cost efficiency, and management ease.</p><p><strong>Scalability and Availability</strong> With Amazon S3, our website can automatically scale to handle traffic spikes, ensuring consistent high availability. Amazon CloudFront complements this by caching content globally, reducing the load on our S3 bucket and providing fast access for users worldwide.</p><p><strong>Improved Performance</strong> Amazon CloudFront reduces latency by serving content from the nearest edge location. Features like Gzip compression and HTTP/2 support ensure faster content delivery and a smoother user experience.</p><p><strong>Enhanced Security</strong> AWS Certificate Manager (ACM) encrypts data in transit using SSL/TLS certificates and ensures secure connections. Additionally, CloudFront provides robust DDoS protection through AWS Shield and enforces HTTPS, safeguarding data transmission.</p><p><strong>Cost Optimisation</strong> Our cost efficiency has improved with Amazon S3’s pay-as-you-go pricing model, which reduces storage and data transfer expenses. CloudFront further optimises costs by caching content, thus lowering data transfer fees from S3.</p><p><strong>Ease of Management</strong> Managing our infrastructure has become more straightforward, with the AWS Certificate Manager automating SSL/TLS certificate renewals. Amazon S3 offers an intuitive management console and API for streamlined website hosting. Meanwhile, Amazon Route 53 simplifies domain management with latency-based and geolocation routing features, ensuring our site remains reachable and performs well.</p><p><strong>Search Engine Optimisation (SEO) and User Experience</strong> Finally, adopting HTTPS through ACM enhances our SEO, as secure websites rank higher in search engines. Additionally, CloudFront’s ability to improve page load times positively impacts our SEO efforts and user satisfaction, creating a better web experience.</p><h4>Outcomes</h4><p>Previously, the website suffered from performance bottlenecks, high operational costs, and substantial technical debt. Updates required significant downtime, and the monolithic structure made modifications risky and time-consuming. The transition to a serverless architecture has resolved these issues, providing a more robust and agile platform.</p><p><strong>Reliable and Fast Website Delivery</strong><br>Migrating to AWS infrastructure has transformed our website’s reliability and speed. With AWS’s global presence, our site is now consistently available with minimal downtime and fast load times, enhancing user experience worldwide.</p><p><strong>Enhanced Security</strong><br>All website traffic is encrypted, safeguarding user data and fostering user trust. Utilising Origin Access Control (OAC) combined with strict bucket policies ensures robust protection for our website files.</p><p><strong>Cost Efficiency</strong><br>Optimising the use of AWS services has significantly reduced our hosting costs while maintaining top-tier performance and reliability. This cost-effective approach allows us to allocate resources more efficiently.</p><p><strong>Operational Simplicity</strong><br>Leveraging managed services such as AWS Certificate Manager (ACM) for certificate management and Amazon S3 for static hosting has simplified our operations. This reduction in operational overhead frees us to focus more on content creation and user engagement.</p><p><strong>Scalability</strong><br>Our website now effortlessly handles traffic spikes without requiring additional configuration or infrastructure adjustments. AWS’s scalable services ensure we can accommodate growth seamlessly.</p><p>By integrating Amazon S3, Route 53, ACM, and CloudFront, we have achieved a high-performing, secure, scalable, and cost-effective website. These tools collectively provide an outstanding web experience for our users.</p><h3>Summary: Transforming from a Monolithic to an AWS Cloud Serverless Architecture</h3><p>The transition from a monolithic architecture to a serverless cloud-based infrastructure represents a significant transformation for the start-up. Initially, the website operated on a single server at the company’s headquarters, causing performance issues, increased operational expenses, and significant technical debt. The monolithic design posed challenges for updates, making them both risky and time-intensive and often necessitating substantial downtime.</p><p><strong>Phase 1: Transition to Server-Based Architecture</strong> To overcome these challenges, we first shifted to a server-based architecture using AWS technologies. We reorganised the website into microservices and hosted the website files on EC2 instances deployed within a Virtual Private Cloud (VPC). We also incorporated vital AWS services such as CloudFront, Route 53, Lambda, S3, and DynamoDB to enhance security, scalability, and cost efficiency. This transition significantly improved performance, reduced operational expenses and laid a solid foundation for future developments.</p><p><strong>Phase 2: Migration to Serverless Architecture</strong> During the second phase, the company focused on maximising the use of serverless technologies. The start-up restructured its infrastructure to adopt a serverless architecture, using AWS services like S3, Lambda, and DynamoDB. This change eliminated the need to manage servers and associated VPC infrastructure, leading to cost savings and improved scalability. Amazon CloudFront enhances the user experience through effective content delivery, and Route 53 handles domain routing to ensure secure connections with ACM certificates.</p><p><strong>Transformative Impact</strong> The transformation to a serverless architecture has had a profound impact on the start-up’s operations and prospects:</p><ul><li><strong>Security:</strong> Enhanced security measures protect the website and its data by encrypting communications and implementing strict access controls.</li><li><strong>Availability: </strong>The migration to a serverless cloud-based architecture has significantly enhanced the availability of the start-up’s web application.</li><li><strong>Performance:</strong> The serverless architecture scales automatically to handle varying loads, ensuring consistent performance.</li><li><strong>Cost Efficiency:</strong> The start-up significantly reduced operational expenses by paying only for actual usage.</li><li><strong>Scalability:</strong> The architecture can seamlessly scale to meet future growth without the limitations of physical servers.</li><li><strong>Agility:</strong> The development and deployment of new features are faster and more reliable, enabling the start-up to innovate rapidly.</li></ul><p>This transition has resolved existing operational issues and positioned the start-up for future growth and adaptability in a competitive market. The move to a serverless infrastructure exemplifies a strategic evolution that leverages modern cloud technologies to drive efficiency, performance, and innovation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7578f3d26a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of a Web Application]]></title>
            <link>https://medium.com/@darrenbassett67/the-evolution-of-a-web-application-4131e772edbf?source=rss-70a03346aa87------2</link>
            <guid isPermaLink="false">https://medium.com/p/4131e772edbf</guid>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-solutions-architect]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[aws-cloudformation]]></category>
            <dc:creator><![CDATA[Darren Bassett]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 09:39:20 GMT</pubDate>
            <atom:updated>2024-08-12T09:39:20.917Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Journey from Monolithic to Serverless on AWS — Part I</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*45FxmfDyAx75e6wt" /><figcaption>Photo by <a href="https://unsplash.com/@lucabravo?utm_source=medium&amp;utm_medium=referral">Luca Bravo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Scenario</h3><p>This article documents a project I recently worked on to migrate a legacy monolithic website to the Amazon Web Services (AWS) Cloud.</p><p>A start-up had been operating its website as a monolithic application on a single server at its headquarters for ten years. This architecture resulted in performance bottlenecks, high operational costs, and substantial technical debt. Modifying the website necessitated significant downtime, and the monolithic structure made updates risky and time-consuming. To address these issues, the start-up concluded that migrating the application to a scaleable and robust architecture hosted in the AWS cloud would improve performance, reduce operational expenses, and eliminate technical debt.</p><p>For this project, I divided the implementation into two distinct phases.</p><ul><li>The monolithic application initially transitioned to a highly available, server-based architecture using AWS technologies.</li><li>Subsequently, the website was migrated to a fully serverless architecture, leveraging AWS serverless technologies such as S3, Lambda, and DynamoDB.</li></ul><p>Part I of this article focuses on the first migration phase, documenting the website&#39;s design and implementation in the AWS cloud. I will cover the second phase in Part II.</p><h3>The Application</h3><p>The software development team organised the application&#39;s code into fundamental website files comprising HTML, CSS, and server-side JavaScript. The site&#39;s services, including the Latest News Page, Blog Post Addition Service, View Counter, and Contact Form, were restructured into microservices, with individual CloudFormation templates developed for each. These templates utilise Amazon EventBridge, API Gateway, Lambda, S3, and DynamoDB services. Furthermore, I created a custom Amazon Machine Image (AMI), incorporating a webserver and the website files.</p><h4>Static Website Hosting with EC2 and CloudFront</h4><p>The static website files were hosted on EC2-based servers using the custom AMI, providing a reliable and scalable solution. A content delivery network, Amazon CloudFront ensured global distribution with minimal latency. The website&#39;s security is enhanced with SSL/TLS certificates managed by AWS Certificate Manager, ensuring encrypted communication.</p><h4>User Interaction and Data Handling</h4><p>User interactions with the web application trigger various AWS Lambda functions for dynamic operations:</p><ul><li><strong>Contact Form Submission</strong>: A Lambda function processes form submissions. This function is integrated with an Amazon SNS topic to notify the website owner via email when a new contact form is submitted.</li><li><strong>Increment View Counter</strong>: When a user views the website, a Lambda function updates a view counter stored in Amazon DynamoDB, leveraging its low latency and scalability to provide real-time updates.</li><li><strong>Blog Post Management</strong>: Content creators upload blog posts to an S3 bucket, initiating a sequence of Lambda functions to process and transform the blog into a format suitable for display on the website.</li></ul><h4>Dynamic Content Management</h4><p>The architecture efficiently handles dynamic content updates:</p><ul><li><strong>News Updates</strong>: AWS Lambda functions fetch the latest news from the AWS News Blog RSS feed, scheduled by Amazon EventBridge, ensuring the website always displays current news. The fetched news is stored in DynamoDB and served to users via another Lambda function, invoked through an HTTP API to keep content fresh and relevant.</li></ul><h3>AWS Architecture for the Migrated Web Application</h3><p>In this section, I describe the AWS architecture for the migrated web application and focus on the fundamental design principles:</p><p>Design Principles</p><ul><li><strong>Security:</strong> The architecture prioritises security by implementing multiple measures. These include encrypting data in transit and at rest, placing EC2 instances in private subnets, strategically deploying security groups, and implementing IAM policies that follow the principle of least privilege.</li><li><strong>Resiliency:</strong> High availability is ensured by deploying the application across multiple EC2 instances in different availability zones and using an Application Load Balancer to distribute user requests evenly.</li><li><strong>Scalability:</strong> An Auto Scaling Group handles fluctuating traffic levels, enabling the system to scale out or in according to the volume of user requests.</li><li><strong>Performance:</strong> The architecture optimises the application&#39;s performance, ensuring fast and efficient operation.</li><li><strong>Cost Optimisation:</strong> The system minimises costs while balancing performance and resource utilisation.</li></ul><h4>Enhanced Server-Based Architecture on AWS</h4><p>The diagram below depicts the secure, resilient, scalable, high-performing, and cost-optimised web application on AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AzwkNDN5cDZJ4oc2-BquhA.jpeg" /><figcaption>AWS Portfolio Server-Based Website</figcaption></figure><h3><strong>Features of the Architecture</strong></h3><h4>Designing a Secure, Robust, and Resilient Solution</h4><p>The solution integrates a comprehensive suite of security measures to establish a robust and resilient environment. AWS services are deployed in a Virtual Private Cloud (VPC), providing granular control over network configurations, including IP address ranges, subnets, route tables, and network gateways.</p><p>Public and private subnets were created within the VPC, ensuring sensitive data and applications remain isolated from the internet. Network Access Control Lists (ACLs) and Security Groups define strict inbound and outbound traffic rules, fortifying security and ensuring that only authorised traffic can access resources.</p><p>AWS Route 53 directs traffic to a custom domain secured by AWS Certificate Manager, ensuring encrypted communication through Amazon CloudFront. I implemented an HTTP to HTTPS redirection on the CloudFront distribution to ensure data in transit was encrypted, further enhancing security. CloudFront also includes DDoS mitigation via AWS Shield Standard, providing an additional layer of protection. Additionally, I configured CloudFront to insert a custom header into requests, which the Application Load Balancer (ALB) verifies, rejecting any direct requests lacking this header.</p><p>The architecture utilises an ALB to distribute traffic evenly across web server instances in the VPC. This setup separates public and private subnets to limit access, placing web server instances in private subnets within an Auto Scaling group for scalable and reliable performance. Public and private security groups control network traffic to the EC2 instances using strict rules and security group chaining, enhancing security.</p><h4>Leveraging AWS Services for Resiliency, Scalability, and Cost Optimisation</h4><p>The AWS services, including Lambda, S3, SNS, DynamoDB, and EventBridge, employed in this architecture, are engineered for resilience, scalability, and cost-effectiveness:</p><ul><li><strong>AWS Lambda</strong> scales automatically based on demand, running code responding to triggers without provisioning or managing servers. This serverless approach reduces costs by charging only for the compute time used.</li><li><strong>Amazon S3</strong> ensures high durability and availability by replicating data across multiple devices and facilities. S3&#39;s tiered storage options, including S3 Standard, S3 Intelligent-Tiering, and S3 Glacier, allow for cost optimisation based on data access patterns.</li><li><strong>Amazon SNS</strong> is a fully managed messaging service that scales dynamically to handle varying loads, providing cost-effective communication. It charges only for the number of messages published and delivered.</li><li><strong>Amazon DynamoDB</strong> offers fast, predictable performance with seamless scalability. It uses on-demand and provisioned capacity modes, allowing cost optimisation based on application needs. DynamoDB&#39;s automatic scaling adjusts throughput capacity to match traffic, ensuring cost efficiency.</li><li><strong>Amazon EventBridge</strong> enables scalable event-driven architectures with a pay-per-use pricing model, making it cost-effective for handling increasing events without upfront costs.</li></ul><p>This architecture on AWS demonstrates how integrating auto-scaling and load balancing with core serverless services creates a highly resilient, scalable, and cost-efficient web application. By leveraging S3 for static content, CloudFront for content delivery, Lambda for serverless computing, and additional services like SNS, DynamoDB, and EventBridge, this architecture ensures a seamless and responsive user experience. Including an ALB and Auto Scaling group within a VPC further enhances reliability and scalability. It is well-suited for handling dynamic web content and varying traffic loads while optimising costs.</p><h3>Implementation</h3><p>This section outlines the steps to construct the infrastructure as detailed in the preceding Architecture section. The implementation was automated using Infrastructure as Code (IaC) via AWS CloudFormation.</p><h4>Step 1: Deploying the AMI</h4><p>I launched an EC2 instance based on Amazon Linux 2023, which I then customised with a pre-installed Apache web server and the web application code deployed in the web server&#39;s document root. I then created a custom AmazonMachine Image (AMI) from the modified EC2 instance and launched an EC2 instance using this custom AMI to create a base for constructing the microservices.</p><h4>Step 2: Deploying Microservices using CloudFormation Stacks</h4><p>I created a CloudFormation stack for each of the four microservices to build out the AWS services for the microservices:</p><ul><li><strong>Blog Post: </strong>This template creates the S3 upload and transformation buckets, the upload and fetch blog Lambda functions, and IAM policies and roles that provide permissions to the Lambda functions to get, list and put objects into the S3 buckets and also to create and write to CloudWatch log streams.</li><li><strong>View Counter:</strong> This microservice template creates a DynamoDB table to store the view counter, a Lambda function to increment it, and IAM policies that grant the Lambda function the required permissions to access the DynamoDB table and CloudWatch logs.</li><li><strong>Contact Form: </strong>The template for this microservice creates a Lambda function for the Contact Form microservice, an SNS topic with an email subscription. This template also defines IAM policies and roles, allowing Lambda to publish a message on the topic and write to CloudWatch logs.</li><li><strong>AWS Latest News:</strong> This template creates a Lambda function that reads news stories from the RSS feed and stores them in a DynamoDB table. The template defines IAM policies and roles for the Lambda function to access DynamoDB and CloudWatch. This template also creates an EventBridge rule that is scheduled to run daily at 9 a.m.</li></ul><h4>Step 3: Create the Microservices Endpoints and AMI</h4><p>In this step, I manually configure the endpoints for each microservice and update the application code on the EC2 instance with the corresponding invoke URL.</p><p><strong>Blog Post: </strong>I configured an HTTP API endpoint using the AWS API Gateway service, integrated a GET method with the Fetch Post Lambda Function, and enabled CORS with these settings:</p><p>- Access-Control-Allow-Origin: *<br>- Access-Control-Allow-Headers: *<br>- Access-Control-Allow-Methods: *<br>- Access-Control-Expose-Headers: *</p><p>I then updated the endpoint URL in the application code with the Invoke URL from the newly created API Gateway endpoint.</p><p>Next, I configured the S3 Upload Bucket event notifications to trigger the Create Post Lambda function when a user uploads a Blog post to the site. I configured the event notification to trigger specifically on .txt files.</p><p>Finally, I tested the microservice by uploading .txt files through the API.</p><p><strong>View Counter: </strong>I set up a Lambda function URL, with CORS enabled, as a trigger for the View Counter Lambda function. I set the authentication type to &quot;NONE&quot; for each Lambda function URL in the solution. I also configured CORS to expose all headers and allow all headers and methods.</p><p>I then updated the function URL in the application code and tested the function URL.</p><p><strong>Contact Form: F</strong>or the View Counter, I created a Function URL for the Contact Form Lambda function with the same settings. Then, I updated the code with the new URL endpoint and tested it to ensure the function URL works correctly.</p><p><strong>AWS Latest News: </strong>I repeated the steps outlined for the View Counter and Contact Form microservices to create a Function URL for the AWS Latest News microservice.</p><p>At this point, the news page was not populated with articles, as the EventBridge rule had yet to run. Before integrating the microservice into the webpage, I tested the AWS Latest News Lambda function, which populated the DynamoDB table with news stories.</p><p>After configuring and testing the microservices, I created a new custom AMI from the EC2 instance. I will use this AMI in the next step when configuring a launch template for an Auto Scaling Group.</p><h4>Step 4: Deploy Network Infrastructure Stacks</h4><p>In this final implementation step, I automated the deployment of the remaining AWS services with CloudFormation. I created CloudFormation templates using nested stacks to group related resources logically, making the templates more straightforward to manage and improving reusability.</p><ul><li><strong>VPC Stack:</strong> Creates the VPC, an Internet Gateway, and attaches the Gateway to the VPC.</li><li><strong>Subnets Stack:</strong> This Stack creates two public and two private subnets in the VPC created in the VPC Stack.</li><li><strong>Route Tables Stack:</strong> I configured public and private route tables and associated them with the corresponding subnets in this Stack. In the public route table, I added a route to the Internet Gateway.</li><li><strong>Security Groups Stack:</strong> This Stack creates two security groups. The first group allows inbound HTTP and HTTPS traffic from anywhere. I placed the ALB into this security group. The second security group only allows inbound HTTP traffic from the ALB security group. I put the EC2 instances into this group. This technique is known as security group chaining, and it enhances security by providing a layered defence, minimising the attack surface.</li><li><strong>Load Balancing Stack:</strong> I configured this Stack to create a launch template, a target group, an ALB, an HTTP listener for the ALB, a listener rule, an auto-scaling group, and a target tracking policy that tracks the <em>ALBRequestCountPerTarget</em> metric. The listener rule is configured only to forward requests that have the <em>X-Custom-Header</em> present with the expected value.</li><li><strong>CloudFront Stack:</strong> This creates a CloudFront distribution and an alias record in the Route53 hosted zone that points to the CloudFront distribution DNS name.</li></ul><h3>Testing and Validation</h3><p>To ensure the reliability and functionality of the migrated web application, I implemented a comprehensive testing strategy that included unit, integration, and end-to-end testing.</p><ul><li><strong>Unit tests:</strong> I employed unit testing to verify the correctness of individual components, such as Lambda functions and API Gateway endpoints, ensuring each function performed as expected in isolation.</li><li><strong>Integration tests:</strong> I performed integration testing to validate the interactions between the microservices, confirming that data flows seamlessly between components like S3, DynamoDB, and SNS.</li><li>E<strong>nd-to-end tests: </strong>Finally, I executed end-to-end testing to simulate real-world user scenarios, ensuring the entire system operated cohesively from user interaction to data processing and response, providing a robust and resilient website experience.</li></ul><h3>Leveraging Generative AI in the Project</h3><p>I used Generative AI extensively in this project in the following ways:</p><h4>Research and Planning</h4><p><strong>Generative AI for Research:</strong> I used AI tools to research best practices for design and implementation, including architectural patterns, service configurations, and deployment strategies.</p><h4>Generating CloudFormation Templates</h4><p><strong>Template Generation:</strong> I used generative AI to create the CloudFormation templates for various AWS resources initially, including VPC, subnets, security groups, EC2 instances, Load Balancers, Lambda functions, and DynamoDB tables. I then customised these templates to match my solution better and fix some issues with the generated code.</p><h4>Project Documentation and Write-Up</h4><p><strong>Assistance in Write-Up:</strong> I used AI tools to structure and refine the project documentation, ensuring clarity and comprehensiveness.</p><p><strong>Proofreading and Editing:</strong> AI tools provided grammar and style checks, enhancing the overall quality of the write-up.</p><h3>Summary</h3><p>This article outlines the process of migrating a legacy monolithic web application to a modern, scalable architecture on Amazon Web Services (AWS). Initially, the application was hosted on a single server, leading to performance issues and high operational costs. The migration was executed in two phases: transitioning to a highly available, server-based architecture using AWS services like EC2, CloudFront, and Lambda and then moving to a fully serverless architecture.</p><p>The article focuses on the first phase, where the monolithic application was restructured into microservices using CloudFormation templates. These microservices managed various aspects of the website, such as blog posts, view counters, and contact forms, with AWS services like S3, Lambda, DynamoDB, and API Gateway. I designed the AWS architecture with security, scalability, and cost optimisation in mind, employing features like Auto Scaling, VPC, and security group chaining.</p><p>I used Generative AI tools for research, CloudFormation template generation, and documentation throughout the project.</p><p>The article emphasises the importance of a secure, resilient, and scalable architecture for modern web applications, highlighting the benefits of moving from a monolithic to a serverless model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4131e772edbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>