<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Blake Romano on Medium]]></title>
        <description><![CDATA[Stories by Blake Romano on Medium]]></description>
        <link>https://medium.com/@blakeromano?source=rss-9667524f196a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JFF_TMKmMjrYwIokMAK9aQ.jpeg</url>
            <title>Stories by Blake Romano on Medium</title>
            <link>https://medium.com/@blakeromano?source=rss-9667524f196a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:38:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@blakeromano/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Building Your Own Load Testing Platform. Use This Instead.]]></title>
            <link>https://medium.com/@blakeromano/stop-building-your-own-load-testing-platform-use-this-instead-52ade17da80a?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ade17da80a</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[load-testing]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Sat, 30 May 2026 15:52:55 GMT</pubDate>
            <atom:updated>2026-05-30T15:52:55.278Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAcJUe2Y4N9qkM780mBi8A.png" /></figure><p>If you’re a platform engineer thinking about building a performance testing system for your team, I want to save you two to four weeks of work.</p><p><a href="https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/">Distributed Load Testing on AWS</a> already exists, it’s free, it’s production ready, and it supports JMeter, k6, and Locust out of the box. My team adopted it in hours, and it’s done a job we’d been planning to spend a quarter building.</p><p>This post is the recommendation I wish I’d gotten earlier. Here’s what the tool does, why it’s better than the thing you’re about to build, and how to know if it’s the right fit.</p><h3>What it actually is</h3><p>Distributed Load Testing on AWS is one of the <a href="https://aws.amazon.com/solutions/">AWS Solutions Library</a> implementations, which is a set of free, production ready CloudFormation deployments that AWS maintains. They sit in the corner of the AWS console and most engineers I talk to have never opened them. That’s a mistake. They aren’t reference architectures or marketing material. They are real, working systems.</p><p>The Distributed Load Testing solution gives you:</p><ul><li>A web UI where developers can upload their own test scripts and configure runs.</li><li>Native support for Apache JMeter, Grafana k6, and Locust. You don’t pick one. You support all three.</li><li>Automatic orchestration of distributed test execution across AWS Regions, with the ability to simulate millions of transactions.</li><li>Built in dashboards for latency, throughput, and error rates. No additional observability tooling required.</li><li>An API for triggering tests programmatically, plus on demand and scheduled execution.</li><li>Test result collection, parsing, logging, and analysis, all automated.</li></ul><p>You deploy it from a CloudFormation template. That’s the install.</p><h3>Why you shouldn’t build this yourself</h3><p>The build it yourself version of this is a real project. I know because we scoped it.</p><p>You need to provision and manage test runner infrastructure. You need to write the orchestration layer that distributes work across runners. You need to handle result aggregation. You need a way for developers to submit and schedule tests without bothering the platform team. You need dashboards. You need an API.</p><p>Our internal estimate was two to four weeks for v1, and v1 supported exactly one framework. Adding JMeter, k6, and Locust support all in the same system was a separate project after that.</p><p>Distributed Load Testing on AWS does all of it. In hours, not weeks.</p><p>The deeper cost of building your own isn’t the initial two to four weeks. It’s the years of maintenance. Every system your platform team owns is a system you patch, document, and migrate when the underlying tech changes. Performance testing is undifferentiated infrastructure. There is no business reason to own it. Spending platform team headcount on it means you’re not spending it on something that actually moves your company.</p><h3>What surprised us once we adopted it</h3><p>I want to be honest about this part because “we adopted an AWS solution” sounds boring and the reality was almost suspiciously good.</p><ul><li><strong>The observability came for free.</strong> We’d been planning to build latency, throughput, and error rate dashboards on top of our homegrown system. We didn’t have to. They were already there.</li><li><strong>CI/CD integration was trivial.</strong> Because tests can be triggered via API, we wired performance testing into our pipelines without writing glue code.</li><li><strong>It works for resilience testing too.</strong> We used DLT to simulate production grade load while intentionally taking application components offline, to verify that previous reliability fixes actually held under stress. We got that for free.</li><li><strong>Onboarding new developers got dramatically faster.</strong> A developer on our team can now operationalize and run a custom performance testing script in two to three hours. Before, performance testing was a small project anyone wanting to do it had to plan around.</li></ul><h3>When it might not be the right fit</h3><p>I don’t want to sell this as a universal answer. A few cases where building or buying something else makes more sense:</p><ul><li>You need a framework DLT doesn’t support. JMeter, k6, and Locust cover most of what teams use, but if you’re committed to Gatling, k6 Cloud, or something more exotic, you’ll want to look elsewhere.</li><li>You need very specific custom orchestration that the solution doesn’t expose. The solution is configurable but not infinitely so. If you have unusual requirements around how tests are sharded or how results are processed, you may hit limits.</li><li>You need fine grained tenancy controls beyond what the solution provides. For most platform teams this isn’t an issue, but worth checking against your specific requirements.</li></ul><p>If none of those apply, the answer is almost certainly to adopt the solution and move on.</p><h3>How to actually evaluate it</h3><p>Here’s the half day version of evaluating Distributed Load Testing on AWS for your team:</p><ul><li>Deploy the CloudFormation template into a sandbox account. It takes about 30 minutes to come up.</li><li>Upload one of your team’s existing test scripts. Pick whichever framework you already use. Run it.</li><li>Check the dashboards. Confirm the metrics you care about are there.</li><li>Trigger a test via the API. Confirm you can integrate it into a CI pipeline if that’s relevant to you.</li><li>Run a longer, larger test that approximates real production load. Confirm it scales.</li></ul><p>That’s enough to know whether the tool fits. If it does, you’ve just bought yourself a quarter back. If it doesn’t, you’ve spent half a day and you know exactly what you’d need to build instead.</p><h3>Wrapping up</h3><p>Platform engineering has a builder instinct. When we see a system, we want to own it. Sometimes that’s right. For performance testing infrastructure, in 2026, on AWS, it isn’t. The tool exists, it works, and the trade off is overwhelmingly in favor of adoption.</p><p>If you’re about to scope a load testing project, please open the AWS Solutions Library first. The half day might save you a quarter.</p><p><em>Want to see how this played out in practice? AWS published </em><a href="https://aws.amazon.com/solutions/case-studies/imagine-learning/"><em>a case study</em></a><em> on how my team at Imagine Learning adopted Distributed Load Testing on AWS, including the specific outcomes and how we integrated it into our internal developer platform.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52ade17da80a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Powerful AI Agents Quickly with Strands Agents SDK]]></title>
            <link>https://medium.com/imaginelearning/building-powerful-ai-agents-quickly-with-strands-agents-sdk-479c21a94237?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/479c21a94237</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 19:54:22 GMT</pubDate>
            <atom:updated>2025-09-22T19:54:22.400Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Harnessing Generative AI for Faster, Smarter, and More Efficient Workflows</em></strong></p><p>Generative AI is rapidly transforming the way we interact with technology, from chatbots assisting customers to innovative tools for educators and students. Its impact is not only about staying ahead of the curve — it’s about building software faster, cheaper, and easier, while delivering greater value to users. Whether you’re focused on retaining customers, helping students, or streamlining development, integrating AI into your projects is essential for modern software development.</p><h3>Understanding AI Agents and Their Importance</h3><p>In AI, an “agent” is an autonomous system that perceives its environment, reasons, and takes actions to achieve specific goals. These agents power many contemporary AI applications, automating repetitive tasks, responding to user queries, orchestrating workflows, and making decisions with minimal human input. Their versatility makes them central to creating smarter, more responsive technology solutions.</p><h3>Strands Agents SDK: Your Toolkit for AI Agents</h3><p>AWS’ Strands Agents SDK is an open-source framework designed to simplify the process of building, deploying, and managing AI agents and agentic workflows. Supporting a wide variety of large language models (LLMs) and providers — including AWS Bedrock, Anthropic, Llama, OpenAI, and more — the SDK offers unmatched flexibility and scalability. With Strands Agents SDK, you can mix and match models to fit your unique needs and leverage generative AI efficiently with clear, easy-to-read code.</p><p><strong>Key Benefits of Strands Agents SDK</strong></p><ul><li><strong>Rapid Prototyping:</strong> Bring your ideas to life in minutes with minimal code needed!</li><li><strong>Open Source and Extensible:</strong> Easily build AI agents in a vendor-agnostic format and extend it for more advanced use cases.</li><li><strong>Model Provider Agnostic:</strong> Choose the best LLM for your application, whether it’s through AWS, OpenAI, Anthropic, etc.</li><li><strong>Open Source Protocols:</strong> Native support for Model Context Protocol (MCP) and Agent to Agent (A2A) protocol accelerates development.</li><li><strong>Deploy Anywhere:</strong> Strands Agents can run on any compute environment, including Bedrock AgentCore, Kubernetes, AWS Lambda, and more!</li></ul><h3>Getting Started: What You Need</h3><p>To begin, ensure you have:</p><ul><li>Basic Python programming knowledge</li><li>Credentials for your LLM provider (this example uses AWS Bedrock)</li><li>Access to a terminal and your preferred code editor</li></ul><h3>Step-by-Step: Build a Terminal-Based Math Tutor for K-12 Students</h3><h4>Step 1: Install Strands Agents SDK</h4><p>Install the SDK via your terminal:</p><pre>pip install strands-agents-sdk strands-agents-tools</pre><h4>Step 2: Import Required Modules</h4><p>Import the essential modules to set up your agent:</p><pre>from strands import Agent<br>from strands.models.bedrock import BedrockModel<br>from strands_tools import calculator, handoff_to_user</pre><p>In this case we are going to import the core Agent class from Strands as well as the BedrockModel class to configure our model provider and add two tools from the Strands Tools library which will be a calculator and a hand off to user.</p><h4>Step 3: Create a System Prompt for Guided Learning</h4><p>Shape your agent’s behavior by configuring a system prompt. For guided learning, the prompt should instruct the agent to help students work through problems, ask questions, and avoid giving direct answers.</p><pre>system_prompt = &quot;&quot;&quot;<br>You are a helpful math tutor for K-12 students. <br>Use the calculator tool to help with math problems. <br>Do not give the student the answer directly but try to guide them through the problem.<br>Hand off to the user for clarification or if you want them to do the calculation themselves.<br>&quot;&quot;&quot;</pre><h4>Step 4: Initialize the Agent with Tools</h4><p>Set up your agent, specifying the model from AWS Bedrock (e.g., “us.anthropic.claude-sonnet-4–20250514-v1:0”), name, description, and adding the calculator and handoff tools.</p><pre>provider = BedrockModel(<br>  region_name=&quot;us-east-1&quot;,<br>  model_id=&quot;us.anthropic.claude-sonnet-4–20250514-v1:0&quot;,<br>  streaming=True,<br>)<br>agent = Agent(<br>  tools=[calculator, handoff_to_user],<br>  model=provider,<br>  system_prompt=system_prompt<br>)</pre><p>The <strong>calculator tool</strong> lets your agent perform calculations, while the <strong>handoff to user tool</strong> enables prompting the student to solve or clarify steps themselves.</p><h4>Step 5: Interact with the Agent via Terminal</h4><p>Use a simple loop to interact with your math tutor agent:</p><pre>while True:<br>  question = input(&quot;Ask a math question (or type &#39;exit&#39; to quit): &quot;)<br>  if question.lower() == &#39;exit&#39;: break<br>  response = agent.ask(question)<br>  print(response)</pre><p>Now, students can type math problems and receive hints, guiding questions, or partial steps from the agent.</p><p>The full code can be seen below:</p><pre>from strands import Agent<br>from strands.models.bedrock import BedrockModel<br>from strands_tools import calculator, handoff_to_user<br><br>system_prompt = &quot;&quot;&quot;<br>You are a helpful math tutor for K-12 students. <br>Use the calculator tool to help with math problems. <br>Do not give the student the answer directly but try to guide them through the problem. <br>Hand off to the user for clarification or if you want them to do the calculation themselves. <br>&quot;&quot;&quot;<br><br>provider = BedrockModel(<br>  region_name=&quot;us-east-1&quot;,<br>  model_id=&quot;us.anthropic.claude-sonnet-4-20250514-v1:0&quot;,<br>  streaming=True,<br>)<br><br>agent = Agent(<br>  tools=[calculator, handoff_to_user],<br>  model=provider,<br>  system_prompt=system_prompt<br>)<br><br>while True:<br>  question = input(&quot;Ask a math question (or type &#39;exit&#39; to quit): &quot;)<br>  if question.lower() == &#39;exit&#39;: break<br>  response = agent(question)<br>  print(response)</pre><p>The system prompt ensures that the agent acts as a coach — asking guiding questions, pointing out errors, and encouraging students to try calculations themselves. If a student is stuck or needs clarification, the agent uses the handoff tool to let them take the lead or prompt them to perform the next step.</p><ul><li><strong>Guided Tutoring:</strong> Promotes understanding by breaking problems into steps and checking comprehension.</li><li><strong>Handoff Tool:</strong> Empowers students to try calculations, reinforcing active learning and problem-solving skills.</li></ul><h4>Next Steps and Customization</h4><p>With the example above, you can create a terminal-based math tutor in just around 25 lines of code that helps students build confidence and mathematical skills. The Strands Agents SDK enables easy customization — add new tools, refine prompts for different math topics, or deploy your agent and integrate it into your application. With just a few lines of code, you’re supporting students through guided discovery and AI-powered learning.</p><h3><strong>Best Practices and Tips</strong></h3><ul><li><strong>Experiment with Different Models:</strong> Test multiple LLM providers to see which works best for your needs. Consider cost, latency, and accuracy.</li><li><strong>Start Simple:</strong> Begin with a basic agent and add features as you go.</li><li><strong>Monitor and Iterate:</strong> Track agent performance and adjust based on user feedback.</li><li><strong>Respect User Privacy:</strong> Always be mindful of the data your agents handle. Leverage things like guardrails to protect users.</li></ul><h3>Conclusion</h3><p>The Strands Agents SDK from AWS is a powerful solution for building AI agents and workflows quickly and efficiently. By supporting a wide range of LLMs and lowering the development barrier, it enables developers, educators, and businesses to innovate and automate like never before. Ready to add generative AI to your products or processes? Give the Strands Agents SDK a try — you might be surprised by how much you can accomplish with just a few lines of code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=479c21a94237" width="1" height="1" alt=""><hr><p><a href="https://medium.com/imaginelearning/building-powerful-ai-agents-quickly-with-strands-agents-sdk-479c21a94237">Building Powerful AI Agents Quickly with Strands Agents SDK</a> was originally published in <a href="https://medium.com/imaginelearning">Imagine Learning Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Developer Efficiency with Model Context Protocol (MCP)]]></title>
            <link>https://medium.com/imaginelearning/unlocking-developer-efficiency-with-model-context-protocol-mcp-fd03e9831d3d?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd03e9831d3d</guid>
            <category><![CDATA[model-context-protocol]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 00:06:56 GMT</pubDate>
            <atom:updated>2025-07-16T00:06:56.743Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.imaginelearning.com/">Imagine Learning</a>, we’ve invested heavily in building tools that support our developers across every stage of the software development lifecycle. But as those tools grew in number, spanning services like AWS, ArgoCD, Jira, Confluence, and more, we began noticing a familiar pattern. Engineers were spending too much time searching for answers and not enough time building software.</p><p>Support channels were filled with questions that documentation already answers. Developers paused their work to hunt through internal documentation to retrieve information. The tools existed, but we didn’t have a cohesive experience to use them, and it was costing our engineers time and focus.</p><p>We needed a better way to unify access to the systems our developers rely on without asking them to hop between different documentations or install new interfaces. Instead of pulling developers into tools, we focused on delivering tools into their existing workflows.</p><h3><strong>Model Context Protocol</strong></h3><p>We turned to the <a href="https://modelcontextprotocol.io/introduction">Model Context Protocol (MCP)</a>, an open protocol designed to bridge the gap between large language models (LLMs) and external tools. MCP gives structure to how tools expose functionality by defining a shared contract for inputs, context, and interactions.</p><p>MCP gives us a reliable way to expose internal tools to AI systems without building custom integrations for each one. We can leverage open source MCP servers like <a href="https://github.com/awslabs/mcp">AWS’</a> in a way that makes it easy for us to integrate new services into our proxy without having to write the servers ourselves. Leveraging MCP makes it possible to build scalable, reliable integrations with minimal overhead.</p><p>Importantly, MCP makes it possible to connect directly to where developers are already working. With integrations into their IDEs like Visual Studio Code or IntelliJ, engineers don’t need to switch tools to get help. They are able stay in their editor, and the system can adapt to their context.</p><h3><strong>Making It Work in Practice</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nBQeNI9dpqdQoImxPgxduQ.png" /></figure><p>We built a containerized system that runs multiple MCP STDIO servers behind a single HTTP interface. Each MCP server handles one integration and is converted to Streamable HTTP servers using <a href="https://github.com/supercorp-ai/supergateway/">SuperGateway</a>, and <a href="https://nginx.org/">NGINX</a> routes requests based on the endpoint, allowing agents and developers to access tools without managing the software to run STDIO servers locally. Everything is packaged in a single Docker container, with no local setup required. This lightweight approach ensures the tools are usable even on low-powered developer machines or in browser-based IDEs. Access is internal-only, and the current setup supports read-only operations to maintain high security standards while still providing value.</p><h3><strong>The Impact on Developer Experience</strong></h3><p>Engineers can retrieve documentation, review deployment metadata, or check ticket history directly from their development environment. This allows for fewer interruptions, shorter onboarding times, and a more centralized access point for knowledge.</p><p>Platform teams’ benefit, too. With MCP, we are able to leverage open source MCP servers instead of having to develop each one ourselves. As a result, we only need to maintain one proxy instead of many different sets of containers. From there, the proxy can be accessed programmatically by agents, queried by developers, or embedded into different interfaces without having to rebuild the same logic across interfaces.</p><h3><strong>Where We’re Headed</strong></h3><p>This work is still evolving. We’re exploring how to extend access control, improve observability for agent usage, and expand support to additional tools. For now, read-only access ensures that we maintain strong security boundaries while laying the groundwork for more advanced capabilities.</p><p>MCP has helped us simplify access to internal knowledge and services, reduce support overhead, and improve developer focus. It’s a straightforward pattern that scales well, integrates cleanly, and enhances the way our teams build.</p><p><a href="https://www.linkedin.com/in/blakeromano/"><em>Blake Romano</em></a><em>, Senior Software Engineer</em></p><p><a href="https://www.linkedin.com/in/mason-price-ab4553282/"><em>Mason Price</em></a><em>, Associate Software Engineer</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd03e9831d3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/imaginelearning/unlocking-developer-efficiency-with-model-context-protocol-mcp-fd03e9831d3d">Unlocking Developer Efficiency with Model Context Protocol (MCP)</a> was originally published in <a href="https://medium.com/imaginelearning">Imagine Learning Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proactively Detecting Manual Changes in AWS with a No-Code Solution]]></title>
            <link>https://medium.com/@blakeromano/proactively-detecting-manual-changes-in-aws-with-a-no-code-solution-fe4df6bb1792?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4df6bb1792</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[aws-security]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 01:02:05 GMT</pubDate>
            <atom:updated>2025-03-18T01:02:05.535Z</atom:updated>
            <content:encoded><![CDATA[<p>When working in an organization that embraces Infrastructure as Code (IaC) tools like Terraform, Pulumi, or Crossplane, ensuring that cloud resource configuration is managed exclusively through these tools is critical. Manual changes by developers outside of IaC pipelines can introduce inconsistencies, security risks, and operational challenges.</p><p>To maintain control and enforce best practices, organizations need a way to detect unauthorized manual actions in their AWS environments. Time to present a no-code solution to proactively monitor and alert on such activity, ensuring adherence to established IaC workflows.</p><h3>Solution Overview</h3><p>This solution leverages AWS-native services to detect and notify teams when manual changes are made.</p><figure><img alt="Diagram of an automated alerting system. AWS CloudTrail sends events to EventBridge, which filters them based on rules. Matching events are forwarded to an SNS Topic, triggering a notification to a Developer for Chat Applications which will send a message to Slack." src="https://cdn-images-1.medium.com/max/1024/0*sinQzOy5upoBTC-Y" /><figcaption>This is an architecture diagram showcasing how we can utilize these AWS Services to build a no-code solution for alerting to manual changes to resources</figcaption></figure><h3>How It Works</h3><ol><li><strong>Monitoring AWS API Activity with CloudTrail</strong></li></ol><ul><li>AWS CloudTrail continuously logs API actions across the AWS environment.</li><li>Events related to resource creation or modification (e.g., Create* and Put* API calls) are recorded.</li></ul><p><strong>2. Filtering Events in EventBridge</strong></p><ul><li>An EventBridge rule is created to detect these Create and Put actions.</li><li>The rule specifically looks for actions performed by an assumed role or a federated user logging in with a personal email (e.g., a gmail.com address).</li><li>This helps detect unauthorized manual actions in organizations using AWS IAM Identity Center.</li></ul><p><strong>3. Forwarding Alerts via SNS and AWS Q</strong></p><ul><li>When a matching event is detected, EventBridge forwards it to an Amazon SNS topic.</li><li>AWS Q Developer for Chat Applications subscribes to the SNS topic and forwards the message to Slack.</li><li>This ensures real-time notifications to the team, enabling quick intervention.</li></ul><h3>Benefits of This Approach</h3><ul><li><strong>Proactive Alerting</strong> — Detects unauthorized manual changes before they become operational issues.</li><li><strong>No-Code Implementation</strong> — Fully managed through AWS services without requiring any custom code.</li><li><strong>Seamless Integration</strong> — Works with existing AWS security frameworks and communication channels.</li><li><strong>Improved Compliance</strong> — Helps enforce organizational policies by tracking deviations from IaC pipelines.</li></ul><h3>Conclusion</h3><p>By leveraging this solution organizations can implement a proactive, no-code solution to detect and respond to unauthorized manual changes in AWS environments. This approach ensures that all infrastructure modifications occur within approved IaC workflows, maintaining security, compliance, and operational efficiency.</p><p>You can see an example terraform module of this implementation here <a href="https://github.com/blakeromano/terraform-aws-iac-alerting">https://github.com/blakeromano/terraform-aws-iac-alerting</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4df6bb1792" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI Meets Internal Developer Platforms: Unleashing AWS Bedrock for Developer Efficiency —…]]></title>
            <link>https://medium.com/imaginelearning/generative-ai-meets-internal-developer-platforms-unleashing-aws-bedrock-for-developer-efficiency-01122fcfbc57?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/01122fcfbc57</guid>
            <category><![CDATA[bedrock]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 17:46:32 GMT</pubDate>
            <atom:updated>2025-02-06T17:46:32.893Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generative AI Meets Internal Developer Platforms: Unleashing AWS Bedrock for Developer Efficiency — Part One</h3><p><em>How a small team is using AI to reclaim time to maintain and enhance the internal developer platform without sacrificing exceptional support for developers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dyw43OSvfEHTsVHcfuR-Tg.jpeg" /></figure><p>After building the next-gen internal developer platform (IDP) for Imagine Learning, the excitement our team felt as we saw wider adoption was tempered as we began fielding questions on the support channel: “Hello team! I’m trying to deploy to the staging environment, but the build failed, and I couldn’t find any other error message. Can you guys help debug this failed build please?”</p><p>At first it was easy to dive in and engage with our IDP’s new users. However, as more users engaged with the platform, the number of questions increased, and we started seeing repeats of questions we had answered previously. Our team then had questions of our own: “Doesn’t anyone read the documentation?” How can we answer all these questions and still find time to develop the platform?” “How can we enhance the developer experience?”</p><p>Enter Artificial Intelligence (AI) — a transformative force in modern software creation. AWS Bedrock, Amazon’s fully managed service for Generative AI, provides a powerful way to embed Generative AI capabilities into an Internal Developer Platform (IDP), enabling faster, smarter, and more efficient developer workflows.</p><p>With AWS Bedrock, developers at Imagine Learning can access pre-trained, scalable AI models that integrate seamlessly with existing workflows, reducing the amount of time they need to spend searching through documentation.</p><p>The AWS Bedrock Agents, using retrieval-augmented generation (RAG), empowers our development teams to focus on what truly matters — building and shipping great software. Below, we share how we implemented AWS Bedrock to support our software engineers.</p><h3>Step 1: Setting Up AWS Bedrock</h3><p>To begin integrating Generative AI into the IDP, first set up an AWS Bedrock Agent. While AWS Bedrock provides access to pre-trained foundation models, the real power comes from configuring an agent tailored to your development ecosystem, retrieving relevant information, and delivering actionable insights.</p><h4>Provisioning AWS Bedrock</h4><p>First, ensure you have access to the AWS Bedrock service, as it’s available only in specific AWS regions. Once access is confirmed:</p><ol><li><strong>Navigate to AWS Bedrock</strong>: Open the AWS Management Console and search for “Bedrock.”</li><li><strong>Explore Foundation Models</strong>: AWS Bedrock offers various models (e.g., Claude, Jurassic-2, Titan). Select a model that aligns with your organization’s needs. Generative AI models like Anthropic’s Claude are well-suited for processing large volumes of natural language data, making them ideal for retrieval tasks.</li><li><strong>Select an Embedding Model</strong>: Choose a model for vectorizing data, which will be stored in a vector database (like OpenSearch).</li></ol><p>After requesting access to these models, proceed with setting up the Bedrock Agent and Knowledge Base.</p><ol><li><strong>Set up the Bedrock Agent</strong>: In the Agents tab, create a new agent. Choose the model you accessed and define the agent’s role, such as “You are a support agent responsible for answering questions about our Internal Developer Platform using documentation.” Click “Save and Exit.”</li><li><strong>Test the Agent</strong>: Test the agent with a sample question like, “How do you create an S3 bucket in our platform?” If the response isn’t accurate, proceed to set up the Bedrock Knowledge Base.</li></ol><h3>Step 2: Configuring a Bedrock Knowledge Base</h3><p>For effective retrieval-augmented generation (RAG) with AWS Bedrock, establish a structured knowledge base as the main reference point for developers. In this setup, the knowledge base syncs with Confluence, a widely used documentation platform, allowing the Bedrock Agent to quickly search relevant content and answer developer questions.</p><ol><li><strong>Syncing Confluence with Bedrock</strong>: Use Confluence as the primary documentation source for your platform. Generate an API token in Confluence and store it in AWS Secrets Manager as a “username” and “password” for secure access.</li><li><strong>Create the Knowledge Base</strong>: In the AWS Console, go to the Knowledge Bases tab to create a new knowledge base. For the data source, select Confluence, enter the URL, choose “Basic Authentication,” and link the stored secret from AWS Secrets Manager.</li><li><strong>Vector Database Configuration</strong>: Select an embedding model and opt for AWS OpenSearch Serverless to create a vector database. OpenSearch is cost-effective and easy to manage, storing vectorized data from Confluence for quick retrieval.</li></ol><p>After syncing Confluence data with the knowledge base, attach it to the Bedrock Agent. When developers ask questions like, “How do you create an S3 bucket in our platform?” the agent pulls information from Confluence. When new information is added to Confluence, simply re-sync to keep the agent’s responses up to date.</p><h3>Step 3: Integrating the Agent as a Chatbot</h3><p>With the Bedrock Agent configured to answer developer questions, the next step is to make it easily accessible. Instead of requiring developers to log into the AWS Console, integrate the agent with Slack using AWS Chatbot, allowing developers to get answers in their existing workspace.</p><ol><li><strong>Setting Up AWS Chatbot with Slack</strong>: In the AWS Chatbot console, link your Slack workspace, add a channel (using the channel ID from Slack), and assign necessary permissions (e.g., AmazonBedrockFullAccess) to the agent.</li><li><strong>Adding the Agent to Slack</strong>: Add the AWS app to the Slack channel and type “@aws connector add idp-support AGENT-ARN” in the channel. Developers can now ask questions like, “@aws ask idp-support How do you create an S3 bucket in our platform?” directly in Slack. Follow-up questions, such as “@aws can you tell me how to configure a lifecycle policy?”, are answered contextually within the same thread.</li></ol><h3>Conclusion</h3><p>Integrating AWS Bedrock into an IDP is a powerful way to boost developer productivity with Generative AI. By setting up a Bedrock Agent and linking it to a Confluence-powered knowledge base, we created a responsive, intelligent support system that saves Imagine Learning developers time and effort. The retrieval-augmented generation (RAG) approach enables engineers to access precise, context-aware responses aligned with our organization’s specific infrastructure.</p><p>The combination of AWS Bedrock’s scalability, Confluence’s structured documentation, and seamless integrations like AWS Chatbot opens the door to a future where developers have AI-driven support within their everyday tools. With no code maintenance required from the platform team, valuable development time is freed up, allowing teams to focus on their core mission — building and delivering innovative software.</p><p>As this journey continues, refining the knowledge base and exploring further AI integrations will ensure that the IDP remains a cutting-edge tool for development teams.</p><p><em>In Part Two of this article, we’ll demonstrate how to implement a feedback loop using Slack messages and replies to improve the Bedrock model’s accuracy and reliability. This feedback loop enables the chatbot to deliver even more precise, effective responses, better tailored to developers’ specific needs.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01122fcfbc57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/imaginelearning/generative-ai-meets-internal-developer-platforms-unleashing-aws-bedrock-for-developer-efficiency-01122fcfbc57">Generative AI Meets Internal Developer Platforms: Unleashing AWS Bedrock for Developer Efficiency —…</a> was originally published in <a href="https://medium.com/imaginelearning">Imagine Learning Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Platform Engineer’s Takeaways from AWS Re:Invent 2024]]></title>
            <link>https://medium.com/@blakeromano/a-platform-engineers-takeaways-from-aws-re-invent-2024-439db4367757?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/439db4367757</guid>
            <category><![CDATA[aws-reinvent-2024]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Sun, 08 Dec 2024 00:06:04 GMT</pubDate>
            <atom:updated>2024-12-08T00:06:04.082Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FnGeLq6j291sHZGvCmTdvA.png" /></figure><p>AWS Re:Invent 2024 was an exciting event, full of impactful announcements and engaging conversations. As someone who embraces Platform Engineering and understands its potential to empower developers to build faster, safer, and more reliably, I believe there’s a lot to unpack from this year’s Re:Invent that will also help improve platform engineering practices.</p><p>When it comes to building platforms on Kubernetes, there were some particularly exciting announcements for AWS EKS, specifically <a href="https://aws.amazon.com/eks/auto-mode/"><strong>EKS Auto Mode</strong></a> and <a href="https://aws.amazon.com/eks/hybrid-nodes/"><strong>EKS Hybrid Nodes</strong></a>. EKS Auto Mode is a fantastic new feature that simplifies the management of storage, ingress, compute, and networking. This should make onboarding easier for people new to Kubernetes, a system that can be quite complex. Another valuable use case is for those building on Kubernetes as a control plane, using tools like <a href="https://www.crossplane.io/"><strong>Crossplane</strong></a> or <a href="https://aws-controllers-k8s.github.io/community/"><strong>AWS Controllers for Kubernetes (ACK)</strong></a>, where the complexity of cluster management could be helped by AWS’ support.</p><p><strong>EKS Hybrid Nodes</strong> is another interesting announcement. It allows AWS to manage your Kubernetes control plane while you bring your own nodes from on-premises or edge locations into your compute environment. This is particularly exciting for organizations needing to run compute at the edge or operate in both cloud and on-premises environments. Additionally, it supports adopting CNCF technologies like <a href="https://cilium.io/"><strong>Cilium</strong></a> as your CNI, providing increased network visibility using eBPF technology, as the AWS VPC CNI cannot run on Hybrid Nodes.</p><p>There were also some noteworthy AI-related announcements, such as the new <a href="https://aws.amazon.com/sagemaker/unified-studio/"><strong>SageMaker Unified Studio</strong></a>, which will bridge the gap for users by offering the right data access in a single, unified interface. Additionally, <a href="https://aws.amazon.com/q/"><strong>Amazon Q</strong></a> received some interesting updates, making it a compelling tool for software engineers and others.</p><p>As for the talks during the week, I wanted to highlight a few key takeaways, starting with <a href="https://www.youtube.com/watch?v=aim5x73crbM">Werner Vogel’s keynote</a>. The concept of “Simplexity” — the idea that complexity cannot be destroyed, only moved — is central to Platform Engineering. Our goal is to shift as much of the complexity related to running distributed services (such as reliability, security, cost-efficiency, and maintenance) from application teams to platform teams. This enables application developers to focus on building products and simplifies the process of running applications at scale.</p><p>There were also excellent sessions, such as <a href="https://www.youtube.com/watch?v=PDb9nLsjFzY"><strong>Christina Andonov and Amine Hilaly “Infrastructure as Kubernetes APIs”</strong></a>, which offered a deep dive into building platforms using the Kubernetes API. They explored how leveraging Kubernetes for self-healing, composability, and state management can unlock significant power and flexibility. Another standout talk was <a href="https://www.youtube.com/watch?v=EV0bukPS_zw"><strong>Gunnar, Thorsten and Johannes “The modern CI/CD toolbox: Strategies for consistency and reliability”</strong></a><strong>. </strong>This session highlighted key patterns like GitOps, Supply Chain Security, Canary Deployments, Feature Flagging, and more. They also gave a great shoutout to <a href="https://cnoe.io/"><strong>CNOE’s</strong></a><strong> IDPBuilder</strong>, which allows you to run your platform locally and develop CI/CD pipelines on your own machine. While there were many other great talks, I highly recommend watching these sessions and exploring the rest of the content available on the <a href="https://www.youtube.com/@AWSEventsChannel"><strong>AWS Events YouTube Channel</strong></a> as well.</p><p>Overall, Re:Invent 2024 demonstrated the ongoing evolution of AWS in powering next-gen platforms, driving developer productivity, and fostering more reliable, scalable architectures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=439db4367757" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Imagine Learning uses Crossplane & Composition Functions]]></title>
            <link>https://medium.com/imaginelearning/how-imagine-learning-uses-crossplane-composition-functions-25532e577538?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/25532e577538</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[internaldeveloperplatform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Thu, 07 Mar 2024 16:57:53 GMT</pubDate>
            <atom:updated>2024-08-09T17:05:13.295Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*UEWSZUhigiwmUGcTbjubug.png" /></figure><p>Since Crossplane Composition Functions were promoted to beta in Crossplane version 1.14, the community has been adopting the new system. Some in the community are leaning into using open-source Composition Functions like <a href="https://github.com/crossplane-contrib/function-patch-and-transform/">Patch and Transform</a> or <a href="https://github.com/crossplane-contrib/function-go-templating/">Go Templates</a>. However, we at Imagine Learning have been building our own Composition Functions due to the troubles we have had with patch and transform with complex compositions.</p><p><a href="https://imaginelearning.com/">Imagine Learning</a> empowers educators to inspire breakthrough moments in every student’s unique learning journey with digital-first, K-12 education solutions. We use Crossplane in our own internal developer platform (IDP) to deploy resources into Amazon Web Services (AWS). Building out abstractions through compositions with native patch and transform was common. Composition files for resources like AWS RDS Clusters regularly contained more than 1,000 lines of repetitive code. We also needed several compositions to be able to specify different instance counts because native patch and transform does not support conditionally rendering resources. This repetition and the large files made testing changes to the compositions nearly impossible and authoring compositions difficult.</p><h3>Implementing Composition Functions</h3><p>After the release of Crossplane 1.14, Imagine Learning moved the existing patch and transform compositions to Functions. As part of that effort, we had to determine if we wanted to use a monolithic Composition Function for all compositions or to have a unique Function for each composition. We chose a monolithic approach. The monolithic approach allows us to route dynamically to the rendering logic for a specific composition and results in significantly fewer idle workloads running in our cluster. Another win associated with this model is that we only need to manage one code base.</p><p>Composition Functions have opened a lot of doors-allowing us to make our control plane easier to test and to run locally and they have improved our ability to manage complex compositions. For example, the composition YAML file for an RDS Cluster composition in the patch and transform framework was about 1,000 lines of code for a single instance cluster and 1,250 lines for a two-instance cluster. Two compositions that looked almost identical to each other were required to support provisioning a different number of database instances. With Crossplane Functions we were able to reduce that to a simple for loop reducing complexity and repetitiveness.</p><p>Here is a code snippet showing how, in an RDS Cluster Composition, we can use a coding practice like a loop to keep code clean and simple:</p><pre>instanceCount, _ := oxr.Resource.GetInteger(&quot;spec.resourceConfig.instanceCount&quot;) <br>for i := int64(0); i &lt; instanceCount; i++ { <br>  desired[resource.Name(fmt.Sprintf(&quot;instance-%v&quot;, i))] = resource.NewDesiredComposed() <br>  GenerateInstanceResource(desired[resource.Name(fmt.Sprintf(&quot;instance-%v&quot;, i))], oxr, observed, tags, i) <br>}</pre><p>Running Crossplane functions locally with the crossplane beta render CLI command allows us to validate that the rendering of resources is correct before deploying to a Kubernetes cluster. The ability to unit test and eventually integration test our compositions saves us from errors caused by contracts breaking between what is expected in the Composite Resource (XR) or in EnvironmentConfigs and what we have deployed.</p><p>Here is the output from crossplane beta render command for an S3 Bucket Composition:</p><pre>apiVersion: imaginelearning.engineering/v1alpha1 <br>kind: XS3Bucket <br>metadata: <br>  labels: <br>    cluster: platform-production <br>    domain: platform <br>    product: sandbox <br>  managedFields: null <br>  name: sandbox-dev-sample-python-bucket-12easda <br>spec: <br>  claimRef: <br>    apiVersion: imaginelearning.engineering/v1alpha1 <br>    kind: S3Bucket <br>    name: sandbox-dev-sample-python-bucket <br>    namespace: sandbox-dev <br>  resourceConfig: {} <br>status: <br>  bucketArn: arn:aws:s3:::sandbox-dev-sample-python-bucket <br>  bucketName: sandbox-dev-sample-python-bucket <br>--- <br>apiVersion: s3.aws.upbound.io/v1beta1 <br>kind: BucketServerSideEncryptionConfiguration <br>metadata: <br>  annotations: <br>    crossplane.io/composition-resource-name: bucket-ses <br>  generateName: sandbox-dev-sample-python-bucket-12easda- <br>  labels: <br>    crossplane.io/claim-name: sandbox-dev-sample-python-bucket <br>    crossplane.io/claim-namespace: sandbox-dev <br>    crossplane.io/composite: sandbox-dev-sample-python-bucket-12easda <br>  managedFields: null <br>  name: sandbox-dev-sample-python-bucket <br>  ownerReferences: <br>  - apiVersion: imaginelearning.engineering/v1alpha1 <br>    blockOwnerDeletion: true <br>    controller: true <br>    kind: XS3Bucket <br>    name: sandbox-dev-sample-python-bucket-12easda <br>    uid: &quot;&quot; <br>spec: <br>  deletionPolicy: Orphan <br>  forProvider: <br>    bucketSelector: <br>      matchControllerRef: true <br>    region: us-east-2 <br>    rule: <br>    - applyServerSideEncryptionByDefault: <br>      - sseAlgorithm: AES256 <br>  providerConfigRef: <br>    name: aws-provider <br>--- <br>apiVersion: s3.aws.upbound.io/v1beta1 <br>kind: Bucket <br>metadata: <br>  annotations: <br>    crossplane.io/composition-resource-name: bucket <br>  generateName: sandbox-dev-sample-python-bucket-12easda- <br>  labels: <br>    crossplane.io/claim-name: sandbox-dev-sample-python-bucket <br>    crossplane.io/claim-namespace: sandbox-dev <br>    crossplane.io/composite: sandbox-dev-sample-python-bucket-12easda <br>  managedFields: null <br>  name: sandbox-dev-sample-python-bucket <br>  ownerReferences: <br>  - apiVersion: imaginelearning.engineering/v1alpha1 <br>    blockOwnerDeletion: true <br>    controller: true <br>    kind: XS3Bucket <br>    name: sandbox-dev-sample-python-bucket-12easda <br>    uid: &quot;&quot; <br>spec:<br>  deletionPolicy: Orphan <br>  forProvider: <br>    region: us-east-2 <br>    tags: <br>      cluster: platform-production <br>      domain: platform <br>      env: dev <br>      name: sandbox-dev-sample-python-bucket <br>      namespace: sandbox-dev <br>      product: sandbox <br>  providerConfigRef: <br>    name: aws-provider</pre><h3>Conclusion</h3><p>Building your internal developer platform (IDP) control plane with Crossplane using Composition Functions allows you to build with confidence. Imagine Learning has leveraged Composition Functions to reduce bugs and the toil of maintaining thousands of lines of YAML configuration. Adding more resources to Composition Function coverage, evaluating how to automate day two operations, and having a more thorough end-to-end test suite with a local Kubernetes cluster are the next steps we will take to bring our internal developer platform to the next level.</p><p>Blake Romano is a Software Engineer at Imagine Learning who leads the Internal Developer Platform Team to build an Internal Developer Platform to serve hundreds of engineers.</p><p><em>Originally published at </em><a href="https://blog.crossplane.io/building-crossplane-composition-functions-to-empower-your-control-plane/"><em>https://blog.crossplane.io</em></a><em> on March 7, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25532e577538" width="1" height="1" alt=""><hr><p><a href="https://medium.com/imaginelearning/how-imagine-learning-uses-crossplane-composition-functions-25532e577538">How Imagine Learning uses Crossplane &amp; Composition Functions</a> was originally published in <a href="https://medium.com/imaginelearning">Imagine Learning Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS OpenSearch Serverless VS Provisioned]]></title>
            <link>https://medium.com/@blakeromano/aws-opensearch-serverless-vs-provisioned-b5385d6b20d8?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5385d6b20d8</guid>
            <category><![CDATA[amazon-opensearch]]></category>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[aws-opensearch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Sun, 02 Jul 2023 17:56:38 GMT</pubDate>
            <atom:updated>2023-07-02T17:56:38.113Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lzmfxXPK0I1HdXP_.png" /></figure><p>AWS OpenSearch is an AWS Managed Service to provision OpenSearch clusters. OpenSearch is the open source AWS fork of ElasticSearch. Commonly used for log analytics, search it is a powerful tool to store, query, and analyze data. There is more use cases emerging for things like trace observability using OpenTelemetry, machine learning for vector storage and anomaly detection and more.</p><p>Before Re:Invent 2022, AWS OpenSearch was only able to be deployed as a Domain which is an OpenSearch cluster, defining things like master nodes, volume settings like IOPS, encryption settings, etc;. OpenSearch and ElasticSearch are fairly complex to manage yourself. Thankfully, AWS does a fairly good job with provisioned domains with giving you the data you need in order to manage the cluster as well as opening up the ability to use packages to add additional features.</p><p>At Re:Invent 2022, AWS OpenSearch Serverless was announced, with some skepticism. One of the big things people were talking about is the minimum around $700/mo that OpenSearch Serverless would start at. Now that the dust has settled the question is, what is the value of Serverless for OpenSearch, what is the right use case?</p><p>Let’s start with the pros:</p><ol><li>Really easy to setup, you start with a collection and use policies to configure the security, networking and encryption.</li><li>Can use IAM or SAML for access control, the lack of allowing username/passwords really a positive to see from a security point of view.</li><li>OCUs can be shared across clusters. If you have multiple collections this can lower the cost of specific collections.</li><li>CloudWatch Metrics just focused on the performance of indexing, search, and errors. This makes understanding performance really simple.</li></ol><p>Now the cons:</p><ol><li>Can’t use packages. If you have a use case for using the ML plugin or other things then that will not be a good use for Serverless. <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html#serverless-limitations">There are other limitations as well.</a></li><li>Large datasets can be not so performant with Serverless. When you start going over 1TB or so with OpenSearch Serverless performance can be lack luster, this is something being worked on however as of right now for large datasets Serverless may not be the best choice.</li></ol><p>Now that we have gone through the pros and cons what is the right answer? Like everything else it really does depend on your use case. For me if I have several clusters with datasets under 1TB that don’t need to use plugins, I’d go OpenSearch Serverless. The reduction in complexity and autoscaling features are quite compelling. However for more advanced or larger use cases I think for now provisioned domains do still win out. If you are using AWS OpenSearch Domains, ensure you are using OpenSearch Reserved Instances to reduce cost. The pros and cons could definitely be changed as more features are released for Serverless and performance/cost is improved.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5385d6b20d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Trio — EKS, Karpenter and Fargate]]></title>
            <link>https://medium.com/@blakeromano/the-trio-eks-karpenter-and-fargate-9b8f1383a640?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b8f1383a640</guid>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[karpenter]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Sat, 10 Jun 2023 16:33:21 GMT</pubDate>
            <atom:updated>2023-06-10T16:33:21.986Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Trio — EKS, Karpenter and Fargate</h3><figure><img alt="EKS, Fargate and Karpenter Logos" src="https://cdn-images-1.medium.com/max/1024/1*xaN4W-Jq1mzr-2yQvk6vlA.png" /><figcaption>EKS, Fargate and Karpenter</figcaption></figure><p>AWS has continued to push the business case as to why to adopt Kubernetes. The continued contributions like IRSA (IAM Roles for Service Accounts), CSI Drivers, AWS IAM Authenticator and more recently Karpenter, has continued to make the integration between Kubernetes and the AWS ecosystem much more cohesive.</p><p>Karpenter is an Open Source tool built by AWS to do more efficient autoscaling by provisioning instances by themselves based on the current need of the cluster and not by using autoscaling groups like the default cluster autoscaler will do. This has made node provisioning faster but also much more cost efficient. Karpenter has also added features to integrate with AWS Health to listen for node health events, spot interruptions, to deal with things before they happen. There is also configuration you can setup to set TTL for nodes to ensure nodes are not long lived and to be able to shuffle nodes around for cost savings.</p><p>You may be asking yourself why not just use Fargate for my entire cluster, and there is a number of reasons in my mind why Fargate may not be the best choice for your entire cluster, however this really depends on your use case. Fargate doesn’t support Daemonsets, if you use things like Fluentd, Cilium, etc; they are not going to work on Fargate nodes, one workaround is to run them as sidecars. Sidecars at scale can be messy so this is something you’ll want to be careful of. If you do need access to the host for things like eBPF, a service mesh, etc; Fargate will not allow those systems to get access to the parts of the host it needs to operate. Take a look at some of the other limitations of Fargate first, and ensure that your use case will or won’t work and be cost efficient before making a decision.</p><p>Karpenter runs on normal instances and <strong>not</strong> on the control plane, this means the way you deploy Karpeneter has to be resilient. What is the best practice is to deploy it on to either Fargate nodes, or ASG based EC2 instances, and what is a better use case for Fargate then workloads that don’t need to be integrated with a service mesh, or some of the other tooling you’d use on an EC2 instance. Fargate will be reliable, and for each pod has a 99.5% uptime per month, which means at most you have a four hour outage per month. You should ensure you have a Disaster recovery plan if Fargate fails to work or does have an outage however for the majority of the time Fargate works well and as expected.</p><p>There are even more ways to optimize compute with tools like Kube Descheuler. One of those being a Kubernetes SIG (special interest group) project called Kube Descheduler. This tool allows you to setup policies on when it should try and optimize (desired amount of CPU/Memory, if pod has high amount of restarts, etc;).</p><p><strong><em>More Resources:</em></strong></p><ul><li><a href="https://github.com/kubernetes-sigs/descheduler">GitHub - kubernetes-sigs/descheduler: Descheduler for Kubernetes</a></li><li><a href="https://karpenter.sh/">Karpenter</a></li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/fargate.html">Simplify compute management with AWS Fargate</a></li><li><a href="https://aws.amazon.com/ecs/sla/">Amazon Elastic Container Service (Amazon ECS) and AWS Fargate SLA</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b8f1383a640" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability First Development]]></title>
            <link>https://medium.com/@blakeromano/observability-first-development-5afe29a5c429?source=rss-9667524f196a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5afe29a5c429</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Blake Romano]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 15:08:49 GMT</pubDate>
            <atom:updated>2023-01-02T15:08:49.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*TvePa-OWBWMn_SF7fYTWCw.png" /></figure><p>How many times do you write code and think this will work as expected. It works locally, it has unit tests, maybe even integration tests and it all passes. When your new code goes to a live environment like production there is some new weird unexpected behavior going on with your software, and you can’t figure out why it may be going on. Is it an infrastructure issue that the DevOps team needs to examine? Is it a code issue unable to handle a real-world workload? What line is causing the issue? Is it an issue with a downstream service that is causing the issue? All of these questions become so difficult to solve if you don’t develop your software with observability in mind. Let’s see how we can build our software with observability first.</p><p>For a little pre-context let’s explore what is observability. There are three main pillars of observability, logs, metrics, and traces. Logs are the messages that are output by the software. Metrics are points of measurement that are exported by the software. Finally, traces are the specific instances of execution of a request through your software. All three can be used to help debug issues and are things that we need to have in mind when writing code.</p><p>Let’s start off with logs. How many times have you looked at logs that are either way too verbose, have data that isn’t valuable, has way too little information on what is going on, or is formatted in such a manner that you can’t find a specific set of logs that you need? These are all issues modern software should try and overcome. Typically logs formatted in a JSON format are great as software like ElasticSearch can parse that JSON and you can search for specific fields which can make debugging through logs much easier. To fix issues about verbosity having levels of logging that can be changed quickly can be vital for either removing verbosity when it isn’t needed or added when you need it for a specific problem.</p><p>Next let’s look at metrics. Engineers typically can use metrics to understand, how many requests per second is my application receiving, how much CPU/Memory are my applications using, etc;. However custom metrics can provide lots of value and underutilized. For example if you are developing an authentication server, you may want to emit metrics for the amount of users logging in based on location, how many authentication requests failed for bad passwords, how many successful logins there were, and much more. These things can not only help you detect when a problem is happening, but where it may be happening, and could even provide the business side much more understanding about how your user base is interacting with the product.</p><p>Last but not least we can explore traces. Traces are valuable to understand a workflow through an application or even a product with distributed tracing. However you need a way to propagate that trace’s context throughout not just one team’s application but potentially several. This is where a trace context propagation mechanism is vital. For example, <a href="https://www.w3.org/TR/trace-context/">W3C has created a standard for trace context and how to define it</a>. Having an organization wide approach to trace context propagation can be vital in helping understand in a distributed environment where issues are stemming from.</p><p>Now that we understand why the three pillars are vital, ways to implement them for the most observability, how do we actually do it in practice.</p><ul><li>Talk about observability with the engineering organization, come up with a plan on how you will execute on it.</li><li>Ensure custom metrics are valuable and continually review to see if metrics become less valuable and should be removed.</li><li>Review logs to determine if verbosity levels are appropriate, and ensure that development teams are emitting logs that are parsable, and easy to query</li><li>Meet with product managers, and other stakeholders as to why engineers need to dedicate time to observability, how it will reduce MTTR, improve detection of issues, and reduce the number of incidents.</li><li>Explore how sharing observability data outside of the engineering department may provide value to product teams, customer support, or potentially other teams.</li></ul><p>I’d recommend you review <a href="https://prometheus.io/">Prometheus</a>, <a href="https://opentelemetry.io/">OpenTelemetry</a> for more information about how you may instrument your applications, and send that data to whichever observability tool your company uses (DataDog, New Relic, Grafana, etc;).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5afe29a5c429" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>