<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Protagona - Medium]]></title>
        <description><![CDATA[Protagona was founded by passionate technologists and practitioners who have been helping organizations adopt the cloud for over a decade. Our focus on the role people play in the long-term success of a cloud transformation has led to amazing outcomes for our clients. - Medium]]></description>
        <link>https://medium.com/protagona?source=rss----4ad8bc182044---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Protagona - Medium</title>
            <link>https://medium.com/protagona?source=rss----4ad8bc182044---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:04:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/protagona" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Brief Guide to Amazon SageMaker Algorithms: Time Series & Computer Vision]]></title>
            <link>https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-time-series-computer-vision-d15ea380e3d7?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/d15ea380e3d7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-data]]></category>
            <category><![CDATA[sagemaker-pipelines]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[sagemaker]]></category>
            <dc:creator><![CDATA[Aaron West]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 16:35:01 GMT</pubDate>
            <atom:updated>2025-02-04T16:35:01.570Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IG9ye9IIWHp6zamX4Nj15w.jpeg" /><figcaption>Image produced with Amazon Bedrock</figcaption></figure><p>This article is the fourth and final installment in a series exploring Amazon SageMaker’s machine learning algorithms, focusing on those tailored for time series forecasting and computer vision. It covers key concepts, functionalities, and practical applications of algorithms such as I<strong>mage Classification</strong>, <strong>Object Detection</strong>, <strong>Semantic Segmentation</strong>, and<strong> DeepAR</strong>. From leveraging recurrent neural networks for precise time series predictions to utilizing convolutional neural networks for advanced image analysis, this guide provides a comprehensive overview of SageMaker’s capabilities in these domains.</p><p>Whether you’re aiming to forecast demand trends across products or enable machines to interpret and act on visual data, these tools offer innovative solutions within the AWS ecosystem. As with the previous entry, actionable code examples accompany the explanations. If you missed earlier parts, <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-text-processing-95eb5aa5301a"><strong>Part 1</strong></a> covers text processing, <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-tabular-data-27e15605e497"><strong>Part 2</strong></a> delves into structured data, and <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-unsupervised-learning-31577b5b65a7"><strong>Part 3</strong></a> highlights unsupervised learning. This final chapter concludes the series, offering a focused resource for professional development and practical applications in cloud-based machine learning.</p><h3>Image Classification</h3><p>SageMaker offers two supervised learning algorithms for image classification, one based on MXNet and another utilizing TensorFlow. Both algorithms accept images in .jpg, .jpeg, or .png formats and outputs probabilities for each class label. They support training from scratch or through transfer learning, allowing fine-tuning of pre-trained models even with limited datasets.</p><h4><strong>MXNet Image Classification</strong></h4><p>This algorithm employs convolutional neural networks (CNNs) for classifying images. It supports multi-label classification, enabling the assignment of multiple labels to a single image. Training data can be provided in Apache MXNet RecordIO format or as individual image files. For optimal performance, GPU instances such as ml.p2 and ml.p3 are recommended for training, while both CPU and GPU instances can be used for inference.</p><h4><strong>TensorFlow Image Classification</strong></h4><p>Leveraging pre-trained models from TensorFlow Hub, this algorithm facilitates transfer learning by attaching a classification layer to the selected model. The classification layer includes a dropout layer and a dense layer with L2 regularization. Users can fine-tune the entire network or just the classification layer based on their specific datasets. The algorithm supports all CPU and GPU instances for training, with a recommendation for GPU instances with more memory when using large batch sizes.</p><h3>Object Detection</h3><p>SageMaker also provides object detection algorithms using MXNet and TensorFlow, designed to identify and localize objects within images by outputting bounding boxes, class labels, and confidence scores for detected objects. Both algorithms are designed to identify and localize objects within an image by outputting bounding boxes, class labels, and confidence scores for detected objects. These algorithms support training from scratch or transfer learning, allowing users to adapt pre-trained models to their specific datasets.</p><h4><strong>MXNet Object Detection</strong></h4><p>The MXNet Object Detection algorithm is based on the Single Shot Multibox Detector (SSD) framework and uses convolutional neural networks (CNNs) like VGG-16 and ResNet-50 as base networks. This algorithm supports tasks where identifying the position and scale of multiple objects within a single image is critical. Input data can be provided in Apache MXNet RecordIO format or as individual image files in .jpg or .png formats.</p><p>To achieve optimal training performance, AWS recommends using GPU instances such as ml.p2 and ml.p3. For inference, both CPU and GPU instances are supported, providing flexibility for different deployment needs. This algorithm’s design allows it to scale well for complex object detection use cases.</p><h4><strong>TensorFlow Object Detection</strong></h4><p>The TensorFlow Object Detection algorithm supports pre-trained models from the TensorFlow Model Garden, such as MobileNet, ResNet, Inception, and EfficientNet. These models can be fine-tuned for object detection tasks using custom datasets, enabling precise identification and localization of objects. The algorithm adds a detection head to the base model, allowing it to output bounding boxes alongside class labels and confidence scores.</p><p>The TensorFlow algorithm accepts input images in .jpg, .jpeg, or .png formats and supports all GPU instances for training, with a recommendation to use higher-memory GPUs for large batch sizes. Both CPU and GPU instances can be used for inference, making it adaptable to various environments and workflows.</p><h3>Semantic Segmentation</h3><p>SageMaker’s Semantic Segmentation algorithm is a solution designed for <strong>detailed image analysis tasks</strong>, offering a <strong>pixel-level approach</strong>, assigning each pixel a class label from a predefined set. This method is ideal for applications requiring a detailed understanding of the scene, such as autonomous driving, medical images, and robot vision. They leverage deep neural network architectures; among the available selections are <strong>Fully-Convolutional Network (FCN)</strong>, <strong>Pyramid Scene Parsing (PSP)</strong>, and <strong>DeepLabV3</strong>.</p><p>Because the algorithm classifies every pixel in an image, it also provides information about the shapes and boundaries of objects in the scene. The output is called a segmentation mask, which is a grayscale image where each pixel corresponds to a class label. The architecture consists of a backbone (encoder) that extracts feature maps and a decoder that generates the segmentation mask. Supported backbones include ResNet50 and ResNet101, which can be fine-tuned using pretrained weights from ImageNet or trained from scratch with your data. Segmentation masks can be output as PNG images or probability maps, making the algorithm flexible for various computer vision applications.</p><ul><li>🧠 Uses deep learning algorithms to analyze images and identify objects.</li><li>🖼️ Tags each pixel in an image with a class label using predefined classes, enabling fine-grained image analysis.</li><li>🤖 Supports deep neural network architectures: Fully-Convolutional Network (FCN), Pyramid Scene Parsing (PSP), and DeepLabV3.</li><li>📊 Supports both file and pipe input modes for efficient data processing during training.</li><li>📂 Requires data in S3, organized into train and validation channels, with PNG format for images and annotations.#### Algorithm Choices and Components</li></ul><h3>DeepAR</h3><p>SageMaker’s DeepAR is a supervised learning algorithm tailored for time series forecasting, offering a robust solution for predicting scalar time series data. By leveraging recurrent neural networks (RNNs), DeepAR captures temporal dependencies and shared patterns across multiple related time series, enabling highly accurate forecasts. Unlike traditional methods that treat each time series independently, DeepAR trains a single model on related datasets, making it particularly effective in scenarios where multiple time series share similar dynamics, such as retail demand forecasting, server load predictions, or financial analysis.</p><ul><li>📊 <strong>DeepAR excels in scenarios with numerous related time series</strong>, such as predicting demand across multiple products or server loads.</li><li>🔄 <strong>Uses RNNs to process sequential data</strong>, capturing temporal dependencies and long-term patterns.</li><li>📚 <strong>Trains on related time series datasets</strong>, learning shared underlying patterns and relationships.</li><li>🔮 <strong>During prediction</strong>, DeepAR generates forecasts for both existing and new time series that exhibit similar characteristics to the training data.</li></ul><h3>SageMaker Workflows</h3><p>SageMaker Workflows automate machine learning pipelines, ensuring a repeatable and scalable process for tasks such as data preprocessing, model training, and deployment. Key benefits include:</p><ul><li><strong>Automation</strong>: Simplifies end-to-end ML workflows, reducing manual steps.</li><li><strong>Scalability</strong>: Enables efficient handling of large datasets and multiple ML processes.</li><li><strong>Reproducibility</strong>: Tracks and manages pipeline runs, ensuring consistent results.</li></ul><h4><strong>SageMaker Workflow Example</strong></h4><p>The following pipeline demonstrates a complete SageMaker workflow for time series forecasting using the DeepAR algorithm. This example includes data preprocessing, model training, and pipeline execution.</p><pre>import sagemaker<br>from sagemaker.workflow.steps import ProcessingStep, TrainingStep<br>from sagemaker.workflow.pipeline import Pipeline<br>from sagemaker.processing import ScriptProcessor, ProcessingOutput, ProcessingInput<br>from sagemaker.estimator import Estimator<br>from sagemaker.inputs import TrainingInput<br>from sagemaker.workflow.parameters import ParameterString<br>from sagemaker.amazon.amazon_estimator import get_image_uri<br>from sagemaker.experiments.run import Run<br><br># Initialize SageMaker session and define role<br>session = sagemaker.Session()<br>role = &quot;arn:aws:iam::your-account-id:role/forecasting-role&quot;<br>bucket_uri = &quot;s3://your-bucket-name&quot;<br><br># Define input data parameter<br>input_data_uri = ParameterString(<br>    name=&quot;input_data_uri&quot;,<br>    default_value=f&quot;{bucket_uri}/AAPL.csv&quot;,<br>)<br><br># Processing step for data preparation<br>processor = ScriptProcessor(<br>    role=role,<br>    image_uri=&quot;763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:1.2-1-cpu-py3&quot;,<br>    instance_count=1,<br>    instance_type=&quot;ml.m5.large&quot;,<br>    command=[&quot;python3&quot;],<br>)<br><br>processing_step = ProcessingStep(<br>    name=&quot;TimeSeriesDataProcessing&quot;,<br>    processor=processor,<br>    inputs=[<br>        ProcessingInput(<br>            source=input_data_uri,<br>            destination=&quot;/opt/ml/processing/input/&quot;,<br>        )<br>    ],<br>    outputs=[<br>        ProcessingOutput(<br>            output_name=&quot;processed_data&quot;,<br>            source=&quot;/opt/ml/processing/output&quot;,<br>            destination=f&quot;{bucket_uri}/processed_data&quot;,<br>        )<br>    ],<br>    code=&quot;scripts/processing.py&quot;,<br>)<br><br># Training step using DeepAR<br>estimator = Estimator(<br>    image_uri=get_image_uri(session.boto_region_name, &quot;forecasting-deepar&quot;),<br>    role=role,<br>    instance_count=1,<br>    instance_type=&quot;ml.m5.large&quot;,<br>    output_path=f&quot;{bucket_uri}/output&quot;,<br>    hyperparameters={<br>        &quot;time_freq&quot;: &quot;H&quot;,<br>        &quot;context_length&quot;: &quot;40&quot;,<br>        &quot;prediction_length&quot;: &quot;20&quot;,<br>        &quot;num_cells&quot;: &quot;40&quot;,<br>        &quot;num_layers&quot;: &quot;2&quot;,<br>        &quot;likelihood&quot;: &quot;gaussian&quot;,<br>        &quot;epochs&quot;: &quot;100&quot;,<br>        &quot;mini_batch_size&quot;: &quot;32&quot;,<br>        &quot;learning_rate&quot;: &quot;0.001&quot;,<br>    },<br>)<br><br>training_step = TrainingStep(<br>    name=&quot;TimeSeriesTraining&quot;,<br>    estimator=estimator,<br>    inputs={<br>        &quot;train&quot;: TrainingInput(<br>            s3_data=f&quot;{bucket_uri}/processed_data&quot;,<br>        )<br>    },<br>)<br><br># Create pipeline<br>pipeline = Pipeline(<br>    name=&quot;TimeSeriesPipeline&quot;,<br>    steps=[processing_step, training_step],<br>    parameters=[input_data_uri],<br>)<br><br># Execute the pipeline<br>pipeline.upsert(role_arn=role)<br>execution = pipeline.start(<br>    parameters={&quot;input_data_uri&quot;: f&quot;{bucket_uri}/AAPL.csv&quot;}<br>)<br>print(execution.arn)</pre><p>The ScriptProcessor is responsible for preprocessing raw time series data, including tasks such as cleaning, formatting, and optimizing the input data into a structure suitable for training. It’s important to align the preprocessing steps with your specific inference goals, as these goals often dictate how the data should be encoded and structured to meet situational requirements effectively.</p><p>The ProcessingStep encapsulates the data preprocessing logic, linking the raw input data from S3 to the ScriptProcessor and storing the processed output back in S3. This step ensures the pipeline can handle raw datasets in a repeatable and modular fashion, making it easy to adapt to changes in data formats or preprocessing requirements.</p><p>The TrainingStep defines the training process using SageMaker’s built-in DeepAR algorithm. The Estimator specifies the training job’s configuration, including the algorithm container, compute instance type, and hyperparameters. The processed data from the ProcessingStep is used as the training input, ensuring a seamless handoff between steps in the pipeline.</p><h3><strong>Further Resources</strong></h3><p>For those looking to deepen their understanding of SageMaker workflows and algorithms, the official <a href="https://sagemaker.readthedocs.io/en/stable/algorithms"><strong>Amazon SageMaker SDK documentation</strong></a> is going to have the most up-to-date information. It provides detailed guidance on configuring pipelines, tuning model hyperparameters, and deploying scalable solutions tailored to real-world machine learning tasks.</p><p>The <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html">a<strong>lgorithms section</strong></a> offers an in-depth look at all of SageMaker’s built-in algorithms, including Image Classification, Object Detection, Semantic Segmentation, and DeepAR. This resource covers critical information such as input/output specifications, parameter details, and implementation strategies essential for production-ready solutions.</p><p>For hands-on examples, the <a href="https://github.com/aws/amazon-sagemaker-examples"><strong>Amazon SageMaker Examples repository</strong></a> contains several Jupyter notebooks showcasing how to implement SageMaker workflows, train models, and integrate them into pipelines.</p><p>More examples of SageMaker workflows and algorithms, including implementations of the DeepAR algorithm, can be found in the <a href="https://github.com/aws/amazon-sagemaker-examples"><strong>Amazon SageMaker Examples repository</strong></a>. If you’re interested in exploring some of my work, feel free to check out my <a href="https://github.com/aphexlog"><strong>GitHub</strong></a>. Here is the <a href="https://github.com/aphexlog/sagemaker-examples/blob/main/examples/deep-ar/pipeline.py#L1-L1">code</a> that that I wrote to demonstrate the workflow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d15ea380e3d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-time-series-computer-vision-d15ea380e3d7">Brief Guide to Amazon SageMaker Algorithms: Time Series &amp; Computer Vision</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Production Line to Pipeline: A Cloud Engineer’s Story]]></title>
            <link>https://medium.com/protagona/from-production-line-to-pipeline-a-cloud-engineers-story-1c414edf462e?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/1c414edf462e</guid>
            <category><![CDATA[career-growth]]></category>
            <category><![CDATA[cloud-engineering]]></category>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[tech-journey]]></category>
            <category><![CDATA[professional-development]]></category>
            <dc:creator><![CDATA[Paul Clark IV]]></dc:creator>
            <pubDate>Fri, 13 Dec 2024 18:08:27 GMT</pubDate>
            <atom:updated>2024-12-13T18:08:27.199Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*Eq7iuCf7lVuzEoGZ2HmSEw.png" /></figure><h3>How I Went from Assembly Line to Cloud Nine in Six Months</h3><p>Hey there! I’m Paul Clark, and I’m about to share the journey that took me from the assembly line to the cloud in just six months. It’s a story about determination, late-night study sessions, and proving that with the right mindset, any career pivot is possible.</p><p>Picture this: September 2022, I’m fresh out of my assembly line job, staring at AWS documentation like it’s written in ancient Greek. Fast forward six months, and I’m building cloud architectures with the same confidence I used to have putting together cars — except this time, I actually get excited about Monday mornings.</p><p>Let’s be honest — when I first stepped into cloud engineering, I felt like I’d walked into a foreign movie without subtitles. Everyone around me was throwing around terms like “containerization” and “infrastructure as code” while I was still trying to figure out what EC2 actually meant (fun fact: I may have googled “What is the cloud?” more times than I’d like to admit).</p><p>But here’s the thing — if someone who used to spend their days on an assembly line can transform into a cloud engineer in less time than it takes to plan a wedding, you can too. This isn’t just another success story to add to your LinkedIn feed. This is your behind-the-scenes look at what it really takes to break into tech and actually make it stick. But one thing I can not stress enough… IT WILL TAKE CONSISTENCY AND HARD WORK!</p><p>So grab your coffee (trust me, you’ll need it), get comfortable, and let me show you how I completely transformed my career. Whether you’re just starting out, thinking about making the switch, or just curious about what cloud engineers actually do all day (hint: we do more than stare at clouds), I’ve got something for you.</p><p>And I promise you this — I’m not going to gloss over the hard parts. We’re talking real struggles, those heart-stopping moments when you think you’ve broken something important, and the kind of victories that make you want to do the happy dance. Yeah, I’ve done that. Multiple times. No regrets at all.</p><p>Ready to see how determination and an endless supply of patience turned an assembly line worker into a cloud engineer? Let’s dive in! 🚀</p><h3>Where It All Started: The Assembly Line Days</h3><p>Remember that feeling when you know you’re meant for something different? That was me, standing on the assembly line, watching the same parts move past me day after day. Don’t get me wrong — it was a stable job at one of the top car manufacturers in the US, and I’m grateful for that experience. But something just wasn’t clicking (and I’m not talking about the car parts). Here’s what really got to me: no matter how hard I worked or how many innovative ideas I brought to the table, my growth was tied to time served, not effort given. It was like being in a video game where you can only level up by waiting, not by actually playing better. That didn’t sit right with me.</p><p>The real turning point came during a family gathering when I got into a conversation with my cousin, who happens to be a cloud engineer. Listening to him talk about his work was like getting a glimpse into a completely different world — one where every day brought new challenges and opportunities to learn.</p><p>Funny enough, after that conversation, cloud computing started showing up everywhere in my YouTube feed (talk about timing, right?). One particularly afternoon, while watching the millionth part (okay, I might be exaggerating) move down the assembly line, I found myself diving deeper into these videos on my phone. The more I learned about cloud computing, the more it grabbed me — the constant evolution, the problem-solving aspects, and honestly, the fact that it was about as different from assembly line work as you could get. It felt like the universe was dropping some not-so-subtle hints about my next career move.</p><h3>The Prep Work: Building My Launch Pad</h3><p>Everyone says “preparation is key”? And they’re not wrong, but they usually skip the part about how overwhelming it can be. When I first started preparing for my cloud journey, I felt like I was standing in front of a massive buffet with a tiny plate — where do you even start?</p><p>I decided to treat learning cloud like learning a new language. You don’t start by writing poetry; you begin with the basics. My first step? Diving into online courses. After trying out several learning platforms, I found one that really clicked with me. The right platform makes all the difference — I needed instructors who could break down complex concepts and an active community where I could get help when I was stuck (which, let’s be honest, was pretty often in the beginning).</p><p>Considering I was on the night shift (10pm-6am) Here’s what my daily routine looked like:</p><ul><li>Wake up around 11am-12am (painful, but worth it)</li><li>Play with my 2 year old son while my wife was at work.</li><li>Squeeze in 2–3 hours of focused study before going into work later that night (During my sons naps of course!)</li><li>Weekends split between hands-on practice and actually having a life</li><li>Late-night documentation reading sessions (my phone’s night mode got a serious workout)</li></ul><p>Let’s talk sacrifices because nobody likes to mention this part. Yes, I missed some social events. Yes, my Netflix watch list grew longer without being touched. And yes, my friends and family probably thought I’d gone a bit crazy with all the cloud talk. But here’s the thing — temporary sacrifices for permanent upgrades aren’t really sacrifices at all, they are investments.</p><p>And those investments? They paid off bigger than I could have imagined. After months of juggling night shifts, daddy duties, and non-stop learning, an amazing opportunity landed in my lap. A fantastic team saw the potential in my dedication and took a chance on me, offering me my first role as a cloud engineer. Looking back now, every missed social event, every late-night study session, and every moment spent reading docs while my son napped — it all led to that life-changing moment. Sometimes the best opportunities come when you’re busy proving to yourself that you deserve them.</p><h3>The First Steps: Stumbling Through the Clouds</h3><p>Remember that feeling of getting a new video game without reading the tutorial? That was me in my first few weeks as a junior cloud engineer. Except this wasn’t a game, and there was no reset button.</p><p>Let me tell you about my first major face-palm moment. Picture this: I’m sitting at my desk, feeling pretty confident after successfully setting up my first Visual Studio Code server. Everything’s going great until I refresh the browser and — poof! — all my terminal data vanishes like my morning coffee. Cue the internal panic mode.</p><p>After hours of trying everything I could think of (and maybe some things I shouldn’t have), my mentor casually takes a look and finds the solution in seconds. The missing piece? A simple “stickiness” configuration in Terraform. I felt like I’d been trying to open a door by pushing when there was a “pull” sign right in front of me.</p><h3>Quick Knowledge Drop: The Stickiness Situation</h3><p>Okay, let me break this down in a way I wish someone had explained it to me:</p><p>Think of an Application Load Balancer (ALB) as a really efficient restaurant host. Their job is to make sure customers (user requests) get seated at tables (servers) in a way that keeps the restaurant running smoothly. Now, “stickiness” is like telling that host, “Hey, when someone comes in, give them their own dedicated waiter for their entire meal.”</p><p>Without stickiness, it’s like getting a different waiter every time you ask for something — nobody remembers your order, and you have to start over each time. With stickiness, you get the same waiter who remembers everything about your meal. That’s exactly what my terminal needed — a consistent connection to the same server.</p><h3>The Growth Spurt: From Junior to Journey-man</h3><p>December 2022 marked a turning point. My company didn’t just keep me in training wheels — they threw me into the deep end by sending me to work with a client company. Talk about a “sink or swim” moment!</p><p>This wasn’t just about playing around in a test environment anymore. I was working on real platforms, solving real problems, and (maybe once or twice) creating real headaches that needed real solutions. But you know what? Those challenges were like protein shakes for my brain — they helped me grow stronger and more confident.</p><p>Each day brought new puzzles to solve:</p><ul><li>Monday: Why is this deployment failing?</li><li>Tuesday: Sweet, deployments passing now!</li><li>Wednesday: Wait, why is this deployment failing again?</li><li>Thursday: Time to explain specific cloud concepts to entirely different tech teams (confidently of course 🙂)</li><li>Friday: Debugging day (with extra coffee)</li></ul><h3>The Big Break: March 2023</h3><p>Remember that moment in movies when everything just clicks? That was March 2023 for me. After six months of grinding, learning, and probably drinking too much coffee, I got offered a position as a full cloud engineer.</p><p>It felt surreal — like someone had fast-forwarded through the hard parts. But looking back, every late-night study session, every failed deployment, and every “aha” moment had led to this. I wasn’t just the assembly line guy anymore; I was a cloud engineer.</p><h3>The Real Talk: Lessons Learned</h3><p>Let me drop some knowledge that I wish I’d known when I started:</p><ol><li><strong>Embrace the Confusion</strong>: If you’re not confused, you’re not learning anything new. Confusion is just your brain upgrading its software.</li><li><strong>Ask Questions Early</strong>: Don’t wait until you’ve spent five hours going down the wrong rabbit hole. Your team would rather spend five minutes explaining something than five hours fixing your “creative” solution.</li><li><strong>Build Your Support Network</strong>: Find people who get excited about cloud tech. Trust me, you’ll need someone to celebrate with when you finally figure out that tricky configuration.</li><li><strong>Keep Side Projects Going</strong>: They’re like your playground — a safe space to try new things without worrying about breaking production.</li></ol><h3>Looking Ahead: The Sky’s Not the Limit</h3><p>Here I am now, still learning, still growing, and still occasionally breaking things (but fixing them much faster!). The cloud industry moves fast, and that’s exactly why I love it. Every day brings something new to learn, a new problem to solve, or a new technology to explore.</p><p>To anyone thinking about making a similar jump: Do it. Seriously. Yes, it’s challenging. Yes, you’ll have moments of doubt. And yes, you’ll probably have to explain to your family why you’re so excited about something they can’t see or touch.</p><p>But if a guy who used to put together cars can make it in the cloud, so can you. The tech world needs more people who aren’t afraid to learn, who bring different perspectives, and who can explain complex things in simple terms (because trust me, that’s a superpower in this field).</p><h3>The Thank You Note</h3><p>Before I wrap this up, I’ve got to give credit where it’s due:</p><ul><li>To Protagona: You took a chance on someone whose only cloud experience was looking up at the sky. Thank you for believing in potential over experience.</li><li>To my mentors: Thanks for answering my endless questions and not laughing (too hard) at my rookie mistakes.</li><li>To my family and friends: Sorry for all the times I talked about cloud computing during dinner. Thanks for nodding along anyway.</li></ul><h3>The Sign-Off</h3><p>So there you have it — my journey from assembly line to cloud nine. It’s been quite a ride, and it’s nowhere near over. The cloud is constantly evolving, and so am I.</p><p>To anyone reading this who’s thinking about making a similar leap: Take that first step. Just do it. A year from now, you’ll wish you had started today.</p><p>Keep building, keep learning, and most importantly, keep pushing yourself beyond your comfort zone. That’s where the real magic happens.</p><p>Catch you in the clouds! ☁️</p><p><em>Paul Clark — Cloud Engineer (still can’t believe I get to write that)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c414edf462e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/from-production-line-to-pipeline-a-cloud-engineers-story-1c414edf462e">From Production Line to Pipeline: A Cloud Engineer’s Story</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brief Guide to Amazon SageMaker Algorithms: Unsupervised Learning]]></title>
            <link>https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-unsupervised-learning-31577b5b65a7?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/31577b5b65a7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws-sagemaker]]></category>
            <dc:creator><![CDATA[Aaron West]]></dc:creator>
            <pubDate>Tue, 10 Dec 2024 22:17:32 GMT</pubDate>
            <atom:updated>2025-01-17T23:30:29.261Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*wnzEfenCpo62d-XogbWIMQ.png" /><figcaption>Image produced with Amazon Bedrock</figcaption></figure><p>Imagine uncovering hidden customer segments without prior labels, detecting fraudulent network activity in real-time, or reducing a mountain of data into actionable insights — all without explicit human guidance. This is the power of <strong>unsupervised learning</strong>.</p><p>Welcome to the third installment in our series exploring Amazon SageMaker’s machine-learning algorithms. In the earlier entries, we tackled natural language processing and tabular data. Now, we venture into unsupervised learning, where data “speaks for itself,” revealing patterns, clusters, and anomalies that might otherwise go unnoticed.</p><p>From identifying anomalies in network traffic with <strong>IP Insights</strong> to grouping data with <strong>K-means</strong>, unsupervised learning unlocks transformative solutions for complex data challenges. This installment includes actionable code examples to get you up and running with SageMaker’s Estimator objects for these algorithms, offering a practical jumpstart to your implementations.</p><p>If you’re new to this series, check out <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-text-processing-95eb5aa5301a"><strong>Part 1</strong></a> on text processing and <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-tabular-data-27e15605e497"><strong>Part 2</strong></a> on tabular data. In <a href="https://medium.com/@aphexlog/brief-guide-to-amazon-sagemaker-algorithms-time-series-computer-vision-d15ea380e3d7">P<strong>art 4</strong></a>, we tackle time series data and computer vision.</p><h3>IP Insights</h3><p>Amazon SageMaker IP Insights is an unsupervised learning algorithm designed to understand the usage patterns of IPv4 addresses. The algorithm learns associations between IP addresses and entities like user IDs, revealing how these entities interact with online resources. This information is crucial for various applications, including detecting unusual activities, understanding user behavior, and enhancing security measures.</p><ul><li>🌐 Learns usage patterns of IP addresses to understand how entities interact with online resources.</li><li>🤖 Unsupervised learning algorithm that identifies patterns without explicit labels.</li><li>🔍 Provides insights into IP address usage, aiding anomaly detection, user behavior analysis, and security improvements.</li></ul><h4><strong>Practical Applications</strong></h4><p>IP Insights is ideal for applications requiring anomaly detection and behavioral analysis involving IP addresses. Here are the most significant use cases:</p><ol><li><strong>Fraud Detection</strong>: Identifying suspicious login attempts or unusual access patterns by analyzing associations between IP addresses and user IDs.</li><li><strong>Security Monitoring</strong>: Detecting unusual traffic patterns, potential bot activity, and unauthorized access to online resources to improve system security.</li><li><strong>Preprocessing for Anomaly Detection Pipelines</strong>: Using IP Insights to generate embeddings or insights that serve as input features for downstream supervised anomaly detection algorithms, such as XGBoost or logistic regression.</li></ol><h4>Hyperparameters</h4><p>The IP Insights algorithm has two required hyper parameters:</p><ul><li><strong>num_entity_vectors: </strong>Specifies the number of unique entity embeddings the model should learn. A higher value is needed for datasets with many unique entities (e.g., user IDs). (e.g., num_entity_vectors=10000 creates embeddings for 10,000 unique entities)</li><li><strong>vector_dim</strong>: Defines the size (dimensionality) of each entity embedding vector. Larger dimensions allow the model to capture more complex patterns but increase computation time. (e.g., vector_dim=128 creates embeddings with 128 dimensions)</li></ul><p>It also supports additional optional hyperparameters, used to tune performance. Here are a couple:</p><ul><li><strong>epochs</strong>: The number of training iterations over the dataset. Increasing this value improves embedding accuracy but may lead to overfitting. (e.g., epochs=10 runs 10 full passes over the training data)</li><li><strong>learning_rate</strong>: Controls the step size for updating model weights during training. A smaller value improves convergence stability, while a larger value speeds up training. (e.g., learning_rate=0.001 uses a learning rate of 0.001)</li></ul><h4>Usage</h4><p>The following example demonstrates how to set up and train an IP Insights model in Amazon SageMaker to detect unusual patterns in IP address usage. The process includes initializing a SageMaker session, defining training and validation datasets stored in S3, and configuring the IP Insights algorithm with essential hyperparameters like <strong>entity embeddings</strong>, <strong>batch size</strong>, and <strong>learning rate</strong>.</p><p>Once trained, the model artifacts are stored in S3, making them ready for deployment and integration into real-world applications. Here’s a code example:</p><pre>import sagemaker<br>from sagemaker import get_execution_role<br>from sagemaker.amazon.amazon_estimator import get_image_uri<br><br># Initialize a SageMaker session<br>session = sagemaker.Session()<br><br># Get the execution role - this is used to give SageMaker access to your AWS resources<br>role = get_execution_role()<br><br># Specify the data paths in S3<br>training_data = &quot;s3://your-bucket/path/to/train/data&quot;  # persumably 80% of your data<br>validation_data = (<br>    &quot;s3://your-bucket/path/to/validation/data&quot;  # persumably 20% of your data<br>)<br><br># Get the IP Insights container image<br>container = get_image_uri(session.boto_region_name, &quot;ipinsights&quot;)<br><br># Create the IP Insights estimator<br>ip_insights = sagemaker.estimator.Estimator(<br>    container,<br>    role,<br>    instance_count=1,  # Number of instances to use for training<br>    instance_type=&quot;ml.c5.xlarge&quot;,  # Instance type for training<br>    output_path=&quot;s3://your-bucket/output&quot;,  # Output path for model artifacts<br>    sagemaker_session=session,<br>    # Hyperparameters for IP Insights<br>    hyperparameters={<br>        &quot;num_entity_vectors&quot;: &quot;10000&quot;,  # Number of entity embeddings<br>        &quot;vector_dim&quot;: &quot;128&quot;,  # Size of embeddings<br>        &quot;epochs&quot;: &quot;10&quot;,  # Number of training epochs<br>        &quot;learning_rate&quot;: &quot;0.001&quot;,  # Learning rate<br>    },<br>)<br><br># Specify the data channels for training<br>ip_insights.fit({&quot;train&quot;: training_data, &quot;validation&quot;: validation_data})</pre><p>In this example, the IP Insights algorithm processes training data stored in Amazon S3 to learn associations between IP addresses and entities. A validation dataset is used to evaluate the model’s generalization performance, ensuring reliable detection of unusual patterns in network behavior.</p><p>The final model artifacts are stored in S3 and can be deployed for real-world applications such as <strong>fraud detection</strong>, <strong>security monitoring</strong>, and <strong>behavioral analysis</strong>. The resulting embeddings can also be integrated into downstream machine learning models for further anomaly detection or classification tasks.</p><h3>K-means</h3><p>Amazon SageMaker’s K-means algorithm is an unsupervised learning technique <strong>designed</strong> to identify and group data into discrete clusters. It works by organizing data points so that members of the same cluster are as similar as possible, while members of different clusters are as distinct as possible. Users can customize the <strong>attributes the algorithm uses</strong> to determine similarity, making it a flexible option for various use cases.</p><ul><li>🎯 The K-means algorithm is an unsupervised learning algorithm.</li><li>🔍 The algorithm attempts to find discrete groupings within data based on similarity.</li><li>⚙️ Users define the attributes used to determine similarity.</li></ul><h4><strong>Practical Applications</strong></h4><p>K-means is widely used for clustering and grouping data into discrete clusters, making it a powerful tool for identifying patterns, segmenting data, and preparing datasets for other machine learning tasks. Here are the most practical examples:</p><ol><li><strong>Retail</strong>: Customer segmentation for personalized marketing, product recommendation engines, and identifying purchasing behavior patterns.</li><li><strong>Healthcare</strong>: Grouping patients based on medical records, identifying disease subtypes, and analyzing treatment response patterns.</li><li><strong>Preprocessing for Supervised Learning</strong>: Grouping data into clusters as a preprocessing step to improve performance in supervised learning tasks like classification or regression. For example, cluster assignments can be used as additional features for algorithms such as XGBoost or logistic regression.</li></ol><h4>Hyperparameters</h4><p>The K-means algorithm in SageMaker has two required hyperparameters:</p><ul><li><strong>k</strong>: Specifies the number of clusters to create. The algorithm groups data points into <strong>k</strong> clusters, where members of the same cluster are as similar as possible, and those in different clusters are distinct. (e.g., k=10 creates 10 clusters)</li><li><strong>feature_dim</strong>: Defines the total number of input features (dimensions) in the dataset. This must match the number of features in the input data. (e.g., feature_dim=784 assumes the dataset has 784 input features)</li></ul><p>It also supports additional optional hyperparameters that are used to optimize performance. Here are a couple:</p><ul><li><strong>mini_batch_size</strong>: The number of samples processed in each training batch. Larger batch sizes improve training speed but require more memory. (e.g., mini_batch_size=500 processes 500 samples per batch)</li><li><strong>init_method</strong>: Determines how the initial cluster centers are chosen. Options include random (random selection of initial points) and kmeans++ (optimized initialization). (e.g., init_method=’kmeans++’ uses the K-means++ initialization method for better convergence)</li></ul><h4>Usage</h4><p>Clustering algorithms like K-means are unsupervised, meaning they don’t rely on labeled data for training. In K-means clustering, each cluster has a defined center, and during training, the algorithm groups data points based on their distance to these cluster centers. The number of clusters, represented as <strong>k</strong>, is a key hyperparameter that users define.</p><p>For example, in the code below, we specify <strong>k = 10</strong> to create 10 clusters:</p><pre>container = get_image_uri(session.boto_region_name, &quot;kmeans&quot;)<br>sagemaker.estimator.Estimator(<br>        container,<br>        role,<br>        instance_count=1,<br>        instance_type=&quot;ml.g4dn.xlarge&quot;,<br>        output_path=output_path,<br>        sagemaker_session=session,<br>        hyperparameters={<br>            &quot;k&quot;: &quot;10&quot;,<br>            &quot;feature_dim&quot;: &quot;784&quot;,<br>            &quot;mini_batch_size&quot;: &quot;500&quot;,<br>            &quot;init_method&quot;: &quot;kmeans++&quot;,<br>        },<br>    )</pre><p>Here, only two hyperparameters are required: <strong>k</strong> (the number of clusters) and <strong>feature_dim</strong> (the number of features in the dataset). The dataset contains 784 features in this example, which the K-means model will process. The other parameters, like mini-batch size, ensure efficient training by dividing the dataset into manageable chunks.</p><h3><strong>Principal Component Analysis (</strong>PCA)</h3><p>The <strong>Amazon SageMaker PCA algorithm</strong> is an unsupervised machine learning algorithm that reduces the number of features in a dataset while retaining as much information as possible. This process, known as <strong>dimensionality reduction</strong>, is widely used in machine learning pipelines where feature reduction improves model performance and speeds up computation.</p><p>By leveraging Principal Component Analysis (PCA), you can transform high-dimensional datasets into a lower-dimensional space while preserving <strong>maximum variance</strong>. SageMaker allows you to easily train and deploy PCA models for scalable dimensionality reduction.</p><ul><li><strong>🚀 Improves computational efficiency</strong>: Reduces the size of input features for faster model training and inference.</li><li><strong>🔍 Avoids overfitting</strong>: Reducing dimensions minimizes noise and redundancy in data.</li><li><strong>📊 Better visualization</strong>: Visualizing lower-dimensional data (e.g., 2D or 3D) is easier for analysis.</li><li><strong>🛠 Flexible input formats</strong>: Supports both numpy arrays and Amazon RecordIO protobuf files stored in S3.</li></ul><h4>Practical Applications</h4><p>PCA is widely used in various fields where reducing dimensionality while retaining critical information enhances performance and interpretability. Here are some practical examples:</p><ol><li><strong>Financial Services</strong>: Detecting anomalies in transaction data, optimizing risk models by reducing correlated features, and improving fraud detection systems.</li><li><strong>Healthcare</strong>: Simplifying high-dimensional medical imaging data (e.g., MRI scans), analyzing gene expression datasets, and predicting disease outcomes.</li><li><strong>Retail</strong>: Reducing redundant features to improve product recommendation engines, customer segmentation, and sales predictions.</li></ol><h4>Hyperparameters</h4><p>The PCA algorithm in SageMaker requires three mandatory hyperparameters:</p><ul><li><strong>num_components</strong>: Specifies the number of principal components to retain. Lower values reduce dimensionality but may result in information loss. (e.g., num_components=10 retains the top 10 principal components)</li><li><strong>mini_batch_size</strong>: Defines the number of samples to process in each batch during training. Larger batch sizes improve training speed but require more memory. (e.g., mini_batch_size=500 processes 500 samples per batch)</li><li><strong>feature_dim</strong>: Specifies the total number of input features (dimensions) in the dataset. This must match the input feature size. (e.g., feature_dim=1000 assumes the input data has 1,000 features)</li></ul><h4>Usage</h4><p>Principal Component Analysis (PCA) is an unsupervised algorithm that reduces the number of input features in a dataset while retaining as much information as possible. PCA projects data onto a set of orthogonal components that explain the most variance.</p><p>The number of principal components to retain is a key hyperparameter, defined as num_components. Additionally, feature_dim specifies the number of input features, and mini_batch_size ensures efficient processing during training.</p><p>For example, in the code below, we use a <strong>P2 instance</strong> (ml.p2.xlarge) for training. PCA supports both <strong>CPU</strong> and <strong>GPU</strong> instances, with GPU instances such as <strong>P2, P3, G4dn, and G5 </strong>often being more performant for large, high-dimensional datasets:</p><pre>container = get_image_uri(session.boto_region_name, &quot;pca&quot;)    <br>estimator = sagemaker.estimator.Estimator(<br>    container,<br>    role,<br>    instance_count=1,<br>    instance_type=&quot;ml.p2.xlarge&quot;,<br>    output_path=output_path,<br>    sagemaker_session=session,<br>    hyperparameters={<br>        &quot;feature_dim&quot;: &quot;1000&quot;,<br>        &quot;num_components&quot;: &quot;10&quot;,<br>        &quot;mini_batch_size&quot;: &quot;500&quot;,<br>    },<br>)</pre><p>In this example, the dataset contains <strong>1,000 features</strong> (feature_dim=1000), and PCA will project the data onto the top <strong>10 principal components</strong> (num_components=10). The <strong>mini-batch size</strong> ensures that data is processed efficiently by dividing it into manageable chunks.</p><h3>Random Cut Forest (RCF)</h3><p>The Amazon SageMaker Random Cut Forest algorithm is an unsupervised algorithm used for detecting anomalies in datasets. It identifies data points that deviate from well-structured or patterned data, such as spikes in time series data or unclassifiable data points. The algorithm works by creating a model that learns the underlying patterns in the data and then assigns an anomaly score to each data point based on how well it fits the learned patterns.</p><h4>Practical Applications</h4><p>RCF is widely used for <strong>unsupervised anomaly detection</strong>, making it ideal for scenarios where identifying unusual patterns or outliers is critical. Here are its most significant applications:</p><ol><li><strong>Time Series Monitoring</strong>: Detecting anomalies such as sudden spikes, drops, or irregular patterns in sequential data. RCF is particularly effective for applications in <strong>financial services</strong>, such as identifying fraudulent transactions, and in <strong>IoT systems</strong>, where it can flag sensor irregularities for predictive maintenance.</li><li><strong>Operational Anomaly Detection</strong>: Monitoring real-time performance of systems to identify failures, faults, or unexpected behavior. RCF is commonly used in <strong>manufacturing</strong> to detect machine performance issues and equipment faults, as well as in <strong>IT infrastructure</strong> for identifying server load anomalies or potential system outages.</li><li><strong>Event-Based Outlier Detection</strong>: Identifying data points that deviate from expected patterns in non-sequential data. This includes applications in <strong>cybersecurity</strong> for detecting unusual access patterns or potential intrusions, and in <strong>healthcare</strong> for identifying irregular patient records or test results that may require further investigation.</li></ol><h4>Hyperparameters</h4><p>RCF only has a single required hyperparameter:</p><ul><li><strong>feature_dim</strong>: The total number of input features (dimensions) in the dataset. (e.g., feature_dim=100 assumes the input data has 100 features)</li></ul><p>RCF also supports additional optional hyperparameters, used to tune performance:</p><ul><li><strong>num_trees</strong>: Specifies the number of trees in the forest. More trees improve accuracy but increase computation time. (e.g., num_trees=50 builds 50 trees)</li><li><strong>num_samples_per_tree</strong>: Defines the number of samples each tree uses during training. Larger values capture finer patterns. (e.g., num_samples_per_tree=512 uses 512 samples per tree)</li></ul><h4>Usage</h4><p>The Random Cut Forest (RCF) algorithm in SageMaker is used for <strong>unsupervised anomaly detection</strong>. It identifies data points that deviate significantly from patterns in the input data and assigns an <strong>anomaly score</strong> based on how well each point aligns with the learned structure.</p><p>For RCF, the only required hyperparameter is feature_dim, which specifies the total number of input features in the dataset. Optional hyperparameters like num_trees and num_samples_per_tree can be used to tune the model’s performance.</p><p>For example, in the code below, we configure an RCF model with feature_dim=100 using a <strong>P2 instance</strong> for training:</p><pre>container = get_image_uri(session.boto_region_name, &quot;randomcutforest&quot;)<br>estimator = sagemaker.estimator.Estimator(<br>    container,<br>    role,<br>    instance_count=1,<br>    instance_type=&quot;ml.p2.xlarge&quot;,    # GPU instance for large datasets<br>    output_path=output_path,<br>    sagemaker_session=session,<br>    hyperparameters={<br>        &quot;feature_dim&quot;: &quot;100&quot;,        # Total number of input features<br>        &quot;num_trees&quot;: &quot;50&quot;,           # Optional: Number of trees<br>        &quot;num_samples_per_tree&quot;: &quot;512&quot; # Optional: Samples per tree<br>    },<br>)</pre><p>In this example, the RCF model processes data with <strong>100 input features</strong> and builds a forest of <strong>50 trees</strong>, each trained on <strong>512 samples</strong>. A <strong>P2 instance </strong>is used to accelerate training, especially for larger datasets.</p><h3>Further Resources</h3><p>For those looking to deepen their understanding of these algorithms and their implementation in SageMaker, the official <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html"><strong>Amazon SageMaker documentation</strong></a><strong> </strong>is the most up-to-date and reliable resource. It provides the guidance needed to fine-tune models, navigate SageMaker’s machine learning tools, and integrate solutions seamlessly into your workflows.</p><p>While this blog serves as a practical, hands-on guide to get you started, AWS documentation remains the definitive reference for production-level implementation. Whether you’re exploring unsupervised learning techniques, such as anomaly detection with <strong>IP Insight</strong> or clustering with <strong>K-means</strong>, or building more complex machine learning workflows, this resource will help you scale your SageMaker expertise effectively. Examples using the SageMaker estimator can be found <a href="https://github.com/aphexlog/sagemaker-examples">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31577b5b65a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-unsupervised-learning-31577b5b65a7">Brief Guide to Amazon SageMaker Algorithms: Unsupervised Learning</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Exhaustion to Expansion: How We Solved a Top 200 Fortune Company’s IP Crisis]]></title>
            <link>https://medium.com/protagona/from-exhaustion-to-expansion-how-we-solved-a-top-200-fortune-companys-ip-crisis-409f50e54b09?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/409f50e54b09</guid>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[ip-addressing]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Josh Sternfeld]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 19:49:43 GMT</pubDate>
            <atom:updated>2024-11-05T20:58:51.056Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Exhaustion to Expansion: How We Solved a Fortune 200 Company’s IP Crisis</h3><h3>Situation</h3><p>You have a large organization that’s utilizing a ton of IP space inefficiently. Let’s face it, nobody’s perfect when it comes to planning and predicting the use of proper CIDR blocks. In a world where data centers are considered old school and everything is moving to the cloud, how do you solve this problem? How do you avoid running out of routable IP space? Even more so, how do you leverage your space perfectly while maintaining credibility in an environment where the size of workloads and the levels of on-premise clusters transitioning to the cloud are only increasing? How can you run 10,000 Fargate containers at once while maintaining enough IP space?</p><p>Better yet, how can you create a service mesh network that can enable other teams to do a Centralized Fabric approach? All of these questions we at Protagona recently asked ourselves at an Enterprise level, to help enable and accelerate a large organization’s move to the cloud. And it turns out, the solution isn’t completely straightforward and required a lot of planning, architecture, and shared services.</p><h3>Task</h3><p>To move the organization and enable many different teams — from billing to planning — to move to AWS, we had to create a solution that would support and deploy very easily to help teams adopt this new process. If you have been in the field for a while, you will see that while the planning, architecture, coding, and approvals are difficult, the hardest part is adoption. Knowing this, we decided to make a fully automated approach to this new style of cloud networking.</p><p>So here’s what we developed: AWS PrivateLink Endpoint Services as a pattern to expose internal workloads, or PLES for short. Diving in, the pattern that we must create to solve this looming IP consumption issue is essentially twofold:</p><ol><li>First is the Service Provider, or the creator of the connection. This service provider will have to create an NLB in their VPC and initiate the PrivateLink endpoint services.</li><li>Second is the Service Consumer, and in this case the “Fabric VPC”. This shared services fabric VPC is a way that all the teams can connect to each other in a centralized manner.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPRl-Fqg7Z1BcV5RaS6pUA.png" /><figcaption>Private Link Endpoint Services Diagram: SP/SC</figcaption></figure><h3>Action</h3><p>Here’s what we developed for this solution: a multi-step function CloudFormation custom resource Lambda-triggered approach. Initially, we went with an event-based approach leveraging both EventBridge and an API Gateway upon deletion, but it turned out that was more complex to manage and promote adoption.</p><p>After learning from this experience, we instead chose to pivot and leverage a CloudFormation custom resource Lambda that, upon creation, would:</p><ol><li>Look up information in the account for things like the PrivateLink endpoint VPC name and NLB details</li><li>Kick off a step function in the fabric account</li><li>Perform a series of lookups and validations on VPC ownership</li><li>Create the PrivateLink endpoint service endpoints in the VPC</li></ol><p>Leveraging this architecture style, we were able to automate the process so that when a team creates a PrivateLink endpoint in their account with the right tags, it automatically:</p><ul><li>Kicks off the creation of the corresponding PrivateLink in the fabric account</li><li>Similarly handles deletion by removing it from the fabric account when requested</li></ul><p>Down the road, the same method can be used to directly connect services together without even using a fabric VPC, if desired.</p><h3>Result</h3><p>With the PLES automation, teams are now able to create routable CIDR blocks as large as the RFC allocations allow. The solution makes it possible to avoid using IPv6 when many apps aren’t compatible, while allowing you to virtually use as many IP addresses as you would like. This is achieved by only needing to reserve a very small routable subnet, while the rest are technically private but become routable via PLES.</p><h3>Key Learnings</h3><h3>Adoption is Everything</h3><p>The most important lesson learned is that adoption is paramount. Sometimes the best solution isn’t the most technically “perfect” one, but rather the solution that teams will actually use. While in an ideal world, we might use Terraform exclusively without custom resources to trigger Lambda functions, the reality is that many organizations don’t use Terraform at all. You need to meet organizations where they are and use the cloud-native tools they’re most familiar with to get the job done, even if it means more steps.</p><h3>Access Control Strategy</h3><p>While ABAC (Attribute-Based Access Control) initially seemed like a natural fit for our PLES implementation, several critical operational and governance requirements led us to implement a more controlled approach:</p><h3>Backup and Restore Operations</h3><p>The backup and restore process was a major factor in our decision. Our centralized approach allows us to:</p><ul><li>Maintain consistent backup schedules</li><li>Implement unified retention policies</li><li>Coordinate cross-account restores</li><li>Ensure proper sequencing during disaster recovery</li><li>Track dependencies between endpoints during restore operations</li></ul><h3>Uniformity and Control Requirements</h3><p>Standardization was crucial for operational efficiency:</p><ul><li>All endpoints follow the same naming conventions</li><li>Security groups are configured consistently</li><li>Routing policies remain uniform</li><li>Monitoring and alerting maintain the same baselines</li><li>Documentation stays synchronized with actual implementations</li></ul><h3>Approval Workflow Benefits</h3><p>The approval process needed to be more nuanced than simple tag-based permissions:</p><ul><li>Teams need to demonstrate compliance with network policies</li><li>Resource quotas must be checked before endpoint creation</li><li>Security teams need visibility into new connections</li><li>Cost allocations need to be verified</li><li>Cross-team dependencies must be documented</li></ul><h3>Technical Limitations and Considerations</h3><p>While VPC Lattice is a newer AWS service for service networking, it currently lacks UDP and TCP routing support. AWS PrivateLink, being a mature and battle-tested service, provides the stability and feature set we need for production traffic.</p><p>Our decision to use PrivateLink over VPC Lattice was driven by:</p><ul><li>PrivateLink’s proven track record in production environments</li><li>Complete feature set for our networking requirements</li><li>Mature tooling and documentation</li><li>Extensive operational experience across the industry</li></ul><p>Though VPC Lattice offers some interesting capabilities, both its cost structure and current limitations made it unsuitable for our immediate needs.</p><h3>Bonus Resources</h3><p>I’ve open-sourced my SIGV4 tokenization self-signing automation script that was initially built for the event-based architecture approach (though we didn’t end up using it). Feel free to check out my GitHub repo if you want to use it for your own projects!</p><p><a href="https://github.com/hammy414/sigv4-token-signer/tree/main">https://github.com/hammy414/sigv4-token-signer/tree/main</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=409f50e54b09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/from-exhaustion-to-expansion-how-we-solved-a-top-200-fortune-companys-ip-crisis-409f50e54b09">From Exhaustion to Expansion: How We Solved a Top 200 Fortune Company’s IP Crisis</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brief Guide to Amazon SageMaker Algorithms: Tabular Data]]></title>
            <link>https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-tabular-data-27e15605e497?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/27e15605e497</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sagemaker]]></category>
            <dc:creator><![CDATA[Aaron West]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 16:27:40 GMT</pubDate>
            <atom:updated>2024-12-12T16:38:01.963Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3t-vKufCle1PjRXsqn3Giw.jpeg" /><figcaption>Image produced with Amazon Bedrock</figcaption></figure><p>This article is the second in a four-part series exploring Amazon SageMaker’s machine learning algorithms, focusing on those designed for tabular data processing. It delves into key concepts, functionalities, and practical applications of algorithms such as XGBoost, AutoGluon, CatBoost, Factorization Machines, K-Nearest Neighbors, Linear Learner, LightGBM, and TabTransformer. This overview highlights aspects essential for the AWS Certified Machine Learning Specialty exam, making it a valuable resource for both certification candidates and professionals seeking to enhance their understanding of SageMaker’s structured data capabilities. Be sure to check out <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-text-processing-95eb5aa5301a">Part 1</a> on Natural Language Processing and Text if you haven’t yet — <a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-unsupervised-learning-31577b5b65a7">Part 3</a> in the series focuses on unsupervised learning models and provides some code examples.</p><h3>XGBoost</h3><p>XGBoost is an open-source, efficient, and popular implementation of the gradient boosted trees algorithm. It is known for its performance in machine learning competitions due to its ability to handle various data types and relationships, and its wide range of hyperparameters for fine-tuning. Amazon SageMaker offers a managed XGBoost environment, providing an estimator to execute training scripts.</p><ul><li>🤖<strong> Supervised Learning:</strong> XGBoost is a supervised learning algorithm that combines estimates from multiple simpler models to make accurate predictions.</li><li>💪 <strong>Versatility: </strong>It excels in handling different data types, relationships, and distributions, making it robust and versatile.</li><li>💻 <strong>Portable Deployment:</strong> XGBoost can be statically compiled, allowing you to create a lightweight build that can run on resource-constrained environments, such as a Raspberry Pi. This flexibility makes it possible to deploy models even in edge computing scenarios or devices with limited hardware resources.</li></ul><h4>Memory Efficiency and Performance</h4><ol><li><strong>In-Memory Processing: </strong>XGBoost can load the entire dataset into memory when resources allow, enabling extremely fast processing and training times.</li><li><strong>Out-of-Core Computing:</strong> For datasets too large to fit in memory, XGBoost employs out-of-core computing techniques, allowing it to handle large datasets efficiently even with limited RAM.</li><li><strong>Cache-Aware Access:</strong> It uses cache-aware access patterns to optimize data processing, further enhancing performance.</li></ol><h4>Practical Applications</h4><p>XGBoost is widely used in various domains due to its versatility and performance:</p><ol><li><strong>Financial Services: </strong>Predicting credit risk and fraud detection</li><li><strong>Retail:</strong> Customer segmentation and product recommendation</li><li><strong>Healthcare:</strong> Disease prediction and patient outcome analysis</li><li><strong>Marketing:</strong> Click-through rate prediction and customer churn analysis</li><li><strong>Manufacturing:</strong> Predictive maintenance and quality control</li></ol><h4>Hyperparameters</h4><p>XGBoost in SageMaker requires just two mandatory hyperparameters: num_class and num_round. num_class is only required if the objective is set to multi:softmax or multi:softprob and specifies the number of classes or categories, while num_round dictates the number of boosting rounds for training. These parameters are fundamental to configuring XGBoost for classification tasks or setting up the overall training process.</p><p>Like other algorithms in this series, XGBoost offers a variety of optional hyperparameters that allow for further tuning. Here are a few valuable ones:</p><ul><li><strong>alpha</strong>: L1 regularization term on weights. A higher value makes the model more conservative.</li><li><strong>base_score</strong>: Initial prediction score of all instances. This can be useful for setting the global bias.</li><li><strong>booster</strong>: Determines the boosting algorithm to use, with gbtree, gblinear, and dart as available options.</li><li><strong>colsample_bytree</strong>: Specifies the subsample ratio of columns when constructing each tree, controlling feature selection.</li></ul><h3>AutoGluon</h3><p>AutoGluon-Tabular is an open-source automated machine learning (AutoML) framework designed for generating high-performing machine learning models from raw tabular datasets. It distinguishes itself from other AutoML frameworks by employing a unique approach that emphasizes model ensembling and stacking, rather than solely focusing on model and hyperparameter selection.</p><ul><li>💻 <strong>Open-Source Framework:</strong> AutoGluon-Tabular is an open-source AutoML framework for training machine learning models on tabular datasets.</li><li>🤖 <strong>Automation:</strong> It automates the process of feature engineering, model selection, hyperparameter tuning, and model ensembling.</li><li>🏆 <strong>Ensembling:</strong> It excels in ensembling multiple models and stacking them in multiple layers to achieve high accuracy.</li></ul><h4>Practical Applications</h4><p>AutoGluon-Tabular is particularly useful in scenarios where quick, automated model development is needed:</p><ol><li><strong>Rapid Prototyping: </strong>Quickly generating baseline models for new projects</li><li><strong>Kaggle Competitions:</strong> Achieving competitive results with minimal manual tuning</li><li><strong>Business Intelligence:</strong> Automating predictive analytics for various business metrics</li><li><strong>Research:</strong> Comparing automated approaches with manual model development</li><li><strong>Educational Purposes: </strong>Teaching students about AutoML concepts and practices</li></ol><h4>Hyperparameters</h4><p>One of the strengths of AutoGluon-Tabular is its ability to automatically tune hyperparameters, allowing it to adapt to different types of tasks without requiring the user to manually set specific values. This flexibility makes it ideal for automated machine learning (AutoML) workflows, where manual tuning can often be a bottleneck. However, AutoGluon-Tabular offers several optional hyperparameters for users who wish to exert more control over the process. Here are a few key examples:</p><ul><li><strong>eval_metric</strong>: Determines the evaluation metric used for validation, with options such as root_mean_squared_error for regression, roc_auc for binary classification, and accuracy for multi-class classification. By default, it’s set to auto, letting the algorithm select the metric based on the task.</li><li><strong>presets</strong>: Provides a set of predefined configurations for various training needs. For instance, best_quality achieves high predictive accuracy with slower inference, while optimize_for_deployment reduces model size for faster inference and deployment.</li><li><strong>auto_stack</strong>: Automatically enables bagging and multi-layer stacking, which can significantly boost predictive accuracy at the cost of longer training times. This is a useful feature for those looking to maximize performance.</li><li><strong>num_bag_folds</strong>: Configures the number of folds used for bagging models. Bagging can help to reduce model variance, and adjusting the number of folds allows for balancing between bias and variance.</li><li><strong>num_stack_levels</strong>: Specifies the number of stacking levels in the ensemble model. Higher stacking levels can improve model accuracy but may increase training time, with a typical recommended range of 1 to 3 levels.</li></ul><h3>CatBoost</h3><p>CatBoost is a powerful open-source machine learning algorithm known as Gradient Boosting Decision Tree (GBDT), which combines multiple simple models to make accurate predictions. It is specifically designed to address prediction shift issues commonly found in GBDT implementations.</p><ul><li>🌳 <strong>Categorical Boosting:</strong> CatBoost is a gradient boosting algorithm known for its performance and ability to handle categorical features effectively.</li><li>⚙️<strong> Innovative Improvements:</strong> It introduces two major improvements: ordered boosting, which mitigates target leakage, and an innovative algorithm for handling categorical features.</li></ul><h4>Practical Applications</h4><p>CatBoost is particularly useful in scenarios involving categorical data and where prediction accuracy is crucial:</p><ol><li><strong>E-commerce:</strong> Product categorization and recommendation systems</li><li><strong>Finance:</strong> Credit scoring and risk assessment</li><li><strong>Healthcare:</strong> Patient diagnosis and treatment outcome prediction</li><li><strong>Marketing:</strong> Customer segmentation and targeted advertising</li><li><strong>Natural Language Processing:</strong> Text classification and sentiment analysis</li></ol><h4>Hyperparameters</h4><p>CatBoost in SageMaker is designed with default hyperparameters that allow for quick and easy setup. While there are no strictly required hyperparameters, CatBoost offers several customizable options to improve model performance depending on your specific needs. Here are a few key hyperparameters you can adjust:</p><ul><li><strong>iterations</strong>: This hyperparameter controls the maximum number of trees that will be built during training. By default, it’s set to 500, but increasing or decreasing this value can help tune the model for longer or shorter training times depending on your dataset.</li><li><strong>early_stopping_rounds</strong>: If the model’s performance stops improving, training will halt after a set number of rounds. The default value is 5, and this parameter helps prevent overfitting while speeding up the training process.</li><li><strong>eval_metric</strong>: The evaluation metric used for validation during training. CatBoost automatically selects the appropriate metric based on the type of problem (RMSE for regression, AUC for binary classification, or MultiClass for multi-class classification). However, you can customize this based on your specific needs by setting values such as logloss or accuracy.</li><li><strong>learning_rate</strong>: Controls how quickly the model adapts during training. Smaller values lead to slower, more precise training, while larger values speed up training but risk overshooting the optimal solution. The default value is 0.009.</li><li><strong>depth</strong>: Refers to the depth of the trees used in the model. Deeper trees can capture more complex patterns in the data but may also lead to overfitting. The default value is 6, and it can be adjusted between 1 and 16.</li><li><strong>l2_leaf_reg</strong>: A regularization parameter for the L2 regularization of leaf scores, which helps avoid overfitting. The default value is 3.</li></ul><h3>Factorization Machines</h3><p>Amazon SageMaker Factorization Machines is a supervised learning algorithm used for classification and regression tasks. It combines the advantages of Support Vector Machines with factorization models, effectively capturing interactions between features in high-dimensional, sparse datasets.</p><ul><li>🤖 <strong>Supervised Learning:</strong> Used for both classification and regression tasks.</li><li>📊<strong> Feature Interaction:</strong> Leverages the strengths of Support Vector Machines and factorization models to capture complex feature interactions.</li><li>📈 <strong>High-Dimensional Data: </strong>Designed for high-dimensional sparse datasets, making it effective for problems like recommender systems.</li></ul><h4>Practical Applications</h4><p>Factorization Machines are particularly useful in scenarios involving sparse data and complex feature interactions:</p><ol><li><strong>Recommender Systems:</strong> Personalized product or content recommendations</li><li><strong>Click-Through Rate Prediction: </strong>Improving online advertising effectiveness</li><li><strong>User Behavior Analysis:</strong> Understanding and predicting user interactions</li><li><strong>Feature Extraction:</strong> Generating meaningful features from sparse data</li><li><strong>Social Network Analysis:</strong> Predicting connections or interactions between users</li></ol><h4><strong>Hyperparameters</strong></h4><p>Factorization Machines in SageMaker require three key hyperparameters that must be set by the user: feature_dim, num_factors, and predictor_type. These are crucial for estimating model parameters from data, especially for sparse inputs.</p><ul><li>feature_dim: This defines the dimensionality of the input feature space, which can be very high with sparse input data. Valid values are positive integers within a suggested range of [10000,10000000].</li><li>num_factors: Determines the number of factors used in factorization. A range of [2,1000] is suggested, with 64 being a good starting point.</li><li>predictor_type: Defines whether the task is binary classification or regression. Valid options are binary_classifier for classification tasks and regressor for regression.</li></ul><p>In addition to these required parameters, there are several optional hyperparameters that can be tuned for specific needs:</p><ul><li>bias_init_method: Controls how the bias term is initialized, with options including normal (default), uniform, or constant.</li><li>bias_init_scale: Specifies the scale range for initializing bias when bias_init_method is set to uniform.</li><li>bias_init_sigma: Defines the standard deviation for initializing bias when using the normal method.</li><li>bias_init_value: Sets a constant value for the bias term when the initialization method is set to constant.</li></ul><h3>K-Nearest Neighbors (KNN)</h3><p>The K-Nearest Neighbors (k-NN) algorithm is an index-based, non-parametric method used for classification and regression tasks. It predicts based on the majority vote or average of the k-nearest data points.</p><ul><li>🧮 <strong>Versatile Algorithm:</strong> KNN is used for both classification and regression tasks.</li><li>📊 <strong>Non-Parametric Method: </strong>It makes no assumptions about the underlying data distribution.</li><li>👨‍🏫 <strong>Classification Approach:</strong> In classification, it assigns the most frequent class label among the k-nearest neighbors.</li><li>📈 <strong>Regression Approach: </strong>In regression, it calculates the average of the k-nearest neighbors’ target values.</li></ul><h4>Practical Applications</h4><p>KNN is widely used in various domains due to its simplicity and effectiveness:</p><ol><li><strong>Image Recognition:</strong> Classifying images based on similar known images</li><li>R<strong>ecommendation Systems:</strong> Suggesting items based on user similarities</li><li><strong>Finance:</strong> Credit scoring and fraud detection</li><li><strong>Healthcare: </strong>Disease diagnosis based on patient similarities</li><li><strong>Anomaly Detection:</strong> Identifying outliers in datasets</li></ol><h4>Hyperparameters</h4><p>There are a several several core parameters must be defined to ensure the KNN algorithm is appropriately configured:</p><ul><li>feature_dim: This defines the number of features in the input data. It is crucial for setting up the dimensionality of the data points.</li><li>k: Represents the number of nearest neighbors the algorithm will consider when making predictions.</li><li>predictor_type: Specifies whether the task is classification or regression, using the values classifier for classification tasks or regressor for regression tasks.</li><li>sample_size: Determines how many data points will be sampled from the training dataset for the k-NN algorithm to process.</li><li>dimension_reduction_target: Required when dimension reduction is applied, this parameter sets the target dimension for the reduction.</li></ul><p>There are also several optional hyperparameters that can be adjusted to optimize performance, especially when dealing with large datasets or specific use cases:</p><ul><li>dimension_reduction_type: Controls the type of dimension reduction, such as sign for random projection or <a href="https://www.stat.berkeley.edu/~mmahoney/f13-stat260-cs294/Lectures/lecture08.pdf">fjlt</a> for the <a href="https://cs.stanford.edu/people/mmahoney/cs369m/Lectures/lecture1.pdf">Johnson-Lindenstrauss </a>transform.</li><li>faiss_index_ivf_nlists: For the FAISS library, this specifies the number of <a href="https://engineeringstatics.org/Chapter_07-centroids.html">centroids</a> to use when building the index, which is essential for large-scale nearest neighbor search.</li><li>faiss_index_pq_m: This defines the number of sub-vector components in FAISS when applying Product Quantization (PQ) indexing, helping to optimize memory usage during nearest neighbor search.</li></ul><h3>Linear Learner</h3><p>Amazon SageMaker Linear Learner is a versatile supervised machine learning algorithm used for both classification and regression problems. It allows users to explore different training objectives and choose the best solution based on a validation set.</p><ul><li>🤖 <strong>Dual Functionality:</strong> Solves both classification (binary and multiclass) and regression problems.</li><li>📊 <strong>Flexible Input:</strong> Accepts labeled examples (x, y) as input, where x is a high-dimensional vector and y is a numeric label.</li><li>📈 <strong>Linear Mapping:</strong> Learns a linear function or threshold function to map input vectors to approximate labels.</li><li>🎯<strong> Objective Exploration:</strong> Allows exploring different training objectives and choosing the best solution using a validation set.</li></ul><h4>Practical Applications</h4><p>Linear Learner is useful in a wide range of scenarios where linear relationships between features and targets are assumed:</p><ol><li><strong>Financial Forecasting: </strong>Predicting stock prices or economic indicators</li><li><strong>Marketing Analytics:</strong> Customer lifetime value prediction</li><li><strong>Healthcare:</strong> Patient length of stay prediction</li><li><strong>Manufacturing: </strong>Quality control and defect prediction</li><li><strong>Environmental Science:</strong> Climate variable prediction</li></ol><h4>Hyperparameters</h4><p>For the Linear Learner algorithm, the following are the key required hyperparameters:</p><ul><li><strong>num_classes</strong>: The number of classes for the response variable. This parameter is required if you’re performing multi-class classification and determines how many classes the algorithm will predict (e.g., if you have three classes, this would be set to 3).</li><li><strong>predictor_type</strong>: Specifies the type of problem, whether it’s binary classification, multi-class classification, or regression.</li></ul><p>There are several optional ones, allowing for greater flexibility and optimization based on the specific characteristics of your dataset and learning task:</p><ul><li><strong>accuracy_top_k</strong>: Defines the top-K accuracy metric, which evaluates how well the model ranks the true label among the top K predictions. It is useful for multi-class classification tasks.</li><li><strong>optimizer</strong>: Determines which optimization algorithm will be used. The valid options include auto, which is the default and selects adam as the optimizer; sgd, which stands for stochastic gradient descent; adam, an adaptive momentum estimation technique; and rmsprop, a method that uses a moving average of squared gradients to normalize the gradient. The default value is auto.</li><li><strong>beta_1</strong> and <strong>beta_2</strong>: Control the decay rates for first and second-moment estimates in optimization. These are specific to the Adam optimizer.</li><li><strong>bias_lr_mult</strong> and <strong>bias_wd_mult</strong>: Allow for different learning rates and regularization for the bias term, which can help fine-tune the model’s performance.</li></ul><h3>LightGBM</h3><p>LightGBM is an open-source implementation of the Gradient Boosting Decision Tree (GBDT) algorithm, known for its efficiency and use in supervised learning. It excels at predicting target variables by combining estimates from simpler models.</p><ul><li>💻 <strong>Efficient Implementation:</strong> LightGBM is an open-source, efficient implementation of the Gradient Boosting Decision Tree (GBDT) algorithm.</li><li>📈 <strong>Ensemble Learning: </strong>GBDT combines estimates from a set of simpler, weaker models to predict a target variable accurately.</li><li>🚀 <strong>Enhanced Performance: </strong>LightGBM improves the efficiency and scalability of traditional GBDT algorithms.</li></ul><h4>Practical Applications</h4><p>LightGBM is widely used in various domains due to its efficiency and performance:</p><ol><li><strong>Online Advertising: </strong>Click-through rate prediction</li><li><strong>Retail: </strong>Sales forecasting and inventory optimization</li><li><strong>Finance: </strong>Risk assessment and fraud detection</li><li><strong>Healthcare:</strong> Disease progression prediction</li><li><strong>Energy Sector:</strong> Energy demand forecasting</li></ol><h4>Hyperparameters</h4><p>For LightGBM, there are no strictly required hyperparameters, as the algorithm can automate much of the tuning process. However, several optional hyperparameters can significantly impact the performance and behavior of the model:</p><ul><li><strong>num_leaves</strong>: Controls the maximum number of leaves in one tree. A larger number can improve accuracy but also increases the risk of overfitting. Default value is 31.</li><li><strong>learning_rate</strong>: Adjusts how quickly the model learns. A smaller value may lead to better generalization but requires more training iterations. Default value is 0.1.</li><li><strong>max_depth</strong>: Limits the depth of a tree to prevent overfitting, especially with smaller datasets. Setting it to -1 removes any limit. Default value is -1.</li></ul><h3>TabTransformer</h3><p>TabTransformer is a novel deep tabular data modeling architecture for supervised learning, leveraging self-attention-based Transformers to transform categorical feature embeddings into robust contextual embeddings for improved prediction accuracy.</p><ul><li>🤖 <strong>Transformer-Based:</strong> Built on self-attention-based Transformers, a powerful architecture from natural language processing.</li><li>📊 <strong>Tabular Data Focus: </strong>Specifically designed for deep tabular data modeling.</li><li>🎯<strong> Improved Accuracy:</strong> Aims for higher prediction accuracy in supervised learning tasks compared to traditional methods.</li><li>💪 <strong>Robust Embeddings: </strong>Creates robust contextual embeddings, improving prediction accuracy.</li><li>🛡️ <strong>Data Handling:</strong> Effectively handles missing and noisy data, a common issue in real-world datasets.</li></ul><h4>Practical Applications</h4><p>TabTransformer is particularly useful in scenarios involving complex tabular data:</p><ol><li><strong>Financial Services: </strong>Credit scoring and fraud detection with improved accuracy</li><li><strong>Healthcare:</strong> Patient outcome prediction considering complex interactions between features</li><li><strong>Marketing: </strong>Customer behavior prediction for targeted campaigns</li><li><strong>Human Resources: </strong>Employee performance prediction and talent management</li><li><strong>IoT and Sensor Data: </strong>Analyzing and predicting patterns in sensor data with temporal aspects</li></ol><h4>Hyperparameters</h4><p>Like AutoGluon, CatBoost and LightGBM, TabTransformer doesn’t have strictly required hyperparameters, and like every other algorithm, it offers various optional settings that enable users to fine-tune training and performance based on their data. Here are three key optional hyperparameters to consider when configuring the model:</p><ul><li><strong>n_blocks</strong>: Specifies the number of transformer blocks used in the architecture, which determines the depth of the model. Valid values range from 1 to 12, and the default is 4.</li><li><strong>learning_rate</strong>: Controls the rate at which the model weights are updated during training. Lower values lead to more gradual updates and can help in achieving better convergence. Valid values range from 0.0 to 1.0, with a default of 0.001.</li><li><strong>mlp_dropout</strong>: Applies dropout to the feedforward network between layers, helping to prevent overfitting by randomly setting a fraction of input units to 0 during training. This parameter has valid values from 0.0 to 1.0, with a default of 0.1.</li></ul><h3><strong>Choosing the Right Algorithm and Complementary Use Cases</strong></h3><p>The selection of a machine learning algorithm depends on the data structure, the problem you’re addressing, and the specific requirements for your output. Although various algorithms can handle a range of tasks, some are more suitable for certain scenarios than others. Below is a guide on when to choose one algorithm over another and how they can be combined for better outcomes.</p><h4><strong>XGBoost vs. CatBoost for Tabular Data</strong></h4><ul><li><strong>When to Choose XGBoost:</strong> XGBoost is a highly efficient algorithm suited for structured tabular data. It excels in handling missing values, non-linear relationships, and unbalanced datasets. If your goal is to perform classification or regression tasks with high control over hyperparameter tuning and fine-grained adjustments, XGBoost provides robust performance and flexibility.</li><li><strong>When to Choose CatBoost:</strong> CatBoost is an ideal choice when dealing with categorical features, as it handles categorical variables natively without requiring extensive preprocessing. It also shines in tasks requiring interpretable models, such as finance or healthcare, where transparency and interpretability are crucial. CatBoost often reduces the need for hyperparameter tuning and automatically optimizes many parameters, making it easier to set up and use effectively.</li></ul><h4><strong>LightGBM vs. Linear Learner for High-Speed Processing</strong></h4><ul><li><strong>LightGBM</strong>: If you are working with extremely large datasets and need a model that can train quickly while consuming minimal memory, LightGBM is an excellent choice. It’s optimized for both speed and performance, making it ideal for high-dimensional data where computational efficiency is essential.</li><li><strong>Linear Learner</strong>: On the other hand, if you are dealing with linear classification or regression tasks where the model’s simplicity and interpretability are key, Linear Learner is an effective and straightforward option. It’s particularly useful for smaller datasets where more complex models may not be necessary or when interpretability is a priority.</li></ul><h4><strong>AutoGluon vs. TabTransformer for Automating Model Tuning</strong></h4><ul><li><strong>When to Choose AutoGluon</strong>: AutoGluon automates much of the machine learning pipeline, from model selection to hyperparameter optimization, making it ideal for those who need rapid prototyping with minimal manual intervention. If you’re looking for an approach that quickly delivers high-quality results without requiring expertise in model tuning, AutoGluon is your best bet.</li><li><strong>When to Choose TabTransformer</strong>: TabTransformer, on the other hand, is ideal when your tabular data contains both categorical and continuous features that can benefit from deep learning-based feature representations. It’s a strong choice for complex classification tasks where capturing interactions between features is critical for model performance.</li></ul><h4><strong>Combining Algorithms for Enhanced Tabular Data Workflows</strong></h4><ul><li><strong>XGBoost for Classification + TabTransformer for Feature Representation</strong>: You can use TabTransformer to learn richer feature representations from tabular data, followed by XGBoost for classification or regression tasks. This combination is effective for complex problems where feature interaction plays a significant role in model performance.</li><li><strong>AutoGluon for Rapid Prototyping + LightGBM for Optimization</strong>: AutoGluon can be employed for rapid prototyping and model selection, while LightGBM can be used to optimize performance on large datasets for more demanding production tasks.</li></ul><h3>Implementation in SageMaker</h3><p>The tabular data algorithms we’ve explored in this article all benefit from Amazon SageMaker’s unified implementation framework. Each algorithm, from XGBoost to TabTransformer, is integrated into SageMaker as an Estimator class, providing a consistent interface for training and deployment. Data preprocessing for these tabular algorithms often involves converting your dataset into the SageMaker-compatible format, typically using the record_set() utility or similar data handling methods specific to each algorithm.</p><p>Training these models in SageMaker involves configuring algorithm-specific hyperparameters, which allow you to fine-tune performance for your particular use case. Whether you’re dealing with gradient boosting algorithms like XGBoost, CatBoost, and LightGBM, or more specialized algorithms like Factorization Machines or TabTransformer, SageMaker provides a standardized approach to model training and evaluation.</p><p>Once trained, these tabular data models can be deployed to SageMaker Endpoints for real-time inference, or used for batch predictions on large datasets. The deployment process is streamlined across all algorithms, allowing for easy integration into your existing data pipelines and applications.</p><h3>Further Resources</h3><p>For those looking to dive deeper into these algorithms or seeking the most up-to-date information, I recommend visiting the official <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html">Amazon SageMaker SDK documentation</a>. The Algorithms section <a href="https://sagemaker.readthedocs.io/en/stable/algorithms">here</a> also provides comprehensive details on all the algorithms we’ve discussed here, as well as others available in SageMaker. This resource offers in-depth explanations of algorithm parameters, input/output specifications, and implementation details that are crucial for fine-tuning your models. While this article serves as an overview and study aid, the official AWS documentation should be your go-to source for the most current and detailed information as you work with these machine learning tools in your projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27e15605e497" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-tabular-data-27e15605e497">Brief Guide to Amazon SageMaker Algorithms: Tabular Data</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brief Guide to Amazon SageMaker Algorithms: Text Processing]]></title>
            <link>https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-text-processing-95eb5aa5301a?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/95eb5aa5301a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aaron West]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 21:04:14 GMT</pubDate>
            <atom:updated>2025-01-18T13:51:38.188Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OlaUO0kPW8Pu2-p5MzXR1w.jpeg" /><figcaption>Image produced with Amazon Bedrock</figcaption></figure><p>This article is the first in a four-part series exploring Amazon SageMaker’s machine learning algorithms, with a focus on those used for natural language processing. It covers key concepts, functionalities, and practical applications of algorithms such as BlazingText, Latent Dirichlet Allocation (LDA), Neural Topic Modeling (NTM), Object2Vec, and Sequence to Sequence. The notes provided offer a concise overview of SageMaker’s capabilities within the AWS ecosystem. For further insights, check out <a href="https://medium.com/@aphexlog/brief-guide-to-amazon-sagemaker-algorithms-tabular-data-27e15605e497">Part 2 on Tabular Data</a> and stay tuned for upcoming articles. Whether you’re preparing for the AWS Certified Machine Learning Specialty exam or seeking to deepen your knowledge of machine learning on AWS, this guide serves as a focused resource for professional development in cloud-based machine learning.</p><h3>BlazingText</h3><p>Amazon SageMaker BlazingText is a highly optimized algorithm designed for natural language processing (NLP) tasks. It excels in two primary areas:</p><ol><li>Generating high-quality word embeddings using the Word2Vec algorithm</li><li>Performing efficient text classification</li></ol><p>What sets BlazingText apart is its ability to leverage both multi-core CPUs and GPUs, enabling it to handle large datasets and achieve state-of-the-art performance in tasks like sentiment analysis, named entity recognition, and machine translation.</p><p>BlazingText offers several key features that make it a powerful choice for NLP tasks:</p><ul><li>💻 <strong>Optimized Implementations:</strong> Provides optimized implementations of Word2Vec and text classification algorithms, ensuring efficient processing of text data.</li><li>🚀 <strong>Scalability:</strong> Enables scaling to large datasets with ease, making it suitable for big data applications.</li><li>🤖 <strong>Flexible Architectures:</strong> Offers Skip-gram and Continuous Bag of Words (CBOW) training architectures, similar to Word2Vec, allowing for versatile word embedding generation.</li><li>💡 <strong>GPU Acceleration:</strong> Extends the fastText text classifier for GPU acceleration using custom CUDA kernels, significantly speeding up training and inference.</li><li>💪 <strong>Competitive Performance: </strong>Achieves performance comparable to advanced deep learning text classification algorithms, making it a strong contender in various NLP tasks.</li></ul><h4>Word2Vec Capabilities</h4><p>Word2Vec is a crucial component of BlazingText, providing a powerful method for understanding and representing language in a machine-readable format. This technique transforms words into numerical vectors, capturing the semantic essence of language in a way that computers can process efficiently. By learning from vast amounts of text data, Word2Vec creates a rich, multidimensional space where words with similar meanings cluster together, enabling sophisticated natural language processing tasks.</p><ul><li>📝 <strong>Semantic Mapping:</strong> Maps words to distributed vectors, capturing semantic relationships between words.</li><li>📊<strong> Improved Generalization:</strong> Generates word embeddings that improve the generalizability of NLP models, enhancing performance across various tasks.</li><li>📚 <strong>Large-Scale Learning: </strong>Learns word embeddings from vast document collections, allowing it to capture nuanced language patterns.</li></ul><h4>Text Classification Strengths</h4><p>BlazingText’s text classification capabilities build upon its strong foundation in word representation, offering a robust solution for categorizing and analyzing text data. This feature is particularly valuable in the age of big data, where organizations need to quickly and accurately process large volumes of textual information. By leveraging GPU acceleration, BlazingText can handle these large-scale classification tasks with impressive speed and accuracy, making it an invaluable tool for a wide range of applications in natural language processing.</p><ul><li>🔍 <strong>Versatile Applications: </strong>Plays a crucial role in applications like web search, information retrieval, and document classification.</li><li>🎯 <strong>Accelerated Training: </strong>Benefits from BlazingText’s GPU acceleration for faster training, enabling quick model development and iteration.</li></ul><h4>Practical Applications</h4><p>BlazingText’s combination of Word2Vec and text classification capabilities makes it particularly useful for:</p><ol><li><strong>Sentiment Analysis:</strong> Determining the sentiment of customer reviews or social media posts.</li><li><strong>Named Entity Recognition:</strong> Identifying and classifying named entities in text.</li><li><strong>Machine Translation:</strong> Improving the quality of language translation systems.</li><li><strong>Document Classification:</strong> Automatically categorizing documents into predefined classes.</li><li><strong>Information Retrieval Systems:</strong> Enhancing search engines and recommendation systems.</li></ol><p>BlazingText stands out as a versatile and powerful tool in Amazon SageMaker’s arsenal of NLP algorithms. Its ability to handle large datasets efficiently, coupled with state-of-the-art performance in various NLP tasks, makes it an excellent choice for both word embedding generation and text classification tasks.</p><h4>Hyperparameters:</h4><p>BlazingText requires only one mandatory hyperparameter, mode, which can be set to a value of batch_skipgram, skipgram, or cbowand determines the Word2Vec architecture used for training. However, like the other algorithms we’ll discuss, it also offers several optional hyperparameters. Here are a few examples:</p><ul><li>batch_size: The size of each batch when mode is set to batch_skipgram. Set to a number between 10 and 20.</li><li>vector_dim: The dimension of the word vectors that the algorithm learns.</li><li>epochs: The number of complete passes through the training data.</li></ul><h3>Latent Dirichlet Allocation (LDA)</h3><p>Latent Dirichlet Allocation (LDA) is an unsupervised learning algorithm available within Amazon SageMaker. It is primarily used for topic modeling, aiming to categorize a set of observations into distinct topics. The algorithm represents each observation as a mixture of these topics, commonly applied to analyze text documents within a corpus.</p><p>LDA in SageMaker offers several key features that make it a robust choice for topic modeling tasks:</p><ul><li>🤖 <strong>Topic Discovery:</strong> The algorithm identifies a user-defined number of topics present in a dataset, allowing for flexible analysis based on the complexity of your corpus.</li><li>📚 <strong>Text Data Analysis:</strong> LDA is particularly useful for analyzing text data, where each observation represents a document, and features correspond to the presence or count of words. This makes it ideal for tasks like content categorization or document summarization.</li></ul><h4>How LDA Works</h4><p>LDA operates on the principle that documents are mixtures of topics, and topics are mixtures of words. The algorithm:</p><ol><li>Assumes documents are produced from a mixture of topics.</li><li>Topics are probability distributions over words.</li><li>Uses statistical inference to back out the topics that likely generated the documents.</li></ol><h4>Hyperparameters:</h4><ul><li>num_topics: Defines the number of distinct topics to be uncovered in the dataset.</li><li>feature_dim: The size of the vocabulary of the input document corpus.</li><li>mini_batch_size: The total number of documents in the input document corpus.</li></ul><h3>Neural Topic Modeling (NTM)</h3><p>Amazon SageMaker NTM is an unsupervised learning algorithm used to discover topics within a corpus of documents. It operates by analyzing the statistical distribution of words, grouping those that frequently appear together into distinct topics. For instance, documents containing words like “bike,” “car,” “train,” “mileage,” and “speed” would likely share a “transportation” topic.</p><p>NTM offers several key features that make it a powerful choice for topic modeling tasks:</p><ul><li>📚 <strong>Statistical Analysis:</strong> The algorithm excels at organizing documents into topics based on the statistical occurrences of word groupings, providing insights into document collections.</li><li>🧠 <strong>Neural Network Approach:</strong> Utilizes neural networks for topic discovery, potentially capturing more complex relationships than traditional methods.</li><li>🔍 <strong>Intuitive Topic Discovery: </strong>For instance, documents containing words like “bike,” “car,” “train,” “mileage,” and “speed” would likely share a “transportation” topic.</li></ul><h4>Practical Applications</h4><p>NTM finds applications in various domains:</p><ol><li><strong>Content Recommendation:</strong> Suggesting articles or products based on discovered topics.</li><li><strong>Document Summarization:</strong> Identifying key themes in large text corpora.</li><li><strong>Trend Analysis: </strong>Tracking evolving topics in social media or news articles over time.</li><li><strong>Customer Feedback Analysis:</strong> Uncovering common themes in customer reviews or support tickets.</li></ol><h4>Hyperparameters:</h4><p>NTM requires two critical hyperparameters for effective training: feature_dim, which defines the vocabulary size of the dataset, and num_topics, determining the number of topics to be extracted. These foundational settings are crucial to ensure the model works effectively on the document corpus.</p><p>Here are a few optional hyperparameters that allow for fine-tuning:</p><ul><li>encoder_layers: Specifies the number of layers in the neural network encoder for deeper topic discovery.</li><li>epochs: Sets the number of iterations through the dataset, controlling the duration and depth of training.</li><li>learning_rate: Determines the step size during optimization, influencing how quickly the model converges.</li></ul><h3>Object2Vec</h3><p>Amazon SageMaker Object2Vec is a highly customizable neural embedding algorithm. It learns low-dimensional dense embeddings of high-dimensional objects while preserving semantic relationships. This algorithm is particularly useful for anomaly detection tasks and can be deployed to a SageMaker Endpoint for inference.</p><p>Object2Vec offers several key features that make it a flexible choice for embedding tasks:</p><ul><li>⚙️ <strong>Customizability:</strong> Highly customizable with numerous hyperparameters, allowing for fine-tuning to specific use cases.</li><li>🧠<strong> Semantic Preservation:</strong> Learns embeddings that capture semantic relationships, enabling sophisticated analysis of object relationships.</li><li>📈 <strong>Anomaly Detection:</strong> Particularly suitable for anomaly detection tasks, leveraging learned embeddings to identify outliers.</li><li>🔌<strong> SageMaker Integration:</strong> Seamless integration with the SageMaker ecosystem, facilitating easy deployment and management.</li></ul><h4>Functionality</h4><p>Object2Vec’s core functionality revolves around its embedding capabilities:</p><ul><li>🤖 <strong>General-Purpose Algorithm:</strong> Object2Vec is a general-purpose neural embedding algorithm, applicable to a wide range of data types.</li><li>📉<strong> Dimensionality Reduction:</strong> It learns low-dimensional dense embeddings of high-dimensional objects, facilitating more efficient processing and analysis.</li><li>🔎 <strong>Relationship Preservation:</strong> The embeddings preserve the semantics of relationships between object pairs, maintaining important contextual information.</li><li>⚠️ <strong>Anomaly Identification:</strong> Object2Vec is particularly useful for anomaly detection, leveraging learned embeddings to identify unusual patterns or objects.</li></ul><h4>Hyperparameters:</h4><p>Object2Vec requires two key hyperparameters for proper embedding configuration: enc0_max_seq_len, which controls the maximum length of input sequences, and enc0_vocab_size, which defines the size of the vocabulary for the embedding model. These parameters ensure the embeddings are created with the appropriate scope for the dataset.</p><p>In addition to these required hyperparameters, several optional ones can help with fine-tuning the model:</p><ul><li>epochs: Specifies the number of layers in the neural network encoder for deeper topic discovery.</li><li>learning_rate: Determines the step size during optimization, influencing how quickly the model converges.</li><li>dropout: The dropout probability for network layers. <em>Dropout</em> is a form of regularization used in neural networks that reduces overfitting by trimming codependent neurons.</li></ul><h3>Sequence to Sequence</h3><p>Amazon SageMaker Sequence to Sequence is a supervised learning algorithm that processes input sequences of tokens, such as text or audio, to generate corresponding output sequences. This algorithm finds applications in various tasks, including machine translation, text summarization, and speech-to-text conversion.</p><ul><li>🤖 <strong>Versatile Input Processing:</strong> Processes input sequences of tokens, such as text or audio, making it suitable for various data types.</li><li>🗣️ <strong>Sequence Generation:</strong> Generates corresponding output sequences of tokens, enabling complex transformations.</li><li>📝<strong> Wide Application Range:</strong> Finds applications in machine translation, text summarization, and speech-to-text conversion, among others.</li><li>🧠 <strong>Advanced Architecture:</strong> Utilizes deep neural networks, including RNNs and CNNs with attention mechanisms, for sophisticated sequence processing.</li><li>🚀<strong> Performance Improvements:</strong> Achieves significant performance improvements over traditional methods in sequence-to-sequence tasks.</li></ul><h4>Functionality</h4><ol><li><strong>Input Processing: </strong>The algorithm takes in a sequence of tokens (e.g., words in a sentence, audio frames).</li><li><strong>Encoding: </strong>The input sequence is encoded into a fixed-dimensional representation.</li><li><strong>Decoding: </strong>The encoded representation is then decoded into the target sequence.</li><li><strong>Attention Mechanism: </strong>Utilizes attention mechanisms to focus on relevant parts of the input sequence during decoding.</li></ol><h4>Practical Applications</h4><p>Sequence to Sequence finds applications in various domains:</p><ol><li><strong>Machine Translation:</strong> Translating text from one language to another.</li><li><strong>Text Summarization: </strong>Generating concise summaries of longer texts.</li><li><strong>Speech-to-Text Conversion:</strong> Transcribing spoken language into written text.</li><li><strong>Code Generation:</strong> Translating natural language descriptions into programming code.</li><li><strong>Chatbots and Dialogue Systems:</strong> Generating appropriate responses in conversational AI.</li></ol><p>By leveraging SageMaker’s implementation of Sequence to Sequence, developers and data scientists can tackle complex sequence transformation tasks with improved efficiency and accuracy.</p><h4>Hyperparameters:</h4><p>The Sequence-to-Sequence algorithm in Amazon SageMaker offers a wide range of optional hyperparameters for customization. Here are just a few examples of the available parameters you can fine-tune:</p><ul><li>batch_size: Defines the mini-batch size for gradient descent during training. The default value is 64, but it can be adjusted based on data size and system resources.</li><li>beam_size: Specifies the length of the beam for beam search, used during inference to control the number of candidate sequences considered. The default is 5.</li><li>bleu_sample_size: The number of instances to select from the validation dataset to compute the BLEU score during training. You can set this to -1 for full validation or any positive integer. The default is 0.</li><li>bucket_width: Adjusts the source and target bucket widths when handling variable-length sequences, which helps improve training efficiency.</li><li>checkpoint_frequency_num_batches: Sets how often to save checkpoints during training, allowing for early stopping and model retrieval. The default is every X batches.</li></ul><h3><strong>Choosing the Right Algorithm and Complementary Use Cases</strong></h3><p>When working with machine learning algorithms, the choice between different models depends on the problem you’re solving, the data you have, and your desired output. While many algorithms can handle a range of tasks, some are better suited for specific use cases. Here’s a guide to help you decide when to choose one algorithm over another or how they might work together:</p><h4><strong>Sequence to Sequence vs. Object2Vec</strong></h4><ul><li><strong>When to Choose Sequence to Sequence:</strong> If you’re dealing with tasks like <strong>machine translation</strong>, <strong>text summarization</strong>, or <strong>speech-to-text conversion</strong>, Sequence to Sequence is ideal because it can map input sequences (e.g., sentences or audio frames) to output sequences with complex transformations. It’s particularly strong in capturing and generating contextual information.</li><li><strong>When to Choose Object2Vec:</strong> Object2Vec excels at <strong>embedding high-dimensional objects</strong> and preserving relationships between them. This makes it perfect for tasks such as <strong>anomaly detection</strong>, <strong>recommendation systems</strong>, and <strong>semantic analysis</strong>, where you need to create dense representations of objects while maintaining their relationships.</li></ul><h4><strong>NTM vs. LDA for Topic Modeling</strong></h4><ul><li><strong>NTM</strong>: Neural Topic Modeling (NTM) is powerful when you need to uncover complex, nonlinear relationships between words in large text corpora. It leverages deep learning and can capture intricate patterns, making it suitable for <strong>large datasets</strong> where patterns may not be immediately obvious.</li><li><strong>LDA</strong>: Latent Dirichlet Allocation (LDA) is more interpretable and works well for simpler topic modeling tasks where the topics are more distinct and linear in nature. It’s best used when you need quick insights into text without the computational overhead of neural networks.</li></ul><p>LDA can serve as a great starting point for exploring your data and identifying basic topics. Once those initial insights are uncovered, NTM can be applied for more in-depth analysis, allowing you to refine the topics or uncover more complex patterns within the data.</p><h4><strong>BlazingText vs. Object2Vec</strong></h4><ul><li><strong>When to Choose BlazingText</strong>: If your focus is on fast, scalable training of word embeddings (e.g., for <strong>text classification</strong>, <strong>entity recognition</strong>, or <strong>semantic similarity tasks</strong>), BlazingText provides an efficient and highly scalable solution, particularly for text-heavy tasks.</li><li><strong>When to Choose Object2Vec</strong>: Object2Vec is better suited when you need to embed objects other than just text (e.g., <strong>graph nodes</strong>, <strong>products</strong>, or <strong>images</strong>), preserving their relationships in lower-dimensional space. It’s perfect for use cases like <strong>anomaly detection</strong> and <strong>recommendation systems</strong> that require embeddings of non-textual data.</li></ul><p>BlazingText can be used to embed text-related data, while Object2Vec can handle embeddings for objects like products or entities. By combining both, you can build a richer, multi-dimensional model that supports tasks like personalized recommendations or multi-modal analysis.</p><h4><strong>Combining Algorithms for Complex Workflows</strong></h4><ul><li><strong>Sequence to Sequence for Text Summarization</strong> + <strong>Object2Vec for Semantic Preservation</strong>: You could use Sequence to Sequence to summarize large blocks of text and Object2Vec to embed the summarized content for downstream tasks like <strong>document retrieval</strong> or <strong>recommendation systems</strong>.</li><li><strong>NTM for Topic Discovery</strong> + <strong>BlazingText for Text Classification</strong>: After using NTM to discover underlying topics in a dataset, BlazingText can be used to classify those topics into categories or entities, offering a two-step approach to content analysis.</li></ul><h3>Implementation in SageMaker</h3><p>Across the algorithms we’ve discussed, Amazon SageMaker provides a consistent pattern for implementation, utilizing SageMaker-specific classes and methods. Each algorithm is typically available as a SageMaker Estimator class, which is a SageMaker construct that simplifies the training process. Data preparation often involves using the SageMaker-specific record_set() utility method, which streamlines the data ingestion process by handling the upload to S3 and creating the necessary RecordSet objects. Once trained, models can be easily deployed using the SageMaker deploy() method, which creates a SageMaker Endpoint for real-time inference. This deployment process returns a SageMaker Predictor object, allowing for straightforward interaction with the deployed model. Additionally, each algorithm offers various hyperparameters that can be adjusted within the SageMaker framework to optimize performance for specific use cases. This consistent pattern of SageMaker-specific tools across algorithms makes it easier for developers and data scientists to work with different models within the SageMaker ecosystem, reducing the learning curve when transitioning between algorithms.</p><h3>Further Resources</h3><p>For those looking to dive deeper into these algorithms or seeking the most up-to-date information, I recommend visiting the <a href="https://sagemaker.readthedocs.io/en/stable/algorithms">official Amazon SageMaker SDK documentation</a>. The Algorithms section <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html">here</a> also provides comprehensive details on all the algorithms we’ve discussed here, as well as others available in SageMaker. This resource offers in-depth explanations of algorithm parameters, input/output specifications, and implementation details that are crucial for fine-tuning your models. While this article serves as an overview and study aid, the official AWS documentation should be your go-to source for the most current and detailed information as you work with these machine learning tools in your projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95eb5aa5301a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/brief-guide-to-amazon-sagemaker-algorithms-text-processing-95eb5aa5301a">Brief Guide to Amazon SageMaker Algorithms: Text Processing</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The ABCs of Profitability: How Pre-Series A Founders Can Leverage AWS to Build Scalable…]]></title>
            <link>https://medium.com/protagona/the-abcs-of-profitability-how-pre-series-a-founders-can-leverage-aws-to-build-scalable-db7841fb53db?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/db7841fb53db</guid>
            <dc:creator><![CDATA[Shaye Moessner]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 15:57:43 GMT</pubDate>
            <atom:updated>2024-05-02T18:03:55.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>The ABCs of Profitability: How Pre-Series A Founders Can Leverage AWS to Build Scalable, Cost-Effective Startups</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOHzMyvSnuBcy9sqNLUyMA.jpeg" /></figure><p>It’s no secret — the venture capital (VC) landscape has undergone significant changes post-pandemic. As a founder, it begs the question — how do you set yourself and your team up for success from launch?</p><h3>Product-First Is Out and Customer-First Is In</h3><p>It sounds obvious, but it wasn’t long ago that the product-first mindset had Silicon Valley in a chokehold. What was considered new and cool took precedence, regardless of whether or not customers actually wanted to use it. There were products galore that sounded really amazing and innovative in theory but in reality, ended up falling flat when the time came for customers to pay. The value simply wasn’t there and multiple recessions inside of a decade left the market unwilling to part with their cash for another feature-bloated subscription.</p><p>Soon enough, investors started realizing that their odds of ROI were getting slimmer and slimmer. So the stakes skyrocketed. 100X became the standard. All of the sudden, it took a unicorn to get a return.</p><p>And then, the pandemic happened. Obvious economic impact aside, the effect on the startup community was a sharp contrast to what we’d observed in years past. To get a check, you had to prove results. You had to have a little something called product-market fit. Headcount? Irrelevant. Downloads? Forget it. Even total revenue was on the chopping block.</p><p>The market has made a significant pivot towards products that are directly in line with what customers are asking for, and they’re not afraid to dig deep to find the metrics that matter to assess product-market fit. Taking a page out of Amazon’s book, it was time to <em>get customer obsessed. </em>Whether that’s through user research, focus groups, or running ads against a landing page to gauge interest — product inception starts with real, tangible challenges your future buyers experience today.</p><h3>Traction Trumps Potential</h3><p>No longer are VCs willing to invest in startups based on their potential alone. Previously, a great idea, a strong team, and a solid business plan were enough to secure funding, particularly in the early stages. However, the tide has turned, and investors are prioritizing traction.</p><p>Even angel investors, who traditionally back early-stage startups, are increasingly looking for evidence of traction before writing a check. This means that founders need to focus on validating their ideas and gaining initial customers or users before seeking investment.</p><h3>Metrics Matter More Than Ever</h3><p>VCs are also becoming more data-driven in their investment decisions. They want to see concrete metrics that demonstrate your startup’s growth potential and viability. This goes beyond just revenue and includes factors such as:</p><ol><li><strong>Net Retention: </strong>Starting strong with net retention, or how much you’ll grow or shrink without any new customer action. Frightening? Potentially. Enlightening? Certainly.</li><li><strong>Burn Ratio:</strong> If you’re a startup on fire (in a good way) and cruising through cash to grow, this is your friendly reality check. Just divide your Net New ARR by your FCF Burn, and voila 🔥</li><li><strong>Cash Conversion Cycle:</strong> CCC is all about how long it takes your company to turn investments into cold, hard cash from sales. It’s like a stopwatch for your inventory, receivables, and bills.</li><li><strong>LTV by Cohort:</strong> Lifetime value is basically how much you expect to make from a customer before they churn. It’s all the sales they bring in after your cost of acquisition.</li><li><strong>Contribution Margin:</strong> In a nutshell, it’s your product line or sales team’s bookings minus their costs. Knowing how much each department is pitching in is clutch for making smart investment moves.</li><li><strong>CAC Payback Period:</strong> This is what you spend on sales and marketing to a new customer. It’s crucial to show that your go-to-market game is on point.</li><li><strong>Sales Quick Ratio: </strong>Want a quick and dirty way to see if your growth is on track? Divide your Sales by your Churn. It’s like a health check for your sales machine.</li><li><strong>User Activation Rate:</strong> Also known as the product activation rate, this tells you how many trial users hit the activation milestone. Just divide the number of users who made it by the total number who signed up.</li><li><strong>LTM Revenue vs Gross Margin vs Profit Margin</strong>: Your P&amp;L statement has three key components: Revenue, Gross Margin, and Profit Margin. Revenue and Profit Margin are the top and bottom lines, while Gross Margin sits in between. By tracking these metrics over the last 12 months, you can get a clear picture of your company’s financial health and spot any trends or changes in performance.</li><li><strong>Customer Concentration:</strong> Landing big deals is awesome, but relying too heavily on a handful of customers can be risky business. Keep an eye on individual customer risk, so one big churn doesn’t tank your business.</li><li><strong>IRR and ROE for Projects: </strong>The biggest mistake leaders make after investing in a specific part of their business is not measuring the return. Make sure to calculate Internal Rate of Return and Return on Equity every six months to stay on top of it.</li><li><strong>Developer Productivity:</strong> Every CTO wants their dev team to crush it. High-quality, speedy application deployment means faster time to market (and more revenue). The key is knowing where to focus those efforts.</li><li><strong>User Conversion Rates:</strong> Not all sales and marketing pipelines are created equal. Sure, you want to max out your funnel’s potential, but at the end of the day, it’s all about the conversion rate.</li><li><strong>Gross Margin:</strong> This is your revenue minus the cost of goods. Higher margins mean more room to invest and grow.</li><li><strong>Revenue Growth:</strong> Last but definitely not least, and probably the most important metric of ’em all. Here’s a handy rule of thumb: 6% month-over-month growth equals 100% year-over-year growth.</li></ol><p>The bottom line is you <em>must</em> be prepared to present a clear picture of your key performance indicators (KPIs) and how you plan to scale these metrics over time. Having a solid understanding of your data and being able to communicate it effectively can make all the difference in securing funding.</p><h3>The Path to Profitability Starts with the Right Infrastructure and Long Term Roadmap</h3><p>Here’s how Protagona partners with founders and their teams to help them scale effectively (and profitably):</p><ol><li><strong>Cloud Infrastructure Optimization</strong>: We help you optimize your cloud infrastructure, ensuring that it’s not just a cost center, but a revenue driver. By leveraging AWS’s scalable, secure, and cost-effective solutions, we enable startups to focus on innovation and growth while minimizing technical debt and operational overhead. The result? Improved efficiency, faster time-to-market, and better overall performance that directly impacts your bottom line.</li><li><strong>Data Analytics and Insights</strong>: We help you harness the power of data, transforming it from a mere byproduct of your operations into a strategic asset. Our team works with yours to assess customer behavior, market trends, and operational performance. These insights inform strategic decision-making, product development, and go-to-market strategies, enabling you to stay ahead of the competition and drive revenue growth.</li><li><strong>Artificial Intelligence and Machine Learning</strong>: We help you identify and implement profitable AI and ML use cases. By enabling startup teams to build intelligent applications, automate processes, and create personalized customer experiences, we help you differentiate your offerings and tap into new revenue streams.</li><li><strong>Cybersecurity and Compliance</strong>: We provide guidance on best practices for securing your cloud environments, implementing access controls, and meeting compliance requirements such as GDPR, HIPAA, or SOC. By ensuring your product is properly protected and compliant, we help you safeguard your valuable data assets and maintain the trust of your customers.</li><li><strong>Due Diligence and Technical Assessments</strong>: Our technical due diligence services help you assess your tech and infrastructure.. By understanding the strengths, weaknesses, and scalability of your stack, you can make informed investment decisions and identify areas for improvement as prepare to scale, raise, or merger / acquisition.</li><li><strong>Support, Mentorship, and Funding</strong>: As technical advisors and mentors, we provide guidance on architecture design, performance optimization, and cloud best practices to help your portfolio companies scale efficiently and cost-effectively. We also facilitate access to AWS funding programs, resources, training, and support designed specifically for startups, ensuring your teams have the tools and knowledge they need to succeed.</li><li><strong>Strategic Partner Network</strong>: Our extensive network of technology partners allows us to recommend specialists in various fields, ensuring that you have access to the expertise for whatever needs may arise.</li><li><strong>Co-Selling</strong>: We work with startup founders to help co-sell their products to the AWS network and make introductions to other clients or contacts who might benefit from their offerings.</li></ol><h3>Navigating the New Normal</h3><p>Things are certainly different, but we’re confident that the fresh emphasis on actionable metrics is a good thing in the long run. By focusing on traction and choosing the right infrastructure, you can position yourself to attract investors and secure the capital you need to grow.</p><p>It also puts you in a great position for alternatives, like acquisition or simply running a profitable business without having to be continually reliant on capital injection.</p><p>As an AWS consulting partner, we specialize in helping startup teams build on AWS and leverage its full potential to drive growth and profitability. Whether you’re just starting out or looking to scale, we can help you build a solid foundation on AWS and set your startup up for long-term success.</p><p><em>If you’re a startup founder or CTO, we want to help! Reach out to us at </em><a href="/service/mailto:contact@protagona.com"><em>contact@protagona.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db7841fb53db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/the-abcs-of-profitability-how-pre-series-a-founders-can-leverage-aws-to-build-scalable-db7841fb53db">The ABCs of Profitability: How Pre-Series A Founders Can Leverage AWS to Build Scalable…</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Protagona’s AWS Expertise Can Help VCs and Startups Thrive]]></title>
            <link>https://medium.com/protagona/how-protagonas-aws-expertise-can-help-vcs-and-startups-thrive-729995a0b59d?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/729995a0b59d</guid>
            <dc:creator><![CDATA[Shaye Moessner]]></dc:creator>
            <pubDate>Mon, 08 Apr 2024 16:37:23 GMT</pubDate>
            <atom:updated>2024-04-02T21:41:32.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAjrCZal2rfjZSEy98hppA.png" /><figcaption><em>She invented Post-its.</em></figcaption></figure><p>As VCs face unprecedented challenges in 2024, with a difficult fundraising environment and increased competition, it’s more crucial than ever for firms to differentiate themselves and support their portfolio companies effectively.</p><p>At Protagona, we understand the unique needs of VCs and startups in this rapidly evolving landscape. Our deep expertise in Amazon Web Services (AWS) and our comprehensive range of technical services are designed to help VCs make informed investment decisions, optimize their portfolio companies’ cloud infrastructure, and drive growth and innovation in the face of adversity.</p><p>So let’s explore the current state of venture capital, the characteristics of high-performing funds, and how Protagona can be a valuable partner in navigating these times.</p><h3><strong>The State of Venture Capital in 2024</strong></h3><p><strong>2023–2024: The Most Challenging Fundraising Environment in Years</strong></p><p>VC performance in 2023 was hindered by the collapse of Silicon Valley Bank, a difficult market for exits, and a tough market for fundraising.</p><ol><li><strong>Market regulation after post-Covid digitization hype:</strong> During the COVID-19 pandemic, there was a surge in investment activity, particularly in digital technologies, as businesses and consumers rapidly adopted online solutions. However, this hype led to overvaluation, market saturation, and unrealistic expectations for returns. As a result, the market is now undergoing a correction, with investors becoming more cautious and selective in their investments.</li><li><strong>Investors scaling back their VC investments proportionately:</strong> Just like any asset class, VC requires commitment to the long term outlook. In 2023 and 2024, we see fund investors that were unfamiliar and uncomfortable with startup markets (understandably) pulling back their investments. In 2023, there was $170.6 billion of VC invested in 15,766 deals, which was well below the $242.2 billion in VC invested across 17,592 deals in 2022.</li><li><strong>Allocators into venture are trying to figure out where they want to concentrate their dollars:</strong> With the market correction and increased uncertainty, investors who allocate capital to venture funds (such as institutional investors and high-net-worth individuals) are reassessing their investment strategies.</li><li><strong>VCs are being forced to course correct:</strong> VCs are having to adapt to the new market realities by revising their investment strategies, focusing on capital efficiency, and supporting their portfolio companies through the challenges. This means some difficult decisions are on the horizon — cutting costs, restructuring, or merging underperforming startups to ensure their survival and growth prospects.</li><li><strong>In order to get more investment, you have to have a healthy portfolio today:</strong> In this environment, the performance of a firm’s existing portfolio companies is mission critical. Investors are paying close attention to how well these startups are weathering the downturn, managing their cash flow, and adapting their business models. VCs that have invested in resilient and innovative companies are better positioned to attract new capital. Sounds simple, but this wasn’t always the case.</li><li><strong>2024 will look a lot like 2023:</strong> The competition between VCs is expected to intensify due to a confluence of factors. One major reason is that funds from previous years are still actively seeking to raise capital, leading to a crowded market. This situation is compelling VCs to differentiate themselves and their investment strategies to attract limited partners (LPs) and secure funding. As a result, VCs are carefully calibrating their fund sizes to align with their strategic objectives, sector focus, and stage preferences. Those who can demonstrate a clear and compelling vision, backed by a strong track record and a well-articulated plan for deploying capital, will be best positioned to stand out in the crowded field and attract the necessary funding to fuel their investment activities.</li></ol><h3><strong>The Highly Effective Habits of High Performing Funds</strong></h3><p><strong>Habit #1: Stay agile (but don’t lose focus)</strong></p><p>Smaller funds are winning. VCs in the $50 million to $200 million range are outperforming their larger, more tenured competitors.</p><p>This phenomenon can be attributed to several factors:</p><ol><li><strong>Agility and focus</strong>: Smaller funds often have a more focused investment strategy, targeting specific sectors, stages, or geographies. This allows them to develop deep expertise and networks in their chosen areas, leading to better deal sourcing and value-add for their portfolio companies. They can also make quicker investment decisions and adapt to changing market conditions more easily than larger funds.</li><li><strong>Alignment of interests:</strong> Smaller funds typically have a higher proportion of the fund manager’s personal capital invested, aligning their interests more closely with those of their LPs. This can lead to a stronger commitment to maximizing returns and a more hands-on approach to supporting portfolio companies.</li><li><strong>Lower entry valuations:</strong> Smaller funds often invest in earlier-stage startups or in less competitive markets, where entry valuations are lower. This allows them to acquire significant ownership stakes in companies with high growth potential, leading to outsized returns if these startups succeed.</li><li><strong>Niche opportunities:</strong> Smaller funds can target niche or emerging sectors that may be overlooked by larger funds, which often focus on more established industries. By identifying and investing in these untapped opportunities, they can generate higher returns.</li><li><strong>Lack of mega-deals:</strong> Larger funds may feel pressure to deploy capital in bigger deals or later-stage companies to maintain their fund size and management fees. However, these mega-deals can be highly competitive and may not always generate the best returns. Smaller funds can be more selective and disciplined in their investment approach.</li></ol><p><strong>Habit #2: Don’t learn on your allocators’ time and dime</strong></p><p>Allocators expect fund managers to have a proven track record, deep expertise, and a well-defined investment strategy before entrusting them with their capital.</p><p>They don’t want to bear the cost or risk associated with inexperienced or unproven VCs who are still learning the ropes. They prefer to invest in funds managed by seasoned professionals who have demonstrated their ability to identify promising startups, provide strategic guidance, and generate strong returns.</p><p>You’ve got to have a strong investment thesis, a solid network, and the right partners to make the cut.</p><p><strong>Habit #3: Know what allocators are <em>really </em>looking for</strong></p><p>When allocators are considering investing in a fund, they typically focus on four key factors:</p><ol><li><strong>Rationalizing existing relationships with managers:</strong> Relationships matter. Allocators often prioritize maintaining and strengthening their relationships with fund managers they have previously invested with. They evaluate the performance of these existing relationships and assess whether the fund managers continue to align with their investment objectives and deliver satisfactory returns.</li><li><strong>Networks and introductions from other VCs or trusted partner:</strong> Allocators value recommendations and introductions from trusted VC peers. They rely on their networks to source new fund managers and gain insights into their reputation, track record, and investment approach. Positive endorsements from respected VCs can significantly influence an allocator’s decision to invest in a particular fund.</li><li><strong>Grassroots — finding funds that specialize:</strong> Product-market fit isn’t just for portfolio companies. Allocators are increasingly interested in funds that have a specialized focus, such as targeting specific sectors, technologies, or geographies. These specialized funds often have a deeper understanding of their chosen domains, enabling them to identify unique investment opportunities and provide value-add to their portfolio companies. <em>However, there’s a fine line between being specialized and being tied to the whims of a particular market.</em> For example, if your fund was all-in on web3 and crypto in 2022, you likely experienced significant losses as that sector cratered amidst the “crypto winter.” Overspecialization and concentration risk can leave a fund vulnerable to downturns in its focus area. Maintaining some diversification, even within a specialized strategy, is prudent to mitigate risk.</li><li><strong>Unique value proposition: </strong>Allocators look for funds that offer a compelling and differentiated value proposition. This can include a proven investment strategy, exceptional team expertise, proprietary deal flow, or a strong track record of successful exits. Funds that can articulate and demonstrate their unique edge in the competitive venture capital landscape are more likely to attract allocator interest and investment.</li></ol><h3>How Protagona Can Help VCs and Startups</h3><p>You can’t be everything to everyone. Cliché, but true. There’s a lot expected out of fund managers, and it’s challenging to keep up with it all.</p><p>You’re expected to possess a wide range of skills and knowledge, from financial analysis and portfolio management to industry expertise and networking. However, attempting to excel in every area can lead to burnout and subpar performance.</p><p>Instead, focus on your core competencies and consider trusted partners to help you fill the gaps.</p><p>At Protagona, we work with VCs as an extension of their leadership by offering tailored support in areas where they need it most. Our team has deep domain knowledge and can help you navigate the rapidly evolving tech landscape, identify promising startups, and make informed investment decisions.</p><p><strong>How we partner with fund managers and founders:</strong></p><ol><li><strong>Cloud infrastructure optimization</strong>: We help VCs assess and optimize the cloud infrastructure of their portfolio companies. By leveraging AWS’s scalable, secure, and cost-effective solutions, startups can focus on innovation and growth while minimizing technical debt and operational overhead. This means improved efficiency, faster time-to-market, and better overall performance.</li><li><strong>Data analytics and insights:</strong> We help VCs and their portfolio companies harness the power of data to gain valuable insights into customer behavior, market trends, and operational performance. These insights inform strategic decision-making, product development, and go-to-market strategies.</li><li><strong>Artificial Intelligence and Machine Learning:</strong> We help VCs identify and implement AI/ML use cases across their portfolio, enabling startups to build intelligent applications, automate processes, and create personalized customer experiences. This can be a significant differentiator in competitive markets.</li><li><strong>Cybersecurity and compliance:</strong> With the increasing importance of data privacy and security, VCs need to ensure that their portfolio companies are properly protected and compliant with relevant regulations. Protagona provides guidance on best practices for securing cloud environments, implementing access controls, and meeting compliance requirements such as GDPR, HIPAA, or SOC.</li><li><strong>Due diligence and technical assessments</strong>: When evaluating potential investments, VCs need to assess the technical capabilities and infrastructure of target companies. We provide technical due diligence services, helping VCs understand the strengths, weaknesses, and scalability of a startup’s technology stack. This can inform investment decisions and help identify areas for improvement post-investment.</li><li><strong>Portfolio company support and mentorship:</strong> We act as technical advisors and mentors to VC portfolio companies, providing guidance on architecture design, performance optimization, and cloud best practices. We can also facilitate access to AWS funding programs, resources, training, and support designed for startups.</li><li><strong>Strategic partner network:</strong> We can act as a resource for VCs to recommend other technology partners with various specialities.</li><li><strong>Co-selling:</strong> We can work with startup founders to help co-sell their products to the AWS network and make introductions to other clients or contacts who might need their product.</li></ol><p>By collaborating with external experts, you can free up time and energy to focus on what you do best, whether that’s fundraising, relationship building, or strategic planning.</p><p>Successful fund managers aren’t afraid to ask for help when they need it. By surrounding yourself with a strong team of internal and external experts, you can deliver better results for your investors (and maintain your sanity).</p><p><em>If you’re curious to learn more, reach out to us at </em><a href="/service/mailto:contact@protagona.com"><em>contact@protagona.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=729995a0b59d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/how-protagonas-aws-expertise-can-help-vcs-and-startups-thrive-729995a0b59d">How Protagona’s AWS Expertise Can Help VCs and Startups Thrive</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When it comes to AI in the enterprise, it’s fear that’s driving the frenzy to try to get projects…]]></title>
            <link>https://medium.com/protagona/when-it-comes-to-ai-in-the-enterprise-its-fear-that-s-driving-the-frenzy-to-try-to-get-projects-6f321807527f?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f321807527f</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Jordan Wester]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 22:35:32 GMT</pubDate>
            <atom:updated>2024-03-28T22:35:32.831Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Overlooked Keys to Building Successful AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5obvB1h1HsaFoC9CZiFbg.png" /></figure><p>When it comes to AI in the enterprise, it’s fear that’s driving the frenzy to try and get projects off the ground. There’s widespread panic among every board director and C-Suite executive that a rival is building a blockbuster GenAI application. One that is going to help them leapfrog the competition and secure more market share.</p><p>As a result, most executives now have a mandate to simply do something with AI — and quickly. But often, it’s not clear what that is. So while there’s intense urgency to begin, few have any actual idea how to start.</p><p>Of course, speed is important. However, putting AI into production for the sake of claiming technology superiority is a fast way to failure. There are critical steps that organizations must take before they’re capable of achieving the outcomes they want with AI.</p><p>Below are a few of the most common challenges that our customers face when considering how to get started with AI.</p><h3>The ROI Paradox</h3><p>There’s one thing everyone wants from their AI: a return on investment.</p><p>The days of throwing real dollars at theoretical tech with no outcome in mind are over. Now, projects have to produce value. It’s the only way to ensure continued investment. But given how new AI is, organizations are struggling to define what value is in the context of the tech.</p><p>Businesses should approach planning with an open mind. And when it comes to initial AI investments, the smaller the better. Document summarization is usually enough to wow the executive team and get them energized for more.</p><p>Naturally, businesses will want to take on more sophisticated use cases. For these, they should first build a test model to ensure the desired outcomes are even possible. While it takes just a few days to construct these rudimentary systems, they only work in test settings. However, success will give companies the confidence to embark on the hard work of actually putting the model into production.</p><h3>Data Governs All</h3><p>The phrase has now become a staple in corporate lexicon: AI is only as good as the data behind it. However, it’s so overused because it’s that important.</p><p>Some leaders still think that AI will suddenly give every employee access to new levels of business intelligence and automation. But rarely are data environments actually ready for Generative AI.</p><p>While it might look like magic, AI is anything but. Behind the scenes, these models must be continually trained on hyper-relevant data. Often, that information must come from several different endpoints. And tracking down and unifying all that data can be one of the hardest steps in the AI journey. This is especially true for organizations that might still be running 20-year-old software.</p><h3>Bring in the Business</h3><p>IT may be in charge of the technical work in bridging a divided data landscape. But they must rely on domain experts to actually identify the relevant information that will make the AI useful for employees.</p><p>Often, those closest to the business understand the problems employees are facing. And solving those pain points requires their expertise. They’re also probably more knowledgeable about all the different data sources available.</p><p>Enterprises must remove any barriers that separate CIOs from the rest of the executive team. They must also encourage technology and business leaders to work together on developing the data and AI strategy.</p><p>AI takes realism to succeed. Companies must understand the gap between where they are and where they want to be. And then, start now to take the steps to close it. It’s natural to feel competitive pressure to move quickly, but speed shouldn’t come at the sacrifice of quality.</p><p>The risks that come with AI are very real. But so are the benefits. And taking the time now to build the right foundation can ensure that the former is a rarity, and the latter a guarantee.</p><p><em>If you’re curious to learn more, reach out to us at </em><a href="/service/mailto:contact@protagona.com"><em>contact@protagona.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f321807527f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/when-it-comes-to-ai-in-the-enterprise-its-fear-that-s-driving-the-frenzy-to-try-to-get-projects-6f321807527f">When it comes to AI in the enterprise, it’s fear that’s driving the frenzy to try to get projects…</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easy Health Checks With SocketPing Test]]></title>
            <link>https://medium.com/protagona/socketping-test-83f346630055?source=rss----4ad8bc182044---4</link>
            <guid isPermaLink="false">https://medium.com/p/83f346630055</guid>
            <dc:creator><![CDATA[Dana Gibson]]></dc:creator>
            <pubDate>Thu, 14 Mar 2024 15:08:08 GMT</pubDate>
            <atom:updated>2024-03-14T15:08:08.829Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of cloud computing, maintaining a vigilant eye on our systems is crucial. Continuous improvement in our monitoring and alerting practices is a fundamental philosophy. Fortunately, AWS provides valuable tools which simplify the automation of these processes. These tools play an important role in detecting and addressing potential issues promptly.</p><p><em>...2 months ago…</em><br>“How long has this been non-functional?” I’m sure they had never been more thankful that their cameras were off, because no one was really sure. This was how a meeting ended about 2 months ago with a company I was supporting.</p><p>The next morning, I looked on my JIRA, and there it was, a new shiny story called SocketPing Test. I had no idea what this was but as I opened up the architecture diagram, it all started to come into focus. A way for us to do a connectivity test periodically on different applications and be alerted if it became unavailable.</p><p>The concept was simple, trigger a Lambda function to ping a specific IP address or host name, if the connection was a success, no further action needed. If the connection failed, a message would be sent to Microsoft Teams (although it could be sent anywhere) indicating an outage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_MwNufEqa7_-D-4hoyReg.png" /></figure><p>Lets work through code. I deployed using a stack in CloudFormation, hence a yaml file.</p><h3>Triggering the Lambda</h3><p>The Lambda function will be triggered in two different ways. A scheduled event using EventBridge, and the putObject action on a S3 Bucket.<br> <strong><em><br></em></strong> When creating a rule in EventBridge, your rule can be triggered at either a scheduled rate, or when a specific event occurs (i.e. “CreateUser”). In this case, we have chosen to trigger the Lambda function every 15 minutes so we will use a scheduled rate. It is also required that you list the Lambda function as your target.</p><pre>EventRule:<br>    Type: &quot;AWS::Events::Rule&quot;<br>    Properties:<br>      Description: Event rule to trigger Lambda every 15 minutes<br>      Name: EventRule<br>      ScheduleExpression: rate(15 minutes)<br>      State: ENABLED<br>      Targets:<br>        - Arn: !GetAtt &#39;SocketPingTestFunction.Arn&#39;<br>          Id: SocketPingTestFunction</pre><p>That takes care of our scheduled trigger, lets look at how we want to trigger the Lambda function when we upload a new manifest file. Manifest file? I had the same question. Our manifest file is simply going to be a json file where we list the different IPs or hostnames we want to ping.</p><p>Below is a sample of what a manifest file could look like. In this example, the first line reaches to google on port 443, this results in a positive ping test. The second line reaches to google on port 111, this results in a negative ping test, which will send an alert to to Microsoft teams.</p><pre>{<br>    &quot;manifest&quot;: [<br>      {&quot;host&quot;:&quot;www.google.com&quot;,&quot;port&quot;:&quot;443&quot;,&quot;env&quot;:&quot;dev&quot;,&quot;description&quot;:&quot;google-success&quot;},<br>      {&quot;host&quot;:&quot;www.google.com&quot;,&quot;port&quot;:&quot;111&quot;,&quot;env&quot;:&quot;dev&quot;,&quot;description&quot;:&quot;google-fail&quot;}<br>    ]<br>}</pre><p>We will create the s3 bucket, following AWS best practices but also include the NotificationConfiguration property which will notify the Lambda that when an object is created (or the manifest file is updated and uploaded to the bucket), the Lambda should be triggered and ran.</p><pre>  S3Bucket:<br>    Type: &#39;AWS::S3::Bucket&#39;<br>    Properties:<br>      BucketName: socketpingtest<br>      PublicAccessBlockConfiguration:<br>        BlockPublicAcls: true<br>        BlockPublicPolicy: true<br>        IgnorePublicAcls: true<br>        RestrictPublicBuckets: true<br>      NotificationConfiguration:<br>        LambdaConfigurations:<br>          - Function: !GetAtt SocketPingTestFunction.Arn<br>            Event: &#39;s3:ObjectCreated:*&#39;<br>      Tags:<br>        - Key: Owner<br>          Value: &lt;Owner&#39;s Name&gt;<br>        - Key: Environment<br>          Value: &lt;Environment&gt;</pre><p>Before we jump to the Lambda function, I’m going to go ahead and share the secret sauce…the part that makes it all work…also the part I forgot. <em>Permissions. </em>YES! In order for the EventBridge rule and the manifest file upload to be able to trigger the Lambda function, you have to grant them permission to communicate with the Lambda.</p><pre>EventBridgeLambdaPermission:<br>    Type: AWS::Lambda::Permission<br>    Properties:<br>      FunctionName: !Ref &#39;SocketPingTestFunction&#39;<br>      Action: lambda:InvokeFunction<br>      Principal: events.amazonaws.com<br>      SourceArn: !GetAtt &#39;EventRule.Arn&#39;<br><br>  S3LambdaPermission:<br>    Type: AWS::Lambda::Permission<br>    Properties:<br>      FunctionName: !Ref SocketPingTestFunction<br>      Action: lambda:InvokeFunction<br>      Principal: s3.amazonaws.com<br>      SourceArn: arn:aws:s3:::socketpingtest</pre><h3>Lambda Function</h3><p>What exactly is it that we want the Lambda function to do? After it is triggered, we want it to retrieve the manifest file from our s3 bucket, iterate through the file and send a ping to each entry. If it connects successfully, no further action needed. If the connection is not successful, a message should be sent. You can customize your message to include whatever information you want, the example below sends the account id, the host we pinged, the port, and the Description. We use a function inside the Lambda code to pull the account id, and the other information is retrieved from the manifest file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*rDXLoS4uIpg6-Nso3VcgiQ.png" /></figure><p>One suggestion I will share, as you are testing your Lambda code, include a timeout period, Lambda’s default timeout period is 15 minutes, that makes for a long debug session :) spoken from experience!</p><p>I am not going to go into depth on sending a message to teams and using a Webhook URL, that is a future article…stay tuned. But you can see the code I used to send the teams message below.</p><pre>SocketPingTestFunction:<br>    Type: &quot;AWS::Lambda::Function&quot;<br>    Properties:<br>      Code:<br>        ZipFile: |<br>          import json<br>          import logging<br>          import os<br>          import http.client<br>          import boto3<br>          import socket<br>          <br>          s3 = boto3.client(&#39;s3&#39;)<br>          <br>          from urllib.request import Request, urlopen<br>          from urllib.error import URLError, HTTPError  <br>                     <br>          #Get account id for Teams message<br>          def get_aws_account_id():<br>            client = boto3.client(&#39;sts&#39;)<br>            response = client.get_caller_identity()<br>            return response[&#39;Account&#39;]<br><br>          def lambda_handler(event, context):<br>            aws_account_id = get_aws_account_id()<br>            print(aws_account_id)<br>    <br>            bucket_name = f&#39;socketpingtest&#39;<br>            object_key = &#39;manifest.json&#39;<br>          <br>            # Read the JSON object from S3<br>            response = s3.get_object(Bucket=bucket_name, Key=object_key)<br>            data = json.loads(response[&#39;Body&#39;].read())<br>            print(data)<br>            <br>            # Iterate through each manifest item<br>            for item in data.get(&quot;manifest&quot;, []):<br>              host = item.get(&quot;host&quot;)<br>              port = item.get(&quot;port&quot;)<br>              env  = item.get(&quot;env&quot;)<br>              description = item.get(&quot;description&quot;)<br>              print (host)<br>                 <br>              # Perform socket test<br>              s = socket.socket()<br>              s.settimeout(1)<br>              try:<br>                  s.connect((host, int(port)))<br>                  print(&quot;Success connecting&quot; )<br>              except:<br>                  print(&quot;Timed out connecting&quot; )<br>              <br>              <br>              # Send Teams message for timed out connection<br>              HookUrl = os.environ.get(&#39;HookUrl&#39;)<br>              message = {<br>                &quot;@context&quot;: &quot;https://schema.org/extensions&quot;,<br>                &quot;@type&quot;: &quot;MessageCard&quot;,<br>                &quot;themeColor&quot;: &quot;#FF0000&quot;,<br>                &quot;title&quot;: &quot;SocketPingTest&quot;,<br>                &quot;text&quot;: f&quot;A connectivity test against Host: {host}, Port: {port} failed. More details below...&quot;,<br>                &quot;sections&quot;: [<br>                    {<br>                        &quot;facts&quot;: [<br>                            {<br>                                &quot;name&quot;: &quot;Account&quot;,<br>                                &quot;value&quot;: f&#39;{aws_account_id}&#39;<br>                            },<br>                            {<br>                                &quot;name&quot;: &quot;Host:&quot;,<br>                               &quot;value&quot;: f&quot;{host}&quot;<br>                            },<br>                            {<br>                                &quot;name&quot;: &quot;Port&quot;,<br>                               &quot;value&quot;: f&quot;{port}&quot;<br>                           },<br>                            {<br>                                &quot;name&quot;: &quot;Description&quot;,<br>                               &quot;value&quot;: f&#39;{description}&#39;<br>                            }<br>                        ]<br>                   }<br>                ]<br>            }<br><br>            message_json = json.dumps(message).encode(&#39;utf-8&#39;)<br>            req = Request(HookUrl, data=message_json, headers={&#39;Content-Type&#39;: &#39;application/json&#39;}, method=&#39;POST&#39;)<br>          <br>            try:<br>              teamsresponse = urlopen(req)<br>              teamsresponse.read()<br>              logger.info(&quot;Message posted&quot;)<br>              print(&quot;Teams message sent&quot;)<br>            except HTTPError as e:<br>              print(f&quot;Request failed: {e.code} {e.reason}&quot;)<br>            except URLError as e:<br>              print(f&quot;Server connection failed: {e.reason}&quot;)<br>            except Exception as e:<br>              print(f&quot;Socket error: {str(e)}&quot;)<br>            finally:<br>              s.close()<br>        <br>                       <br>      Handler: index.lambda_handler<br>      FunctionName: SocketPingTestFuntion<br>      Runtime: python3.11<br>      TracingConfig:<br>        Mode: Active<br>      Timeout: 60<br>      Role: !GetAtt &#39;LambdaRole.Arn&#39;<br>      Environment:<br>        Variables:<br>          S3BucketName: socketpingtest<br>          HookUrl: &lt;insert WebHookUrl&gt; </pre><p>The last step — creating a role for the Lambda to use. Technically the only actions you need to include are allowing your lambda to assume the role and allowing the lambda access to s3 so that it can pull the manifest file. However, I always include permissions to allow for logs so that I can debug and track down errors as I am attempting to deploy my stack.</p><pre>LambdaRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: lambda.amazonaws.com<br>            Action: sts:AssumeRole<br>      Path: /<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole<br>      Policies:<br>        - PolicyName: SocketTestPolicy<br>          PolicyDocument:<br>            Version: &quot;2012-10-17&quot;<br>            Statement:<br>                - Effect: Allow<br>                  Resource: &quot;*&quot;<br>                  Action:<br>                    - logs:CreateLogGroup<br>                    - logs:CreateLogStream<br>                    - logs:PutLogEvents<br>                    - xray:PutTraceSegments<br>                    - xray:PutTelemetryRecords<br>                - Effect: Allow<br>                  Action:<br>                    - sts:AssumeRole<br>                  Resource: &quot;*&quot;<br>                - Effect: Allow<br>                  Action:<br>                   - s3:GetObject<br>                   - s3:ListBucket<br>                  Resource:<br>                   - arn:aws:s3:::socketpingtest/*&#39;<br>                   - arn:aws:s3:::socketpingtest&#39;</pre><p>Thats it! Now I realize scheduled, periodic checks are suboptimal, but this is a quick and easy way to actively <em>know</em> when something you are expecting to be up, isn’t. Plus now, you get to be the attentive SRE that discovered the outage in 15 minutes or less and not have to watch your client sit in awkward silence when the CEO asks “How long has this been down?”.</p><p>You can find the full <a href="https://github.com/dzyrowski/SocketPingTest.git">template</a> in my GitHub account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83f346630055" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/socketping-test-83f346630055">Easy Health Checks With SocketPing Test</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>