<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jason Mihalopoulos on Medium]]></title>
        <description><![CDATA[Stories by Jason Mihalopoulos on Medium]]></description>
        <link>https://medium.com/@jason.mihalopoulos?source=rss-be8135a6ce68------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j5lg49hhS635BaI6qixkWg.jpeg</url>
            <title>Stories by Jason Mihalopoulos on Medium</title>
            <link>https://medium.com/@jason.mihalopoulos?source=rss-be8135a6ce68------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:30:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jason.mihalopoulos/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless Data Processing with AWS Step Functions — An Example]]></title>
            <link>https://medium.com/weareservian/serverless-data-processing-with-aws-step-functions-an-example-6876e9bea4c0?source=rss-be8135a6ce68------2</link>
            <guid isPermaLink="false">https://medium.com/p/6876e9bea4c0</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[step-functions]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Jason Mihalopoulos]]></dc:creator>
            <pubDate>Tue, 12 Feb 2019 05:15:04 GMT</pubDate>
            <atom:updated>2019-07-08T05:24:46.473Z</atom:updated>
            <content:encoded><![CDATA[<h3>Serverless Data Processing with AWS Step Functions — An Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fUjEDqwvSNMlPDt7" /><figcaption>Photo by <a href="https://unsplash.com/@bruno_nascimento?utm_source=medium&amp;utm_medium=referral">Bruno Nascimento</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Building on the pros and cons of AWS Step Functions as evaluated by Nikola Milicic <a href="https://medium.com/weareservian/aws-step-functions-the-power-of-simplicity-10e8395af4f3">here</a>, this article presents our working example of the conditional and potentially complex data pipelines that can be achieved serverlessly via AWS Step Functions and their integration with several AWS cloud products, such as AWS Lambda.</p><p><strong>The Use Case</strong></p><p>In this example use case, we are processing the customer reviews of a company which provides many products and services. The company wants to handle negative reviews quickly and appropriately, but does not have the resources to read each review.</p><p>By using an AWS Step Function in the data pipeline for customer reviews, each review can be categorised by metadata, sent to the appropriate sentiment analysis (implemented with AWS Comprehend), send out automatic notifications if negative feedback is detected, write relevant data to the review database, and appropriately handle errors at each stage of this process, all achieved serverlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WV0pDDoCsivu5w8tLKMgnQ.png" /><figcaption>A summarized flowchart demonstrating the serverless integration of AWS products used in this example.</figcaption></figure><p><strong>The State Machine</strong></p><p>Step Functions allow us to define a state machine using Amazon States Language (ASL), which is essentially a JSON object that defines the available states of the state machine, as well as the connections between them.</p><p>AWS generates a flowchart from our ASL code, which allows us to better visualise the machine, as seen here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MbvmA9OrEOSVGRpE" /></figure><p>AWS Automatically generates and updates your flowchart as you modify your state machine.</p><p>Additionally, each state machine execution provides a colour-coded flowchart of status and outcome, improving the visibility of failure points and paths taken through the machine. An example of this shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IzYypPEqkT04R9DC" /><figcaption>Better check what’s going on in that WriteToDB block!</figcaption></figure><p>The full ASL code is accessible from our GitHub repository, available <a href="https://github.com/servian/aws-step-function-example/blob/master/state-machine">here</a>.</p><p><a href="https://github.com/servian/aws-step-function-example">servian/aws-step-function-example</a></p><p><strong>The Trigger</strong></p><p>We can use a Lambda function to trigger the state-machine and pass through the file details, upon its landing in the S3 bucket. In our example, we trigger the Lambda function each time a CSV file is dropped in the /reviews folder of our S3 bucket, which in turn triggers the Step Function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/972db3a25ec658c7f07462069f1ae663/href">https://medium.com/media/972db3a25ec658c7f07462069f1ae663/href</a></iframe><p>Step machines can also be triggered using other trigger conditions and other methods. For example, periodic triggers can be set using Amazon CloudWatch Events rules.</p><p><strong>Reading the Input</strong></p><p>In our example, reviews are submitted as single-line CSV files with parameter headers. However, this implementation could easily be extended to other input data formats, perhaps utilising alternative storage solutions from AWS, such as Kinesis Streams or SQS.</p><p>The Lambda function below reads the input file and outputs its data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49b134e8b59647d712a36d9ec95ab3a5/href">https://medium.com/media/49b134e8b59647d712a36d9ec95ab3a5/href</a></iframe><p><strong>Product or Service?</strong></p><p>Our company wants the capability to address reviews differently based on whether the review is for a product or service. To implement this functionality we use a “Choice” block of the Step Function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d91efc8f8cf9948baafed4ad0d32e8f4/href">https://medium.com/media/d91efc8f8cf9948baafed4ad0d32e8f4/href</a></iframe><p>More complex use cases could employ multiple branching classifications, or even complex ML models for classification, stored in S3 and evaluated using Lambda functions.</p><p><strong>Sentiment Analysis</strong></p><p>We perform sentiment analysis on the review data, returning ‘POSITIVE’, ‘NEGATIVE’, or ‘NEUTRAL’. Calling Amazon Comprehend in a Lambda function, we can achieve this with not much more than one line of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11b6a2cf20c53021a75a27772edfc3ed/href">https://medium.com/media/11b6a2cf20c53021a75a27772edfc3ed/href</a></iframe><p>A more detailed implementation could use other features of Amazon Comprehend to further delineate the review procedure, perhaps by detecting entities or named entities in the customer’s review to determine the most appropriate company department or staff member to alert.</p><p><strong>Positive or Negative?</strong></p><p>As our use case is focused on handling negative reviews, our pipeline only sends a notification via SNS if the result of the sentiment analysis is “NEGATIVE”. This is implemented using a second “Choice” block in the state machine:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0621d6ec50b0b43ac76f9d78ea41024d/href">https://medium.com/media/0621d6ec50b0b43ac76f9d78ea41024d/href</a></iframe><p>Using a choice block allows easy and transparent error handling, as indicated by the default choice states of <em>SentimentFail</em> (unexpected result from sentiment analysis) and <em>CategorisationFail</em> (unable to categorise the input).</p><p><strong>SNS Notification</strong></p><p>In our example, a notification is sent to the company via email whenever a negative review is placed. Using AWS’s Simple Notification Service (SNS), this is achieved in a single Lambda function call. Additionally, the message contains dynamic information depending on the details of the data being processed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92b976ad7f461d0e0be0dc4caf6c67bd/href">https://medium.com/media/92b976ad7f461d0e0be0dc4caf6c67bd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*E9VKAX1k98OIZQWzO8US-w.png" /><figcaption>An example notification, sent via email.</figcaption></figure><p><strong>Writing results to the Database</strong></p><p>Whether the result from the sentiment analysis is positive or negative, we want to store the results in a database for future reference. In our example, we use a short Lambda function to write to the database and pass input variables through to the output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbd705606771064e4fda201d0f06b7b1/href">https://medium.com/media/bbd705606771064e4fda201d0f06b7b1/href</a></iframe><p>AWS’s DynamoDB is well-integrated into both Step Functions and Lambda, meaning that similar functionality can also be achieved directly through AWS Step Functions.</p><p><strong>Conclusion</strong></p><p>Using AWS Step Functions and Lambda, we have demonstrated how a serverless data pipeline can be achieved with only a handful of code, with a relatively high complexity ceiling, in-built scaling and parallelisation, and integration with many diverse AWS products. The above example is only a toy example for demonstration purposes, but hopefully it will serve as inspiration for other ideas which can now be achieved serverlessly.</p><p>Find out more about Servian’s data and analytics capabilities <a href="https://www.servian.com/data-and-analytics/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6876e9bea4c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/weareservian/serverless-data-processing-with-aws-step-functions-an-example-6876e9bea4c0">Serverless Data Processing with AWS Step Functions — An Example</a> was originally published in <a href="https://medium.com/weareservian">Cognizant Servian</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>