<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manishadutt on Medium]]></title>
        <description><![CDATA[Stories by Manishadutt on Medium]]></description>
        <link>https://medium.com/@manishadutt90?source=rss-cfc610576d65------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-fRZS5-JRMs-hOez</url>
            <title>Stories by Manishadutt on Medium</title>
            <link>https://medium.com/@manishadutt90?source=rss-cfc610576d65------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 00:12:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manishadutt90/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Efficiently Scanning and Processing DynamoDB with AWS Step Functions]]></title>
            <link>https://medium.com/@manishadutt90/efficiently-scanning-and-processing-dynamodb-with-aws-step-functions-ac57bebd0f26?source=rss-cfc610576d65------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac57bebd0f26</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Manishadutt]]></dc:creator>
            <pubDate>Sun, 07 Sep 2025 11:58:31 GMT</pubDate>
            <atom:updated>2025-09-07T11:58:31.588Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently in one of my projects, we had a requirement to <strong>scan a DynamoDB table and process each row</strong>.</p><p>Our initial approach was to use <strong>AWS Lambda</strong>, since it’s cost-effective and doesn’t require heavy infrastructure — you simply pay for what you use. At first glance, this seemed like the perfect solution. However, Lambda functions have a <strong>maximum execution time of 15 minutes</strong>. If your table has a large number of rows, the function may time out before completing the scan, leaving the job unfinished.</p><p>Clearly, this was not an efficient approach. So, what’s next?</p><p>That’s when we explored <strong>AWS Step Functions</strong> — another serverless feature that allows us to orchestrate long-running processes reliably while still paying only for what we use.</p><h3>What Are AWS Step Functions?</h3><p>AWS Step Functions is a <strong>serverless orchestration service</strong> that helps coordinate multiple AWS services into workflows using a visual interface.</p><p>It allows you to break down complex business processes into <strong>states</strong> and <strong>transitions</strong>, making workflows easier to design, manage, and monitor. Common use cases include:</p><ul><li>Data processing</li><li>Automation workflows</li><li>Microservice orchestration</li></ul><p>Step Functions also provides <strong>automatic retries, error handling, and parallel execution</strong>, ensuring reliability and scalability.</p><h3>Requirement: Scan DynamoDB and Process Each Row</h3><p>Let’s break down the solution step by step.</p><p>DynamoDB Batch Scan with Step Functions — Detailed Explanation</p><p>This Step Functions state machine processes all items in a DynamoDB table by scanning in <strong>batches</strong> and processing each item individually. It also handles <strong>pagination</strong> automatically to ensure all items are processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*y4-oEEmjBjqhSjR6ahTCEA.png" /></figure><h3>State-by-State Breakdown</h3><h4>1. CheckStartKey (Choice State)</h4><ul><li><strong>Purpose</strong>: Determines whether this is the first scan or a continuation from a previous batch.</li><li><strong>Logic</strong>: If $.LastEvaluatedKey exists → go to <strong>ScanWithKey</strong>. If not → go to <strong>ScanWithoutKey</strong>.</li><li><strong>Why needed</strong>: DynamoDB scans return <strong>paginated results</strong>, so we must handle initial scans differently from continuation scans.</li></ul><h4>2. ScanWithoutKey (Task State)</h4><ul><li><strong>Purpose</strong>: Performs the initial scan of the DynamoDB table.</li><li><strong>Resource</strong>: Lambda function scan-dynamoDB.</li><li><strong>Output</strong>: Returns items and possibly a LastEvaluatedKey.</li><li><strong>Next</strong>: Always goes to <strong>ProcessItems</strong>.</li></ul><h4>3. ScanWithKey (Task State)</h4><ul><li><strong>Purpose</strong>: Continues scanning from the previous batch.</li><li><strong>Resource</strong>: Same Lambda function scan-dynamoDB.</li><li><strong>Parameters</strong>:ExclusiveStartKey.$: Uses the LastEvaluatedKey from the last scan.</li><li><strong>Output</strong>: Returns the next batch and possibly a new LastEvaluatedKey.</li><li><strong>Next</strong>: Always goes to <strong>ProcessItems</strong>.</li></ul><h4>4. ProcessItems (Map State)</h4><ul><li><strong>Purpose</strong>: Processes each item from the scan results in <strong>parallel</strong>.</li><li><strong>Configuration</strong>:</li><li>ItemsPath: $.Items (array of scan results).</li><li>MaxConcurrency: 2 (processes 2 items at a time to avoid overwhelming downstream systems).</li><li>Iterator: Calls the processing Lambda function (xyz) for each item.</li><li><strong>Next</strong>: Moves to <strong>HasMore</strong> once all items are processed.</li></ul><h4>5. HasMore (Choice State)</h4><ul><li><strong>Purpose</strong>: Checks if there are more items to scan.</li><li><strong>Logic</strong>: If $.LastEvaluatedKey exists → go back to <strong>ScanWithKey</strong>.If not → go to <strong>Done</strong>.</li><li><strong>Why needed</strong>: Creates the <strong>loop</strong> that ensures all items are scanned.</li></ul><h4>6. Done (Succeed State)</h4><ul><li><strong>Purpose</strong>: Ends execution once all items are processed.</li></ul><h3>Data Flow</h3><ul><li><strong>Input</strong>: May or may not include LastEvaluatedKey.</li><li><strong>Between scans</strong>: Each scan returns an Items array and possibly a LastEvaluatedKey.</li><li><strong>Item processing</strong>: Each item is processed individually by the Lambda function.</li><li><strong>Looping</strong>: If a LastEvaluatedKey is present, the process continues until the entire table is scanned.</li></ul><h3>Key Design Patterns</h3><p>✅ <strong>Pagination Handling</strong></p><ul><li>Automatically handles DynamoDB’s 1 MB scan limit.</li><li>Uses LastEvaluatedKey to continue seamlessly.</li><li>Ensures no items are missed or duplicated.</li></ul><p>✅ <strong>Parallel Processing</strong></p><ul><li>Uses Map state with MaxConcurrency = 2.</li><li>Processes items independently.</li><li>Failures in one item don’t block others.</li></ul><h3><strong>Lambda Code for Handling </strong><strong>LastEvaluatedKey</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aff4bcdc94eb4114742a10ed8015b068/href">https://medium.com/media/aff4bcdc94eb4114742a10ed8015b068/href</a></iframe><p>This Java Lambda function scans a DynamoDB table and converts the items into a plain Java map structure. If a LastEvaluatedKey is present in the input, it continues scanning from the previous batch; otherwise, it starts fresh. The function then returns the scanned items and, if available, the new LastEvaluatedKey so Step Functions can decide whether to keep scanning.</p><h3>Use Cases</h3><p>This pattern is ideal for:</p><ul><li><strong>ETL operations</strong> on large DynamoDB tables</li><li><strong>Data migration</strong> or transformation</li><li><strong>Batch data processing</strong></li><li><strong>Cache warming operations</strong> (e.g., website cache preloading)</li></ul><p>✅ With Step Functions, you can efficiently scan even large DynamoDB tables without worrying about timeouts, missed records, or unnecessary infrastructure overhead.</p><h3>Error Handling Note</h3><ul><li>The current implementation doesn’t include explicit error handling.</li><li>In production, you’d typically add retry logic and error states.</li></ul><h3>Conclusion</h3><p>In my project, I initially tried using AWS Lambda to scan DynamoDB, but the <strong>15-minute execution limit</strong> made it impractical for large tables. By switching to <strong>AWS Step Functions</strong>, I was able to break the task into smaller steps, handle pagination seamlessly, and process each item efficiently in parallel.</p><p>This approach not only solved my problem but also gave me a <strong>scalable, cost-effective, and serverless solution</strong> for large-scale DynamoDB processing. If you’re dealing with similar challenges, I highly recommend combining Step Functions with Lambda — it’s simple, reliable, and powerful.</p><p>References:</p><p><a href="https://aws.amazon.com/step-functions/">Workflow Orchestration - AWS Step Functions - AWS</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac57bebd0f26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to solve the problem of cold start AWS Lambda?]]></title>
            <link>https://medium.com/@manishadutt90/how-to-solve-the-problem-of-cold-start-aws-lambda-98adee0497b8?source=rss-cfc610576d65------2</link>
            <guid isPermaLink="false">https://medium.com/p/98adee0497b8</guid>
            <dc:creator><![CDATA[Manishadutt]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 03:22:48 GMT</pubDate>
            <atom:updated>2025-08-20T03:22:48.152Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi everyone! This is my first article on Medium. I wrote about serverless and how to solve the cold start problem in AWS Lambda. If you’re not a Medium member, you can still read it here: <a href="https://devonblog.com/software-development/how-to-solve-the-problem-of-cold-start-aws-lambda/">https://devonblog.com/software-development/how-to-solve-the-problem-of-cold-start-aws-lambda/</a>.</p><p>Recently, while working with AWS Lambda, our biggest challenge has been improving its performance, specifically reducing the cold start time.</p><p><strong>What is AWS Lambda?</strong><br>AWS Lambda is a serverless compute function, that allows developers to run code without having to worry about managing servers. When you use AWS Lambda, you simply upload your code, and it automatically handles all the details needed to run and scale your code with high availability. This means you don’t need to provision, scale, or manage servers. Instead, you can focus on writing your code, and AWS Lambda takes care of the rest, including the infrastructure and maintenance. This makes it easier and more efficient to develop and deploy applications.</p><p><strong>What is the cold start problem?</strong><br>The cold start problem in AWS Lambda refers to the delay that occurs when a Lambda function is invoked for the first time after it has been idle, as the function’s runtime environment is initialized. Cold starts happen because AWS Lambda is a serverless service, meaning that the underlying infrastructure is managed automatically. When there’s no traffic, AWS doesn’t keep any running instances of the function. Instead, it spins up new instances as needed. A cold start results in the Lambda function taking more time to respond, increasing the function’s latency.</p><p>Here are the approaches our team has taken to address this issue.</p><ol><li><strong><em>Provisioned concurrency</em></strong></li></ol><p><strong>What do you mean by provisioned concurrency?</strong><br>Provisioned concurrency enables serverless functions to adapt to sudden bursts of traffic and significant scaling events. You can use provisioned concurrency to build scalable serverless applications with predictable latency. The feature lets you set a desired concurrency for all aliases and versions of each function.</p><p><strong>How to use provisioned concurrency?</strong><br>Depending on the project requirements, you can use provisioned concurrency. There is no definite count you can choose for the provisioned concurrency but after some hit and trial plus depending upon the traffic, you can achieve a decent count.<br>1. Create a “publish” version (inside versions).<br>2. Create an “alias” (inside Aliases).<br>3. Inside Configuration –&gt; Concurrency –&gt; assign the count for concurrency<br>For the first time, you should follow the above steps. But once the count is finalized, for deploying the lambda next time follow the below steps:-<br>1. Create the “publish” version.<br>2. Inside aliases, select the alias and edit it to the latest “publish” version.</p><p><strong>Question: How is this going to reduce the cold start time?</strong><br>To answer this question, let’s understand how lambda works, suppose the lambda function receives 10 requests, each request will be handled by new, on-demand execution environments. Lambda will initialize an execution environment (init) and then invoke the function (invoke).<br>In short for every request, Lambda must “init and invoke”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*TW4-AraMod3QaSfm.png" /></figure><p>In the above figure, on-demand lambda will follow both init and invoke, hence this will take time to start the lambda. All 10 on-demand lambda follow the same process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/929142c137f8a5bf53dd759394540105/href">https://medium.com/media/929142c137f8a5bf53dd759394540105/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*8yYuTnMd2hwMMJ-L.png" /></figure><p>As shown in the figure, for the 2nd request, the execution environment is already set up, only invoke (handler method) needs to be executed. AWS Lambda uses provisioned concurrency to pre-prepare containers for your functions. This helps ensure that once the containers are invoked, they respond with double-digit millisecond latency.</p><p><strong><em>2. Initialization of 3rd Party libraries</em></strong></p><p>Initialize the properties file, database, etc before calling the handler method. Like the example below, lambda starts invoking code that is written inside the handler methods(handleRequest) as we are initializing all the parameters before, this helps us to reduce the initialization time of 3rd party libraries. For our project, we are initializing elastic search, DynamoDB, and properties cache before the function invokes the handler method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*r7V19bYEdnHdDGKL.png" /></figure><p><strong><em>3. Keep poking lambda</em></strong></p><p>Create an event bridge rule that will ping the lambda every minute or after every 30 seconds, this will keep lambda warm for most of the time. This won’t make any difference regarding the usage and billing of lambda. The lambda will hardly spawn for 2 or 3 milliseconds. For example, below is the screenshot where lambda will start and end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*6PGbnOyE4UavSEHE.png" /></figure><p><strong>Conclusion</strong></p><p>Using the above approaches, our team has solved the problem of cold start, bringing down the response time from 10 to 15 seconds to 1 to 2 seconds.</p><p>References:</p><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html">https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98adee0497b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>