<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zied Ben Tahar on Medium]]></title>
        <description><![CDATA[Stories by Zied Ben Tahar on Medium]]></description>
        <link>https://medium.com/@zied-ben-tahar?source=rss-b3a9f7e87b27------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*U9AHQ8z6sGIxYZdYGUJSvg.jpeg</url>
            <title>Stories by Zied Ben Tahar on Medium</title>
            <link>https://medium.com/@zied-ben-tahar?source=rss-b3a9f7e87b27------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:13:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zied-ben-tahar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless RAG Chat with AppSync Events and Bedrock Knowledge Bases]]></title>
            <link>https://levelup.gitconnected.com/serverless-rag-chat-with-appsync-events-and-bedrock-knowledge-bases-5104d96b1409?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/5104d96b1409</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[appsync-event]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[lambda-powertools]]></category>
            <category><![CDATA[bedrock-knowledge-bases]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Fri, 09 May 2025 14:39:17 GMT</pubDate>
            <atom:updated>2025-05-12T14:07:14.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tfppozDb2pdXtLbM" /><figcaption>Photo by <a href="https://unsplash.com/@ninjason?utm_source=medium&amp;utm_medium=referral">Jason Leung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When it comes to building serverless WebSocket APIs on AWS, there’s no shortage of options: API Gateway, IoT Core, AppSync GraphQL subscriptions, and now AppSync Events. Each option comes with its own level of control and complexity. I’ve found that AppSync Events to be simplest to work with.</p><p>One of the interesting features of AppSync Events is its data sources capability. It lets you directly integrate to resources like DynamoDB, OpenSearch, Bedrock and Lambda. You can interact with these data sources using AppSyncJS (appsync’s own flavor of javascript). But to be totally fair, I lean toward direct lambda integration as it gives more control and makes the development and testing workflow more familiar, standard and manageable.</p><p>Currently, Bedrock data source supports only the InvokeModel and Converse APIs. So, if you want to integrate with Knowledge Bases, the viable approach is to create a custom data source using Lambda.</p><p>And that’s exactly what this blog post is about, we’ll walk through how to build this RAG-based chat application with AppSync Events and bedrock knowledge based using nodejs, TypeScript and Terraform.</p><h3>Solution overview</h3><p>Let’s take a look at how the whole setup fits together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omOfcDXecfXZtOH-NuwbZQ.png" /><figcaption>Architecture overview</figcaption></figure><p>The Knowledge Base is configured to use PostgreSQL as its vector store where we store the embeddings as well as the associated metadata of the documents we want to index. Using Postgres gives us control over the schema, the indexing strategy, and embedding format, all of which come in handy when fine-tuning a vector-based RAG setup.</p><p>We’ve got the <strong>handleAppSyncEvents</strong> function directly integrated as a data source for the AppSync Events API. Its role is to process incoming events from AppSync and to invoke the <strong>retrieveAndGenerate</strong> from the Knowledge Base. This function is configured to be asynchronous (with the <strong>Event</strong> invocation type), which means AppSync doesn&#39;t wait for the function to complete before returning a response to the client. Once we receive a result from bedrock this function publishes a response back to the client’s response channel.</p><p>AppSync Events supports multiple authorization methods to secure Event APIs, including API keys, Lambda authorizers, IAM, OpenID Connect, and Amazon Cognito user pools. In this setup, I’m using both Cognito user pools and IAM:</p><ul><li>Web clients use cognito for authentication</li><li>And I chose IAM over using API key for publishing events from the <strong>handleAppSyncEvents</strong> function to AppSync, as it offers better security posture.</li></ul><p>One thing I appreciate in this setup: AppSync Events supports Web ACLs. That means you can easily layer in protections like rate limiting and IP filtering. It’s a nice edge over API Gateway WebSockets, which still doesn’t offer native WAF support.</p><p>And tying it all together, the browser connects via WebSocket to AppSync, giving us a real-time, bidirectional channel, ideal for sending the models responses back to users in conversational interfaces.</p><p>Let’s dive into the details of the solution; but if you’d like to jump straight to the complete implementation, you can find it here 👇</p><p><a href="https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases">https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases</a></p><h3>Setting up the knowledge base</h3><p>Let’s first take a look at how we can use Aurora PostgreSQL as a vector store.</p><h4>Creating the vector store on Postgres</h4><p>When using PostgreSQL as a vector store, Knowledge Base requires an Aurora Serverless cluster with the Data API enabled. The database must include a vector table with specific columns:</p><ul><li>An embedding column to store the vector representation of the content,</li><li>A chunk column for the actual text tied to each embedding,</li><li>And a metadata column that holds references, which are useful for pointing back to the original source during retrieval.</li></ul><p>The Knowledge Base keeps this table up to date automatically whenever content is synced from the source bucket.</p><p>Since I like to keep everything automated, I trigger the db init script right after the database cluster is created. This script sets up everything we need: a role, schema, table, and indexes; all in one go, wrapped in a single transaction. It’s run by a function once the cluster is deployed:</p><pre>export const handler = async (_: unknown): Promise&lt;void&gt; =&gt; {<br>    const bedrockKnowledgeBaseCreds = {<br>        username: &quot;bedrock_user&quot;,<br>        password: generatePostgresPassword(),<br>    };<br><br>    let schema = &quot;knowledge_base&quot;;<br>    let vectorTable = &quot;bedrock_kb&quot;;<br><br>    const queries = [<br>        `CREATE EXTENSION IF NOT EXISTS vector`,<br>        `CREATE SCHEMA IF NOT EXISTS ${schema}`,<br>        `CREATE ROLE ${bedrockKnowledgeBaseCreds.username} WITH PASSWORD &#39;${bedrockKnowledgeBaseCreds.password}&#39; LOGIN`,<br>        `GRANT ALL ON SCHEMA ${schema} to ${bedrockKnowledgeBaseCreds.username}`,<br>        `CREATE TABLE IF NOT EXISTS ${schema}.${vectorTable} (id uuid PRIMARY KEY, embedding vector(1024), chunks text, metadata json, custom_metadata jsonb)`,<br>        `CREATE INDEX IF NOT EXISTS bedrock_kb_embedding_idx ON ${schema}.${vectorTable} USING hnsw (embedding vector_cosine_ops) WITH (ef_construction=256)`,<br>        `CREATE INDEX IF NOT EXISTS bedrock_kb_chunks_fts_idx ON ${schema}.${vectorTable} USING gin (to_tsvector(&#39;simple&#39;, chunks))`,<br>        `CREATE INDEX IF NOT EXISTS bedrock_kb_custom_metadata_idx ON ${schema}.${vectorTable} USING gin (custom_metadata)`,<br>        `GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA ${schema} TO ${bedrockKnowledgeBaseCreds.username}`<br>    ];<br><br>    await executeTransaction(databaseArn, databaseSecretArn, databaseName, queries, bedrockKnowledgeBaseCreds);<br>};</pre><p><strong>executeTransaction</strong> leverages aurora’s data api in order to execute these statements. In this function, I also set the secret containing a dedicated database user and password, which will later be used when setting up the data source for the Knowledge Base.</p><h4>Creating The knowledge base</h4><p>Quite straightforward in terraform:</p><pre>resource &quot;aws_bedrockagent_knowledge_base&quot; &quot;this&quot; {<br><br>  name     = &quot;${var.application}-${var.environment}-kb&quot;<br>  role_arn = aws_iam_role.kb_role.arn<br><br>  knowledge_base_configuration {<br>    vector_knowledge_base_configuration {<br>      embedding_model_arn = local.embedding_model_arn<br>    }<br>    type = &quot;VECTOR&quot;<br>  }<br><br>  storage_configuration {<br>    type = &quot;RDS&quot;<br>    rds_configuration {<br>      credentials_secret_arn = aws_secretsmanager_secret.kb_creds.arn<br>      database_name          = aws_rds_cluster.rds_cluster.database_name<br>      resource_arn           = aws_rds_cluster.rds_cluster.arn<br>      table_name             = &quot;${local.db_schema}.${local.vector_table}&quot;<br>      field_mapping {<br>        primary_key_field = &quot;id&quot;<br>        metadata_field    = &quot;metadata&quot;<br>        text_field        = &quot;chunks&quot;<br>        vector_field      = &quot;embedding&quot;<br>      }<br>    }<br>  }<br><br>  depends_on = [<br>    aws_rds_cluster_instance.rds_cluster_instance,<br>    aws_lambda_invocation.seed_db_function,<br>    aws_secretsmanager_secret.kb_creds<br>  ]<br>}<br><br>resource &quot;aws_bedrockagent_data_source&quot; &quot;this&quot; {<br>  knowledge_base_id = aws_bedrockagent_knowledge_base.this.id<br>  name              = &quot;kb_datasource&quot;<br><br>  vector_ingestion_configuration {<br>    chunking_configuration {<br>      chunking_strategy = &quot;FIXED_SIZE&quot;<br>      fixed_size_chunking_configuration {<br>        max_tokens         = 300<br>        overlap_percentage = 20<br>      }<br>    }<br>  }<br><br>  data_source_configuration {<br><br>    type = &quot;S3&quot;<br>    s3_configuration {<br><br>      bucket_arn         = aws_s3_bucket.kb_bucket.arn<br>      inclusion_prefixes = [&quot;${local.kb_folder}&quot;]<br>    }<br>  }<br>}</pre><p>When configuring RDS as a data source there are a few key parameters you’ll need to provide:</p><ul><li>The vector table and the field mappings that define which columns in the table should be used by the Knowledge Base.</li><li>The IAM role associated with the Knowledge Base must have the following permissions on the RDS cluster: rds-data:ExecuteStatement, rds-data:BatchExecuteStatement and rds:DescribeDbClusters</li><li>The Secret arn that holds the database credentials that we created in the previous step.</li></ul><p>Once the knowledge base is deployed, here’s what it looks like in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ftylbyNoSjNREd3ftcnXA.png" /><figcaption>Knowledge base overview</figcaption></figure><p>We can already start testing it right from the console. In this example, I synced the knowledge base with a dataset containing texts about the Roman Empire.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cBqWjktY8wUH8i0ryTnCVQ.png" /><figcaption>Testing the knowledge base</figcaption></figure><p>Alights, let’s see how to setup the AppSync Events integration</p><h3>Setting up AppSync integration</h3><p>As mentioned earlier, I’ll be using Cognito User Pools as the default auth mode as well as IAM:</p><ul><li>AppSync will handle validating both subscribe and publish requests from clients, as long as they provide a valid Cognito token.</li><li>IAM auth will be used <strong>handleAppSyncEvent</strong> function as needs to publish responses back to the clients</li></ul><pre>resource &quot;awscc_appsync_api&quot; &quot;this&quot; {<br>  name          = &quot;${var.application}-${var.environment}-events-api&quot;<br>  owner_contact = &quot;${var.application}-${var.environment}&quot;<br>  event_config = {<br>    auth_providers = [<br>      {<br>        auth_type = &quot;AMAZON_COGNITO_USER_POOLS&quot;<br>        cognito_config = {<br>          aws_region   = data.aws_region.current.name,<br>          user_pool_id = var.user_pool_id,<br>        }<br>      },<br>      {<br>        auth_type = &quot;AWS_IAM&quot;<br>      }<br>    ]<br>    connection_auth_modes = [<br>      {<br>        auth_type = &quot;AMAZON_COGNITO_USER_POOLS&quot;<br>      }<br>    ]<br>    default_publish_auth_modes = [<br>      {<br>        auth_type = &quot;AMAZON_COGNITO_USER_POOLS&quot;<br>      }<br>    ]<br>    default_subscribe_auth_modes = [<br>      {<br>        auth_type = &quot;AMAZON_COGNITO_USER_POOLS&quot;<br>      }<br>    ]<br>  }<br>}</pre><p>Next up, I’ll need to create a data source and associated to handleAppSyncEvents function. Unfortunately, this part isn’t supported in the terraform provider <em>yet, </em>so for now, I’m using the sdk to create these resources in a function hat runs once right after the AppSync Events API resource is created:</p><pre>export const handler = async (event: {<br>    tf: { action: string };<br>    apiId: string;<br>    dataSourceName: string;<br>    lambdaFunctionArn: string;<br>    serviceRoleArn: string;<br>    channelName: string;<br>}) =&gt; {<br>    if (<br>        event.apiId == null ||<br>        event.dataSourceName == null ||<br>        event.lambdaFunctionArn == null ||<br>        event.serviceRoleArn == null ||<br>        event.channelName == null<br>    ) {<br>        throw new Error(&quot;SourceArn, TargetArn, RoleArn and channel name are required&quot;);<br>    }<br><br>    if (event.tf.action === &quot;create&quot;) {<br>        const client = new AppSyncClient({ region: process.env.AWS_REGION });<br><br>        const createDataSourceCommand = new CreateDataSourceCommand({<br>            apiId: event.apiId,<br>            name: event.dataSourceName,<br>            type: &quot;AWS_LAMBDA&quot;,<br>            serviceRoleArn: event.serviceRoleArn,<br>            lambdaConfig: {<br>                lambdaFunctionArn: event.lambdaFunctionArn,<br>            },<br>        });<br><br>        await client.send(createDataSourceCommand);<br><br>        const createChannelCommand = new CreateChannelNamespaceCommand({<br>            apiId: event.apiId,<br>            name: event.channelName,<br>            subscribeAuthModes: [<br>                {<br>                    authType: &quot;AMAZON_COGNITO_USER_POOLS&quot;,<br>                },<br>            ],<br>            publishAuthModes: [<br>                {<br>                    authType: &quot;AMAZON_COGNITO_USER_POOLS&quot;,<br>                },<br>                {<br>                    authType: &quot;AWS_IAM&quot;,<br>                },<br>            ],<br>            handlerConfigs: {<br>                onPublish: {<br>                    behavior: &quot;DIRECT&quot;,<br>                    integration: {<br>                        dataSourceName: event.dataSourceName,<br>                        lambdaConfig: {<br>                            invokeType: &quot;EVENT&quot;,<br>                        },<br>                    },<br>                },<br>                onSubscribe: {<br>                    behavior: &quot;DIRECT&quot;,<br>                    integration: {<br>                        dataSourceName: event.dataSourceName,<br>                        lambdaConfig: {<br>                            invokeType: &quot;EVENT&quot;,<br>                        },<br>                    },<br>                },<br>            },<br>        });<br><br>        await client.send(createChannelCommand);<br><br>        return;<br>    }<br><br>    // ... handle resource update and deletion<br>};</pre><p>Note the direct integration of the default channel with the function. The <strong>handleAppSyncEvents</strong> function is invoked directly in <strong>EVENT</strong> invocation mode.</p><p>Here how it looks in the console once the AppSync resources are created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BcBab5nH6ir1XJkj3OGO8g.png" /><figcaption>Chat channel namespace</figcaption></figure><p>With the Lambda function as a data source defined in this screen</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGIxFHryDfz8q67hOJbkDw.png" /><figcaption>Lambda data source</figcaption></figure><h4>Using Lambda Powertools to handle AppSync realtime events</h4><p>Now let’s get to the interesting part: Handling events from AppSync and putting the knowledge base to work. Lambda Powertools offers a handy utility that makes it easier to integrate Lambda functions with AppSync events. It allows defining clear, dedicated handler methods for publish/subscribe interactions, so less messy if-else blocks. Routing is handled automatically based on namespaces and channel patterns, keeping the code clean and easy to maintain.</p><p>Here’s how it works in practice:</p><p><strong>Setting things up and handing subscription</strong></p><p>We start by initializing the resolver using the Lambda Powertools Event Handler utility, then define a handler for new subscriptions with <strong>app.onSubscribe</strong>:</p><pre>import { AppSyncEventsResolver, UnauthorizedException } from &quot;@aws-lambda-powertools/event-handler/appsync-events&quot;;<br>// ...<br><br>const app = new AppSyncEventsResolver();<br><br>app.onSubscribe(&quot;/chat/responses/*&quot;, (payload) =&gt; {<br>    const identity = payload.identity ? (payload.identity as { sub: string; username: string }) : null;<br>    const sub = identity?.sub;<br><br>    if (!sub || (payload.info.channel.segments.length != 3 &amp;&amp; !payload.info.channel.path.endsWith(`/${sub}`))) {<br>        throw new UnauthorizedException(&quot;You cannot subscribe to this channel&quot;);<br>    }<br>});</pre><p>Here, I ensure that users can only subscribe to their own chat responses channel. This check prevents accidental or unauthorized access to other users’ channels. Since we’re using a Cognito User Pool, the subscription payload contains decoded user information, including the sub (user ID) and username.</p><p><strong>Validating and handling messages</strong></p><p>Let’s now see how we handle messages from users:</p><pre>const messageSchema = z.object({<br>    id: z.number(),<br>    content: z.string(),<br>    type: z.enum([&quot;chat&quot;]).optional(),<br>    sessionId: z.string().optional(),<br>});<br><br>app.onPublish(&quot;/chat/request/*&quot;, async (payload, event) =&gt; {<br>    const identity = event.identity ? (event.identity as { sub: string; username: string }) : null;<br>    const sub = identity?.sub;<br><br>    if (!sub || (event.info.channel.segments.length != 3 &amp;&amp; !event.info.channel.path.endsWith(`/${sub}`))) {<br>        throw new UnauthorizedException(&quot;You cannot publish to this channel&quot;);<br>    }<br><br>    const message = messageSchema.safeParse(payload);<br><br>    if (!message.success) {<br>        return {<br>            result: { text: &quot;I don&#39;t understand what you mean, your message format seems invalid&quot; },<br>            error: &quot;Invalid message payload&quot;,<br>        };<br>    }<br><br>    const { content, sessionId } = message.data;<br><br>    const result = await bedrockClient.send(<br>        new RetrieveAndGenerateCommand({<br>            input: {<br>                text: content,<br>            },<br>            retrieveAndGenerateConfiguration: {<br>                type: &quot;KNOWLEDGE_BASE&quot;,<br>                knowledgeBaseConfiguration: {<br>                    knowledgeBaseId: process.env.KB_ID,<br>                    modelArn: process.env.KB_MODEL_ARN,<br>                },<br>            },<br>            sessionId,<br>        })<br>    );<br><br>    const signedRequest = await signRequest(`https://${process.env.EVENTS_API_DNS}/event`, &quot;POST&quot;, {<br>        channel: `/chat/responses/${sub}`,<br>        events: [<br>            JSON.stringify({<br>                result: result.output,<br>                sessionId: result.sessionId,<br>            }),<br>        ],<br>    });<br><br>    await fetch(`https://${process.env.EVENTS_API_DNS}/event`, {<br>        method: signedRequest.method,<br>        headers: signedRequest.headers,<br>        body: signedRequest.body,<br>    });<br><br>    return {<br>        processed: true,<br>    };<br>});</pre><p>As with the subscribe handler, when a message is published to the <strong>/chat/request/{userId}</strong> channel <strong>app.onPublish</strong> gets invoked, I extract the user’s identity from the event and enforce that they can only publish to their own channel. We then validate the payload using zod to ensure it has the expected structure. If validation succeeds we then call bedrock <strong>RetrieveAndGenerate</strong> endpoint.</p><p>As I am using IAM auth to let the function publish the response back to the client via <strong>chat/responses/{userId}</strong> channel. I need to sign the request with sigv4 and pass the signed request headers when calling AppSync event endpoint. This function needs to have <strong>appsync:EventPublish</strong> permission on the api channels.</p><p><strong>☝️ Some notes:</strong></p><ul><li>In this solution we waits for the full response from the model, we’re not using bedrock’s response streaming feature. I could have used the <strong>RetrieveAndGenerateStream</strong> API and then send chunks by publishing incrementally each chunk to the client. However, depending on the model’s response, this could lead to multiple calls to the Events API potentially increasing costs (since each call counts as a separate operation). One possible solution would be to buffer the response chunks and send them in batches, striking a better balance between responsiveness and cost. Handling lambda response streaming natively is a feature I’d love to see supported by AppSync in the future.</li><li>The <strong>RetrieveAndGenerate</strong> endpoint returns a sessionId that we’ll need to reuse in messages within the same conversation. This sessionId is what allows Amazon Bedrock to maintain context. We simply return that sessionId along with the result, and include it in every subsequent chat message to keep the conversation context-aware.</li><li>You can check out the Cognito user pool terraform resource definition at the following <a href="https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases/blob/main/infra/modules/auth/user-pool.tf">link</a>.</li></ul><p>You can find the complete function code following <a href="https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases/blob/main/src/chat/lambda-handlers/handle-appsync-events.ts">this link</a>.</p><h3>Creating a Client</h3><p>Here, I’m building a small React client to chat with the Knowledge Base via AppSync. I’m using Amplify because it makes it easy to connect to the AppSync API and handle authentication through the user pool to retrieve an access token.</p><p>First, we need to configure Amplify by providing the AppSync Events API endpoint along with the Cognito User Pool and Identity Pool ids:</p><pre>Amplify.configure({<br>  API: {<br>    Events: {<br>      &quot;endpoint&quot;: &quot;https://&lt;api-id&gt;.appsync-api.eu-west-1.amazonaws.com/event&quot;,<br>      &quot;region&quot;: &quot;eu-west-1&quot;,<br>      &quot;defaultAuthMode&quot;: &quot;userPool&quot;,<br>    },<br>  },<br>  Auth: {<br>    Cognito: {<br>      userPoolId: &quot;&lt;user-pool-id&gt;&quot;,<br>      userPoolClientId: &quot;&lt;user-pool-client-id&gt;&quot;,<br>      identityPoolId: &quot;&lt;identity-pool-id&gt;&quot;,<br>    },<br>  },<br>});</pre><p>To handle real-time chat with AppSync Events, the client is connected to two endpoints:</p><ul><li><strong>events.connect(&#39;/chat/responses/${userId}&#39;)</strong> to <strong>subscribe</strong> to responses</li><li><strong>events.post(&#39;/chat/request/${userId}&#39;)</strong> to <strong>send</strong> user messages</li></ul><pre>  useEffect(() =&gt; {<br>    const connectToChannel = async () =&gt; {<br>      try {<br>        if(!userId) {<br>          return;<br>        }<br><br>        const channel = await events.connect(`/chat/responses/${userId}`);<br><br>        const subscription = channel.subscribe({<br>          next: (data: any) =&gt; {<br>            if(data.type !== &quot;data&quot;) return;<br><br>            const botMessage: Message = {<br>              id: Date.now(),<br>              sender: &quot;bot&quot;,<br>              content: data.event.result.text,<br><br>            };<br>            setMessages((prev) =&gt; [...prev, botMessage]);<br>            setSessionId(data.event.sessionId);<br>          },<br>          error: (err: any) =&gt; console.error(&quot;subscription error&quot;, err),<br>        });<br><br>        return () =&gt; {<br>          subscription.unsubscribe?.();<br>        };<br>      } catch (error) {<br>        console.error(&quot;connection error&quot;, error);<br>      }<br>    };<br><br>    connectToChannel();<br>  }, [userId]);<br><br>  const sendMessage = async ({userId }:{userId : string}) =&gt; {<br>    if (input.trim() === &quot;&quot;) return;<br>    setInput(&quot;&quot;);<br>    const newMessage: Message = {<br>      id: Date.now(),<br>      sender: &quot;user&quot;,<br>      content: input,<br>      sessionId: sessionId,<br>    };<br><br>    setMessages([...messages, newMessage]);<br><br>    await events.post(`/chat/request/${userId}`, newMessage);<br>  };</pre><p>When the React component mounts, we use <strong>useEffect</strong> to set up a connection to the <strong>/chat/responses/{userId}</strong> channel. This subscribes to responses coming from AppSync Events. Once we start getting responses, we add that message to the chat and update the <strong>sessionId</strong> to maintain context.</p><p>To send a message, the <strong>sendMessage</strong> function posts to the same <strong>/chat/request/${userId}</strong> endpoint.</p><p>The userId in this example is provided by Amplify’s Authenticator component, which handles user authentication and exposes the signed-in user&#39;s details.</p><p>Which gives this chat interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4F_hKzBL0dBD9dS1l_kTCw.gif" /></figure><p>If you look at the browser’s WebSocket connection, you’ll see real-time responses coming through as AppSync Events sends data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJXbW1NmLJwlC5eS8BKkTA.png" /></figure><h3>Wrapping up</h3><p>In this post, we walked through building a serverless WebSocket API using AppSync Events, Lambda, and Bedrock Knowledge Bases. We explored how to handle real-time communication securely with Cognito and IAM. Lambda Powertools makes working with AppSync Events an absolute breeze.</p><p>As usual you can find the complete repo with the solution ready to be adapted and deployed here 👉 <a href="https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases">https://github.com/ziedbentahar/rag-chat-with-appsync-events-and-bedrock-knowledge-bases</a></p><p>I will make sure to update the repo once AppSync gets better terraform support 👌</p><p>Thanks for making it all the way here !</p><h3>Resources</h3><p><a href="https://docs.powertools.aws.dev/lambda/typescript/latest/features/event-handler/appsync-events/">https://docs.powertools.aws.dev/lambda/typescript/latest/features/event-handler/appsync-events/</a></p><ul><li><a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.VectorDB.html#AuroraPostgreSQL.VectorDB.PreparingKB">Using Aurora PostgreSQL as a Knowledge Base for Amazon Bedrock</a></li><li><a href="https://docs.aws.amazon.com/appsync/latest/eventapi/supported-datasources.html">Supported data sources</a></li><li><a href="https://www.ranthebuilder.cloud/post/aws-appsync-events-and-powertools-for-aws-lambda">Build Serverless WebSockets with AWS AppSync Events and Powertools for AWS Lambda</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5104d96b1409" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/serverless-rag-chat-with-appsync-events-and-bedrock-knowledge-bases-5104d96b1409">Serverless RAG Chat with AppSync Events and Bedrock Knowledge Bases</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scheduled queries  in Amazon Timestream for LiveAnalytics]]></title>
            <link>https://levelup.gitconnected.com/using-scheduled-queries-with-amazon-timestream-for-liveanalytics-fba47dd4f325?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/fba47dd4f325</guid>
            <category><![CDATA[time-series-database]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[scheduled-queries]]></category>
            <category><![CDATA[amazon-timestream]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Sun, 16 Mar 2025 17:26:39 GMT</pubDate>
            <atom:updated>2025-03-19T13:37:39.038Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scheduled queries in Amazon Timestream for LiveAnalytics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QGDZ5fB0Bx0pvK27" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Timestream for LiveAnalytics is a serverless, purpose-built database for processing time-series data. While it efficiently ingests and stores data, querying large raw datasets frequently can be inefficient, especially for use cases like aggregations, trend analysis, or anomaly detection.</p><p>Scheduled Queries is a feature of timestream that helps addressing this by running recurring SQL queries at specified intervals, rolling-up raw data into aggregated results, and storing them in another Timestream table. This approach improves performance on target tables, and optimises storage by retaining only the aggregated data.</p><p>In this post, we’ll walk through setting up Timestream Scheduled Queries to automate data rollups. We’ll also explore how this setup helps you analyze and detect trends in your data over time.</p><h3>What are we going to build ?</h3><p>As a use case, we’ll have a clickstream kinesis topic where producers can push events such as clicks, views, and user actions. These events will be ingested into a Timestream table<strong> </strong>and made available for downstream consumption. A scheduled query will run hourly to roll up raw event data, which will then be used to detect trending products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QrIG72UpguZLR92M08IMPQ.png" /><figcaption>Solution overview</figcaption></figure><p>Let’s dive into this:</p><ul><li><strong>Time Series store</strong>: With EventBridge Pipes supporting Timestream as a direct target, integrating with Kinesis stream source is simple. This eliminates much of the custom glue code needed for ingestion. In this setup, EventBridge Pipes polls the Kinesis stream, converts the received events into records, and writes them directly to the Timestream table.</li><li><strong>A scheduled aggregation query</strong> runs at predefined intervals to process raw events, performing an hourly rollup and storing the results in a dedicated table. Once it successfully completes, it publishes a notification event that triggers a function to query the table and detect the top N trending products, which are then published to a topic. We can imagine that these events can be used for various purposes: adjusting ad spend or fine-tuning A/B tests. Additionally, the rolled-up data can be fed into dashboards like QuickSight, Tableau, or Grafana for visualizing and monitoring product performance.</li></ul><p><strong>☝️Note: </strong>While I won’t go into detail in this post, it’s worth noting that a common way to reduce writes to the raw table is by pre-aggregating data in small batches, such as using a 5-minute tumbling window. This groups events into fixed, non-overlapping intervals before writing, effectively lowering the write frequency. <a href="https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/blob/main/python/GettingStarted/README.md">Managed Apache Flink</a> can help achieve this.</p><h3>Let’s see the code</h3><p>In this section, I will mainly focus on the integration with EventBridge Pipes as well as the scheduled query configuration. You can find the complete end-to-end solution at the following link 👇</p><p><a href="https://github.com/ziedbentahar/scheduled-queries-with-amazom-timestreamdb">https://github.com/ziedbentahar/scheduled-queries-with-amazom-timestreamdb</a></p><h4><strong>Simplifying time-series data ingestion with event bridge pipes</strong></h4><p>The events ingested into Kinesis are quite basic; they contain only the productId, pageId, eventType, and the event timestamp. Configuring a Timestream table as a target in EventBridge Pipes requires defining the mapping from the source event to the selected measurements and dimensions in the target table. It also involves specifying the time field type and format:</p><pre>resource &quot;awscc_pipes_pipe&quot; &quot;kinesis_to_timestream&quot; {<br>  name     = &quot;${var.application}-${var.environment}-kinesis-to-timestream&quot;<br>  role_arn = awscc_iam_role.pipe.arn<br>  source   = var.source_stream.arn<br>  target   = aws_timestreamwrite_table.events_table.arn<br><br>  source_parameters = {<br>    kinesis_stream_parameters = {<br>      starting_position      = &quot;TRIM_HORIZON&quot;<br>      maximum_retry_attempts = 5<br>    }<br><br>    filter_criteria = {<br>      filters = [{<br>        pattern = &lt;&lt;EOF<br>{<br>  &quot;data&quot;: {<br>    &quot;eventType&quot;: [&quot;pageViewed&quot;, &quot;productPageShared&quot;, &quot;productInquiryRequested&quot;]<br>  }<br>}<br>EOF<br>      }]<br>    }<br>  }<br><br>  target_parameters = {<br>    timestream_parameters = {<br>      timestamp_format = &quot;yyyy-MM-dd&#39;T&#39;HH:mm:ss.SSS&#39;Z&#39;&quot;<br>      version_value    = &quot;1&quot;<br>      time_value       = &quot;$.data.time&quot;<br>      time_field_type  = &quot;TIMESTAMP_FORMAT&quot;<br><br>      single_measure_mappings = [{<br>          measure_name      = &quot;$.data.eventType&quot;<br>          measure_value     = &quot;$.data.value&quot;<br>          measure_value_type = &quot;BIGINT&quot;<br>      }]<br><br>      dimension_mappings = [<br>        {<br>          dimension_name       = &quot;id&quot;<br>          dimension_value      = &quot;$.data.id&quot;<br>          dimension_value_type = &quot;VARCHAR&quot;<br>        },<br>        {<br>          dimension_name       = &quot;pageId&quot;<br>          dimension_value      = &quot;$.data.pageId&quot;<br>          dimension_value_type = &quot;VARCHAR&quot;<br>        },<br>        {<br>          dimension_name       = &quot;productId&quot;<br>          dimension_value      = &quot;$.data.productId&quot;<br>          dimension_value_type = &quot;VARCHAR&quot;<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>Here, I am filtering and ingesting only the pageViewed, productPageShared, and productInquiryRequested events into the table from the stream.</p><p>We can view the details of the deployed pipe in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BhTubUpZiQGZJuDqmGq67Q.png" /><figcaption>Kinesis to timestream, filtering events</figcaption></figure><p>You can also view and edit the configured mapping for the target data model in the console… but when it comes to editing, you should always use IaC! Unless, of course, you enjoy the thrill of undocumented changes 😉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pks9HNpSkRheOg1OdCOcVQ.png" /><figcaption>Configuring table data model from the console</figcaption></figure><h4><strong>Automating the creation of the scheduled queries</strong></h4><p>Creating the scheduled query requires defining the schedule configuration, the query statement, and the target mapping, ensuring that the results are properly mapped to the data model for insertion into the destination table. You also need to define an SNS topic to publish notifications about the execution status of scheduled queries.</p><pre>resource &quot;aws_timestreamquery_scheduled_query&quot; &quot;hourly_rollup&quot; {<br>  name = &quot;${var.application}-${var.environment}-hourly-rollup&quot;<br>  schedule_configuration {<br>    schedule_expression = &quot;rate(1 hour)&quot;<br>  }<br><br>  query_string = templatefile(&quot;${path.module}/queries/hourly-rollup.sql.tmpl&quot;, {<br>    table = &quot;\&quot;${aws_timestreamwrite_table.events_table.database_name}\&quot;.\&quot;${aws_timestreamwrite_table.events_table.table_name}\&quot;&quot;<br>  })<br><br><br>  target_configuration {<br>    timestream_configuration {<br>      database_name = aws_timestreamwrite_database.events_db.database_name<br>      table_name    = aws_timestreamwrite_table.hourly_rollup.table_name<br>      time_column   = &quot;time&quot;<br><br>      dimension_mapping {<br>        name                 = &quot;pageId&quot;<br>        dimension_value_type = &quot;VARCHAR&quot;<br>      }<br>      dimension_mapping {<br>        name                 = &quot;productId&quot;<br>        dimension_value_type = &quot;VARCHAR&quot;<br>      }<br>      measure_name_column = &quot;eventType&quot;<br>      multi_measure_mappings {<br>        target_multi_measure_name = &quot;eventType&quot;<br>        multi_measure_attribute_mapping {<br>          source_column      = &quot;sum_measure&quot;<br>          measure_value_type = &quot;BIGINT&quot;<br>        }<br>      }<br>    }<br>  }<br>  <br>  execution_role_arn = aws_iam_role.scheduled_query_role.arn<br><br>  error_report_configuration {<br>    s3_configuration {<br>      bucket_name = aws_s3_bucket.error_bucket.id<br>      object_key_prefix = local.hourly_rollup_error_prefix<br>    }<br>  }<br><br>  notification_configuration {<br>    sns_configuration {<br>      topic_arn = aws_sns_topic.scheduled_query_notification_topic.arn<br>    }<br>  }<br><br>  depends_on = [<br>    aws_lambda_invocation.seed_raw_events_table<br>  ]<br>}</pre><p>Here is the query:</p><pre>SELECT <br>    SUM(measure_value::bigint) as sum_measure, <br>    measure_name as eventType, <br>    bin(time, 1h) as time, <br>    productId,<br>    pageId<br>FROM <br>    ${table}<br>WHERE<br> time BETWEEN @scheduled_runtime - (interval &#39;2&#39; hour) AND @scheduled_runtime <br>GROUP BY<br>    measure_name, <br>    bin(time, 1h), <br>    productId,<br>    pageId</pre><p>To account for late-arriving events, it processes data in a two-hour window, ensuring delays of up to two hours are included in the roll-up.</p><p>☝️ <strong>Note:</strong> As of the time of writing, in order to create a scheduled query, the source table must contain data; otherwise, the creation will fail. To enable fully automated IaC deployment, I will add a Lambda invocation resource that triggers immediately after the source events table is created. This Lambda function will insert dummy events into the table, ensuring that the scheduled query can correctly infer the target schema during its creation:</p><pre>resource &quot;aws_lambda_invocation&quot; &quot;seed_raw_events_table&quot; {<br><br>  function_name = aws_lambda_function.seed_raw_table.function_name<br>  input         = jsonencode({})<br><br>  depends_on = [<br>    aws_lambda_function.seed_raw_table,<br>    aws_timestreamwrite_table.events_table<br>  ]<br><br>  lifecycle_scope = &quot;CREATE_ONLY&quot;<br><br>}</pre><p>Once deployed, you can head straight to the Timestream schedule queries page . Here’s what it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbJY5kz1sO_oNtE14IP2ng.png" /></figure><h4><strong>Identifying trending products</strong></h4><p>The “<a href="https://github.com/ziedbentahar/scheduled-queries-with-amazom-timestreamdb/blob/main/src/trend-analysis/lambda-handlers/handle-hourly-roll-up.ts">handle hourly roll up</a>” function subscribes to the scheduled query execution events and runs a trend analysis that compares the page views per product of the last hour to the previous hour to identify the top N products that have achieved 2x views. It then publishes an event for these products.</p><p>Below is how this function queries the Timestream table using the Timestream Query Client:</p><pre>const getTopNProduct = async (params: { eventType: string; topN: number }): Promise&lt;TrendingProducts&gt; =&gt; {<br>    const { eventType, topN } = params;<br><br>    const qs = `<br>WITH LastHour AS (<br>    SELECT <br>        productId, <br>        sum_measure AS current_views<br>    FROM &quot;${db}&quot;.&quot;${table}&quot; <br>    WHERE measure_name = &#39;${eventType}&#39;<br>        AND time &gt; ago(1h)<br>),<br>PreviousHour AS (<br>    SELECT <br>        productId, <br>        sum_measure AS previous_views<br>    FROM &quot;${db}&quot;.&quot;${table}&quot; <br>    WHERE measure_name = &#39;${eventType}&#39; <br>        AND time &gt; ago(2h) <br>        AND time &lt;= ago(1h)<br>)<br>SELECT <br>    l.productId, <br>    l.current_views, <br>    COALESCE(p.previous_views, 0) AS previous_views, <br>    (l.current_views - COALESCE(p.previous_views, 0)) AS increase_last_hour<br>FROM LastHour l<br>LEFT JOIN PreviousHour p ON l.productId = p.productId<br>WHERE (l.current_views &gt;= COALESCE(p.previous_views, 0) * 2)<br>ORDER BY increase_last_hour DESC<br>LIMIT ${topN}<br>`;<br><br>    const queryResult = await timestreamQueryClient.send(<br>        new QueryCommand({<br>            QueryString: qs,<br>        })<br>    );<br><br>    const result = parseQueryResult(queryResult);<br><br>    return {<br>        eventType,<br>        time: new Date().toISOString(),<br>        products: result.map((row) =&gt; ({<br>            productId: row.productId,<br>            count: Number(row.current_views),<br>            increaseLastHour: Number(row.increase_last_hour),<br>        })),<br>    };<br>};</pre><p>The Timestream SDK returns query responses in a raw format. To make them more usable, I drew heavy inspiration from <a href="https://github.com/awslabs/amazon-timestream-tools/blob/master/sample_apps/js/query-example.js">this code example provided in an AWS Labs repository</a> to parse the query result.</p><p>For comparison, here’s the equivalent query executed on both the raw events table and the hourly roll-up table for a synthetic dataset of 200,000 ingested events:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JD0hWMed7uucSJG428u03Q.png" /></figure><p>Even with this relatively small dataset and its distribution, we can observe a clear difference in both the query duration and the number of bytes scanned between the two tables.</p><h3><strong>Wrapping up</strong></h3><p>I hope you find this article useful! We explored how to leverage Timestream for live analytics, focusing on automating data rollups to optimise query performance.</p><p>As always, you can find the full code source, ready to be adapted and deployed here:</p><p><a href="https://github.com/ziedbentahar/scheduled-queries-with-amazom-timestreamdb">https://github.com/ziedbentahar/scheduled-queries-with-amazom-timestreamdb</a></p><p>Thanks for reading !</p><h3>Further readings</h3><ul><li><a href="https://aws.amazon.com/fr/blogs/database/build-time-series-applications-faster-with-amazon-eventbridge-pipes-and-timestream-for-liveanalytics/">Build time-series applications faster with Amazon EventBridge Pipes and Timestream for LiveAnalytics | Amazon Web Services</a></li><li><a href="https://repost.aws/questions/QUZXrL3Ww8ScmZLuHcQg-8JA/how-to-simultaneously-deploy-timestream-database-table-and-scheduled-query">How to simultaneously deploy Timestream Database, Table and Scheduled Query</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fba47dd4f325" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/using-scheduled-queries-with-amazon-timestream-for-liveanalytics-fba47dd4f325">Scheduled queries  in Amazon Timestream for LiveAnalytics</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replicate data from DynamoDB to Apache Iceberg tables using Glue Zero-ETL integration]]></title>
            <link>https://levelup.gitconnected.com/replicate-data-from-dynamodb-to-apache-iceberg-tables-using-glue-zero-etl-integration-9397523bfea2?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/9397523bfea2</guid>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[zero-etl]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 14:19:54 GMT</pubDate>
            <atom:updated>2025-10-03T06:37:09.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2foCCX2nK7lsvy9U" /><figcaption>Photo by <a href="https://unsplash.com/@mlenny?utm_source=medium&amp;utm_medium=referral">Alexander Hafemann</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Analyzing data directly from Amazon DynamoDB tables can be tricky since it doesn’t come with built-in analytics features. One approach is to set up ETL pipelines to replicate the data into a data lake or lakehouse. From there, services like Amazon Athena or EMR can take over for analysis and processing. Building and maintaining those ETL pipelines takes time and effort.</p><p>AWS Glue Zero-ETL Integration provides an easy way to replicate data from DynamoDB tables to Apache Iceberg tables in Amazon S3. It’s particularly useful when your DynamoDB table schema isn’t complex.</p><blockquote>Apache Iceberg is an open-source table format designed for high performance and large-scale analytics. It is increasingly recognized as a standard in data lake architectures providing advanced features such as schema evolution, time travel, ACID transactions, and efficient metadata handling, addressing key challenges in data lakes while offering warehouse-like capabilities.</blockquote><p>In this article, I’ll walk you through setting up Glue Zero-ETL Integration using Terraform. Along the way, I’ll share my thoughts on using this service.</p><h4>TL;DR</h4><p>You can find the complete code repository at this link 👇</p><p><a href="https://github.com/ziedbentahar/glue-zero-etl-dynamodb-to-apache-iceberg-table"><strong>https://github.com/ziedbentahar/glue-zero-etl-dynamodb-to-apache-iceberg-table</strong></a></p><h3>Solution overview</h3><p>I’ll use a hypothetical Orders table to demonstrate running analytical queries with Athena across various order-related dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGq9IFMkcinrd4FATliefA.png" /><figcaption>Using Glue Zero-ETL integration</figcaption></figure><p>In this example, I’m using a simplified Orders model, which has the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xllwT_c5pTpiBHS7H9qlUg.png" /></figure><p>We’ll look at how Zero-ETL integration handles nested fields, sets, and lists of maps but first let’s setup the configuration.</p><h3>Integration configuration</h3><p>Here are the steps involved to configure the integration.</p><h4>1- Configuring the DynamoDb source table</h4><p>Before getting started, Point in time recovery (PITR) must be enabled on the source table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xluw69kUTUCgFiH9ManKnQ.png" /><figcaption>Ddb PITR</figcaption></figure><p>We also need to configure the table’s resource policy to allow the integration to export table’s point in time.</p><pre>resource &quot;aws_dynamodb_resource_policy&quot; &quot;this&quot; {<br>  resource_arn = data.aws_dynamodb_table.this.arn<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Principal = {<br>          Service = &quot;glue.amazonaws.com&quot;<br>        },<br>        Action = [<br>          &quot;dynamodb:ExportTableToPointInTime&quot;,<br>          &quot;dynamodb:DescribeTable&quot;,<br>          &quot;dynamodb:DescribeExport&quot;<br>        ],<br>        Resource = &quot;*&quot;,<br>        Condition = {<br>          StringEquals = {<br>            &quot;aws:SourceAccount&quot; = data.aws_caller_identity.current.account_id<br>          },<br>          StringLike = {<br>            &quot;aws:SourceArn&quot; = &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:integration:*&quot;<br>          }<br>        }<br>      }<br>    ]<br>  })<br>}</pre><h4>2- Glue catalog database configuration</h4><p>An IAM role must be created for the Zero-ETL integration target to grant access to the Glue database:</p><pre>resource &quot;aws_glue_catalog_database&quot; &quot;this&quot; {<br>  name         = &quot;${var.application}${var.environment}db&quot;<br>  location_uri = &quot;s3://${aws_s3_bucket.database_bucket.bucket}/&quot;<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;integration_policy&quot; {<br>  name = &quot;${var.application}-${var.environment}-integration-policy&quot;<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;glue:CreateTable&quot;,<br>          &quot;glue:GetTable&quot;,<br>          &quot;glue:UpdateTable&quot;,<br>          &quot;glue:GetTableVersion&quot;,<br>          &quot;glue:GetTableVersions&quot;,<br>          &quot;glue:GetResourcePolicy&quot;<br>        ],<br>        Resource = [<br>          &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog&quot;,<br>          &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}&quot;,<br>          &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/${aws_glue_catalog_database.this.name}/*&quot;<br>        ]<br>      },<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;cloudwatch:PutMetricData&quot;,<br>        ],<br>        Resource = &quot;*&quot;,<br>        Condition = {<br>          StringEquals = {<br>            &quot;cloudwatch:namespace&quot; = &quot;AWS/Glue/ZeroETL&quot;<br>          }<br>        },<br><br>      },<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;logs:CreateLogGroup&quot;,<br>          &quot;logs:CreateLogStream&quot;,<br>          &quot;logs:PutLogEvents&quot;<br>        ],<br>        Resource = &quot;*&quot;<br>      },<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;glue:GetDatabase&quot;,<br>        ],<br>        Resource = [<br>          &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog&quot;,<br>          &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}&quot;,<br>        ]<br>      },<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;s3:ListBucket&quot;<br>        ],<br>        Resource = [<br>          aws_s3_bucket.database_bucket.arn,<br>        ]<br>      },<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Action = [<br>          &quot;s3:GetObject&quot;,<br>          &quot;s3:PutObject&quot;,<br>          &quot;s3:DeleteObject&quot;<br>        ],<br>        Resource = [<br>          &quot;${aws_s3_bucket.database_bucket.arn}/*&quot;,<br>        ]<br>      }<br><br>    ]<br>  })<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;integration_role&quot; {<br>  name = &quot;${var.application}-${var.environment}-integration-role&quot;<br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Effect = &quot;Allow&quot;,<br>        Principal = {<br>          Service = &quot;glue.amazonaws.com&quot;<br>        },<br>        Action = &quot;sts:AssumeRole&quot;<br>      }<br>    ]<br>  })<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;integration&quot; {<br>  role       = aws_iam_role.integration_role.name<br>  policy_arn = aws_iam_policy.integration_policy.arn<br>}</pre><h4><strong>3- Creating the integration</strong></h4><p>Currently, neither CloudFormation nor the AWS Terraform provider supports Glue Zero-ETL. So, I’m using the AWS SDK to create the integration and configure table properties. To handle this, I rely on <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_invocation">aws_lambda_invocation</a> to trigger a lambda function that creates or deletes the integration whenever a database is created or deleted—similar-ish to a CloudFormation custom resource.</p><pre>import { GlueClient, CreateIntegrationCommand, CreateIntegrationResourcePropertyCommand, DeleteIntegrationCommand, CreateIntegrationTablePropertiesCommand } from &quot;@aws-sdk/client-glue&quot;;<br>import { SSMClient, PutParameterCommand, GetParameterCommand } from &quot;@aws-sdk/client-ssm&quot;;<br><br>export const handler = async (event) =&gt; {<br><br>    let glueClient = new GlueClient({ region: process.env.AWS_REGION });<br>    let paramStore = new SSMClient({ region: process.env.AWS_REGION });<br><br>    if(event.sourceArn == null || event.targetArn == null || event.roleArn == null) {<br>        throw new Error(&quot;SourceArn, TargetArn and RoleArn are required&quot;);<br>    }<br><br>    if (event.tf.action === &quot;create&quot;) {<br><br>        const integrationResourcePropertyResult =  await glueClient.send(new CreateIntegrationResourcePropertyCommand({<br>            ResourceArn: event.targetArn,<br>            TargetProcessingProperties: {<br>                RoleArn: event.roleArn<br>            }<br>        }));       <br><br>       const integrationResult = await glueClient.send(new CreateIntegrationCommand({<br>            IntegrationName : event.integrationName,<br>            SourceArn : event.sourceArn,<br>            TargetArn : event.targetArn,<br>    <br>        }));<br>      <br>        await glueClient.send(new CreateIntegrationTablePropertiesCommand({<br>            ResourceArn: integrationResult.IntegrationArn,<br>            TableName: event.tableConfig.tableName,<br>            TargetTableConfig: {<br>                PartitionSpec: event.tableConfig.partitionSpec ? event.tableConfig.partitionSpec : undefined,<br>                UnnestSpec: event.tableConfig.unnestSpec ? event.tableConfig.unnestSpec : undefined,<br>                TargetTableName: event.tableConfig.tableName ? event.tableConfig.tableName : undefined<br>            }<br>            <br>        }));<br><br>        await paramStore.send(new PutParameterCommand({<br>            Name: event.integrationName,<br>            Value: JSON.stringify({<br>                integrationArn:  integrationResult.IntegrationArn,<br>                resourcePropertyArn: integrationResourcePropertyResult.ResourceArn<br>            }),<br>            Type: &quot;String&quot;,<br>            Overwrite: true<br>        }));<br>      <br>        return;<br>    }<br><br>    if (event.tf.action === &quot;delete&quot;) {<br>        const integrationParams = await paramStore.send(new GetParameterCommand({<br>            Name: event.integrationName,<br>        }));<br><br>        const { integrationArn } = JSON.parse(integrationParams.Parameter.Value);<br><br>        await glueClient.send(new DeleteIntegrationCommand({<br>            IntegrationIdentifier: integrationArn<br>        }));<br>     <br>        return;<br>    }<br> <br>};</pre><p>I’m using the <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/glue/command/CreateIntegrationCommand/">@aws-sdk/client-glue</a> to set up the integration, assign the target processing role, and configure table properties such as the target table name, schema unnesting options, and data partitioning for the target Apache Iceberg table. By default, the integration with DynamoDB uses the table’s primary keys.</p><p>Here’s how Lambda invocation is used; I’m passing the parameters I want to use to configure the integration:</p><pre>resource &quot;aws_lambda_invocation&quot; &quot;manage_zero_etl_integration&quot; {<br><br>  function_name = aws_lambda_function.manage_zero_etl_integration_fn.function_name<br>  input = jsonencode({<br>    integrationName = &quot;${var.application}-${var.environment}-zero-etl-integration&quot;,<br>    sourceArn       = data.aws_dynamodb_table.this.arn,<br>    targetArn       = aws_glue_catalog_database.this.arn,<br>    roleArn         = aws_iam_role.integration_role.arn,<br>    tableConfig = {<br>      tableName = data.aws_dynamodb_table.this.name,<br>      partitionSpec = [<br>        {<br>          FieldName    = &quot;orderDate&quot;,<br>          FunctionSpec = &quot;day&quot;<br>        }<br>      ],<br>      unnestSpec : &quot;FULL&quot;<br>    }<br><br>  })<br><br>  lifecycle_scope = &quot;CRUD&quot;<br><br>  depends_on = [aws_glue_resource_policy.this]<br><br>}</pre><p>Very much a happy-path solution here — just a workaround while waiting for proper IaC support. If you’d prefer not to take this route, another option is to create the integration using the <a href="https://docs.aws.amazon.com/cli/latest/reference/glue/create-integration.html">CLI</a>.</p><h4>4- Glue resource policy</h4><p>Since I’m using the Glue catalog for the integration, I made sure to include the following permissions in the glue catalog resource policy. This allows for integration between the source DynamoDB table and the target Iceberg table:</p><pre>data &quot;aws_iam_policy_document&quot; &quot;glue_resource_policy&quot; {<br>  statement {<br>    effect = &quot;Allow&quot;<br><br>    principals {<br>      type = &quot;AWS&quot;<br>      identifiers = [<br>        &quot;arn:aws:iam::${data.aws_caller_identity.current.account_id}:root&quot;,<br>        aws_iam_role.manage_zero_etl_integration_role.arn<br>      ]<br>    }<br><br>    actions = [<br>      &quot;glue:CreateInboundIntegration&quot;,<br>    ]<br><br>    resources = [<br>      &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog&quot;,<br>      &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}&quot;,<br>    ]<br><br>    condition {<br>      test     = &quot;StringLike&quot;<br>      variable = &quot;aws:SourceArn&quot;<br>      values   = [data.aws_dynamodb_table.this.arn]<br>    }<br>  }<br><br>  statement {<br>    effect = &quot;Allow&quot;<br>    principals {<br>      type        = &quot;Service&quot;<br>      identifiers = [&quot;glue.amazonaws.com&quot;]<br>    }<br><br>    actions = [<br>      &quot;glue:AuthorizeInboundIntegration&quot;<br>    ]<br><br>    resources = [<br>      &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:catalog&quot;,<br>      &quot;arn:aws:glue:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:database/${aws_glue_catalog_database.this.name}&quot;,<br>    ]<br><br>    condition {<br>      test     = &quot;StringEquals&quot;<br>      variable = &quot;aws:SourceArn&quot;<br>      values   = [data.aws_dynamodb_table.this.arn]<br>    }<br>  }<br><br>  depends_on = [<br>    aws_iam_role.manage_zero_etl_integration_role,<br>    aws_lambda_function.manage_zero_etl_integration_fn<br>  ]<br>}<br><br><br>resource &quot;aws_glue_resource_policy&quot; &quot;this&quot; {<br>  policy = data.aws_iam_policy_document.glue_resource_policy.json<br>}</pre><p>You can find this configuration in the <a href="https://docs.aws.amazon.com/glue/latest/dg/zero-etl-prerequisites.html">official docs here</a>.</p><h3><strong>Glue Zero-ETL in action</strong></h3><p>Once you’ve deployed all the components, you can go straight to the Glue Zero-ETL list. Here’s what it looks like in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ejB9fEuy1EcBZQg7BaifA.png" /><figcaption>Active Zero-ETL integrations list</figcaption></figure><p>You can view the details of the integration. By default the refresh interval from the source DynamoDb table to the Iceberg table is set to 15 minutes, it is not editable for now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*myxChZOtw-1e5Gm2pAFEfw.png" /><figcaption>Glue Zero ETL integration details</figcaption></figure><p>You can also monitor the integration operations and track the number of items inserted, updated, or deleted directly from CloudWatch Logs. The documentation for the metrics generated during each execution can be found at the <a href="https://docs.aws.amazon.com/glue/latest/dg/zero-etl-monitoring.html#zero-etl-cloudwatch-metrics">following link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OijICtHmWI9tfyUnAqlOvg.png" /><figcaption>Cloudwatch Logs — number of inserted items on SEED</figcaption></figure><p>Once the first insert operation is successful, you can view the inferred Iceberg table schema on the data catalog page on the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOn6wQwtWLemCrbE9Gk5-Q.png" /><figcaption>Table schema following first seed operation</figcaption></figure><p>☝️ Note that the shippingAddress was un-nested and deliveryPreferences was inferred as an array. That’s very convenient. However, items property was inferred as string. Since it’s a list of maps in DynamoDB, I expected it to map cleanly to a list of structs in Apache Iceberg, but it didn’t quite get the schema right.</p><p>The items property ends up as a plain JSON string in this DynamoDb list format, It’s not perfect, but we can work around it by using <a href="https://docs.aws.amazon.com/athena/latest/ug/extracting-data-from-JSON.html">json_extract</a> in Athena to parse the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A3Siu5k2mh4Yzbmi_TWiWw.png" /></figure><h4>Querying with Athena</h4><p>Here’s an example query using Athena to get the number of orders grouped by city:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6oQ3IH_iX6A_je-FoABdJQ.png" /><figcaption>Example of a query: Number of orders by city</figcaption></figure><p>Here’s another example where I want to get the number of orders by city where the delivery preferences include LeaveAtDoor. While this involves some extra steps with DynamoDB, it’s much easier to achieve with Iceberg tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*toOm2d9VtD3OfVsoeX-f4Q.png" /><figcaption>Getting the number of orders by city where delivery prefrence contains LeaveAtDoor</figcaption></figure><h3>Wrapping up</h3><p>I hope you found this article helpful! I’ve found the Glue Zero-ETL integration to be an interesting tool to have in your toolkit, especially for offloading undifferentiated heavy lifting and focusing on what matters most. It’s also useful for teams that aren’t familiar with writing Glue Jobs, as it makes running ad-hoc analytics queries on data originally stored in DynamoDB much easier.</p><p>As ususal, you can find the full code source, ready to be adapted and deployed here 👇</p><p><a href="https://github.com/ziedbentahar/glue-zero-etl-dynamodb-to-apache-iceberg-table"><strong>https://github.com/ziedbentahar/glue-zero-etl-dynamodb-to-apache-iceberg-table</strong></a></p><p>Thank you for reading and may your data be clean, your queries be fast, and your pipelines never break 😉</p><h4>Resources</h4><ul><li><a href="https://docs.aws.amazon.com/glue/latest/dg/zero-etl-using.html">Zero-ETL integrations</a></li><li><a href="https://docs.aws.amazon.com/cli/latest/reference/glue/create-integration.html">create-integration - AWS CLI 1.36.28 Command Reference</a></li><li><a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/glue/command/CreateIntegrationCommand/">AWS SDK for JavaScript v3</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9397523bfea2" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/replicate-data-from-dynamodb-to-apache-iceberg-tables-using-glue-zero-etl-integration-9397523bfea2">Replicate data from DynamoDB to Apache Iceberg tables using Glue Zero-ETL integration</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building smarter RSS feeds for my newsletter subscriptions with SES and Bedrock]]></title>
            <link>https://levelup.gitconnected.com/building-a-smarter-rss-feed-for-my-newsletter-subscriptions-a2e8735d0680?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2e8735d0680</guid>
            <category><![CDATA[claude-3]]></category>
            <category><![CDATA[newsletter]]></category>
            <category><![CDATA[hono]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Sat, 23 Nov 2024 16:24:16 GMT</pubDate>
            <atom:updated>2024-11-29T18:04:03.439Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LrlFqriDDuwEQDbB" /><figcaption>Photo by <a href="https://unsplash.com/@joannakosinska?utm_source=medium&amp;utm_medium=referral">Joanna Kosinska</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, I’ll share a tool I recently built for my personal use, driven largely by intellectual curiosity. Although I was aware of services like <a href="https://kill-the-newsletter.com/"><em>Kill the Newsletter!</em></a>, I wanted to create a service that generates personalized RSS feeds for my newsletter subscriptions. I wanted the feed to do more than just list content — it would provide summaries of featured articles and topics shared in newsletters I am subscribed to and powered by LLM to categorize, summarize, and extract key information. This helps me stay on top of relevant updates and ensure I don’t miss out on insights from the community.</p><p>To achieve this, I used a feature from Amazon Simple Email Service (SES) that allows for handling incoming emails, making it a suitable option for building event-driven automations that process messages as they arrive. In my case, I used this capability to efficiently manage the newsletters I receive, transforming them into personalized RSS feeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLhSt1pprMHnrsfLroBoxg.png" /></figure><h3>Designing the smart RSS feed generator</h3><p>To start, I wanted to be able to set up email addresses that I can create on the fly whenever I want to subscribe to one or many newsletters. Each email address would correspond to a “virtual” inbox, tailored for a specific type of subscription. Although all emails are technically routed to the same location, this approach allows me to manage and organize my subscriptions as if they were separate inboxes, each based on specific interests and topics. I can create as many dedicated inboxes as needed. For example, one could be set up for <strong><em>awesome-serverless-community@my-domain.com</em></strong>, while another could be for <strong><em>awesome-system-design@my-domain.com</em></strong>, each handling different types of newsletters and allowing me to configure separate content filtering rules for each subscription.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5_K-JGvLYmWgbq7gK2amw.png" /><figcaption>High-level overview of smart and personalized RSS feed generation from newsletters</figcaption></figure><p>Whenever a newsletter arrives at one of these “virtual” inboxes, the RSS feed generation starts. The first step involves verifying whether the system should handle the email. To achieve this, I implemented an allow list of trusted newsletter senders, which can be configured as needed. This ensures that only approved sources are processed, adding an extra layer of control to the system. Next, the raw email content is converted into Markdown format. An LLM is then used to create a gist of the newsletter and filter the content based on my interests. Both the filter configurations and the allow list are stored in a dedicated table, with filters configured for each feed.</p><p>Some of the newsletters I subscribe to feature valuable community content, such as links to blog posts or videos. I also use the LLM to generate a structured list of these links, along with their main topics, so I can easily access the content that’s most relevant to me.</p><p>Once the gist of a newsletter is ready, it is finally stored in a dedicated table. Each email address gets its own personalized RSS feed that gets served through an API, so it can be accessed by RSS feed readers.</p><h3>Solution overview</h3><p>Let’s now have a deeper look at the solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKydLFG0g9fA0vL-w3VDxg.png" /><figcaption>Smart RSS Feed generation — Solution overview</figcaption></figure><p>Incoming emails are stored in an S3 bucket. Each time a new email arrives, an event is sent to the default EventBridge bus, triggering a workflow to process the email. The <strong>Process Email</strong> function handles the entire workflow, including content conversion, filtering, and gist generation. It uses Amazon Bedrock with the Claude Sonnet model to create a structured newsletter gist, which is then stored in a DynamoDB table.</p><p>For serving the RSS feeds, I built the api using Hono web framework.</p><h4>Some notes</h4><ul><li>I chose to use a single Lambda function to handle the entire process of generating the newsletter gist, keeping that part self-contained. <a href="https://medium.com/gitconnected/ai-powered-video-summarizer-with-amazon-bedrock-and-anthropics-claude-9f1832f397dc">In a previous article</a>, I explored another approach using a Step Function to interact with an LLM, as it avoids paying for an active Lambda function while waiting for the LLM response, but it requires a more complex setup.</li><li>The API is deployed using a Function URL (FURL). I use CloudFront Origin Access Control (OAC) to restrict access to the Lambda function URL origin.</li></ul><h3>Solution details</h3><p>I built this solution using Nodejs and typescript for functions code and terraform for IaC</p><h4>TL;DR</h4><p>You will find the complete repo of this service here 👉<a href="https://github.com/ziedbentahar/smart-feeds"><strong>https://github.com/ziedbentahar/smart-feeds</strong></a></p><h4>1 — Email handling — Configuring SES</h4><p>To get started with SES handling incoming emails, we’ll add an MX record to our domain’s DNS configuration. Next, we’ll create an SES receipt rule to process all incoming emails sent to @my-domain.com. This rule includes an action to deliver the raw email content to an S3 bucket.</p><p>Here is the how to define this in terraform:</p><pre>...<br><br>resource &quot;aws_route53_record&quot; &quot;email_mx_records&quot; {<br>  zone_id = var.subdomain_zone.id<br>  name    = local.email_subdomain<br>  type    = &quot;MX&quot;<br>  ttl     = &quot;600&quot;<br><br>  records = [<br>    &quot;10 inbound-smtp.us-east-1.amazonses.com&quot;,<br>    &quot;10 inbound-smtp.us-east-1.amazonaws.com&quot;,<br>  ]<br>}<br><br>...<br>  <br>resource &quot;aws_ses_receipt_rule_set&quot; &quot;this&quot; {<br>  rule_set_name = &quot;${var.application}-${var.environment}-newsletter-rule-set&quot;<br>}<br><br>resource &quot;aws_ses_receipt_rule&quot; &quot;this&quot; {<br>  name          = &quot;${var.application}-${var.environment}-to-bucket&quot;<br>  rule_set_name = aws_ses_receipt_rule_set.this.rule_set_name<br>  recipients = [&quot;${local.email_subdomain}&quot;]<br>  enabled       = true<br>  scan_enabled  = true<br><br>  s3_action {<br>    position = 1<br>    bucket_name = aws_s3_bucket.email_bucket.bucket<br>    object_key_prefix = &quot;${local.emails_prefix}&quot;<br>  }<br><br>  depends_on = [<br>    aws_ses_receipt_rule_set.this,<br>    aws_s3_bucket.email_bucket,<br>    aws_s3_bucket_policy.email_bucket_policy<br>  ]<br>}<br><br>resource &quot;aws_ses_active_receipt_rule_set&quot; &quot;this&quot; {<br>  rule_set_name = aws_ses_receipt_rule_set.this.rule_set_name<br>  depends_on = [aws_ses_receipt_rule_set.this]<br>}</pre><p>We also need to update the bucket policy to allow SES to write to the emails bucket:</p><pre>resource &quot;aws_s3_bucket_policy&quot; &quot;email_bucket_policy&quot; {<br>  bucket = aws_s3_bucket.email_bucket.id<br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;,<br>    Statement = [<br>      {<br>        Effect    = &quot;Allow&quot;,<br>        Principal = { Service = &quot;ses.amazonaws.com&quot; },<br>        Action    = &quot;s3:PutObject&quot;,<br>        Resource  = &quot;${aws_s3_bucket.email_bucket.arn}/*&quot;,<br>        Condition = {<br>          StringEquals = {<br>            &quot;aws:Referer&quot; = data.aws_caller_identity.current.account_id<br>          }<br>        }<br>      }<br>    ]<br>  })<br>}</pre><p>After deployment, you will be able to view in the console the SES email receiving receipt rule details :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Kc8xkdQXMvMrHtP7rkL_A.png" /><figcaption>Receipt rule details</figcaption></figure><h4>2— Generating newsletter gist</h4><p>The ‘Process Email’ lambda is invoked by an event bridge rule whenever a new object is created on the inbox bucket. Let’s have a look into the different involved steps in generating the newsletter gist:</p><pre>export const lambdaHandler = async (event: S3ObjectCreatedNotificationEvent) =&gt; {<br>    const rawContent = await getRawEmailContent(event.detail.object.key);<br>    const emailId = basename(event.detail.object.key);<br><br>    if (!rawContent) {<br>        throw new Error(&quot;Email content not found&quot;);<br>    }<br><br>    const { newsletterEmailFrom, newsletterEmailTo, html, date, subject } = await parseEmail(rawContent);<br><br>    const feedsConfigs = await getFeedConfigurationsBySenderEmail(newsletterEmailFrom);<br><br>    if (feedsConfigs.length === 0) {<br>        console.warn(`No feed config found for ${newsletterEmailFrom}`);<br>        return;<br>    }<br><br>    let shortenedLinks = new Map&lt;string, string&gt;();<br>    const markdown = generateMarkdown(html, {<br>        shortenLinks: true,<br>        shortener: (href) =&gt; {<br>            let shortened = nanoid();<br>            shortenedLinks.set(shortened, href);<br>            return shortened;<br>        },<br>    });<br>    for (const [shortened, original] of shortenedLinks) {<br>        await addShortenedLink(original, shortened);<br>    }<br><br>    const output = await generateNewsletterGist(markdown);<br><br>    if (!output) {<br>        throw new Error(&quot;Failed to generate newsletter gist&quot;);<br>    }<br><br>    await Promise.allSettled(<br>        feedsConfigs.map(async (feedConfig) =&gt; {<br>            await addNewItemToFeed(feedConfig.feedId, {<br>                feedId: feedConfig.feedId,<br>                date,<br>                title: subject,<br>                emailFrom: newsletterEmailFrom!,<br>                id: emailId,<br>                ...output,<br>            });<br>        })<br>    ).catch((e) =&gt; {<br>        console.error(e);<br>    });<br>   <br>};</pre><p>🔎 Let’s zoom-in:</p><ul><li>First, the raw email content is retrieved from the inbox S3 bucket. This content is then parsed to extract the HTML content, the sender, and other relevant details used downstream.</li><li>Once we confirm the sender is on the allow list, the generateMarkdown function converts the email content into Markdown. During the transformation, unnecessary elements such as headers and styles are stripped from the email’s HTML content.</li><li>As I am interested in capturing relevant shared content, typically containing links to original sources, the generateMarkdown function extracts these links and transforms them into short Ids. These Ids are used in the prompt instead of the full links, helping to reduce the input context length when invoking the model.</li><li>The short ids are stored in a table, linked to the original URLs, and used in the RSS feed items.</li><li>generateNewsletterGist generates the prompt and invokes the model</li><li>And finally the addNewsletterGistToFeed stores the structured output in the feeds table.</li></ul><p>You can find below the the details of the generateMarkdown function, here I am relying on the <a href="https://github.com/mixmark-io/turndown">turndown</a> lib:</p><pre>import TurndownService from &quot;turndown&quot;;<br><br>const generateMarkdown = (<br>    html: string,<br>    options: <br>      { shortenLinks: true; shortener: (href: string) =&gt; string } |<br>      { shortenLinks: false }<br>): string =&gt; {<br>    const turndownService = new TurndownService({});<br><br>    turndownService.addRule(&quot;styles-n-headers&quot;, {<br>        filter: [&quot;style&quot;, &quot;head&quot;, &quot;script&quot;],<br>        replacement: function (_) {<br>            return &quot;&quot;;<br>        },<br>    });<br><br>    if (options.shortenLinks) {<br>        turndownService.addRule(&quot;shorten-links&quot;, {<br>            filter: &quot;a&quot;,<br>            replacement: function (content, node) {<br>                const href = node.getAttribute(&quot;href&quot;);<br>                if (href) {<br>                    const shortened = options.shortener(href);<br>                    return `[${content}](${shortened})`;<br>                }<br>                return content;<br>            },<br>        });<br>    }<br><br>    const markdown = turndownService.turndown(html);<br><br>    return markdown;<br>};</pre><p>One important part is the prompt I use to generate the newsletter gist:</p><pre>const prompt = `<br>Process the provided newsletter issue content in markdown format and generate a structured JSON output by performing the following tasks and adhering to the constraints:<br><br>&lt;tasks&gt; <br>    * Summarize the most important topics in this newsletter. <br>    * Identify and extract the list of content shared in the newsletter, including: <br>        * Key topics, extracted as paragraphs. <br>        * Articles <br>        * Tutorials. <br>        * Key events<br>    * For shared content, extract the most relevant link. For each link, generate a summary sentence related to it. Do not create a link if one is not provided in the newsletter. <br>    * Exclude any irrelevant content, such as unsubscribe links, social media links, or advertisements. <br>    * Do not invent topics or content that is not present in the newsletter. <br>&lt;/tasks&gt;<br><br>Here is the expected JSON schema for the output: <br>&lt;output-json-schema&gt;<br>{{output_json_schema}}<br>&lt;/output-json-schema&gt;<br><br>Here is the newsletter content: <br>&lt;newsletter-content&gt;<br>{{newsletter_content}}<br>&lt;/newsletter-content&gt;<br>`;<br><br>export const generatePromptForEmail = (newsletterContent: string, outputJsonSchema: string) =&gt; {<br>    return prompt<br>        .replace(&quot;{{newsletter_content}}&quot;, newsletterContent)<br>        .replace(&quot;{{output_json_schema}}&quot;, outputJsonSchema);<br>};</pre><p>To ensure the LLM generates the expected result in JSON, I provide the JSON schema for the output structure. Instead of hardcoding the output schema in the prompt, I define the schema of the newsletter gist using Zod and infer both the TypeScript type and the JSON schema from it. This way, any changes to the schema are also reflected in the LLM output:</p><pre>import { z } from &quot;zod&quot;;<br>import zodToJsonSchema from &quot;zod-to-json-schema&quot;;<br><br>const linkSchema = z.object({<br>    text: z.string(),<br>    url: z.string(),<br>});<br><br>t const newsletterGist = z.object({<br>    summary: z.string(),<br>    topics: z.array(z.string()),<br>    links: z.array(linkSchema),<br>});<br><br>export type NewsletterGist = z.infer&lt;typeof newsletterGist&gt;;<br>export const newsletterGistSchema = zodToJsonSchema(newsletterGist);</pre><p>To invoke the model, I use Bedrock Converse API, this allows me to write code once and use it with different models:</p><pre>const prompt = generatePromptForEmail(markdown, JSON.stringify(newsletterGistSchema), config);<br><br>const result = await bedrockClient.send(<br>    new ConverseCommand({<br>        modelId: process.env.MODEL_ID,<br>        system: [{ text: &quot;You are an advanced newsletter content extraction and summarization tool.&quot; }],<br>        messages: [<br>            {<br>                role: &quot;user&quot;,<br>                content: [<br>                    {<br>                        text: prompt,<br>                    },<br>                ],<br>            },<br>            {<br>                role: &quot;assistant&quot;,<br>                content: [<br>                    {<br>                        text: &quot;{&quot;,<br>                    },<br>                ],<br>            },<br>        ],<br>    })<br>);</pre><p>Since I want to enforce a JSON output, I’ll need to prefill the assistant’s message with an opening <strong>{</strong>. This is specific to Claude models.</p><h4>3 — Serving the newsletters gists as an RSS feed</h4><p>Working with <a href="https://hono.dev/docs/getting-started/aws-lambda">Hono</a> is a breeze. It simplifies many aspects of defining web APIs while supporting Lambda natively. This API serves multiple routes, and I chose to deploy it as a mono-lambda (AKA Lambdalith) to simplify the infrastructure definition:</p><pre>import { Hono } from &quot;hono&quot;;<br>import { handle } from &quot;hono/aws-lambda&quot;;<br>import { feeds } from &quot;./routes/feeds&quot;;<br>import { newsletters } from &quot;./routes/newsletters&quot;;<br>import { links } from &quot;./routes/links&quot;;<br>import { home } from &quot;./routes/home&quot;;<br><br>export const app = new Hono();<br><br>app.route(&quot;/&quot;, home);<br>app.route(&quot;/feeds&quot;, feeds);<br>app.route(&quot;/links&quot;, links);<br>app.route(&quot;/newsletters&quot;, newsletters);<br><br>export const handler = handle(app);</pre><p>Easy! The feeds route generates the RSS feed from the newsletter gists already stored in the feeds table:</p><pre>export const feeds = new Hono().get(&quot;/:id/rss&quot;, async (c) =&gt; {<br>    const feedId = c.req.param(&quot;id&quot;);<br><br>    const feedConfig = await getFeedConfig(feedId);<br><br>    let feedItems: NewsletterIssueGist[] = [];<br><br>    for await (const items of getFeedItems(feedId)) {<br>        feedItems = [...items.map((item) =&gt; item.content), ...feedItems];<br>    }<br><br>    const rssFeedItems = feedItems.reduce(<br>        (acc, item) =&gt; {<br>            acc[item.id] = {<br>                item: {<br>                    title: item.title,<br>                    description: html`&lt;div&gt;<br>                        &lt;section&gt;📩 ${item.emailFrom}&lt;/section&gt;<br>                        &lt;section&gt;📝 ${item.summary}&lt;/section&gt;<br>                        &lt;section&gt;<br>                            &lt;div&gt;📝 Topics&lt;/div&gt;<br>                            &lt;ul&gt;<br>                                ${item.topics.map((t) =&gt; {<br>                                    return `&lt;li&gt;${t}&lt;/li&gt;`;<br>                                })}<br>                            &lt;/ul&gt;<br>                        &lt;/section&gt;<br>                        &lt;section&gt;<br>                            &lt;div&gt;<br>                                &lt;a href=&quot;https://${process.env.API_HOST}/newsletters/${item.id}&quot;<br>                                    &gt;📰 Open newsletter content&lt;/a<br>                                &gt;<br>                            &lt;/div&gt;<br>                        &lt;/section&gt;<br>                        &lt;section&gt;<br>                            &lt;ul&gt;<br>                                ${item.links.map((l) =&gt; {<br>                                    return `<br>                                            &lt;li&gt;<br>                                                &lt;a href=&quot;https://${process.env.API_HOST}/links/${l.url}&quot;<br>                                                    &gt;🔗 ${l.text}&lt;/a<br>                                                &gt;<br>                                            &lt;/li&gt;<br>                                        `;<br>                                })}<br>                            &lt;/ul&gt;<br>                        &lt;/section&gt;<br>                    &lt;/div&gt;`.toString(),<br>                    guid: item.id,<br>                    link: `https://${process.env.API_HOST}/newsletters/${item.id}`,<br>                    author: item.emailFrom,<br>                    pubDate: () =&gt; new Date(item.date).toUTCString(),<br>                },<br>            };<br>            return acc;<br>        },<br>        {} as Record&lt;string, unknown&gt;<br>    );<br><br>    const feed = toXML(<br>        {<br>            _name: &quot;rss&quot;,<br>            _attrs: {<br>                version: &quot;2.0&quot;,<br>            },<br>            _content: {<br>                channel: [<br>                    {<br>                        title: feedConfig?.name,<br>                    },<br>                    {<br>                        description: feedConfig?.description,<br>                    },<br>                    {<br>                        link: `https://${process.env.API_HOST}/feeds/${feedId}/rss`,<br>                    },<br>                    {<br>                        lastBuildDate: () =&gt; new Date(),<br>                    },<br>                    {<br>                        pubDate: () =&gt; new Date(),<br>                    },<br>                    {<br>                        language: &quot;en&quot;,<br>                    },<br>                    Object.values(rssFeedItems),<br>                ],<br>            },<br>        },<br>        { header: true, indent: &quot;  &quot; }<br>    );<br><br>    return c.text(feed);<br>});</pre><p>here I am using the <a href="https://www.npmjs.com/package/jstoxml">jstoxml</a> package to be able to convert the newsletter gist structure to the RSS feed XML format.</p><p>The other routes exposed by this API include <strong>/newsletters</strong>, which renders the HTML of the received email already stored in the inbox bucket, and <strong>/links</strong>, which redirects the caller to the original content link using the short link id.</p><p>And finally, here is the CloudFront OAC configuration for the API exposed via a function URL:</p><pre>resource &quot;aws_cloudfront_origin_access_control&quot; &quot;this&quot; {<br>  name                              = &quot;${var.application}-${var.environment}-api-oac&quot;<br>  origin_access_control_origin_type = &quot;lambda&quot;<br>  signing_behavior                  = &quot;always&quot;<br>  signing_protocol                  = &quot;sigv4&quot;<br>}<br><br><br><br>resource &quot;aws_cloudfront_distribution&quot; &quot;this&quot; {<br><br>  origin {<br>    domain_name              = replace(aws_lambda_function_url.api.function_url, &quot;/https:\\/\\/|\\//&quot;, &quot;&quot;)<br>    origin_access_control_id = aws_cloudfront_origin_access_control.this.id<br>    origin_id                = &quot;api&quot;<br><br>    custom_origin_config {<br>      http_port              = 80<br>      https_port             = 443<br>      origin_protocol_policy = &quot;https-only&quot;<br>      origin_ssl_protocols   = [&quot;TLSv1.2&quot;]<br>    }<br>  }<br><br>  enabled         = true<br>  is_ipv6_enabled = true<br><br>  aliases = [local.api_subdomain]<br><br>  default_cache_behavior {<br>    allowed_methods  = [&quot;DELETE&quot;, &quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;, &quot;PATCH&quot;, &quot;POST&quot;, &quot;PUT&quot;]<br>    cached_methods   = [&quot;GET&quot;, &quot;HEAD&quot;]<br>    target_origin_id = &quot;api&quot;<br><br>    cache_policy_id        = data.aws_cloudfront_cache_policy.disabled.id<br>    viewer_protocol_policy = &quot;allow-all&quot;<br>    min_ttl                = 0<br><br>  }<br><br>  price_class = &quot;PriceClass_200&quot;<br><br>  restrictions {<br>    geo_restriction {<br>      restriction_type = &quot;none&quot;<br>      locations        = []<br>    }<br>  }<br><br>  viewer_certificate {<br>    acm_certificate_arn      = aws_acm_certificate.this.arn<br>    ssl_support_method       = &quot;sni-only&quot;<br>    minimum_protocol_version = &quot;TLSv1.2_2021&quot;<br><br>  }<br><br>  depends_on = [aws_acm_certificate_validation.cert_validation]<br>}</pre><p>Once we deploy the API we can access the RSS feed by using this url <strong>https://&lt;some domain&gt;/feeds/&lt;feed-id&gt;/rss</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKvobK8MXX901ph8iL2MnQ.png" /><figcaption>Getting `Awesome serverless updates` newsletters gists</figcaption></figure><p>Here is an example of how it renders in an RSS reader application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*2qD021vQ2TfTd5_4f3H1Fg.gif" /><figcaption>`Awesome serverless` newsletters gists from an RSS reader</figcaption></figure><p>Pretty neat, isn’t it? This way, I can follow updates from the community all in one place ! 👍</p><h3>Wrapping up</h3><p>I had fun building this tool. In my initial iteration, I intended to leverage Bedrock’s prompt management and prompt flow features. Unfortunately, at the time of writing, these services were not mature enough. But, I might explore them in the future.</p><p>The email automation pattern used here isn’t limited to processing newsletters. It can be applied to a varity of other use cases, such as customer support systems or invoice and receipt handling.</p><p>As always, you can find the full code source, ready to be adapted and deployed here 👇</p><p><a href="https://github.com/ziedbentahar/smart-feeds"><strong>https://github.com/ziedbentahar/smart-feeds</strong></a></p><p>Hope you enjoyed it!</p><h3>Resources</h3><ul><li><a href="https://docs.aws.amazon.com/ses/latest/dg/receiving-email.html">Email receiving with Amazon SES</a></li><li><a href="https://hono.dev/docs/getting-started/aws-lambda">AWS Lambda - Hono</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2e8735d0680" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/building-a-smarter-rss-feed-for-my-newsletter-subscriptions-a2e8735d0680">Building smarter RSS feeds for my newsletter subscriptions with SES and Bedrock</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Alternative to Batch Jobs: Scheduling Events with EventBridge Scheduler]]></title>
            <link>https://levelup.gitconnected.com/an-alternative-to-batch-jobs-scheduling-events-with-eventbridge-scheduler-3512bbf2e9a4?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/3512bbf2e9a4</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[patterns]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[eventbridge]]></category>
            <category><![CDATA[eda]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Tue, 10 Sep 2024 10:25:44 GMT</pubDate>
            <atom:updated>2024-09-12T15:11:28.411Z</atom:updated>
            <content:encoded><![CDATA[<h3>An alternative to batch jobs: scheduling events with EventBridge Scheduler</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GAOT-Q6MTvfw2W2c" /><figcaption>Photo by <a href="https://unsplash.com/@karsten_fuellhaas?utm_source=medium&amp;utm_medium=referral">Karsten Füllhaas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In a previous post, I wrote about EventBridge Scheduler and how we can use it to <a href="https://levelup.gitconnected.com/using-aws-eventbridge-scheduler-to-build-a-serverless-reminder-application-ba3086cf8e">build a reminder application</a>. In this article, we’ll explore how scheduling messages in the future can be also an alternative to batch jobs. This approach is sometimes overlooked; it offers advantages such as reducing system load and improving cost efficiency.</p><p>To illustrate this pattern, we’ll design a system that needs to execute a task after a specified delay whenever a new item is created in a database. This task could involve operations such as resource creation or resource cleanup.</p><p>Here are two approaches to solving this:</p><ul><li>A batch job is triggered on a schedule, this job selects all newly created items that match the creation time criteria and performs the required tasks for each:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0TQLV-FGBdTxCRIRG09oA.png" /><figcaption>Batch job approach</figcaption></figure><ul><li>Or an event-based approach: When an item is created, the system schedules a one-time message for that new item to be triggered in the future. At due time, the message is sent and the associated task is executed:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7WQWh-W6Wkmen7FW2qFIg.png" /><figcaption>Event-based approach</figcaption></figure><p>There are some downsides to using a batch job approach compared to an event-based approach:</p><ol><li><strong>Potential delays</strong>: Batch jobs run on a fixed schedule, causing tasks to be processed at intervals. This can lead to delays, as some tasks will only be executed after the next batch execution. An event-based approach triggers tasks instantly when execution time criteria is met, ensuring timely execution.</li><li><strong>Resource overhead and inefficiency</strong>: Depending on how the data is stored, batch jobs can be resource-intensive, particularly when processing a large number of records at once. For example, if the data is stored in a DynamoDB table, it may require scanning the table to find items matching item’s creation time criteria or adjusting the design of the table. This can lead to sub-optimal resource usage. In contrast, an event-based approach distributes the workload more evenly over time and potentially lowering costs by eliminating the need for periodic processing.</li></ol><p><strong>⚠️ A word on cancellation on the event-based approach</strong>: This pattern can become tricky when handling task cancellations. Possible solutions include adding the ability to delete an existing schedule when a cancellation event occurs or validating in the ‘Run Task’ step whether the scheduled action is still eligible for execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PAa6pymiSSHDUXYLT2BLQ.png" /><figcaption>A strategy for handling task cancellation</figcaption></figure><h3>Possible implementations</h3><p>In the example, I’ll use DynamoDB as the storage layer. However, the same principles apply to other storage systems that support item-level change data capture.</p><h4>❌ Using TTL as a scheduling mechanism</h4><p>One solution is to leverage DynamoDB’s TTL item expiration. When creating the original item, add an associated item in the same transaction, which we’ll call a ‘signal’. This signal is created with a TTL that matches the desired event date. And then, configure a DynamoDB Stream and subscribe the ‘Run Task’ function to the signal <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-streams.html">item expiration event</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gc44y0qLPmcxDrhWoq1WkQ.png" /><figcaption>Using TTL as a schedule</figcaption></figure><p><strong>🛑 There is an issue with this approach</strong>: The item expiration and deletion are not guaranteed to be immediate so tasks for these new items might not run exactly at the due time. The DynamoDB TTL docs do not specify a precise timeframe; item expiration can occur within a few days:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9O3YtdYZHHpuagLKWDKi0A.png" /><figcaption>DynamoDb <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html">TTL documentation page</a></figcaption></figure><p>If there are no strict requirements to trigger the task at the exact expected time, this solution can be suitable. It’s a tradeoff.</p><h4>✅ A better solution</h4><p>There is a better solution with EventBridge Scheduler and one-time schedules capability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cpjcKrx2tArEiir1H_TsHA.png" /><figcaption>Using event bridge scheduler</figcaption></figure><p>The ‘Create One-Time Schedule for New Items’ function filters for newly created items in the table’s stream and creates a new schedule for each. While I could have configured the EventBridge Scheduler to directly trigger a Lambda function as the target, this approach might not be suited when dealing with a high volume of events. In some cases, it’s better to use a queue as target and control the ‘Run Task’ function concurrency. This way, the function can process messages at a manageable rate. It’s about finding the right balance between direct invocation and rate control, and depending on the concrete use case at hand.</p><p>On a side note, I would have loved to see EventBridge Scheduler as a target of EventBridge Pipes; this could have simplified the solution even further.</p><p>Let’s see how to implement this with EventBridge Scheduler. I’ll be using terraform for IaC and Rust for functions code.</p><h3>Scheduling new tasks</h3><p>First, I’ll create a dedicated scheduler group to organise all scheduled tasks within a single group:</p><pre>resource &quot;aws_scheduler_schedule_group&quot; &quot;schedule_group&quot; {<br>  name = &quot;${var.application}-${var.environment}&quot;<br>}</pre><p>The ‘Schedule Tasks’ function handles only new items created on the table. We achieve this by using DynamoDB Stream filtering capabilities provided by the event source mapping. As a result, the function will be invoked only when new items are inserted:</p><pre>resource &quot;aws_lambda_event_source_mapping&quot; &quot;schedule_task_lambda&quot; {<br>  event_source_arn        = aws_dynamodb_table.table.stream_arn<br>  function_name           = aws_lambda_function.schedule_task_lambda.function_name<br>  starting_position       = &quot;TRIM_HORIZON&quot;<br>  function_response_types = [&quot;ReportBatchItemFailures&quot;]<br><br>  filter_criteria {<br>    filter {<br>      pattern = jsonencode({<br>        eventName = [&quot;INSERT&quot;]<br>      })<br>    }<br>  }<br>}</pre><p>To create new one-time schedules on EventBridge Scheduler, this function requires the scheduler:CreateSchedule action on the custom scheduler group, as well as the iam:PassRole action for the role used by the schedule. A role that grants permission to send messages to the Tasks SQS queue:</p><pre>...<br> {<br>    Effect = &quot;Allow&quot;<br>    Action = [<br>      &quot;scheduler:CreateSchedule&quot;<br>    ]<br>    Resource = &quot;arn:aws:scheduler:${region}:${account_id}:schedule/${aws_scheduler_schedule_group.schedule_group.name}/*&quot;<br>  },<br>  {<br>    Effect = &quot;Allow&quot;<br>    Action = [<br>      &quot;iam:PassRole&quot;<br>    ]<br>    Resource = aws_iam_role.scheduler_role.arn<br>  },<br>...</pre><h4><strong>Zooming in on the ‘Schedule Task’ function code</strong></h4><p>For each new item, we’ll create a schedule that will trigger once, two hours after the item’s creation time:</p><pre>async fn process_new_item(<br>    new_item: &amp;SomeItem,<br>    scheduler_client: &amp;aws_sdk_scheduler::Client,<br>    scheduler_group_name: &amp;String,<br>    scheduler_target_arn: &amp;String,<br>    scheduler_role_arn: &amp;String,<br>) -&gt; Result&lt;(), Error&gt; {<br><br>    // as an example, we&#39;ll configure a one-time schedule two hours after the item was created<br>    let now = Utc::now();<br>    let two_hours_later = now + Duration::hours(2);<br>    let two_hours_later_fmt = two_hours_later.format(&quot;%Y-%m-%dT%H:%M:%S&quot;).to_string();<br><br>    let response = scheduler_client<br>        .create_schedule()<br>        .name(format!(&quot;schedule-{}&quot;, &amp;new_item.id))<br>        .action_after_completion(ActionAfterCompletion::Delete)<br>        .target(<br>            Target::builder()<br>                .input(serde_json::to_string(&amp;new_item)?)<br>                .arn(scheduler_target_arn)<br>                .role_arn(scheduler_role_arn)<br>                .build()?,<br>        )<br>        .flexible_time_window(<br>            FlexibleTimeWindow::builder()<br>                .mode(FlexibleTimeWindowMode::Off)<br>                .build()?,<br>        )<br>        .group_name(scheduler_group_name)<br>        .schedule_expression(format!(&quot;at({})&quot;, two_hours_later_fmt))<br>        .client_token(nanoid!())<br>        .send()<br>        .await;<br><br>    match response {<br>        Ok(_) =&gt; Ok(()),<br>        Err(e) =&gt; {<br>            error!(&quot;Failed to create schedule: {:?}&quot;, e);<br>            return Err(Box::new(e));<br>        }<br>    }<br>}</pre><p>The important bits:</p><ul><li>Some parameters are required. When defining the target, you’ll need to provide the message payload and specify the role that EventBridge Scheduler will use to invoke the target as well as the target arn, which is the arn of the SQS queue in this case.</li><li>We also need to ensure that the schedule is deleted after the target invocation is successful by using ActionAfterCompletion::Delete</li></ul><p>And since we configured the function response type to ReportBatchItemFailures, here is how process_new_item is called by the function handler:</p><pre>async fn process_records(<br>    event: LambdaEvent&lt;Event&gt;,<br>    scheduler_client: &amp;aws_sdk_scheduler::Client,<br>    scheduler_group_name: &amp;String,<br>    scheduler_target_arn: &amp;String,<br>    scheduler_role_arn: &amp;String,<br>) -&gt; Result&lt;DynamoDbEventResponse, Error&gt; {<br>    let mut response = DynamoDbEventResponse {<br>        batch_item_failures: vec![],<br>    };<br><br>    if event.payload.records.is_empty() {<br>        return Ok(response);<br>    }<br><br>    for record in &amp;event.payload.records {<br>        let item = record.change.new_image.clone();<br>        let new_item: SomeItem = serde_dynamo::from_item(item)?;<br><br>        let record_processing_result = process_new_item(<br>            &amp;new_item,<br>            scheduler_client,<br>            &amp;scheduler_group_name,<br>            &amp;scheduler_target_arn,<br>            &amp;scheduler_role_arn,<br>        )<br>        .await;<br><br>        if record_processing_result.is_err() {<br>            let error = record_processing_result.unwrap_err();<br>            error!(&quot;error processing item - {}&quot;, error);<br>            response.batch_item_failures.push(DynamoDbBatchItemFailure {<br>                item_identifier: record.change.sequence_number.clone(),<br>            });<br>           <br>        }<br>    }<br><br>    Ok(response)<br>}</pre><p>Once new tasks are correctly scheduled, they will be visible on the schedules page in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5QA4DWGyAVOvtsWRRFHYrQ.png" /><figcaption>Schedules page on the Console</figcaption></figure><p>We can also view the details related to the schedule’s target in the console</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pUXhmSfuPl2gNo07VLP5Gg.png" /><figcaption>Schedule Target configuration</figcaption></figure><p>When the scheduled task time is due, the message associated with the item, will be sent to the SQS queue and then processed by the ‘Run Task’ function. That’s all 👌 !</p><h3>Wrapping up</h3><p>With one-time schedules EventBridge Scheduler enables interesting patterns that can improve application architecture by reducing the overhead associated with batch jobs. But as always, choosing between an event-driven approach or batch jobs depends on your application’s needs and complexity of the use case at hand.</p><p>You can find the complete source code, this time written in Rust and Terraform, ready to adapt and deploy 👇</p><p><a href="https://github.com/ziedbentahar/scheduling-messages-in-future-with-eventbridge-scheduler/tree/main">GitHub - ziedbentahar/scheduling-messages-in-future-with-eventbridge-scheduler</a></p><p>Thanks for reading — I hope you found it helpful!</p><h3>Resources</h3><ul><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html">Using time to live (TTL) in DynamoDB</a></li><li><a href="https://serverlessland.com/serverless/visuals/eventbridge/eventbridge-scheduler">Serverless Land</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzWgqj2OEKX8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzWgqj2OEKX8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzWgqj2OEKX8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f44420712a0499d1b4b9eb7193224a9/href">https://medium.com/media/3f44420712a0499d1b4b9eb7193224a9/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3512bbf2e9a4" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/an-alternative-to-batch-jobs-scheduling-events-with-eventbridge-scheduler-3512bbf2e9a4">An Alternative to Batch Jobs: Scheduling Events with EventBridge Scheduler</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG on media content with Bedrock Knowledge Bases and Amazon Transcribe]]></title>
            <link>https://levelup.gitconnected.com/using-rag-on-media-content-with-bedrock-knowledge-bases-and-amazon-transcribe-92abea166e68?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/92abea166e68</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[amazon-transcribe]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Thu, 01 Aug 2024 15:34:36 GMT</pubDate>
            <atom:updated>2024-08-06T22:11:00.083Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IN9Eb43nUR64B1Q9" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=medium&amp;utm_medium=referral">Pawel Czerwinski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://levelup.gitconnected.com/ai-powered-video-summarizer-with-amazon-bedrock-and-anthropics-claude-9f1832f397dc">a previous article</a>, I wrote about building an application capable of generating summaries from YouTube videos. It was fun to build. However, the solution was limited as it was only handling Youtube videos and relied on YouTube’s generated transcripts. I wanted to improve this solution and make it more versatile as well as supporting other media formats.</p><p>In this article, I take a different approach: Transcribing media files with Amazon Transcribe and using the generated transcripts as a knowledge base, allowing for retrieval-augmented generation with Amazon Bedrock.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*5YKjzEm6p84XeS7Bzwcyrw.png" /></figure><h3>RAG with Amazon Bedrock Knowledge Bases</h3><p>Retrieval-Augmented Generation (RAG) is a technique that improves model responses by combining information retrieval with prompt construction: When processing a query, the system first retrieves relevant data from custom knowledge bases. It then uses this data in the prompt, enabling the model to generate more accurate and contextually relevant answers.</p><p>RAG significantly improves the model’s ability to provide informed, up-to-date answers, bridging the gap between the model’s training data and custom up-to-date information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*qxn1RXNCsbG7QpXKc8YlcQ.png" /><figcaption>Typical retrieve and generate flow</figcaption></figure><h4>How Bedrock can help ?</h4><p>Amazon Bedrock’s Knowledge Bases simplify the RAG process by handling much of the heavy lifting. This includes synchronizing content from Amazon S3, chunking, converting it into embeddings, and storing them in vector databases. It also provides endpoints that allow applications to query the knowledge base while generating responses based on the retrieved data. By handling these tasks, Bedrock allows to focus on building AI-powered applications rather than managing infrastructure.</p><p>In this article, I will use RAG for media transcripts to generate responses based on audio or video content. These contents can be meeting recordings, podcasts, conference talks, and more.</p><h3>Solution overview</h3><p>The architecture of a typical RAG system consists of two components:</p><ul><li>Knowledge base indexing and synchronisation</li><li>Retrieval and generation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qo8AbLkySnAPylgiT1KA5Q.png" /><figcaption>Architecture overview</figcaption></figure><p>Users first request an upload link by invoking the <strong>Request Media upload link</strong> function, which generates an S3 presigned URL. The user’s request includes the media metadata such as the topic, the link to the media and date. This metadata will be stored and used downstream by Bedrock to apply filtering during the retrieval phase. When the upload process completes in the media bucket, the <strong>Start transcription job</strong> function is triggered by media bucket event notification.</p><p>When a transcription job state changes, EventBridge will publish job completion status events (Success or Failure). The <strong>Handle transcription and sync knowledge base</strong> function handles only successful events, extracts the transcription content, stores the extracted text transcript in the knowledge base bucket, and triggers a knowledge base sync.</p><p>The vector database is an important part of a RAG system. It stores and retrieves text representations as vectors (also known as embeddings). allowing for similarity searches when given a query. Bedrock supports various vector databases, including Amazon OpenSearch, PostgreSQL with the pgvector extension, and Pinecone. Each option has its advantages. In this solution, I chose Pinecone as it is a serverless service that allows for quick and easy setup.</p><p>When using Knowledge Bases, the Model used for generating embeddings can differ from the one used for response generation. For example, in this sample, I use “Amazon Titan Text embedding v2” for embedding and “Claude 3 Sonnet” for response generation.</p><p>☝<strong>️Note</strong>: In this article, I did not include the parts that handle transcription job monitoring and asynchronous job progress notifications to the requester. For insights on building an asynchronous REST APIs, you can refer to <a href="https://zied-ben-tahar.medium.com/serverless-asynchronous-rest-apis-on-aws-c48156d790c1">this previous article.</a></p><p>Alright, let’s deep dive into the implementation</p><h3>Solution Details</h3><p>This time, I am taking a different approach compared to my previous articles: I will be using Rust for lambda code and terraform for IaC.</p><h4>1- Creating the Pinecone vector database</h4><p>I try to do IaC whenever possible. The good news is that Pinecone offers a Terraform provider, which simplifies managing Pinecone indexes and collections as code. First we’ll need an API Key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8rC0BmI_c1GNTyBkW5zBg.png" /><figcaption>Creating API Key from pinecone console</figcaption></figure><p>Here, I am using the serverless version of Pinecone. We need to set the PINECONE_API_KEY environment variable to the API key we just created so that it can be used by the provider.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abee557d3eae62e752b13e7250c82373/href">https://medium.com/media/abee557d3eae62e752b13e7250c82373/href</a></iframe><h4>2- Creating the knowledge base</h4><p>Creating the knowledge base involves defining two key components: the vector store configuration, which points to Pinecone, and the data source.</p><p>The data source dictates how the content will be ingested, including the storage configuration and the content chunking strategy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b33b554f83b9878158ef2a47fc93cbb/href">https://medium.com/media/1b33b554f83b9878158ef2a47fc93cbb/href</a></iframe><p>For the data source, I am setting the chunking strategy to FIXED_SIZE</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/677fe7547f72f12bb1d467b631e22f29/href">https://medium.com/media/677fe7547f72f12bb1d467b631e22f29/href</a></iframe><p>After deployment, you will be able to view in the console the data source configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-etm7HYsfA8C53lqLQjaA.png" /><figcaption>Data source configuration on the console</figcaption></figure><h4>3 — Requesting media upload link</h4><p>This function is invoked by the API Gateway to generate a presigned URL for media file uploads. A unique identifier is assigned to the object, which will also serve as the transcription job name and as the reference for the knowledge base document.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d655b46d769d02232b715ac9cae51e3/href">https://medium.com/media/4d655b46d769d02232b715ac9cae51e3/href</a></iframe><p>The request is validated to ensure that the media metadata properties are properly defined. I am using the serde_valid crate to validate the request payload. This crate is very convenient for defining schema validations using attributes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/882795446934b50b0811ea64f8ce1334/href">https://medium.com/media/882795446934b50b0811ea64f8ce1334/href</a></iframe><p>And here is are details of the generate_presigned_request_uri</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4ddc96d3fddd2d3bc5a6ac42030afc5/href">https://medium.com/media/e4ddc96d3fddd2d3bc5a6ac42030afc5/href</a></iframe><h4>4- Handling media upload and starting transcription job</h4><p>This function is triggered by an S3 event whenever a new file is successfully uploaded. As a convention, I am using the media object key as the transcription job name which is the unique identifier of the task.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8f49a3dd9184dae6cfc371aac1eeda1/href">https://medium.com/media/e8f49a3dd9184dae6cfc371aac1eeda1/href</a></iframe><p>This function needs transcribe:startTranscriptionJob permission in order to be able to start a transcription task.</p><p>Once the task is started, we can monitor the transcription job process in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RJVF64sPBKSHYMIL6yrQg.png" /><figcaption>Transcription job details</figcaption></figure><h4>5- Subscribing to transcription success events and syncing knowledge base</h4><p>Let’s first have a look into the event bridge rule definition in terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87c1e31b019b82deff87690a2074a1ad/href">https://medium.com/media/87c1e31b019b82deff87690a2074a1ad/href</a></iframe><p>Which translates to the following configuration in the AWS console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucKkyXfDBNo0zfROI1B6IQ.png" /><figcaption>Transcription success event bridge rule</figcaption></figure><p>Here, the <strong>Handle successful transcription</strong> function is invoked each time a transcription is successfully completed. I am only interested in having the transcription job name, as I will use it as the data source object key:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33183a6df82e8db90b0a63b28f08f8f1/href">https://medium.com/media/33183a6df82e8db90b0a63b28f08f8f1/href</a></iframe><p>This function first retrieves the transcription result content available at the transcript_file_uri, extracts the important part and stores it in the knowledge base bucket as well as its metadata and then triggers a start_ingestion_job. If the operation fails, it will be retried by EventBridge and eventually put into a dead letter queue.</p><p>☝️<strong>Note: </strong>I opted against using a step function for this part since the transcribe output could exceed 256 KB.</p><h4>6- Chatting with the knowledge base in the console</h4><p>Before building the function that queries the knowledge base, we can already test it from the console. I used this <a href="https://www.youtube.com/watch?v=5Qe9mCgr6-Q">awesome first believe in serverless podcast episode</a> as a data source:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9DrVpZGnMj_v7-vRSvsa4Q.png" /><figcaption>Testing the knowledge base from the console</figcaption></figure><p>The console also provides a way to test and adjust the generation configuration, including choosing the model, using a custom prompt, and adjusting parameters like temperature, top-p. This allows you to tailor the configuration to your specific use case requirements.</p><p>Alright, let’s now create the endpoint to see how we can query this knowledge base</p><h4>7- Querying the knowledge base</h4><p>This function requires the bedrock:RetrieveAndGenerate permission for accessing the knowledge base and the bedrock:InvokeModel permission for the Claude 3 sonnet model arn used during the generation phase. It returns an output result along with the source URL associated with the retrieved chunks that contributed to the output:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99789c830edcbf701f209c5e151c32ce/href">https://medium.com/media/99789c830edcbf701f209c5e151c32ce/href</a></iframe><p>The build_retrieve_and_generate_configuration function prepares the necessary parameters for calling the retrieveAndGenerate endpoint.</p><p>As an example, I am applying a retrieval filter to the topic attribute.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee3d0b2446523bfafe95d53d3adf3afd/href">https://medium.com/media/ee3d0b2446523bfafe95d53d3adf3afd/href</a></iframe><p>Et voilà ! Let’s call our api straight from postman</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKuFQiw0gNHd26UBdr2-bA.png" /><figcaption>Testing the RAG endpoint from Postman</figcaption></figure><h3>Wrapping up</h3><p>I’ve only scratched the surface when it comes to building RAG systems. Bedrock simplifies the process considerably. There’s still plenty of room for improvement, such as optimising retrieval methods and refining prompts. I had also fun building this in Rust — using <a href="https://www.cargo-lambda.info/">Cargo Lambda</a> makes creating Lambdas in Rust a breeze, check it-out!</p><p>As always, you can find the full code source, ready to be adapted and deployed here:</p><p><a href="https://github.com/ziedbentahar/rag-on-media-content-with-bedrock-and-transcribe">ziedbentahar/rag-on-media-content-with-bedrock-and-transcribe</a></p><p>Thanks for reading ! Hope you enjoy it</p><h3>Resources</h3><ul><li><a href="https://docs.pinecone.io/integrations/aws">Amazon Web Services (AWS) - Pinecone Docs</a></li><li><a href="https://www.cargo-lambda.info/">Rust functions on AWS Lambda made simple</a></li><li><a href="https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-create.html">Create a knowledge base by connecting to a data source in Amazon Bedrock Knowledge Bases</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92abea166e68" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/using-rag-on-media-content-with-bedrock-knowledge-bases-and-amazon-transcribe-92abea166e68">RAG on media content with Bedrock Knowledge Bases and Amazon Transcribe</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding flexibility to your deployments with Lambda Web Adapter]]></title>
            <link>https://levelup.gitconnected.com/adding-flexibility-to-your-deployments-with-lambda-web-adapter-67aecb0ecbff?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/67aecb0ecbff</guid>
            <category><![CDATA[flexible-architecture]]></category>
            <category><![CDATA[lambda-function]]></category>
            <category><![CDATA[lambda-web-adapter]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[cdk]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Fri, 19 Apr 2024 07:43:53 GMT</pubDate>
            <atom:updated>2024-05-09T14:59:38.608Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XWhUspOFaIMUneQD" /><figcaption>Photo by <a href="https://unsplash.com/@leladesign?utm_source=medium&amp;utm_medium=referral">Lea L</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://github.com/awslabs/aws-lambda-web-adapter">Lambda Web Adapter</a> (LWA) is an open-source project that enables running Web apps on Lambda functions without the need to change or adapt the code.</p><p>In my opinion, LWA opens up interesting pathways for architecture evolution: While it’s an interesting tool that helps lift &amp; shift Web apps and APIs to Lambda functions without a lot of effort, it can also enable another migration path : Start deploying your application in Lambda as a Lambdalith and then transition to a classical container deployment when needed (<em>e.g.</em> ECS Fargate). In some scenarios, it may happen that you don’t have enough data to decide whether it’s better to host on Lambda or on Fargate. LWA contributes by adding portability to your deployments.</p><p>In this article, we’ll explore how to use LWA with CDK to simplify the deployment of your Web apps in Lambda and how to easily transition to ECS Fargate.</p><h3>How Lambda Web adapter works ?</h3><p>LWA is a Lambda extension. It creates an independent process within the Lambda execution environment that listens for incoming events and forwards them to your HTTP server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nxwb4iEtSkvmTvFGzpGjYQ.png" /></figure><p>LWA can integrate with invocations from Lambda function URLs (FURL), ALB, and API Gateway, converting invocation JSON payloads into HTTP requests that web frameworks like fastify or ASP.NET can handle. LWA also supports non-HTTP triggers (e.g. SQS, S3 notifications), but in these cases, it acts as a pass-through without converting the invocation payload.</p><p>LWA supports functions packaged as zip as well as Docker or OCI images.</p><h3>Solution overview</h3><p>Let’s have a look on how we’ll create our flexible deployment using CDK. In this example we’ll be focusing on deploying a public Web application using <a href="https://fastify.dev/">fastify</a> as a Web framework:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZgr0wk-x9Mn0QpP6SFJ4Q.png" /></figure><p>My objective is to create a CDK construct supporting two deployment strategies: <strong>Lambda</strong> or <strong>ECSFargate</strong>. Depending on the selected strategy, only the required components will be deployed:</p><ul><li>When in Lambda mode, we’ll configure the function to use LWA extension. We’ll also configure a REST API Gateway with lambda proxy integration.</li><li>When the deployment mode is ECSFargate, We will deploy our application in an ECS Fargate service that is exposed via an ALB.</li></ul><p>In both of these deployment strategies, users access the Web app through CloudFront. We will associate a WAF Web ACL to restrict access to both the API Gateway and the Application Load Balancer. These origins will only respond to requests that include a custom verification header added by the CloudFront distribution. This approach prevents bypassing the CloudFront distribution to access the origin directly.</p><p>☝️<strong>Some notes:</strong></p><p>When deploying in Lambda, I ruled-out the use of FURL or HTTP API Gateway:</p><ul><li>With FURL, while you can setup Origin Access Control (OAC) with CloudFront, at the time of writing, <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-lambda.html#create-oac-overview-lambda">PUT and </a><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-lambda.html#create-oac-overview-lambda">POST operations require the client to sign the request payload</a>.</li><li>HTTP API Gateway does not support WAF. An alternative solution would involve creating a Lambda@Edge to verify the presence of the custom header in the request.</li></ul><p>To improve security, the custom verification header can be stored in secrets manager with <a href="https://github.com/aws-samples/amazon-cloudfront-waf-secretsmanager">rotation</a> enabled so that the header can be updated as well as the origin WAF and the CloudFront distribution configurations.</p><h3>Let’s see the code</h3><p>Here are the relevant parts of the solution:</p><h4>1- Deploying fastify Web app on Lambda using LWA</h4><p>Creating a new fastify project is a breeze, I generally go with <a href="https://fastify.dev/docs/latest/Reference/TypeScript/">typescript</a>; for the purpose of this article, I will create one super basic api:</p><pre>import fastify from &quot;fastify&quot;;<br><br>const server = fastify();<br><br>server.get(&quot;/health&quot;, async () =&gt; {<br>  return &quot;yup ! I am healthy&quot;;<br>});<br><br>server.get(&quot;/where-are-you-deployed&quot;, async () =&gt; {<br>  return {<br>    &quot;i-am-deployed-on&quot;: process.env.DEPLOYED_ON,<br>  };<br>});<br><br>server.listen({ host: &quot;0.0.0.0&quot;, port: 8080 }, (err, address) =&gt; {<br>  if (err) {<br>    console.error(err);<br>    process.exit(1);<br>  }<br>  console.log(`Server listening at ${address}`);<br>});</pre><p>I will deploy the Lambda function using zip archive and in order to use LWA as a Lambda extension, we’ll need to:</p><ul><li>Attach the LWA layer to the function</li><li>Set the handler to the startup command <a href="https://github.com/ziedbentahar/flexible-deployment-with-lambda-web-adapter/blob/main/src/run.sh">run.sh</a> script. This script starts the fastify Web app. It will be added to the zip package after the code bundling.</li><li>And lastly, define the AWS_LAMBDA_EXEC_WRAPPER environment variable to /opt/bootstrap</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6304c54746026ac6e4aa063894915c53/href">https://medium.com/media/6304c54746026ac6e4aa063894915c53/href</a></iframe><p>The <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_apigateway.RestApi.html">RestApi</a> CDK construct simplifies exposition of the Lambda function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e95ea6620e2a5d32ade5a58369f4234/href">https://medium.com/media/5e95ea6620e2a5d32ade5a58369f4234/href</a></iframe><p>After deployment, you will be able to view in the console the layer associated with the function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2-x_V0rcVbsZMb1_MXIkQ.png" /><figcaption>Lambda with LWA layer configured</figcaption></figure><h4>2- Defining the alternative deployment on ECS Fargate</h4><p>CDK offers an <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.html">L3 construct</a> to deploy a load balanced ECS service. What I find interesting about this construct is that it hides all the complexity and verbosity of defining such a deployment, while allowing a level of flexibility. Another neat feature is that it can build and push <a href="https://github.com/ziedbentahar/flexible-deployment-with-lambda-web-adapter/blob/main/src/Dockerfile">our container</a> image.</p><p>We’ll make sure to enable HTTPS, for that we’ll create a certificate and associated to the load balancer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4371f84b96cea2d73dccbefd06679ea6/href">https://medium.com/media/4371f84b96cea2d73dccbefd06679ea6/href</a></iframe><p>Here, I am using the default configuration, but you will want to adapt it to your own requirements.</p><p>Once deployed, the ECS service looks like this on the AWS console</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HxmV9l15tk2gNoajI1iJIg.png" /></figure><p>You can find the full definition of the construct <a href="https://github.com/ziedbentahar/flexible-deployment-with-lambda-web-adapter/blob/main/infra/lib/WebAppOnECSFargate.ts">here</a>.</p><h4>3- Handling the two different deployment strategies</h4><p>The important bit, the CDK construct that enables flexible deployments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0255bf0a5f0102bb1f7aacac24a48ac2/href">https://medium.com/media/0255bf0a5f0102bb1f7aacac24a48ac2/href</a></iframe><p>This construct handles two deployment strategies Lambda or ECSFargate. For each strategy, we’ll need to provide a factory function that creates the required resources. These two functions need to be injected from a parent construct and they are lazily evaluated given the selected strategy.</p><p>We’ll also make sure that the distribution cache policy is disabled for both of these two origins.</p><p>As an example, the origin of the distribution, should end up looking like this when you select Lambda deployment strategy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kSi63nG4AUSV0HmjDg345Q.png" /><figcaption>Configured origin on “Lambda + RestAPI” mode</figcaption></figure><p>And finally, let’s see how to define the WAF WebACL with a rule that checks the x-origin-header verification header:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fa1518f323766e5635ff3bb65eb9841/href">https://medium.com/media/0fa1518f323766e5635ff3bb65eb9841/href</a></iframe><p>You can follow <a href="https://github.com/ziedbentahar/flexible-deployment-with-lambda-web-adapter/blob/main/infra/lib/WebAppDeployment.ts">this link</a> for the complete WebAppDeployment construct definition</p><h3>Flexible deployment in action</h3><p>Before wapping up, let’s call where-are-you-deployed endpoint defined in the sample web app for each strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fe8iQkPEecXnjLdudm_V2A.png" /><figcaption>calling the “where-are-you-deployed” endpoint from Postman</figcaption></figure><h3>Wrapping up</h3><p>Lambda web adapter is certainly not the only tool that helps running full-fledged web apps on Lambda functions, but it simplifies their deployment while supporting architectural evolution.</p><p>In this article we have seen how to build a CDK construct that offers a way to deploy the same web app on two distinct platforms, we can choose Lambda function or ECS Fargate by specifying a configuration during design time. We can extend this further by enabling the system to automatically redeploy itself, during runtime, on a specific target based on some events or CloudWatch alarms !</p><p>As always, you can find the full code source, ready to be adapted and deployed here:</p><p><a href="https://github.com/ziedbentahar/flexible-deployments-with-lambda-web-adapter">GitHub - ziedbentahar/flexible-deployments-with-lambda-web-adapter</a></p><p>Thanks for reading and hope you enjoyed it !</p><h3>Resources</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FArsTZ2y7u80&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DArsTZ2y7u80&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FArsTZ2y7u80%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce0eb7898dc29cda51af2bf134cd3d34/href">https://medium.com/media/ce0eb7898dc29cda51af2bf134cd3d34/href</a></iframe><ul><li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/restrict-access-to-load-balancer.html">Restrict access to Application Load Balancers</a></li><li><a href="https://github.com/awslabs/aws-lambda-web-adapter">GitHub - awslabs/aws-lambda-web-adapter: Run web applications on AWS Lambda</a></li><li><a href="https://github.com/aws-samples/amazon-cloudfront-waf-secretsmanager">GitHub - aws-samples/amazon-cloudfront-waf-secretsmanager: Enhance Amazon CloudFront Origin Security with AWS WAF and AWS Secrets Manager</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67aecb0ecbff" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/adding-flexibility-to-your-deployments-with-lambda-web-adapter-67aecb0ecbff">Adding flexibility to your deployments with Lambda Web Adapter</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI powered video summarizer with Amazon Bedrock and Anthropic’s Claude]]></title>
            <link>https://levelup.gitconnected.com/ai-powered-video-summarizer-with-amazon-bedrock-and-anthropics-claude-9f1832f397dc?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f1832f397dc</guid>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 10:53:21 GMT</pubDate>
            <atom:updated>2024-06-27T07:15:22.924Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u1aoh6IkniSqBqg8" /><figcaption>Photo by <a href="https://unsplash.com/@benham3160?utm_source=medium&amp;utm_medium=referral">Andy Benham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At times, I find myself wanting to quickly get a summary of a video or capture the key points of a tech talk. Thanks to the capabilities of generative AI, achieving this is entirely possible with minimal effort.</p><p>In this article, I’ll walk you through the process of creating a service that summarizes YouTube videos based on their transcripts and generates audio from these summaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2c9LbDWn59VZaz-GfscIA.png" /><figcaption>AI powered youtube video summarizer</figcaption></figure><p>We’ll leverage Anthropic’s Claude 2.1 foundation model through Amazon Bedrock for summary generation, and Amazon Polly to synthesize speech from these summaries.</p><h3>Solution overview</h3><p>I will use a step functions to orchestrate the different steps involved in the summary and audio generation :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vW39v-yN0WNZJoHiOVfHwA.png" /><figcaption>AI powered youtube video summarizer architecture</figcaption></figure><p>🔍 Let’s break this down:</p><ul><li>The ‘<strong>Get Video Transcript</strong>’ function retrieves the transcript from a specified YouTube video URL. Upon successful retrieval, the transcript is stored in an S3 bucket, ready for processing in the next step.</li><li><strong>‘Generate Model Parameters’</strong> function retrieves the transcript from the bucket and generates the prompt and inference parameters specific to Anthropic’s Claude v2 model. These parameters are then stored in the bucket for use by the Bedrock API in the subsequent step.</li><li>Invoking the Bedrock API is achieved through the step functions’ AWS SDK integration, enabling the execution of the model inferences with inputs stored in the bucket. This step generates a structured JSON containing the summary.</li><li><strong>‘Generate audio form summary’ </strong>relies on Amazon Polly to perform speech synthesis from the summary produced in the previous step. This step returns the final output containing the video summary in text format, as well as a presigned URL for the generated audio file.</li><li>The bucket serves as a state storage used across all the steps of the state machine. In fact, we don’t know the size of generated video transcript upfront; it might reach the Step Functions’ payload size limit of 256 KB in some lengthy videos.</li></ul><h4><strong>On using Anthoropic’s Claude 2.1</strong></h4><p>At the time of writing, Claude 2.1 model supports 200K tokens, an estimated word count of 150K. It provides also a good accuracy over long documents, making it well-suited for summarizing lengthy video transcripts.</p><h3>TL;DR</h3><p>You will find the complete source code here 👇</p><p><a href="https://github.com/ziedbentahar/yt-video-summarizer-with-bedrock">GitHub - ziedbentahar/yt-video-summarizer-with-bedrock</a></p><p>I will use NodeJs, typescript and CDK for IaC.</p><h3>Solution details</h3><h4><strong>1- Enabling Anthropic’s Claude v2 in your account</strong></h4><p>Amazon Bedrock offers a range of foundational models, including Amazon Titan, Anthropic’s Claude, Meta Llama2, etc., which are accessible through Bedrock APIs. By default, these foundational models are not enabled; they must be enabled through the console before use.</p><p>We’ll request access to Anthropic’s Claude models. But first we’ll need to submit a use case details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WwRSrEGHnqbZ2CCOh-kLeA.png" /><figcaption>Request Anthropic’s Claude access</figcaption></figure><h4>2- Getting transcripts from Youtube Videos</h4><p>I will rely on <a href="https://github.com/Kakulukian/youtube-transcript">this lib</a> for the video transcript extraction (It feels like a cheat code 😉) ; in fact, this library makes use of an unofficial YouTube API without relying on a headless Chrome solution. For now, it yields good results on several YouTube videos, but I might explore a more robust solutions in the future :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62e9f8a429c7b8c4b80c663e6d8ade87/href">https://medium.com/media/62e9f8a429c7b8c4b80c663e6d8ade87/href</a></iframe><p>The extracted transcript is then stored on the s3 bucket using ${requestId}/transcript as a key.</p><p>You can find the code for this lambda function <a href="https://github.com/ziedbentahar/yt-video-summarizer-with-bedrock/blob/main/src/lambda-handlers/get-youtube-video-transcript.ts">here</a></p><h4>3- Finding the adequate prompt and generating model inference parameters</h4><p>At the time of writing, Bedrock currently only supports Claude’s Text Completions API. Prompts must be wrapped in \n\nHuman: and \n\nAssistant: markers to let Claude understand the conversation context.</p><p>Here is the prompt; I find that it produces good results for our use case:</p><pre>You are a video transcript summarizer. <br>Summarize this transcript in a third person point of view in 10 sentences. <br>Identify the speakers and the main topics of the transcript and add them in the output as well. <br>Do not add or invent speaker names if you not able to identify them.<br>Please output the summary JSON format conforming to this JSON schema:<br>{<br>  &quot;type&quot;: &quot;object&quot;,<br>  &quot;properties&quot;: {<br>    &quot;speakers&quot;: {<br>      &quot;type&quot;: &quot;array&quot;,<br>      &quot;items&quot;: {<br>        &quot;type&quot;: &quot;string&quot;<br>      }<br>    },<br>    &quot;topics&quot;: {<br>      &quot;type&quot;: &quot;string&quot;<br>    },<br>    &quot;summary&quot;: {<br>      &quot;type&quot;: &quot;array&quot;,<br>      &quot;items&quot;: {<br>        &quot;type&quot;: &quot;string&quot;<br>      }<br>    }<strong><br>  }<br>}<br><br>&lt;transcript&gt;{{transcript}}&lt;/transcript&gt;<br></strong></pre><p>🤖<strong> Helping Claude producing good results</strong>:</p><ul><li>To clearly mark to the transcript to summarize, we use <strong>&lt;transcript/&gt;</strong> XML tags. <a href="https://docs.anthropic.com/claude/docs/constructing-a-prompt#mark-different-parts-of-the-prompt">Claude will specifically focus</a> on the structure encapsulated by these XML tags. I will be substituting <strong>{{transcript}} </strong>string<strong> </strong>with the actual video transcript.</li><li>To assist Claude in generating a reliable JSON output format, I include in the prompt the JSON schema that needs to be adhered to.</li><li>Finally, I also need to inform Claude that I want to generate only a concise JSON response without unnecessary chattiness, meaning without including a preamble and postscript while returning the JSON payload:</li></ul><pre>\n\nHuman:{{prompt}}\n\nAssistant:<strong>{</strong></pre><p>Note that the full prompt ends with a trailing <strong>{</strong></p><p>As mentioned on the section above, we will store this generated prompt as well as the model parameters in the bucket so that It can be used as an input of Bedrock API:</p><pre>  const modelParameters = {<br>    prompt,<br>    max_tokens_to_sample: MAX_TOKENS_TO_SAMPLE,<br>    top_k: 250,<br>    top_p: 1,<br>    temperature: 0.2,<br>    stop_sequences: [&quot;Human:&quot;],<br>    anthropic_version: &quot;bedrock-2023-05-31&quot;,<br>  };</pre><p>You can follow <a href="https://github.com/ziedbentahar/yt-video-summarizer-with-bedrock/blob/main/src/lambda-handlers/generate-model-parameters.ts">this link</a> for the full code of the generate-model-parameters lambda function.</p><h4>4- Invoking Claude Model</h4><p>In this step, we’ll avoid writing custom lambda function to invoke Bedrock API. Instead, we’ll use Step functions direct SDK integration. This state loads from the bucket the model inference parameters that were generated in the previous step:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f761ba58fc621b81ebca57281555f105/href">https://medium.com/media/f761ba58fc621b81ebca57281555f105/href</a></iframe><p><strong>☝️ Note: </strong>As we instructed Claude to generate the response in JSON format, the completion API response misses a leading <strong>{</strong> as Claude outputs the rest of the requested JSON schema.</p><p>We use <a href="https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html">intrinsic functions</a> on the state’s ResultSelector to add the missing opening curly brace and to format the state output in a well formed JSON payload :</p><pre>ResultSelector: {<br>  &quot;id.$&quot;: &quot;$$.Execution.Name&quot;,<br>  &quot;summaryTaskResult.$&quot;:<br>    &quot;States.StringToJson(States.Format(&#39;\\{{}&#39;, $.Body.completion))&quot;,<br>}</pre><p>I have to admit, it is not ideal but this helps get by without writing a custom Lambda function.</p><h4>5- Generating audio from video summary</h4><p>This step is heavily inspired by this <a href="https://levelup.gitconnected.com/building-a-serverless-text-to-speech-application-with-amazon-polly-step-functions-and-websocket-56e9871730b7">previous blog post</a>. Amazon Polly generates the audio from the video summary:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83b7b0298ec33fa03d44fef5afe5446c/href">https://medium.com/media/83b7b0298ec33fa03d44fef5afe5446c/href</a></iframe><p>Here are the details of synthesize function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04ae7aa610bbf80d671cb305100876b4/href">https://medium.com/media/04ae7aa610bbf80d671cb305100876b4/href</a></iframe><p>Once the audio generated, we store it on the S3 bucket and we generate a presigned Url so it can be downloaded afterwards.</p><p>☝️ <strong>On</strong> <strong>language detection : </strong>In this example, I am not performing language detection; by default, I am assuming that the video is in English. You can find in <a href="https://medium.com/gitconnected/building-a-serverless-text-to-speech-application-with-amazon-polly-step-functions-and-websocket-56e9871730b7">my previous article</a> how to perform such a process in speech synthesis. Alternatively, We can also leverage Claude model capabilities to detect the language of the transcript.</p><h4>6- Defining the state machine</h4><p>Alright, let’s put it all together and let’s take a look at the CDK definition of the state machine:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7e3066a86e061951acc8855f8839202/href">https://medium.com/media/d7e3066a86e061951acc8855f8839202/href</a></iframe><p>In order to be able to invoke Bedrock API, we’ll need to add this policy to the workflow’s role (And it’s important to remember granting the S3 bucket read &amp; write permissions to the state machine):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fab4aae04c6f4c45ebac67759b9aeac/href">https://medium.com/media/9fab4aae04c6f4c45ebac67759b9aeac/href</a></iframe><h3>Wrapping up</h3><p>I find creating generative AI based applications to be a fun exercise, I am always impressed by how quickly we can develop such applications by combining Serverless and Gen AI.</p><p>Certainly, there is room for improvement to make this solution production-grade. This workflow can be integrated into a larger process, allowing the video summary to be sent asynchronously to a client, and let’s not forget robust error handling.</p><p>Follow <a href="https://github.com/ziedbentahar/yt-video-summarizer-with-bedrock/tree/main">this link</a> to get the source code for this article.</p><p>Thanks for reading and hope you enjoyed it !</p><h3>Further readings</h3><ul><li><a href="https://docs.anthropic.com/claude/docs/put-words-in-claudes-mouth">Put words in Claude&#39;s mouth</a></li><li><a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html">Anthropic Claude models</a></li><li><a href="https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html">What is Amazon Bedrock?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f1832f397dc" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/ai-powered-video-summarizer-with-amazon-bedrock-and-anthropics-claude-9f1832f397dc">AI powered video summarizer with Amazon Bedrock and Anthropic’s Claude</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database schema migrations on AWS with Custom Resources and CDK]]></title>
            <link>https://levelup.gitconnected.com/database-schema-migrations-on-aws-with-custom-resources-and-cdk-c8a058e5a33b?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8a058e5a33b</guid>
            <category><![CDATA[lambda-function]]></category>
            <category><![CDATA[aurora-serverless-v2]]></category>
            <category><![CDATA[database-schema-migration]]></category>
            <category><![CDATA[aws-custom-resources]]></category>
            <category><![CDATA[cdk]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Wed, 14 Jun 2023 18:34:15 GMT</pubDate>
            <atom:updated>2023-06-25T12:56:31.601Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Yin6EjOtLJ3LYAa" /><figcaption>Photo by <a href="https://unsplash.com/ja/@tomkirvela?utm_source=medium&amp;utm_medium=referral">Tomas Kirvėla</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With AWS <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html">Custom Resources</a> you can manage resources that are not natively supported by CloudFormation. You can execute application-specific provisioning logic as well as custom code during the deployment, the update or the deletion of a Stack.</p><p>In this article, we will focus on using Custom Resources to handle schema migrations on an Aurora Postgres database. We will create a custom database migration resource that executes schema changes during the Stack deployment. To accomplish this, we will associate a Lambda function with our Custom Resource, this function ensures that any new changes to the database are automatically applied when necessary.</p><h3>Solution overview</h3><p>We’ll use Aurora Serverless V2 with Postgres engine, NodeJs and typescript for the Lambda function code and CDK for the IaC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RaSEXAwy2Cvk0uWEbL3ouA.png" /><figcaption>DB schema migrations using custom resources -Solution overview</figcaption></figure><p>Here are the details of this solution:</p><ul><li>We’ll create the <a href="https://github.com/ziedbentahar/db-schema-migration-with-custom-resources/blob/6d783560e1fedeadbfa7fb039d45c1b38e4485eb/lib/database.ts#L60">database Cluster</a> on an isolated subnet as well as a secret that stores the credentials of the database. This secret is accessed by the DB schema migration Lambda function .</li><li>The <strong>DB schema migration Lambda</strong> <strong>function</strong> is responsible for executing the necessary database schema changes. It uses the “<a href="https://salsita.github.io/node-pg-migrate/#/">node-pg-migrate</a>” tool, which allows us to run migration scripts programmatically. We’ll need to configure this lambda function to access the Aurora resource in our VPC.</li><li><strong>Migration scripts</strong> files are included in the zip package of the lambda function : each file contains a set of changes to apply to the database, these files are stored in the same repository as the application. Each new set of changes need to be written into a new distinct file.</li><li><strong>DB migration custom resource</strong> invokes the lambda function during the stack deployment when it detects changes on the migration scripts.</li></ul><h4><strong>TL;DR</strong></h4><p>You can find the complete repo of this solution here:</p><p><a href="https://github.com/ziedbentahar/db-schema-migration-with-custom-resources">GitHub - ziedbentahar/db-schema-migration-with-custom-resources</a></p><h3>Let’s see the code</h3><h4><strong>1 —DB schema migration lambda</strong></h4><p>As mentioned above, we will use “<a href="https://salsita.github.io/node-pg-migrate/#/">node-pg-migrate</a>” to run schema changes.</p><p>One interesting aspect of this library is its flexibility in defining migration scripts: We have the option to write our migration scripts in either ES or TypeScript, allowing us to define database schemas with code. Alternatively, we can also define migration scripts as plain SQL files, providing a more traditional approach to managing database schema changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*0Xfd8Ez_KGdNB846gUxyrQ.png" /><figcaption>Example of a migration script directory</figcaption></figure><p>This code below shows how to use node-pg-migrate to run the migration scripts from the custom resource lambda function:👇</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6516292fee3a9cd4a218d52e1e9baff5/href">https://medium.com/media/6516292fee3a9cd4a218d52e1e9baff5/href</a></iframe><p>Db Schema migration is run by invoking runner function.</p><p>☝️ <strong>Note</strong>: This lib can handle forward and backward migrations but with our solution we will only be supporting forward migrations.</p><p>Here is CDK definition of this lambda function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fd0a01c604c6d77fcc825fc0e940594/href">https://medium.com/media/8fd0a01c604c6d77fcc825fc0e940594/href</a></iframe><p>This Lambda Function requires <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html">VPC access</a> as it needs to access the Aurora Database. We will also make sure to place the network interface associated with the Lambda function in a PRIVATE_WITH_EGRESS subnet as we need to access the secrets manager service.</p><p>Additionally, we’ll associate the security group of the Lambda function to the security group of the Aurora Cluster:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0dc99f28f9e84d3ef1a26085d1849d9a/href">https://medium.com/media/0dc99f28f9e84d3ef1a26085d1849d9a/href</a></iframe><p>📦 <strong>On embedding migration scripts: </strong>In our example, Migration scripts need to be included in the Lambda function package. We use the <a href="https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-lambda-nodejs.ICommandHooks.html">afterBunding</a> hook to copy the content of the migration dir to the bundle output dir:</p><pre>afterBundling: (_, outputDir: string) =&gt; {<br>  return [<br>    `mkdir -p ${outputDir}/migrations &amp;&amp; cp ${migrationDirectoryPath}/* ${outputDir}/migrations`,<br>  ];<br>},</pre><p>You can find <a href="https://github.com/ziedbentahar/db-schema-migration-with-custom-resources/blob/6d783560e1fedeadbfa7fb039d45c1b38e4485eb/lib/database.ts#L101">here</a> the complete definition of this Lambda Function.</p><h4><strong>2 —Defining the custom resource:</strong></h4><p>Straightforward to define with CDK:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/384f9532cac4125bf7795bcec5ce885d/href">https://medium.com/media/384f9532cac4125bf7795bcec5ce885d/href</a></iframe><p><strong>☝️ One important note</strong>: the computeHash function computes a hash of the migration script directory content. This hash is passed as a property of the custom resource. During the stack deployment, whenever this hash changes, the lambda function gets invoked and the new migrations scripts are taken into account.</p><p>You will find <a href="https://github.com/ziedbentahar/db-schema-migration-with-custom-resources/blob/6d783560e1fedeadbfa7fb039d45c1b38e4485eb/lib/database.ts#L185">here</a> the definition of the Custom::DbSchemaMigration custom resource.</p><h4>3-Putting all together</h4><p>And here is how we use this Database construct that supports schema migration:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33ca4faa115a2313a04afc6665e2ca92/href">https://medium.com/media/33ca4faa115a2313a04afc6665e2ca92/href</a></iframe><p>Et voilà! Once you make a deployment with new migration script files, you can see the DB schema migration in action via the Lambda CloudWatch logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPgPcsyHRol84DZYmu_PsA.png" /><figcaption>Db schema migration logs — Lambda logs</figcaption></figure><h3>Wrapping up</h3><p>In this article, we have seen how to use custom resources to run database schema migrations on AWS. CDK makes it a breeze ! You can find a complete sample application repository with the complete github action workflow here:</p><p><a href="https://github.com/ziedbentahar/db-schema-migration-with-custom-resources">GitHub - ziedbentahar/db-schema-migration-with-custom-resources</a></p><p>Hope you find it useful, and thanks for reading !</p><h3>Further readings</h3><ul><li><a href="https://docs.aws.amazon.com/cdk/api/v1/docs/custom-resources-readme.html">@aws-cdk/custom-resources module · AWS CDK</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html">Connecting outbound networking to resources in a VPC</a></li><li><a href="https://salsita.github.io/node-pg-migrate/#/">node-pg-migrate - Postgresql database migration management tool for node.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8a058e5a33b" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/database-schema-migrations-on-aws-with-custom-resources-and-cdk-c8a058e5a33b">Database schema migrations on AWS with Custom Resources and CDK</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Asynchronous REST APIs on AWS]]></title>
            <link>https://levelup.gitconnected.com/serverless-asynchronous-rest-apis-on-aws-c48156d790c1?source=rss-b3a9f7e87b27------2</link>
            <guid isPermaLink="false">https://medium.com/p/c48156d790c1</guid>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[aws-api-gateway]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[eventbridge-pipes]]></category>
            <dc:creator><![CDATA[Zied Ben Tahar]]></dc:creator>
            <pubDate>Thu, 20 Apr 2023 09:29:01 GMT</pubDate>
            <atom:updated>2024-09-27T13:19:33.973Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*38eNt_XxpLuhNQHo" /><figcaption>Photo by <a href="https://unsplash.com/@viazavier?utm_source=medium&amp;utm_medium=referral">Laura Ockel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://medium.com/gitconnected/building-a-serverless-text-to-speech-application-with-amazon-polly-step-functions-and-websocket-56e9871730b7">In a previous article</a>, we explored how to use the API Gateway WebSocket API to provide real-time, asynchronous feedback and responses to its clients. While WebSocket APIs can be an excellent option to build real-time asynchronous APIs between a browser and a server, there are situations where asynchronous REST APIs are a better fit. For example, if you need to ensure compatibility with existing architectures or integrating with backend systems where setting up a messaging system like a queue or an event bus is not an option.</p><p>This article provides a guide on how to build a Serverless Async Request/Response REST API on AWS.</p><h3><strong>First, Let’s design the API</strong></h3><p>Let’s consider an application that lets clients perform long running tasks through HTTP requests, these tasks might require a long amount of time to complete (<em>e.g.</em> text-to-speech or audio-to-text tasks, report generation, etc.). In order to keep clients informed about the status of their requests, we’ll need to create a separate status endpoint that allows clients to poll at regular intervals the current processing status until the task is finished.</p><p>As an example, to initiate the task creation, clients POST this request:</p><pre>POST /task<br>{<br>    &quot;someProperty&quot;: &quot;someValue&quot;,<br>    &quot;anotherPropety&quot;: &quot;anotherValue&quot;,<br>    ...<br>}</pre><p>The API should then return a HTTP 202 Accepted response with a body containing a URI that references the task :</p><pre>HTTP 202 (Accepted)<br><br>{<br> &quot;task&quot;: {<br>  &quot;href&quot;: &quot;/task/&lt;taskId&gt;&quot;,<br>  &quot;id&quot;: &quot;&lt;taskId&gt;&quot;<br> }<br>}</pre><p>Clients can then poll that endpoint to retrieve the status of the task. The API should respond with a status corresponding to the current state of the task, such as inprogress, complete , error. Additionally, the response payload should reflect to the specific status of the task.</p><p>Here is an example of a request with a response containing the payload of a task that is in progress:</p><pre>GET /task/&lt;taskId&gt;<br><br>HTTP 200 (OK)<br><br>{<br>    &quot;status&quot;: &quot;inProgress&quot;,<br>    &quot;taskCreationDate&quot;: &quot;2023-04-04T19:23:42&quot;,<br>    &quot;requestData&quot;: {...},<br>    ...<br>}</pre><p>For server-to-server scenarios, we’ll add a callback/webhook mechanism: When the task completes, the server updates the task state and sends POST request to a callback URL that the client provides on the initial task creation request.</p><p>One solution is to include the callback URL in the header of the request so that the API use it to send the response payload:</p><pre>POST /task<br>Callback: https://foo.example.com/some-callback-resource<br><br>{<br>    &quot;someProperty&quot;: &quot;someValue&quot;,<br>    &quot;anotherPropety&quot;: &quot;anotherValue&quot;,<br>    ...<br>}</pre><p>⚠️ <strong>Important:</strong> Security and reliability are important aspects to be taken into account when using callback URLs:</p><ul><li><strong>Security</strong>: It is important to verify that the callback URLs belong to the client who initiated the requests. We also need to ensure that the callback destination server receives legitimate data from the source server, rather than forged information from malicious actors attempting to spoof the webhook. This can be done by requiring clients to use API Keys when making requests and on the server side associating these keys with well-known callback URLs that the clients must provide. In addition, <a href="https://prismatic.io/blog/how-secure-webhook-endpoints-hmac/">HMAC signature verification</a> can be used to authenticate and validate the payloads of API callbacks.</li><li><strong>Failure handling</strong>: It’s important to account for scenarios where the client may be unavailable or experiencing intermittent faults resulting in errors. To address this, we’ll need to implement a retry mechanism that includes a dead-letter queue. This mechanism allows failed deliveries to be stored in the queue and resent at a later time.</li></ul><p>Alright, let’s see how to implement this on AWS.</p><h3>Architecture Overview</h3><p>In this architecture, my goal was to use AWS integrations wherever possible to minimize the amount of glue code that needs to be written and maintained. This approach not only saves time and effort, but also helps ensure the scalability of the API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Au0YOGop8Q53flHJyhZVLQ.png" /><figcaption>Architecture overview</figcaption></figure><p><strong>🔍 Breaking this down:</strong></p><ul><li>We create direct AWS integration between the API Gateway RestAPI and the DynamoDb table. In this task table we store the context of the request: the client id, the request payload, the task status and the task result once available. This table gets updated by the task processing workflow.</li><li>Since we’re using API Gateway as a proxy for the DynamoDB table, we rely on two Lambda authorizers to authenticate API calls. The first authorizer verifies client request headers: The client’s API key and the callback URL, to authorize POST /task requests. The second authorizer is dedicated to GET /task/&lt;taskId&gt; route.</li><li>To trigger the task workflow, we rely on the table’s DynamoDb Streams. We connect the stream to the task’s state machine by using EventBridge Pipes. This Pipe selects only the newly created tasks.</li><li>The task is then run and coordinated by a Step function workflow. The status of the task is also updated in this workflow by direct AWS SDK integration.</li><li>Once the task is complete, its state gets updated and the result payload gets written into the task table. We use another EventBridge Pipe to trigger the sending of the callback to the client from the stream of the completed tasks.</li></ul><p><strong>☝️Note : </strong>In this architecture<strong>, </strong>Instead of using the <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations.html">Event Bridge API destinations</a>, we’ll write a custom lambda to send the callbacks. I did this to achieve better flexibility and control over how the task result payload is sent to the clients. For example, a client may require multiple registered callback URLs. To accomplish this, we use an EventBus as the target destination for the output Pipe.</p><h4>TL;DR</h4><p>You will find the complete source code with the complete deployment pipeline here 👇</p><p><a href="https://github.com/ziedbentahar/async-rest-api-on-aws">GitHub - ziedbentahar/async-rest-api-on-aws</a></p><p>In this example I will use nodejs, typescript and CDK for IaC.</p><h3><strong>Let’s see the code</strong></h3><h4><strong>1- API Gateway, creating the long running task</strong></h4><p>Let’s zoom into the POST /task integration with the DynamoDb task table. We associate this route with an authorizer that requires two headers to be present in the request: Authorization and Callback.</p><p>Once the request is authorized, this integration maps the request payload to a task entry. We use the API gateway $context.requestId as the identifier of the task. Additionally, we map $context.authorizer.principalId (in our case the client Id) in order to be used down the line. And we extract the request callback URL from the header using $util.escapeJavascript($input.params(‘callback’) :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b25898142e77cf29153a6dc506c84377/href">https://medium.com/media/b25898142e77cf29153a6dc506c84377/href</a></iframe><p>When the task is successfully added to the table, we return HTTP 202 Accepted response with a body containing the task id as well as a reference pointing to the task: by using $context.path/$context.requestId which translates to this URI /&lt;stage-name&gt;/task/&lt;task-id&gt;</p><p>You can find the complete CDK definition of the API Gateway here.</p><h4><strong>2- ‘Create Task’ authorizer Lambda</strong></h4><p>As mentioned above, this authorizer validates the API key and checks whether the callback URL is associated with the API Key that identifies the client:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/758bc87e01dfd41b5f90e75b34487cca/href">https://medium.com/media/758bc87e01dfd41b5f90e75b34487cca/href</a></iframe><p>Here, I do not dive into the implementation of validateTokenWithCallbackUrl , but this is something that can be delegated to another service that owns the responsibility to do the proper checks.</p><h4><strong>2- Defining the ‘create task’ EventBridge Pipe</strong></h4><p>This EventBridge Pipe selects the INSERT events from the DynamoDb stream: The newly created tasks will trigger new state machine executions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20ff25a82bcb840a99930ac1bfa601fa/href">https://medium.com/media/20ff25a82bcb840a99930ac1bfa601fa/href</a></iframe><p>We’ll to set the invocation type to FIRE_AND_FORGET as the execution of this state machine is asynchronous.</p><p>We’ll also need to set these target and destination policies:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2cc58aabe7aee09cb985cb8b8f7ba16/href">https://medium.com/media/f2cc58aabe7aee09cb985cb8b8f7ba16/href</a></iframe><p>When deployed, this Pipe will look like this on the AWS console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2hUdKADQzqS7FAdeA9eQA.png" /><figcaption>DynamoDb stream to step function Pipe</figcaption></figure><h4><strong>3- Defining the ‘handle completed task’ EventBridge Pipe</strong></h4><p>We apply the same principle as in the previous section. However, this Pipe selects the completed tasks from the DynamoDb stream, transforms them, and subsequently forwards them to the EventBus:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6305b3ec6e0b307aa25b628fc3d789f5/href">https://medium.com/media/6305b3ec6e0b307aa25b628fc3d789f5/href</a></iframe><p>We’ll need to attach this role to this Pipe:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53727353b6f69e6c6990f40f9ccaf753/href">https://medium.com/media/53727353b6f69e6c6990f40f9ccaf753/href</a></iframe><p>After deployment, the Pipe resources looks like this in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFqxBtMlQXioLLXwLCx7pQ.png" /><figcaption>DynamoDb stream to EventBus Pipe</figcaption></figure><h4><strong>4- Sending the Callbacks to the clients</strong></h4><p>To send callbacks, we associate an EventBridge rule with the EventBus. This rule matches all the events and defines the ‘<a href="https://github.com/ziedbentahar/async-rest-api-on-aws/blob/main/src/backend/lambdas/callback-lambda.ts">send callback lambda</a>’ as the target:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/797242ba5d8963534926e8c4c78f1fdd/href">https://medium.com/media/797242ba5d8963534926e8c4c78f1fdd/href</a></iframe><p>In some situations, callbacks may fail to be delivered due to temporary unavailability of the client or a network error. We configure EventBridge to retry the operation up to 10 times within a 24-hour time-frame, after which, the message is directed to a dead letter queue to prevent message loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FwDc2FeOUXvjLZeHt0qyJw.png" /><figcaption>EventBridge rule configuration</figcaption></figure><p><strong>Callback lambda</strong></p><p>The callback lambda is a simple function that computes the HMAC of the message to be sent and then sends a POST request to the target callback URL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/639837eeb09fcb74386c7b4b5e22aa46/href">https://medium.com/media/639837eeb09fcb74386c7b4b5e22aa46/href</a></iframe><p>In the computeHMAC function, the clientId is used to retrieve the key necessary for generating the HMAC signature from the response payload.</p><h3>Wrapping up</h3><p>In this article, we’ve seen how to create Serverless, asynchronous REST API on AWS. By leveraging API Gateway AWS integrations and EventBridge Pipes, it is possible to build such an API that requires minimal Lambda glue code. This approach not only simplifies the development process, but it also allows better flexibility and scalability.</p><p>📝 You can find the complete source code and the deployment pipeline here:</p><p><a href="https://github.com/ziedbentahar/async-rest-api-on-aws">GitHub - ziedbentahar/async-rest-api-on-aws</a></p><p>Thanks for reading !</p><h3>Further readings</h3><ul><li><a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html">Event retry policy and using dead-letter queues</a></li><li><a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/async-request-reply">Asynchronous Request-Reply pattern - Azure Architecture Center</a></li><li><a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html">Amazon DynamoDB stream as a source</a></li></ul><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>💰 Free coding interview course ⇒ <a href="https://skilled.dev/?utm_source=luc&amp;utm_medium=article">View Course</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c48156d790c1" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/serverless-asynchronous-rest-apis-on-aws-c48156d790c1">Serverless Asynchronous REST APIs on AWS</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>