<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bart Reyserhove on Medium]]></title>
        <description><![CDATA[Stories by Bart Reyserhove on Medium]]></description>
        <link>https://medium.com/@bartreyserhove?source=rss-918f929b73c5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*P2scLulgMKH0OJQrXX8SIA.jpeg</url>
            <title>Stories by Bart Reyserhove on Medium</title>
            <link>https://medium.com/@bartreyserhove?source=rss-918f929b73c5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:20:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bartreyserhove/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to stream OpenAI responses to the client in a Mendix App]]></title>
            <link>https://medium.com/@bartreyserhove/how-to-stream-openai-responses-to-the-client-in-a-mendix-app-dd35dfe0ff4e?source=rss-918f929b73c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd35dfe0ff4e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[mendix]]></category>
            <dc:creator><![CDATA[Bart Reyserhove]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 14:11:06 GMT</pubDate>
            <atom:updated>2024-03-17T14:39:07.397Z</atom:updated>
            <content:encoded><![CDATA[<p>We are using OpenAI / Azure OpenAI for a couple of use cases in our Mendix app. Overall that works really well, but of course ChatGPT has put the user experience standard quite high by streaming the response back to the user so that you do not have to wait until the complete response comes back. It takes the same amount of time to get the entire result but the user experience is so much better if you can start reading almost immediately.</p><p>My requirements for the solution:</p><ul><li>Stream responses back to the client like ChatGPT does</li><li>No OpenAI API keys on the client: if you just call OpenAI from the client, this would also work, but then you would have to expose your API key in the client.</li><li>Ability to add context to the prompt before sending it over to OpenAI</li></ul><p>I started looking at different options to accomplish this and ended up using an A<a href="https://aws.amazon.com/pm/lambda/?gclid=Cj0KCQjwqdqvBhCPARIsANrmZhMEtDe6waglf3dGS3yjDRcojUnNifDd_nbeNWCUk3xZ3_R065Fhv3caAnLgEALw_wcB&amp;trk=96858a58-f2ac-44e1-8d07-8ec9590efd51&amp;sc_channel=ps&amp;ef_id=Cj0KCQjwqdqvBhCPARIsANrmZhMEtDe6waglf3dGS3yjDRcojUnNifDd_nbeNWCUk3xZ3_R065Fhv3caAnLgEALw_wcB:G:s&amp;s_kwcid=AL!4422!3!651510153333!p!!g!!what%20is%20lambda!19828209774!147081382237">WS Lamba function</a> in combination with a Mendix widget calling the Lambda function. The picture underneath shows what we want to accomplish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uvx_vCz49oZvyS04RoC1g.png" /></figure><p>Let’s dive in and explain how you can do this.</p><p><strong>Lambda function</strong></p><p>From the AWS console, create a new Lambda function and make sure to sett he “Invoke Mode” to “RESPONSE_STREAM”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DsIq_fJ_-L7U0YF6YxGapQ.png" /></figure><p>You also need to add your OpenAI API key to the “Environment Variables” section of the Lambda function and you probably want to increase the timeout in the “General configuration” because by default that is on 30 seconds which can be quite low, but that depends on your use case.</p><p>My Lambda function contains the following code to send a request to OpenAI and stream the response back to the caller.</p><pre>&quot;use strict&quot;;<br><br>import OpenAI from &#39;openai&#39;;<br><br>const openai = new OpenAI();<br><br><br>export const handler = awslambda.streamifyResponse(<br><br><br>  async (event, responseStream, _context) =&gt; {<br><br>     // Parse the incoming JSON payload<br>    console.log(&quot;Body: &quot;, event.body);<br>    let body;<br>    if (event.body) {<br>      body = JSON.parse(event.body);<br>    } else {<br>      console.log(&quot;No event body available&quot;);<br>      // Default values or error handling if necessary<br>      return {<br>        statusCode: 400,<br>        body: JSON.stringify({ message: &quot;Bad request: No data provided&quot; }),<br>    };<br>  }<br><br>    // Extract parameters from the parsed body<br>    const model = body.model;<br>    const prompt = body.prompt;<br>    const temperature = body.temperature;<br>    const max_tokens = body.max_tokens;<br>    const top_p = body.top_p;<br><br>    const stream = await openai.chat.completions.create({<br>      model: model,<br>      messages: [{ role: &#39;user&#39;, content: prompt }],<br>      temperature: temperature,<br>      max_tokens: max_tokens,<br>      top_p: top_p,<br>      stream: true,<br>    });<br>    for await (const chunk of stream) {      <br>      responseStream.write(chunk.choices[0]?.delta?.content || &#39;&#39;);<br>    }<br>    responseStream.end();<br><br>  }<br>);</pre><p>When that is done you need to setup a way for the client to actually call this in an authenticated way. In most use cases you probably want a “logged in user” using a feature that connects to OpenAI.</p><p><strong>Identity Pool</strong></p><p>In my situation I opted to setup an “Identity Pool” on AWS so that I can federate the identities in my app with AWS.</p><ol><li>Go to “Cognito” and create a new identity pool</li><li>An identity pool needs an identity provider. Go to “IAM” to setup a new identity provider. We are using <a href="https://auth0.com/">Auth0</a> to for authentication in our Mendix App. This <a href="https://auth0.com/docs/customize/integrations/aws/amazon-cognito">article</a> guides you through making the link between Auth0 and AWS Cognito. Of course your authentication method in your Mendix app might be different. You would need to research a bit how you can link it to AWS Cognito.</li><li>Check the “User Access” tab of the identity pool and verify which role it authenticates into.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L78ZgwYRJFYjx0Z6t5WkpA.png" /></figure><p>4. Go to the role definition and check which permission policies are linked to the role. The role needs permission to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Authorizing.IAM.LambdaCreatePolicy.html">invoke the lambda function</a>.</p><p>5. In the “Trust relationships” tab check that the StringEquals condition contains the id of the identity pool you created.</p><p><strong>Mendix widget</strong></p><p>Once you have all of that, you can move to creating a Mendix widget that calls the lambda function. I actually first created a really simple React app to test if the React app could call the Lambda function before I moved on creating a Mendix widget.</p><p>The code for that simple React app looks like this:</p><pre>import React, {useState} from &#39;react&#39;;<br>import { InvokeWithResponseStreamCommand, LambdaClient } from &quot;@aws-sdk/client-lambda&quot;<br>import {fromCognitoIdentityPool} from &quot;@aws-sdk/credential-providers&quot;;<br>import &#39;./App.css&#39;;<br><br>function App() {<br>  const [messages, setMessages] = useState([]);<br>  const [isLoading, setIsLoading] = useState(false);<br><br>  const invokeLambdaFunction = async () =&gt; {<br>    try{<br>      const REGION = &#39;your-region&#39;;<br><br>      const lambda = new LambdaClient({<br>        region: REGION,<br>        credentials: fromCognitoIdentityPool({    <br>          clientConfig: { region: REGION }, // Configure the underlying CognitoIdentityClient.      <br>          identityPoolId: &#39;your-identity-pool-id&#39;,<br>          logins: {<br>            your-identity-provider: &#39;the idToken of the logged in user&#39;<br>          },<br>        })<br>      })<br><br>      return await lambda.send(<br>        new InvokeWithResponseStreamCommand({<br>          FunctionName: &#39;lambda_chat&#39;,<br>          Payload: JSON.stringify({<br>            body: JSON.stringify({ <br>              model: &quot;gpt-4-turbo-preview&quot;,<br>              prompt: &quot;Do you know a good joke about developers?&quot;,<br>              temperature: 0.5,<br>              max_tokens: 500,<br>              top_p: 1.0})<br>          })<br>        })<br>      );      <br>    }<br>    catch (error){<br>      console.error(&quot;Error invoking Lambda function&quot;, error);<br>    }<br>  }<br><br>const streamLambdaResponse = async () =&gt; {<br>    setIsLoading(true);<br>    const response = await invokeLambdaFunction();  <br><br>    const decoder = new TextDecoder(&quot;utf-8&quot;);<br>    for await (const event of response.EventStream) { // error here<br>      const text = decoder.decode(event.PayloadChunk?.Payload);<br>      setMessages((prevMessage) =&gt; prevMessage + text);<br>      setIsLoading(false);<br>    }<br>}<br><br>  return (<br>    &lt;div className=&quot;App&quot;&gt;<br>      &lt;header className=&quot;App-header&quot;&gt;<br>        &lt;h2&gt;Conversation:&lt;/h2&gt;<br>        &lt;button onClick={streamLambdaResponse}&gt;Start Stream&lt;/button&gt;   <br>        {isLoading &amp;&amp; &lt;div className=&quot;spinner&quot;&gt;&lt;/div&gt;}     <br>        &lt;div className=&#39;message-container&#39;&gt;              <br>            &lt;p className=&quot;message&quot;&gt;{messages}&lt;/p&gt;<br>        &lt;/div&gt;<br>      &lt;/header&gt;<br>    &lt;/div&gt;<br>  );<br><br><br>}<br><br>export default App;</pre><p>Once I got that working I went on to create the Mendix widget thinking that was going to be fairly easy, but things got complicated because of how the pluggable widget tools build a widget. I am a Mendix developer and I can build a Mendix widget but I am far from a React expert and also not an expert in the different bundling tools of the front end world.</p><p>The following imports were causing all kinds of issues when building the widget. As far as I understand this is because this imports have dependencies on other libraries that are to be used on the server and not in the browser.</p><pre>import { InvokeWithResponseStreamCommand, LambdaClient } from &quot;@aws-sdk/client-lambda&quot;<br>import {fromCognitoIdentityPool} from &quot;@aws-sdk/credential-providers&quot;;</pre><p>Maybe there are other modules I could be using or maybe there is way to have more fine grained control over the imports, but I did not really find those. If you know of a way to use these modules in a Mendix widget, by all means, let me know in the comments.</p><p><strong>The solution for the Mendix widget</strong></p><p>The scope of my Mendix widget is limited to making a call to the Lambda function and returning the answers as a stream. That is mainly because when the user in our app adds a prompt we first want to add context to the prompt before sending that to the Lambda function.</p><p>To make that a bit more clear. The textbox and cancel and send buttons in the screenshot underneath are still part of the Mendix app. When you click on “Send” the widget is activated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ymW_NhS95SpgXPx2nkR6g.png" /></figure><p>To tackle the build issue in the Mendix widget I decided to create a separate package that uses the AWS SDK and then bundle that with webpack. Then I copy the bundle.js to my widget code so that I can call the child component from the parent component (the widget).</p><p>Here is the code of the child component which I on purpose give as little scope as possible. I just need it to resolve the rollup issue I was having in the Mendix widget when I included this in the widget code itself.</p><pre>import React,{ createElement,useEffect, useState } from &quot;react&quot;;<br>import {StreamerProps} from &quot;./StreamerProps&quot;;<br>import { InvokeWithResponseStreamCommand, LambdaClient } from &quot;@aws-sdk/client-lambda&quot;;<br>import { fromCognitoIdentityPool } from &quot;@aws-sdk/credential-providers&quot;;<br><br><br><br><br>export const Streamer = (props: StreamerProps) =&gt; {    <br>    const [initialContext, setInitialContext] = useState&lt;string&gt;(&#39;&#39;);    <br>    const [isLoading, setIsLoading] = useState(false);<br><br>    const preparePromptForOpenAI = (newUserPrompt: string) =&gt; {<br>        // Define the instruction for the AI<br>        const instruction = &quot;Given the prompt of the user, and the given context, answer the question as truthfully as possible based on the context and say &#39;I do not know&#39; when you do not know. Return your answer in Markdown format.&quot;;<br>        <br>        if (props.conversationHistory.length === 0) {<br>            // For the first message, include the initial context, the instruction, and the user&#39;s prompt<br>            return `${props.initialContext}\n\n${instruction}\n\nPrompt: ${newUserPrompt}`;<br>        } else {<br>            // For subsequent messages, compile recent conversation history without repeating initialContext<br>            const conversationContext = props.conversationHistory.map(m =&gt; `${m.sender}: ${m.text}`).join(&#39;\n&#39;);<br>            // Include the compiled context, the instruction, and the new user&#39;s prompt<br>            return `${conversationContext}\n\n${instruction}\n\nYou: ${newUserPrompt}`;<br>        }<br>    };<br><br><br>    const invokeLambdaFunction = async () =&gt; {<br>        try {<br>            const REGION = &quot;your-region&quot;;<br>    <br>            const lambda = new LambdaClient({<br>                region: REGION,<br>                credentials: fromCognitoIdentityPool({<br>                    clientConfig: { region: REGION }, // Configure the underlying CognitoIdentityClient.<br>                    identityPoolId: props.identityPoolId,<br>                    logins: {<br>                        [props.identityProvider]: props.idToken<br>                    }<br>                })<br>            });<br>    <br>            return await lambda.send(<br>                new InvokeWithResponseStreamCommand({<br>                    FunctionName: &quot;lambda_chat&quot;,<br>                    Payload: JSON.stringify({<br>                        body: JSON.stringify({<br>                            model: props.model,<br>                            prompt: preparePromptForOpenAI(props.prompt),<br>                            temperature: props.temperature,<br>                            max_tokens: props.maxTokens,<br>                            top_p: props.topP<br>                        })<br>                    })<br>                })<br>            );<br>        } catch (error) {<br>            console.error(&quot;Error invoking Lambda function&quot;, error);<br>        }<br>    };<br><br>    useEffect(() =&gt; {<br>        const fetchData = async () =&gt; {<br>            setIsLoading(true);<br>            // Directly notify the parent component about the user&#39;s prompt without storing it locally<br>            props.onNewMessage({ sender: &quot;You&quot;, text: props.prompt });<br>    <br>            const response = await invokeLambdaFunction();<br>            if (response &amp;&amp; response.EventStream) {<br>                const decoder = new TextDecoder(&quot;utf-8&quot;);<br>                let isFirstChunk = true; // Flag to track the first chunk<br>                for await (const event of response.EventStream) {<br>                    const text = decoder.decode(event.PayloadChunk?.Payload);<br>                    if (text !== undefined) {<br>                        // Directly notify the parent component about the new message<br>                        props.onNewMessage({ sender: &quot;Klea&quot;, text });<br>                        // Disable the spinner after the first chunk is received and processed<br>                        if (isFirstChunk) {<br>                            setIsLoading(false);<br>                            isFirstChunk = false; // Ensure this only happens once<br>                        }<br>                    }<br>                }<br>                <br>            }<br>            else{<br>                setIsLoading(false);<br>            }<br>        };<br>    <br>        fetchData().catch(console.error);<br>    }, [props.prompt, props.onNewMessage]);<br><br>    return (<br>        &lt;div&gt;<br>            {isLoading &amp;&amp; &lt;div className=&quot;spinner&quot;&gt;&lt;/div&gt;}<br>        &lt;/div&gt;<br>    );<br>};</pre><p>Then I build this using webpack in such a way so that I get one bundle.js file. I then copy that bundle.js file to the Mendix widget code (the parent component). The rest is fairly simple in fact.</p><pre>import { createElement, useState, useMemo,useCallback, useEffect } from &quot;react&quot;;<br>import { KleastreamchatContainerProps } from &quot;../typings/KleastreamchatProps&quot;;<br>import { Streamer, StreamerProps, Message } from &quot;./components/bundle&quot;;<br>import ReactMarkdown from &#39;react-markdown&#39;;<br><br>import &quot;./ui/Kleastreamchat.css&quot;;<br><br>const Kleastreamchat = (props: KleastreamchatContainerProps) =&gt; {<br>    const [messages, setMessages] = useState&lt;Message[]&gt;([]);<br>    const [conversationHistory, setConversationHistory] = useState&lt;Message[]&gt;([]);<br><br><br><br>    const handleNewMessages = useCallback((newMessage: Message) =&gt; {<br>        setMessages((prevMessages) =&gt; {<br>            const lastMessage = prevMessages[prevMessages.length - 1];<br>            <br>            // If the last message is from &quot;Klea&quot; and the new chunk is also from &quot;Klea&quot;,<br>            // append the chunk to the last message&#39;s text.<br>            if (lastMessage &amp;&amp; newMessage.sender === &quot;Klea&quot; &amp;&amp; lastMessage.sender === &quot;Klea&quot;) {<br>                return [<br>                    ...prevMessages.slice(0, -1),<br>                    { ...lastMessage, text: lastMessage.text + newMessage.text },<br>                ];<br>            } else {<br>                // Otherwise, start a new message.<br>                return [...prevMessages, newMessage];<br>            }<br>        });        <br>        setConversationHistory((prevHistory) =&gt; {<br>            // Check if the last entry in the history is from the same sender<br>            const lastEntry = prevHistory[prevHistory.length - 1];<br>            if (lastEntry &amp;&amp; lastEntry.sender === newMessage.sender) {<br>              // If so, append the new text to the existing entry&#39;s text<br>              return [<br>                ...prevHistory.slice(0, -1),<br>                { ...lastEntry, text: lastEntry.text + newMessage.text },<br>              ];<br>            } else {<br>              // Otherwise, add a new entry to the history<br>              return [...prevHistory, newMessage];<br>            }<br>        });<br>    }, []);<br><br>    useEffect(() =&gt; {<br>        // Check if initialContext has a meaningful value before setting the conversation history<br>        if (props.initialContext?.value) {<br>            setConversationHistory([<br>                { sender: &quot;System&quot;, text: props.initialContext.value },<br>            ]);<br>        }<br>    }, [props.initialContext.value]); // Dependency on props.initialContext.value<br><br>    if (<br>        props.prompt?.status === &quot;available&quot; &amp;&amp;<br>        props.idToken?.status === &quot;available&quot;<br>    ) {                <br>        const streamerProps: StreamerProps = useMemo(() =&gt; ({<br>            idToken: props.idToken.value || &quot;&quot;,<br>            prompt: props.prompt.value || &quot;&quot;,<br>            initialContext: props.initialContext.value || &quot;&quot;,<br>            model: props.model.value || &quot;&quot;,<br>            temperature: props.temperature.value?.toNumber() ?? 0, // Assuming these are meant to be numbers and have a default<br>            topP: props.top_p.value?.toNumber() ?? 0,<br>            maxTokens: props.max_tokens.value?.toNumber() || 0,<br>            identityProvider: props.identityProvider.value || &quot;&quot;,<br>            identityPoolId: props.identityPoolId.value || &quot;&quot;,       <br>            onNewMessage: handleNewMessages,<br>            conversationHistory: conversationHistory     <br>        }),[props.idToken.value, props.prompt.value, props.initialContext.value, props.model.value, props.temperature.value, props.top_p.value, props.max_tokens.value, props.identityProvider.value, props.identityPoolId.value]);;<br><br>        return (<br>            &lt;div className=&quot;kleastreamchat-container&quot;&gt;                <br>                &lt;Streamer {...streamerProps} /&gt;              <br>                &lt;div className=&#39;message-container&#39;&gt;              <br>                    {messages.map((message, index) =&gt; (<br>                        &lt;div key={index} className=&quot;message&quot;&gt;<br>                            &lt;strong&gt;{message.sender}:&lt;/strong&gt; &lt;ReactMarkdown&gt;{message.text}&lt;/ReactMarkdown&gt;<br>                        &lt;/div&gt;<br>                    ))}<br>                &lt;/div&gt;                                <br>            &lt;/div&gt;<br>        );<br>    } else {<br>        // Handle loading or unavailable state<br>        return &lt;div&gt;Loading...&lt;/div&gt;;<br>    }<br>};<br><br>export default Kleastreamchat;</pre><p>Because I ask OpenAI to return the response in MarkDown format, I use “react-markdown” to format the response. That is obviously optional but it is a nice addition for my specific use case.</p><p>I added a short demo video of the result in a Mendix app.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F924338538%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F924338538%3Fshare%3Dcopy&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1817161077-86d7a252348cbfd89aef73a57a507fbded35ce5f201f0a7b7ab7f944603e38f9-d_1280&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1716" height="946" frameborder="0" scrolling="no"><a href="https://medium.com/media/f99a5a182fa52d69ad2344d1d79a2390/href">https://medium.com/media/f99a5a182fa52d69ad2344d1d79a2390/href</a></iframe><p>I hope this helps the Mendix community to create similar features in their apps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd35dfe0ff4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>