<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roy Pasternak on Medium]]></title>
        <description><![CDATA[Stories by Roy Pasternak on Medium]]></description>
        <link>https://medium.com/@roy-pstr?source=rss-17dd97a32c1d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*e0xCNAF1kTTELcmSKZhIGg.png</url>
            <title>Stories by Roy Pasternak on Medium</title>
            <link>https://medium.com/@roy-pstr?source=rss-17dd97a32c1d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:21:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@roy-pstr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Python Terminal Copilot — PyPilot]]></title>
            <link>https://medium.com/@roy-pstr/a-python-terminal-copilot-pypilot-979165ff344f?source=rss-17dd97a32c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/979165ff344f</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[copilot-programming]]></category>
            <dc:creator><![CDATA[Roy Pasternak]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 09:41:15 GMT</pubDate>
            <atom:updated>2024-03-25T09:41:15.715Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Python Terminal Copilot — PyPilot</h3><p><em>This one is for all python terminal users</em></p><p>So you already using github’s copilot for coding assistant inside VSCode and sometimes you even go to ChatGPT for a quick coding discussion?</p><p>PyPilot is the missing piece for you — <strong>A coding assistance inside your python terminal.</strong> It looks like a regular python terminal, it smells like a regular python terminal but it has a LLM integration inside of it!</p><p>Here is a quick demo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4IKabTkYamLpqxqaF4EEqQ.gif" /></figure><h3>Quickstart</h3><p><strong>Github: </strong><a href="https://github.com/roy-pstr/python-pilot">https://github.com/roy-pstr/python-pilot</a></p><p><strong>PIP: </strong><a href="https://pypi.org/project/python-pilot/">https://pypi.org/project/python-pilot/</a></p><pre>$ pip install python-pilot</pre><p>Then to start the terminal,</p><pre>$ pypilot --api-key sk-....</pre><h3>How Does PyPilot Work?</h3><p>The idea was to keep everything simple and similar as possible to the normal python terminal. I figured no one is using python comments inside the terminal so let’s repurpose those comments to be instructions/requests to the LLM.</p><p>But of course this is not enough — to get relevant code generation the model needs to get also the terminal context. In this case these are all previous commands (history()) and the local variables (ulocals()). This way the model will be able to use functions, variables and modules that are relevant to the current session.</p><p>The model response can be either code or text. If the model answer was code, you will be prompted to approve the execution of it. Behind the scene there is a router architecture that selects between two different types of prompts — one for code generation and the second for text response.</p><h3>Constraints</h3><ol><li>Context size — As your session grow larger the context size will grow respectively and eventually will hit the model’s maximum context size. You can clear() to clean the history and reduce the context size.</li><li>Security — The terminal by default has access to your operating system. This combined with code generated by the model can be pretty nasty. This is your responsibility to make sure the code the LLM generated (in light blue) is safe before your press ENTER to execute it.</li><li>Cost — For now PyPilot supports only OpenAI models so you need to put your own API Key to use it.</li></ol><p>This is really basic for now, there are a lot more features and improvements to make — feel free to contribute.</p><h3>Similar Projects</h3><p>Here are other projects that let you use LLMs inside the python terminal:</p><ol><li><a href="https://github.com/AlexWiles/pai">https://github.com/AlexWiles/pai</a></li><li><a href="https://github.com/GoodAI/HALLM">https://github.com/GoodAI/HALLM</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=979165ff344f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simple Way To Manage Sessions With AWS Lambda & DynamoDB In Python]]></title>
            <link>https://medium.com/@roy-pstr/a-simple-way-to-manage-sessions-with-aws-lambda-dynamodb-in-python-c7aae1aa7258?source=rss-17dd97a32c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7aae1aa7258</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[session-management]]></category>
            <dc:creator><![CDATA[Roy Pasternak]]></dc:creator>
            <pubDate>Mon, 06 Feb 2023 18:59:00 GMT</pubDate>
            <atom:updated>2023-02-13T09:27:31.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7kRy71yQsfwCtSlJ" /></figure><p>Motivation &amp; code examples for how to use DynamoDB as a session store for AWS Lambda function — in a python environment.</p><h3>Tl;dr</h3><p>I created the session-lambda python package for managing sessions on AWS lambda with DynamoDB — <a href="https://github.com/roy-pstr/session-lambda">https://github.com/roy-pstr/session-lambda</a></p><h3>Motivation</h3><p>AWS Lambdas are <strong>stateless</strong> functions running in a managed cloud environment. The key here is stateless. It means that two subsequent lambda calls have no dependency. This is by design, and due to the fact there is no shared memory between lambda calls (at least not on purpose).</p><p>With that said, there are use cases where you would want those lambdas to have a state. For example when running a backend server with user sessions or for applications that need to have a state — like a chat app where you want to access the recent chat history. Let&#39;s see how this can be done.</p><h3>Why DynamoDB?</h3><p>For a lambda to have a shared state between subsequent calls it needs to have a shared memory of some kind. Then those lambdas will need a way to access this memory and pull the data they want — the session data. But, with restrictions. The lambda should gain access only to the session data of the client that invoked the call. The best way to do this is with key/value storage — DynamoDB!</p><blockquote>I chose DynamoDB over Reddis because DynamoDB is a fully serverless service, while Reddis has no native serverless service in the AWS ecosystem, yet.</blockquote><h3>Session-Lambda Python Package</h3><p>session-lambda is a simple lambda handler wrapper in python that takes care of the session management part! You provide the table name in DynamoDB and the wrapper handles everything else for you.</p><pre>from lambda_session import init_session_store, session, get_session_data<br><br>@session(key_name=&#39;session-id&#39;, update=False, ttl=15*60)<br>def lambda_handler(event, context):<br>  session_data = get_session_data()<br>  ...</pre><p>Its key features are:</p><ul><li>Session id generation (both server-side and client-side).</li><li>Get the session data from the table so you can access it from within the handler function.</li><li>TTL feature (time to live) which is important if you don’t want your session data to live forever in the DynamoDB table.</li><li>Both mutable and immutable session data modes.</li></ul><h3>Code Example</h3><h4>Clone example repository</h4><p>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:roy-pstr/session-lambda-example.git</p><p>cd session-lambda-example</p><h4>Install python requirements</h4><pre>python3 -m venv .venv<br>source ./.venv/bin/activate<br>pip install -r requirements.txt</pre><h4>Create a local store</h4><blockquote>Store refers to a key/value database where we keep the sessions data</blockquote><p>For simplicity, we will work with a local JSON file as the sessions’ store.</p><pre>echo &#39;{}&#39; &gt;&gt; local_store.json</pre><h4>A brief look at the code files</h4><ol><li>lambda_function.py — This is where the handler implementation is. Pay attention to the @session, get_session_data() and set_sessions_data() . The handler parses the user message out of the event body and saves it to the session data.</li></ol><ul><li>@session is a python decorator that pulls the session data (if a session-id is provided and the session exists) from the store and injects it into the handler function.</li><li>get_session_data() is our way to fetch the session data into a local variable inside the handler.</li><li>set_session_data() is our way to put the session data in the store. If the session does not exist in the store it will create it, and if it already exists it can update its’ value, depending on the update flag value.</li></ul><p>2. invoke.py — This is a simple script for local invoking of the lambda handler. It gets two arguments: message [session-id], the second is optional. Then the message is passed as the body value, and the session-id is passed inside the headers.</p><h4>Invoke without session-id</h4><pre>python invoke.py &quot;hello&quot;<br>python invoke.py &quot;hello&quot;</pre><ul><li>Call the script twice</li><li>The store content should be (with different session ids)</li></ul><pre>{<br>    &quot;zpXmm-bkGQCwKwKGmwltJYBbumx_RZfHZrmfCegD1FQ&quot;: null,<br>    &quot;xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot;: null<br>}</pre><ul><li>Two sessions were created in the store, but there is no session data. Only null values.</li></ul><h4>Invoke lambda with session-id</h4><ul><li>Let&#39;s try again to invoke the lambda, but this time with a session-id (this session-id is passed as a header to the lambda handler request)</li></ul><pre>python invoke.py &quot;xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot; &quot;hello&quot;</pre><ul><li>And now you should see it in the local store</li></ul><pre>{<br>    &quot;zpXmm-bkGQCwKwKGmwltJYBbumx_RZfHZrmfCegD1FQ&quot;: null,<br>    &quot;xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot;: &quot;hello&quot;<br>}</pre><ul><li>Nice! now you can access this session data in any lambda call using the same session-id.</li><li>But what if you want to update this session data? For that, you can use the update flag. By default this behavior is disabled and calling set_session_data() does not override existing value.</li></ul><h4>Invoke the lambda and update session data</h4><ul><li>Enable update flag</li></ul><pre>@session(update=True)<br>def lambda_handler(event, context):<br>  ...</pre><ul><li>Invoke the lambda with new data</li></ul><pre>python invoke.py &quot;-xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot; &quot;hello world&quot;</pre><ul><li>Now check the store, you should see:</li></ul><pre>{<br>    &quot;zpXmm-bkGQCwKwKGmwltJYBbumx_RZfHZrmfCegD1FQ&quot;: null,<br>    &quot;xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot;: &quot;hello world&quot;<br>}</pre><h4>Server-side vs client-side generated session ids</h4><ul><li>In the above example, the server was in charge of generating the initial session-id value and passing it to the client. Then the client uses it to access the session data.</li><li>There is another use case, where the client wants to generate its’ own session-id. Let&#39;s take a look at how it can be done.</li><li>The client generated the next key: client-session-id</li><li>Now, use it the same way as before:</li></ul><pre>python invoke.py &quot;client-session-id&quot; &quot;hello&quot;</pre><ul><li>You should see in the store</li></ul><pre>{<br>    &quot;zpXmm-bkGQCwKwKGmwltJYBbumx_RZfHZrmfCegD1FQ&quot;: null,<br>    &quot;xtNGpXFFPl8tgHr9CN8dWlWrF81FbeATx_f2CApIijs&quot;: &quot;hello world&quot;,<br>    &quot;client-session-id&quot;: &quot;hello&quot;<br>}</pre><ul><li>The difference here is that when passing a session-id that does not exist in store, we assume it is on purpose and because the client wants to create the id himself. In this case, a new session will be created in the store, with the corresponding id value.</li></ul><h4>Use DynamoDB table as the session store</h4><ul><li>OK. so for the real thing you better use DynamoDB. To do so, just remove the following line from the code:</li></ul><pre>init_session_store(file_path=&#39;local_store.json&#39;)</pre><ul><li>Now invoke the function again</li></ul><pre>python invoke.py &quot;-dqMD2t1AiiHf95K58q5bg1qectK0XQOeFe57cJEOlc&quot; &quot;hello world&quot;</pre><ul><li>You should get an error message with:</li></ul><pre>session_lambda.session.SessionStoreNotSet: Store not set</pre><ul><li>This is because we did not point the session manager to any store.</li><li>To point to a DynamoDB table store you use SESSION_LAMBDA_DYNAMODB_TABLE_NAME environment variable or use the init_session_store(dynamodb_table_name=TABLE_NAME) like this.</li><li>Your DynamoDB table should have a key attribute named key of type string . session-id will be stored under this attribute, and the session data will be stored under value attribute.</li><li>That’s it! Now your sessions will be saved to your DynamoDB table.</li></ul><h4>Time to live feature</h4><p>It is recommended to use the TTL feature for your table. Enable this from your AWS console and set the attribute name to ttl then you can use it with the session decorator like this: @session(ttl=60*15) . ttl units are seconds.</p><h3>Conclusion</h3><p>session-lambda can help you manage sessions for lambda functions without the overhead of creating the behavior between the function and the session store (DynamoDB).</p><p>The session-lambda package was just released and any feedback and suggestions for improve will be super welcome! Enjoy.</p><h3>References</h3><p><a href="https://github.com/roy-pstr/session-lambda">GitHub - roy-pstr/session-lambda: A simple way to manage sessions for AWS Lambdas</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7aae1aa7258" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FastAPI Server Errors And Logs — Take Back Control]]></title>
            <link>https://medium.com/@roy-pstr/fastapi-server-errors-and-logs-take-back-control-696405437983?source=rss-17dd97a32c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/696405437983</guid>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[server-error]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[logs]]></category>
            <category><![CDATA[https]]></category>
            <dc:creator><![CDATA[Roy Pasternak]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 11:46:18 GMT</pubDate>
            <atom:updated>2023-01-22T13:49:01.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>FastAPI Server Errors And Logs — Take Back Control</h3><p>Keywords: Python, FastAPI, Server Errors, Logs, Exception Handlers, AWS Lambda, AWS Cloud Watch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*pnFwykIqzV5rgBRLJq8wug.png" /></figure><p>We are here to understand how FastAPI handles server errors and logs. After reading this:</p><ul><li>You will better understand HTTP status codes</li><li>You will get to know FastAPIs exception handlers and what happens when your server runs into an error</li><li>You will be able to control this behavior</li><li>You will have an improved version of FastAPI logs — with request time measurement and extra details on Internal Server Error and Request Validation</li><li>You will have better logs for AWS Lambda</li></ul><p>Before reading this make sure you are familiar with the FastAPI documentation on handling errors:</p><ul><li><a href="https://fastapi.tiangolo.com/tutorial/handling-errors/">https://fastapi.tiangolo.com/tutorial/handling-errors/</a></li></ul><p>Plus, if you prefer to read code than words, check the GitHub repository:</p><ul><li><a href="https://github.com/roy-pstr/fastapi-custom-exception-handlers-and-logs">https://github.com/roy-pstr/fastapi-custom-exception-handlers-and-logs</a></li></ul><h3>Agenda</h3><ol><li>HTTP response status code and FastAPI</li><li>FastAPI built-in exceptions handlers</li><li>Examples of FastAPI exception handling</li><li>Code to control FastAPI exception handling and improve logging</li></ol><h3>HTTP Response Status Codes and FastAPI</h3><p>The response status code of an HTTP server gives information on how the server handled a client request. Status codes are grouped into five classes:</p><ol><li>Informational responses (100 – 199)</li><li>Successful responses (200 – 299)</li><li>Redirection messages (300 – 399)</li><li>Client error responses (400 – 499)</li><li>Server error responses (500 – 599)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*E6jvS_Rk2-TT-ZtSlO3r3A.png" /><figcaption>(See <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status">https://developer.mozilla.org/en-US/docs/Web/HTTP/Status</a> for a complete reference)</figcaption></figure><p>When you build an HTTP server it is your responsibility to use the protocol (in other words, you really can set the status code value to anything, like return 500 on success 😱). Anyway, you better stick to the protocol. Of course that any HTTP server framework will have some built-in support in this protocol. For example, with FastAPI, returning a response without mentioning any specific status code will assume success and use status code 200.</p><p>FastAPI has a default behavior also for server and some client errors. It will use status codes 500, 422, and 404 in specific cases for you. Let&#39;s understand when it will happen and how you can control this.</p><h3>FastAPI Built-In Exceptions Handlers</h3><p>FastAPI has built-in exception handlers for HTTPException and ValidationError. It means that whenever one of these exceptions occurs, FastAPI catches it and returns an HTTP response with a corresponding status code and details about the error. Let&#39;s break it down into cases:</p><ol><li>Pydantic Validation Error (422) — This is a client-side error and it returns a response with a status code 422. This error is caused by the data the client sends. Pydantic validates the path parameters, query parameters, and body data — so any validation error there will raise this exception.</li><li>HTTP Exception (≥500)— This is a server-side error and it can happen in two ways:</li></ol><ul><li>You explicitly raise an HTTP Exception from your code. This case is when you use “try and except” block and raise HTTP Exception to indicate that a certain server error occurred. Here you choose the status code value (≥500) that will be returned.</li><li>Implicit raised by FastAPI. It happens when a pythonic exception (E.g. ValueError) is raised from within your code. FastAPI will catch the exception and turn it into a response with status code 500 and message detail of “Internal Server Error”. In this case, you should also see python trace back in your logs (which is super important to understand what happened).</li></ul><p>The last built-in handler is for “404 Not Found” response. This one indicates that a client tried to access a path in your server that does not exist.</p><h3>A Code Example Of FastAPI Exception Handling</h3><p>Let’s walk through a code example that will help us understand how FastAPI handles exceptions.</p><ul><li>Create a python project and install fastapi[&quot;all&quot;] with your favorite python package manager.</li><li>Create main.py with the following code:</li></ul><pre>from fastapi import FastAPI<br>app = FastAPI()<br>@app.get(&quot;/divide&quot;)<br>def do_divide(a: int, b: int):<br>    return a/b<br>@app.get(&quot;/divide_with_explicit_exception&quot;)<br>def do_divide_with_explicit_exception(a: int, b: int):<br>    try:<br>        return a/b<br>    except ZeroDivisionError:<br>        raise HTTPException(status_code=500, detail=&quot;Division by zero is not allowed&quot;)</pre><ul><li>Start the server</li></ul><pre>uvicorn main:app --reload</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlqX_sMwmMspe-NOFzVzzA.png" /><figcaption>Logs in your terminal on server start</figcaption></figure><ul><li>Open another terminal window and run the following requests from the server:</li></ul><pre>curl -X GET &quot;http://localhost:8000/divide?a=1&amp;b=2&quot;<br>curl -X GET &quot;http://localhost:8000/divide?a=1.5&amp;b=2&quot;<br>curl -X GET &quot;http://localhost:8000/divide_with_explicit_exception?a=1&amp;b=0&quot;<br>curl -X GET &quot;http://localhost:8000/divide?a=1&amp;b=0&quot;</pre><ul><li>This should be your server logs:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*sB6LQEJm_LXQASO6d8fVhw.png" /><figcaption>FastAPI logs</figcaption></figure><ul><li>The first log is for a successful response</li><li>The second is for client-side error — invalid request input data (in this case b must be an integer)</li><li>The third is for server-side error that is handled explicitly by you. Here it is on you to add more details so you will understand what caused this because there are no traces.</li><li>The fourth is for server-side error that is handled implicitly by FastAPI. Here the “Internal Server Error” does not tell you much, but the traces should tell the story.</li></ul><p>Looks good right? indeed. But it can be better. For example, a time measurement can be added to each log. “422 Unprocessable Entity” can be more verbose, printing the client input data so we can easily figure out what data was invalid. And “500 Internal Server Error” is worth nothing without further explanation.</p><p>There is one more thing, when deploying FastAPI application on AWS Lambda you will not see any of these logs. It has something to do with how uvicron define its loggers. It will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2Q34it3F7vZ6aIKwQ_WhQ.png" /><figcaption>FastAPI service logs on cloud watch from lambda deployment. Where are all the logs we see in our terminal???</figcaption></figure><p>Let’s see how you can improve all those.</p><h3>Control FastAPI Exception Handling And Improve Logging</h3><p>We are going to replace all FastAPI’s default logs (which are actually uvicorns’) with the exact same logs (+some additions), but this time created by us. This will gain you the ability to customize them in the future.</p><p>Those are the results you will get:</p><ol><li>Add to the regular INFO response log a time measurement in milliseconds:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxAF9Zxa9Eu-oRTJwFmi5g.png" /><figcaption>Original</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UN1eVQF34mFOAhjUfuQT0A.png" /><figcaption>Ours — more verbose about the time it took to process the request</figcaption></figure><p>2. Add INFO log with body data and query parameters when 422 occurs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kZxEloKKNb6F4w2t7wrpA.png" /><figcaption>Original</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7mo72LdtbsaK_St2HqHCA.png" /><figcaption>Ours — more verbose about the client input data</figcaption></figure><p>3. Add error type and message to the response log:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QmlyKzrt_vTNVOUUC4gr1Q.png" /><figcaption>Original</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PRLyS6aWovE1jsIZau9lQw.png" /><figcaption>Ours — more verbose about the reason for the error (this information is found also at the end of the traceback)</figcaption></figure><p>4. I mentioned before the problem with FastAPI’s logs doesn’t appear in AWS Cloud Watch logs — this is solved with this implementation.</p><h4><strong>Let’s start</strong></h4><ul><li>Create logger.py with the following code:</li></ul><pre>import logging<br><br># Disable uvicorn access logger<br>uvicorn_access = logging.getLogger(&quot;uvicorn.access&quot;)<br>uvicorn_access.disabled = True<br><br>logger = logging.getLogger(&quot;uvicorn&quot;)<br>logger.setLevel(logging.getLevelName(logging.DEBUG))</pre><ul><li>You can change the logging level of course ( logging.debug ).</li><li>Create middleware.py with the following code:</li></ul><pre>import http<br>import time<br><br>from fastapi import Request<br><br>from logger import logger<br><br><br>async def log_request_middleware(request: Request, call_next):<br>    &quot;&quot;&quot;<br>    This middleware will log all requests and their processing time.<br>    E.g. log:<br>    0.0.0.0:1234 - GET /ping 200 OK 1.00ms<br>    &quot;&quot;&quot;<br>    logger.debug(&quot;middleware: log_request_middleware&quot;)<br>    url = f&quot;{request.url.path}?{request.query_params}&quot; if request.query_params else request.url.path<br>    start_time = time.time()<br>    response = await call_next(request)<br>    process_time = (time.time() - start_time) * 1000<br>    formatted_process_time = &quot;{0:.2f}&quot;.format(process_time)<br>    host = getattr(getattr(request, &quot;client&quot;, None), &quot;host&quot;, None)<br>    port = getattr(getattr(request, &quot;client&quot;, None), &quot;port&quot;, None)<br>    try:<br>        status_phrase = http.HTTPStatus(response.status_code).phrase<br>    except ValueError:<br>        status_phrase=&quot;&quot;<br>    logger.info(f&#39;{host}:{port} - &quot;{request.method} {url}&quot; {response.status_code} {status_phrase} {formatted_process_time}ms&#39;)<br>    return response</pre><ul><li>Create exception_handlers.py with the following code:</li></ul><pre>import sys<br><br>from typing import Union<br><br>from fastapi import Request<br>from fastapi.exceptions import RequestValidationError, HTTPException<br>from fastapi.exception_handlers import http_exception_handler as _http_exception_handler<br>from fastapi.exception_handlers import (<br>    request_validation_exception_handler as _request_validation_exception_handler,<br>)<br>from fastapi.responses import JSONResponse<br>from fastapi.responses import PlainTextResponse<br>from fastapi.responses import Response<br><br>from logger import logger<br><br>async def request_validation_exception_handler(request: Request, exc: RequestValidationError) -&gt; JSONResponse:<br>    &quot;&quot;&quot;<br>    This is a wrapper to the default RequestValidationException handler of FastAPI.<br>    This function will be called when client input is not valid.<br>    &quot;&quot;&quot;<br>    logger.debug(&quot;Our custom request_validation_exception_handler was called&quot;)<br>    body = await request.body()<br>    query_params = request.query_params._dict  # pylint: disable=protected-access<br>    detail = {&quot;errors&quot;: exc.errors(), &quot;body&quot;: body.decode(), &quot;query_params&quot;: query_params}<br>    logger.info(detail)<br>    return await _request_validation_exception_handler(request, exc)<br><br><br>async def http_exception_handler(request: Request, exc: HTTPException) -&gt; Union[JSONResponse, Response]:<br>    &quot;&quot;&quot;<br>    This is a wrapper to the default HTTPException handler of FastAPI.<br>    This function will be called when a HTTPException is explicitly raised.<br>    &quot;&quot;&quot;<br>    logger.debug(&quot;Our custom http_exception_handler was called&quot;)<br>    return await _http_exception_handler(request, exc)<br><br><br>async def unhandled_exception_handler(request: Request, exc: Exception) -&gt; PlainTextResponse:<br>    &quot;&quot;&quot;<br>    This middleware will log all unhandled exceptions.<br>    Unhandled exceptions are all exceptions that are not HTTPExceptions or RequestValidationErrors.<br>    &quot;&quot;&quot;<br>    logger.debug(&quot;Our custom unhandled_exception_handler was called&quot;)<br>    host = getattr(getattr(request, &quot;client&quot;, None), &quot;host&quot;, None)<br>    port = getattr(getattr(request, &quot;client&quot;, None), &quot;port&quot;, None)<br>    url = f&quot;{request.url.path}?{request.query_params}&quot; if request.query_params else request.url.path<br>    exception_type, exception_value, exception_traceback = sys.exc_info()<br>    exception_name = getattr(exception_type, &quot;__name__&quot;, None)<br>    logger.error(<br>        f&#39;{host}:{port} - &quot;{request.method} {url}&quot; 500 Internal Server Error &lt;{exception_name}: {exception_value}&gt;&#39;<br>    )<br>    return PlainTextResponse(str(exc), status_code=500)<br><br></pre><ul><li>Add these changes to main.py file:</li></ul><pre>from fastapi import FastAPI<br>from starlette.exceptions import HTTPException<br>from fastapi.exceptions import RequestValidationError<br>from exception_handlers import request_validation_exception_handler, http_exception_handler, unhandled_exception_handler<br>from middleware import log_request_middleware<br><br>app = FastAPI()<br><br>app.middleware(&quot;http&quot;)(log_request_middleware)<br>app.add_exception_handler(RequestValidationError, request_validation_exception_handler)<br>app.add_exception_handler(HTTPException, http_exception_handler)<br>app.add_exception_handler(Exception, unhandled_exception_handler)</pre><ul><li>Your files structure should be like this:</li></ul><pre>|fastapi-exceptions-project<br>  |--exception_handlers.py<br>  |--logger.py<br>  |--main.py<br>  |--middleware.py</pre><ul><li>You are all set! Test it with the same curl commands from before and check the server logs (You can remove all the logger.debug later. This is just for… debug).</li><li>By the way, it also solves the AWS lambda logs problem. This time all those logs will also appear in your cloud watch:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xuhU88447kKAXukncD4oQQ.png" /><figcaption>AWS Cloud Watch logs. This time with the FastAPI logs we use to see in our terminal.</figcaption></figure><h3>To Sum It All Up</h3><p>You now should feel much more confident putting FastAPI service in production, knowing that you understand what happens on server errors and what logs you will see.</p><ul><li>You can find all the code in this repository</li></ul><p><a href="https://github.com/roy-pstr/fastapi-custom-exception-handlers-and-logs">https://github.com/roy-pstr/fastapi-custom-exception-handlers-and-logs</a></p><p>Feel free to leave comments and feedback,</p><p>Thanks,</p><p>Roy.</p><h3>Resources</h3><ul><li><a href="https://github.com/tiangolo/fastapi">GitHub - fastapi/fastapi: FastAPI framework, high performance, easy to learn, fast to code, ready for production</a></li><li><a href="https://github.com/encode/uvicorn">GitHub - encode/uvicorn: An ASGI web server, for Python. 🦄</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status">HTTP response status codes - HTTP | MDN</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=696405437983" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using FastAPI In Production For One Year— This Is What I’ve Learned]]></title>
            <link>https://medium.com/@roy-pstr/using-fastapi-in-production-for-one-year-this-is-what-ive-learned-d1ff4a95f373?source=rss-17dd97a32c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1ff4a95f373</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Roy Pasternak]]></dc:creator>
            <pubDate>Thu, 19 Jan 2023 07:56:33 GMT</pubDate>
            <atom:updated>2023-03-14T11:25:46.787Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Keywords:</strong> FastAPI, Serverless, AWS, Monitoring, CICD, Startup, Backend.</p><p>In the last year, I have been leading the technology side of a small startup (Larium). Our team is working on language models that help financial experts gain insights from financial textual data sources. In addition to the AI stuff, we are building a SaaS web application so users can access the data and insights. We are using React (typescript) for the client side and FastAPI (python) for the server side.</p><p>When I started this adventure, my experience was in machine learning and Python development, never built web applications or servers before. Even though, I ended up in charge of our backend service (someone mentioned a startup?). Let’s see how it works for us…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eQonJ16ISPO5J1Q4c1M2jw.jpeg" /><figcaption>Mount Blanc. A picture I took one year ago, from before I knew how to create a backend server.</figcaption></figure><h3>Stages</h3><p>(or, let me choose what to read)</p><ol><li>We need a backend server</li><li>Just make this work</li><li>Scaling up the infrastructure</li><li>Moving to a better CICD</li><li>Monitor, alert, and fix</li><li>To sum it all up</li></ol><h3>We Need A Backend Server</h3><p>(or, why we chose Python and FastAPI?)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*o9cwQXSjr3K1Tbtk8N_wdA.png" /></figure><p>At the time, those were the responsibilities we defined for the backend server:</p><ul><li>Communication layer between the application and the database</li><li>Manage user authentication and authorization</li><li>Connector to external APIs (such as stock data prices)</li><li>Computational unit for algorithmic calculation (E.g. anomaly detection on time series data)</li></ul><p>My first decision was to use Python as the programming language because I wanted the same code base for the algorithms and the rest of the backend.</p><p>The second decision was to go with <a href="https://fastapi.tiangolo.com/">FastAPI</a> as the web API framework. Back in the day, it was relatively new with a growing community and strong positive sentiment. It felt right, and I was also a big fan of the type validation feature with <a href="https://docs.pydantic.dev/">Pydantic</a> (lack of type validation is considered to be one of the major down flaws of using Python as a backend language. FastAPI solved it). It also has great documentation, AWS Lambda integration (<a href="https://mangum.io/">Mangum</a>), and tons of open-source add-ons (<a href="https://github.com/mjhea0/awesome-fastapi">A list of most of them</a>).</p><p>As for the database, I chose to use Postgress as the database system and AWS RDS as a managed database server service. For user authentication and authorization, we use Auth0, which is an external service that handles it for you. Their documentation is not the best, but it works well in the long run.</p><blockquote>In order quick start with Auth0 and FastAPI, I created this GitHub repository, check it out!</blockquote><p><a href="https://github.com/roy-pstr/simple-auth0-fastapi-react-app">GitHub - roy-pstr/simple-auth0-fastapi-react-app: A simple application for authentication, authorization and user management based on Auth0. 🐍 Python ⚡ FastAPI + React.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*BUvRNslmH0RDnUI0D3r2FQ.png" /><figcaption>Application login page. Probably one of the first screens you want your backend server to support.</figcaption></figure><p>Language &amp; Framework: Python, FastAPI</p><p>Infrasrtucre: A Laptop</p><p>New Technologies: REST API, Postgres, JWT, HTTP</p><p>No. of Developers: 1</p><h3>Just Make This Work</h3><p>(or, build with infrastructure that works but does not scale)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*xS5EV9-c_1v0V-eZBs9LMg.png" /></figure><p>In the beginning, with no users, things were pretty straightforward. We used a single EC2 instance (a remote Linux server) for deployment. To make it an actual web server you will need to allow web access. Here you will find out that it is easy to define security policies in AWS (open web access on specific ports), and that it is much harder to configure the SSL layer for an EC2 instance (this makes the data transfer between your server to the client encrypted and more secure, the difference between http to https). This guide can help: <a href="https://marcosantonocito.medium.com/steps-to-install-a-go-daddy-ssl-certificate-on-nginx-on-ubuntu-14-04-ff942b9fd7ff">Steps to install a Go Daddy SSL Certificate with NGINX on Ubuntu 14.04</a></p><p>As I was still a single developer, the deployment process was manual. For each deployment, I would build the docker image locally on my laptop, push it to a docker image repository, then ssh to the EC2 instance, pull the image, and re-run the service container with the new image.</p><p>Both of those decisions worked well for the first month but didn’t hold for long. The EC2 instance start reaching its memory limits, which caused a forced server reboot, and network bottlenecks (we did not have a load balancer). The deployment method started to break when more than one developer worked on the code. There was too much idle time waiting for things to be built and deployed, plus we couldn&#39;t align the version deployed with a specific commit or branch in the git repository.</p><p>Language &amp; Framework: Python, FastAPI</p><p>Infrasrtucre: A single EC2 server, RDS</p><p>New Technologies: Docker, NGIX, SSL/TSL (HTTPS), SSH, AWS security policies</p><p>No. of Developers: 2</p><h3>Scaling Up The Infrastructure</h3><p>(or, when your server is shutting itself down)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*SoGEgK8mW5_0GGLxhLiDcQ.png" /></figure><p>The first thing we changed was the infrastructure — moving from EC2 to Lambda. Deploying the backend on AWS Lambda gave us the ability to scale up in peak hours and down when no one used the API with better costs than “always running server”. Suddenly, there was no server to maintain, no memory to manage and no unexpected EC2 crashes. The trade-off is that AWS manages this service, so there are some limits that you cannot change. Here is a list of limits we encountered and that you should know before moving to Lambda:</p><ul><li>Scale limit: The number of concurrent functions, which is 1000 across all functions in a region per account</li><li>Lambda size limit: “.zip” file size limit is up to 250MB and the container size limit is up to 10GB.</li><li>In-memory (RAM) limit: 128MB-10GB</li><li>CPU limit: You can’t select a specific CPU. It is auto-allocated according to the RAM allocation you choose.</li><li>Timeout: 30 seconds (caused by the API Gateway timeout)</li></ul><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html">See AWS Lambda official quotes here</a></p><p>The bottom line is that for us the simplicity of almost zero management and the out-of-the-box scalability was the killer features here. The only limit that made us uncomfortable was the “.zip” file size (our python environment is heavy) so we moved to a container-based lambda.</p><p>Language &amp; Framework: Python, FastAPI</p><p>Infrastructure: Lambda &amp; API Gateway, AWS RDS</p><p>New Technologies: Lambda</p><p>No. of Developers: 3</p><h3>Moving To A Better CICD</h3><p>(or, when there is more than one developer)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-LPUZ0NgmBHkqFQC.png" /></figure><p>Next, we upgraded our deployment from manual to automatic CICD (Continuous integration and Continuous deployment) pipeline. The CI goal is to enable simultaneous code development on the same repository by using a version control tool (GitHub, GitLab, etc..). And the CD goal is to automate the build and deployment process of new versions of your code to your production environment.</p><p>A key concept here is to have a different and isolated environment for each stage of development and define the process of moving between stages. Those are some of the changes we made:</p><ul><li>Modified our repository branch structuring from a single branch to a branch per stage (“main”, “staging”, and “develop”) and a branch per feature.</li><li>Created “develop” and “staging” environments for our RDS deployments, to enable better environment isolation when testing new changes.</li><li>Added “DB migration” step, so database changes will be made automatically when merging from one environment to another (we use <a href="https://alembic.sqlalchemy.org/en/latest/">alembic</a> for database migration).</li><li>Linked different sub-domains to each deployment environment: https://api.your.domain,https://api-staging.your.domain,https://api-dev.your.domain</li><li>Defined a feature approval process. This is the process of merging a feature branch to “develop”. It includes linter checks, unit tests, integration tests, and a peer code review.</li><li>Defined a version release process. The goal here is to bring new features into “production” without breaking anything. For that, we use “staging” environment, which mimics “production” environment. Its purpose is to test new features in a “production” like environment before actually deploying.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76bzs-mQDp8FKdLVqQD_1A.png" /><figcaption>This is how it actually looks on our Gitlab CI Pipeline. Merging from “staging” to “main”</figcaption></figure><p>The new CICD pipeline ran on “Gitlab CI pipeline”, which is a job manager triggered by events such as pull requests and merges. For automating the deployment to AWS and managing its resources, we use the <a href="https://www.serverless.com/">serverless framework</a>, an IaC tool that lets you describe your infrastructure in a .yml file and then deploy or manage it using a command line interface.</p><ul><li>Here is an example of <a href="https://github.com/roy-pstr/fastapi-serverless-aws-backend-service/blob/master/.github/workflows/cicd.yml">GitHub actions “.yml</a>” file</li><li>Here is an example of the <a href="https://github.com/roy-pstr/fastapi-serverless-aws-backend-service/blob/master/serverless.yml">serverless.yml</a></li></ul><p>Language &amp; Framework: Python, FastAPI</p><p>Infrastructure: Lambda &amp; API Gateway, AWS RDS</p><p>New Technologies: Serverless Framework, GitHub Actions / Gitlab CI, Linters &amp; Formatters (PyLint, Black), Static Type Analysis (MyPy), Database migration (alembic)</p><p>No. of Developers: 5</p><h3>Monitor, Alert, and Fix</h3><p>(or, when real users have real bugs)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*yvZA9h8B8tbZi6R2" /></figure><p>The system starts to get complex, and things break from time to time — the ability to proactively detect that something is not working as expected and fix it fast is needed. You will probably want to:</p><ul><li>Get an alert on 500 status code response since this means unexplained errors occurred on the server and this will lead to unexpected behavior on the client side.</li><li>Have a sense of the response time of your API so you can improve slow-responding endpoints.</li><li>Be able to debug a specific request and reproduce them to test a fix.</li></ul><p>Those abilities are gained by using monitoring and tracing tools, alerts, and verbose logs.</p><p>We chose to stick with the native AWS stack:</p><ul><li>For <strong>monitoring</strong>, we use Cloud Watch, which is a log aggregator. Combining this with informative* logs is the first step to detecting errors and reproducing them.</li><li>Another thing we started monitoring is the <strong>performance</strong>. For this, you can work with the native AWS lambda metrics which cover a few basic metrics such as failures over time, average response time, and more.</li><li>On top of those metrics, you can define <strong>alerts</strong> using AWS cloud watch alerts. For example, you can set an alert that sends a slack message whenever there are more than 3 requests with 500 status codes in a 1-minute interval.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lDIqBgY-c57xCTvCWTnAg.png" /><figcaption>This is 24 hours window. Invocations are the number of lambda function calls per 1-minute intervals, Duration is the average, max, and min duration of all invocations in a 1-minute interval, and Error count and success rate(%) is the absolute error count (red line), how many lambdas ended up with some type of error, and the green line is for the percentage of lambda ended up successfully out of all invocations</figcaption></figure><p>There are many application monitoring tools (such as Datadog, New Relic, and more) that handle those tasks. My advice is to keep it simple and use your native cloud provider tools up to the point you must change. These tools tend to get costly very fast and demand some level of expertise and maintenance from your team.</p><blockquote>(*) A word about informative logs when using FastAPI on Lambda</blockquote><blockquote>For us it took some time until we had a good log coverage on all incoming requests and possible failures. You can check the other post I did on how to better handle server errors and logs with FastAPI: <a href="https://medium.com/@roy-pstr/fastapi-server-errors-and-logs-take-back-control-696405437983">https://medium.com/@roy-pstr/fastapi-server-errors-and-logs-take-back-control-696405437983</a></blockquote><p>Language &amp; Framework: Python, FastAPI</p><p>Infrastructure: Lambda &amp; API Gateway, AWS RDS</p><p>New Technologies: AWS Cloud Watch, AWS XRay, AWS Lambda Metrics</p><p>No. of Developers: 10</p><h3>Final Backend Architecture</h3><p>In case you wondering, this is what the final architecture scheme looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*eCH4XiWbl-ZdI0ORHuGz6Q.png" /><figcaption>Backend Architecture</figcaption></figure><p>The code layer is Python FastAPI, deployed on AWS Lambda which acts as the compute engine (server). API Gateway is in charge on networking to the outside world (clients). DynamoDB, RDS (PostgresSQL) and ElasticSearch are all resources of data. And Auth0 and Polygon (Stock market data API) are external API that the backend communicate with.</p><h3>To Sum It All Up</h3><p>So, where did this journey lead us?</p><ol><li>We deployed our FastAPI backend service on AWS lambda to handle scale and stop maintaining servers.</li><li>We built a strong CICD pipe to sync the parallel code development, and automate testing, code-quality checks, and deployments.</li><li>We start using monitoring tools to detect failures in production and fix them quickly.</li></ol><blockquote>In order to quick start a new project of FastAPI for production, I created a GitHub repository with most of the lessons mentioned above, check it out!</blockquote><p><a href="https://github.com/roy-pstr/fastapi-serverless-aws-backend-service">GitHub - roy-pstr/fastapi-serverless-aws-backend-service: FastAPI python backend server with multi-stage CI/CD deployment</a></p><p>Feel free to leave any comment here or as a GitHub issue on the repository,</p><p>Thanks,</p><p>Roy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1ff4a95f373" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>