<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Asbjorn Bering on Medium]]></title>
        <description><![CDATA[Stories by Asbjorn Bering on Medium]]></description>
        <link>https://medium.com/@asbjorn.bering_39499?source=rss-ba9403b68709------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*btA_IrJHMO8D8Rrl</url>
            <title>Stories by Asbjorn Bering on Medium</title>
            <link>https://medium.com/@asbjorn.bering_39499?source=rss-ba9403b68709------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:20:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asbjorn.bering_39499/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (10/10): Kafka & SAGA Choreography…]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-10-10-kafka-saga-choreography-aeb9781b00a6?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeb9781b00a6</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Wed, 21 May 2025 05:27:28 GMT</pubDate>
            <atom:updated>2025-05-21T15:41:06.844Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Enterprise Python Microservices with FastAPI in 2025 (10/10): Kafka &amp; SAGA Choreography Pattern</h3><h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous article, we implemented the Outbox Table pattern. This allows our microservice to publish messages, using an outbox table and Debezium, to Kafka. In this article we are going to focus on how our microservice can consume these messages from Kafka, and how we can implement the SAGA Choreography pattern to enable distributed transactions across microservices. This is also the final article in this series.</p><h3>SAGA Choreography Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*JBMZ1_NnhD4e16OobMaC1A.png" /><figcaption>Our Pub/Sub that follows the SAGA Choreography Pattern</figcaption></figure><p>To better understand our implementation of our consumer logic, we will first walk through, how we want to implement the SAGA Choreography pattern. We will use an end-user requesting to begin a subscription as our example case.</p><ol><li>End-user sends a POST /subscriptions/create-subscription with his email and subscription_type as request body. A subscription_type could for example be premium_tier or paid_tier depending on which subscription the end-user wants to initiate.</li><li>Subscription Service inserts a Subscription record in auth.subscriptions and an Outbox record in auth.subscriptions_outbox .<br>Publishes an event to Kafka with topic subscriptionservice.subscription and event typesubscription_created_success or subscription_created_failed .<br>The subscription_id and email is the payload, as well as exception should the transaction fail.</li><li>User Service consumes event with topic subscriptionservice.subscription .<br>User Service filters out subscription_created_failed .<br>Inserts a User in auth.users and an Outbox in auth.users_outbox if the event type is subscription_created_success .<br>Publishes an event to Kafka with topic userservice.user and event type user_created_from_new_subscription_success or user_created_from_new_subscription_failed .<br>The payload consist of email , is_active , subscription_id , and if the transaction fails also exception .</li><li>Subscription Service consumes event with topic userservice.user .<br>Subscription Service filters our user_created_from_new_subscription_success .<br>If the event type is user_created_from_new_subscription_failed, the Subscription Service deletes the corresponding subscription from auth.subscriptions.</li></ol><p>This simple example shows how we are able to perform distributed transactions across microservices, when using the SAGA Choreography pattern. In the upcoming section we are going to implement our consumer in our User Service.</p><h3>Kafka Consumer</h3><p>In this section we are going to implement our Kafka consumer. We are going to go through our code step by step. Specifically, we are going through <strong>/src/consumer/kafka.py</strong>:</p><pre>KAFKA_BOOTSTRAP_SERVERS = &quot;kafka:29092&quot;<br>KAFKA_TOPICS = [&quot;subscriptionservice.subscription&quot;]<br>KAFKA_CONSUMER_GROUP = &quot;user_service_group&quot;<br>KAFKA_AUTO_COMMIT = True<br>KAFKA_MAX_POLL_INTERVAL_MS = 300000<br>KAFKA_SESSION_TIMEOUT_MS = 30000</pre><ol><li>Defines Kafka connection and consumer settings.</li><li>Specifies the Kafka server, topics to listen to, consumer group, and various timeouts.</li></ol><pre>class EventHandler:<br>    &quot;&quot;&quot;Base class for event handlers&quot;&quot;&quot;<br>    async def handle(self, payload: Dict[str, Any]) -&gt; None:<br>        raise NotImplementedError(&quot;Subclasses must implement handle method&quot;)</pre><ol><li>Defines an abstract base class for all event handlers.</li><li>Enforces that each handler implements its own handle method.</li></ol><pre>class SubscriptionCreatedSuccessHandler(EventHandler):<br>    async def handle(self, payload: Dict[str, Any]) -&gt; None:<br>        logger.info(f&quot;Processing subscription_created_success event: {payload}&quot;)<br>        try:<br>            async for db_session in get_db_session_for_background():<br>                user_repository = create_user_repository(db_session)<br>                user_service = UserService(user_repository)<br>                email = payload.pop(&quot;email&quot;)<br>                await user_service.create_user(<br>                    User_instance=UserSchemas.User(email=email),<br>                    eventtype_prefix=&quot;user_created_from_new_subscription&quot;,<br>                    payload_add=payload<br>                )<br>        except ResourceAlreadyExistsException:<br>            raise<br>        except Exception as e:<br>            logger.exception(f&quot;Error creating user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Error creating user: {str(e)}&quot;) from e</pre><ol><li>Implements logic for handling a &quot;subscription_created_success&quot; event.</li><li>Extracts the email, creates a user using the service layer, and handles exceptions.</li><li>Creates a new short-lived <em>Session</em> to interact with the database.</li></ol><pre>class KafkaEventManager:<br>    &quot;&quot;&quot;Manages Kafka event consumption and routing to appropriate handlers&quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        self.consumer = None<br>        self.tasks = []<br>        self.event_handlers: Dict[str, EventHandler] = {}<br>        <br>    def register_handler(self, event_type: str, handler: EventHandler) -&gt; None:<br>        &quot;&quot;&quot;Register a handler for a specific event type&quot;&quot;&quot;<br>        self.event_handlers[event_type] = handler<br>        logger.info(f&quot;Registered handler for event type: {event_type}&quot;)</pre><ol><li>Initializes the event manager, which will manage Kafka consumers and event handlers.</li><li>Allows registering handlers for specific event types.</li></ol><pre>class KafkaEventManager:<br>    &quot;&quot;&quot;Manages Kafka event consumption and routing to appropriate handlers&quot;&quot;&quot;<br><br>  (...)<br><br>  async def start(self, topics: List[str], bootstrap_servers: str, group_id: str) -&gt; None:<br>          self.consumer = AIOKafkaConsumer(<br>              *topics,<br>              bootstrap_servers=bootstrap_servers,<br>              group_id=group_id,<br>              value_deserializer=lambda m: json.loads(m.decode(&#39;utf-8&#39;)),<br>              enable_auto_commit=KAFKA_AUTO_COMMIT,<br>              max_poll_interval_ms=KAFKA_MAX_POLL_INTERVAL_MS,<br>              session_timeout_ms=KAFKA_SESSION_TIMEOUT_MS,<br>              auto_offset_reset=&quot;earliest&quot;<br>          )<br>          await self.consumer.start()<br>          logger.info(f&quot;Kafka consumer started for topics: {topics}&quot;)<br>          for i in range(3):<br>              task = asyncio.create_task(self._consume(i))<br>              self.tasks.append(task)</pre><ol><li>Configures and starts the Kafka consumer asynchronously.</li><li>Starts multiple worker tasks for parallel message processing.</li></ol><pre>class KafkaEventManager:<br>    &quot;&quot;&quot;Manages Kafka event consumption and routing to appropriate handlers&quot;&quot;&quot;<br><br>  (...)<br><br>  async def _consume(self, worker_id: int) -&gt; None:<br>          logger.info(f&quot;Starting consumer worker {worker_id}&quot;)<br>          try:<br>              async for msg in self.consumer:<br>                  try:<br>                      event = msg.value<br>                      topic = msg.topic<br>                      event_type = event.get(&quot;payload&quot;, {}).get(&quot;type&quot;)<br>                      payload = event.get(&quot;payload&quot;, {}).get(&quot;payload&quot;, {})<br>                      logger.info(f&quot;Worker {worker_id} received event from topic {topic}: {event_type}&quot;)<br>                      await self._process_event(event_type, json.loads(payload))<br>                  except Exception as e:<br>                      logger.error(f&quot;Error processing message: {e}&quot;, exc_info=True)<br>          except asyncio.CancelledError:<br>              logger.info(f&quot;Consumer worker {worker_id} cancelled&quot;)<br>          except Exception as e:<br>              logger.error(f&quot;Consumer worker {worker_id} error: {e}&quot;, exc_info=True)<br>          finally:<br>              logger.info(f&quot;Consumer worker {worker_id} shutting down&quot;)</pre><ol><li>Each worker continuously consumes messages from Kafka.</li><li>Extracts the event type and payload from each message.</li><li>Routes each event to the appropriate handler.</li><li>Handles errors and logs them.</li></ol><pre>class KafkaEventManager:<br>    &quot;&quot;&quot;Manages Kafka event consumption and routing to appropriate handlers&quot;&quot;&quot;<br><br>  (...)<br><br>  async def _process_event(self, event_type: str, payload: Dict[str, Any]) -&gt; None:<br>          handler = self.event_handlers.get(event_type)<br>          if handler:<br>              try:<br>                  await handler.handle(payload)<br>              except Exception as e:<br>                  logger.error(f&quot;Error in handler for {event_type}: {e}&quot;, exc_info=True)<br>          else:<br>              logger.warning(f&quot;No handler registered for event type: {event_type}&quot;)</pre><ol><li>Looks up and invokes the correct handler for each event type.</li><li>Logs a warning if no handler is registered.</li></ol><pre>class KafkaEventManager:<br>    &quot;&quot;&quot;Manages Kafka event consumption and routing to appropriate handlers&quot;&quot;&quot;<br><br>  (...)<br><br>  async def stop(self) -&gt; None:<br>          logger.info(&quot;Stopping Kafka event manager...&quot;)<br>          for task in self.tasks:<br>              if not task.done():<br>                  task.cancel()<br>          if self.tasks:<br>              await asyncio.gather(*self.tasks, return_exceptions=True)<br>              self.tasks = []<br>          if self.consumer:<br>              await self.consumer.stop()<br>              self.consumer = None<br>          logger.info(&quot;Kafka consumer stopped&quot;)</pre><ol><li>Cancels all running consumer tasks.</li><li>Stops the Kafka consumer gracefully.</li></ol><pre># Create event manager instance<br>event_manager = KafkaEventManager()<br><br><br>async def setup_kafka_handlers():<br>    &quot;&quot;&quot;Initialize and start Kafka event handlers&quot;&quot;&quot;<br>    # Register event handlers<br>    event_manager.register_handler(&quot;subscription_created_success&quot;, SubscriptionCreatedSuccessHandler())<br>    <br>    # Start the Kafka consumer<br>    await event_manager.start(<br>        topics=KAFKA_TOPICS,<br>        bootstrap_servers=KAFKA_BOOTSTRAP_SERVERS,<br>        group_id=KAFKA_CONSUMER_GROUP<br>    )</pre><ol><li>Instantiates the event manager.</li><li>Registers the event handler for &quot;subscription_created_success&quot;.</li><li>Starts the Kafka consumer.</li></ol><p><strong>/src/main.py</strong></p><pre>(...)<br>from src.consumer.kafka import event_manager, setup_kafka_handlers<br><br>setup_logging()<br>logger = logging.getLogger(__name__)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    logger.info(&quot;Running startup tasks...&quot;)<br><br>    (...)<br><br>    # Start Kafka consumer as a background task<br>    await setup_kafka_handlers()<br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    # Shutdown: stop Kafka consumer gracefully<br>    logger.info(&quot;Stopping Kafka consumer...&quot;)<br>    await event_manager.stop()<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>(...)</pre><p>Finally, we start our consumer on start up, and close it on shutdown.</p><p>This wraps up how we can implement the SAGA Choreography pattern. We have now designed a robust event-driven system where our microservices are loosely coupled.</p><p>This also means the end of this series, where we’ve covered a vast amount of concepts when building enterprise-grade microservices. Building a functional FastAPI microservice is extremely easy, but to actually make it robust and scalable enough to work in the real world, a lot of stuff needs to be considered. We’ve also covered external systems: PostgreSQL, DynamoDB, Kafka and Debezium. These external systems are widely used and understanding them, and how they integrate with FastAPI applications, is therefore highly relevant in 2025. Finally, we’ve used Docker for running everything locally so we can develop complex enterprise-grade systems locally, and easily ship our containers to a cloud provider when it’s time to deploy.</p><p>I hope you’ve enjoyed the series!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeb9781b00a6" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-10-10-kafka-saga-choreography-aeb9781b00a6">Building Enterprise Python Microservices with FastAPI in 2025 (10/10): Kafka &amp; SAGA Choreography…</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (9/10): Debezium & Outbox Table…]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-9-10-debezium-outbox-table-76b14710bd71?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/76b14710bd71</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[outbox-pattern]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Tue, 20 May 2025 03:25:17 GMT</pubDate>
            <atom:updated>2025-05-21T15:43:07.163Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Enterprise Python Microservices with FastAPI in 2025 (9/10): Debezium &amp; Outbox Table Pattern</h3><h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous article we discussed event-driven architecture to make our microservices loosely coupled. We also discussed common challenges and patterns to address these. In this article we are going to implement the SAGA Choreography pattern and the Outbox Table pattern using <em>Kafka</em> as our broker and <em>Debezium</em> for CDC. This is a common enterprise-grade stack when dealing with PostgreSQL database backends.</p><p>Debezium is not compatible with DynamoDB, so for CDC with DynamoDB, we would use DynamoDB Streams in combination with AWS Lambda, for event distribution we could then use Kafka or SNS and SQS. In this article, we focus on the Kafka and Debezium stack with PostgreSQL.</p><h3>Outbox Table Definition &amp; OutboxORM</h3><p>Firstly we are going to define our table, and the relating ORM.</p><p><strong>Table Definition</strong></p><pre>-- Create the schema if not exists<br>CREATE SCHEMA IF NOT EXISTS auth;<br><br>-- Create the users table<br>CREATE TABLE IF NOT EXISTS auth.users (<br>    email VARCHAR(50) PRIMARY KEY,<br>    hashed_password VARCHAR(200), -- hashed (including salt)<br>    is_active BOOLEAN<br>);<br><br>-- create the users outbox table<br>CREATE TABLE IF NOT EXISTS auth.users_outbox (<br>    id UUID PRIMARY KEY,<br>    aggregatetype TEXT NOT NULL,<br>    aggregateid TEXT NOT NULL,<br>    eventtype TEXT NOT NULL,<br>    payload JSONB NOT NULL,<br>    created_at BIGINT NOT NULL<br>);</pre><p>The outbox table consist of the following:</p><ol><li><em>id</em> — unique id for each record</li><li><em>aggregatetype</em> — used as topic suffix</li><li><em>aggregateid</em> — user id, in our case the user’s email</li><li><em>eventtype</em> — type of event, events with the same topic might have different event types. This way consumers can subscribe to one topic, and handle events differently based on the event type.</li><li><em>payload</em> — the data related to the event as JSON.</li><li><em>created_at</em> — UNIX timestamp in milliseconds</li></ol><p>This table definition is highly dependent on our choice of tool for CDC. In our case this table definition is heavily inspired by the official <a href="https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html">Debezium documentation</a>.</p><p><strong>/src/repository/implementations/PostgreSQL/models/ORM_User.py</strong></p><pre>import uuid6<br>from datetime import datetime ,timezone<br>from sqlalchemy import Column, String, Boolean, BigInteger, JSON<br>from sqlalchemy.dialects.postgresql import UUID<br>from sqlalchemy.ext.declarative import declarative_base<br><br>Base = declarative_base()<br><br>class UserORM(Base):<br>    __tablename__ = &quot;users&quot;<br>    __table_args__ = {&quot;schema&quot;: &quot;auth&quot;}<br>    email = Column(String, primary_key=True, unique=True, index=True, nullable=False)<br>    hashed_password = Column(String, nullable=True)<br>    is_active = Column(Boolean, default=True)<br><br>class UsersOutboxORM(Base):<br>    __tablename__ = &quot;users_outbox&quot;<br>    __table_args__ = {&quot;schema&quot;: &quot;auth&quot;}  # Specifies the schema<br><br>    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid6.uuid6)<br>    aggregatetype = Column(String, nullable=False)     # e.g., &quot;user&quot; -&gt; TOPIC<br>    aggregateid = Column(String, nullable=False)       # e.g., user_id<br>    eventtype = Column(String, nullable=False)              # e.g., &quot;user_created_success&quot;<br>    payload = Column(JSON, nullable=False)             # Event data as JSON<br>    created_at = Column(BigInteger, nullable=False, default=lambda: int(datetime.now(timezone.utc).timestamp() * 1000))</pre><p>Similar to how we created our UserORM , we must also define a UserOutboxORM to interact with our Outbox table in a pythonic manner. Note that we are using default values for id and created_at , as these should just be set automatically when we create a record.</p><p><strong>/src/schemas/UserSchemas.py:</strong></p><pre>(...)<br><br>class User(BaseModel):<br>    email: str<br>    hashed_password: Optional[str] = None<br>    is_active: Optional[bool] = None<br><br>class UserResponse(BaseModel):<br>    email: str<br>    is_active: Optional[bool] = None<br><br>class Outbox(BaseModel):<br>    aggregatetype: str<br>    aggregateid: str<br>    eventtype_prefix: str<br>    payload: Dict[str, Any]</pre><p>We also create our Outbox model, as we are now not only creating a User , but also an Outbox record when we insert a User in our database.</p><p><strong>/src/service/UserService.py</strong></p><pre>(...)<br><br>class UserService:<br><br>    def __init__(self, user_repository: interface_UserRepository):<br>        self.user_repository = user_repository<br>    <br>    (...)<br>        <br>    async def create_user(<br>            self,<br>            User_instance: UserSchemas.User,<br>            eventtype_prefix: str,<br>            payload_add: Dict[str, Any] = None<br>        ) -&gt; None:<br>        try:<br>            <br>            # Create the user<br>            await self.user_repository.create_user(<br><br>                User_instance = User_instance,<br><br>                Outbox_instance = UserSchemas.Outbox(<br>                    aggregatetype = &quot;user&quot;,<br>                    aggregateid = User_instance.email,<br>                    eventtype_prefix = eventtype_prefix,<br>                    payload = {<br>                        &quot;email&quot;: User_instance.email,<br>                        &quot;is_active&quot;: True if User_instance.is_active else False,<br>                        **(payload_add or {})<br>                    }<br>                )<br>            )<br><br>        except ResourceAlreadyExistsException:<br>            raise<br>        except Exception as e:<br>            logger.exception(f&quot;Error creating user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Error creating user: {str(e)}&quot;) from e<br><br>(...)</pre><p>In our create_user function we are now calling user.repository.create_user with not only the User instance as argument, but also an Outbox instance. This effectively tells our database backend, which records to put in the database.</p><p><strong>/src/repository/implementations/PostgreSQL/postgres_UserRepository.py</strong></p><pre>(...)<br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, db: AsyncSession):<br>        self.db = db<br><br>    (...)<br><br>    async def create_user(<br>            self,<br>            User_instance: UserSchemas.User,<br>            Outbox_instance: UserSchemas.Outbox<br>        ) -&gt; None:<br><br>        try:<br>            db_user = UserORM(<br>                email=User_instance.email,<br>                hashed_password=User_instance.hashed_password,<br>                is_active=True if User_instance.is_active else False<br>            )<br><br>            outbox_event = UsersOutboxORM(<br>                aggregatetype = Outbox_instance.aggregatetype,<br>                aggregateid = Outbox_instance.aggregateid,<br>                eventtype = f&quot;{Outbox_instance.eventtype_prefix}_success&quot;,<br>                payload = Outbox_instance.payload<br>            )<br><br>            # Start transaction<br>            async with self.db.begin():  # This ensures atomicity<br>                self.db.add(db_user)<br>                self.db.add(outbox_event)<br>        <br>        except IntegrityError as e:<br>            if &quot;UniqueViolationError&quot; in str(e.orig):<br>                logger.warning(f&quot;User with email {User_instance.email} already exists&quot;)<br><br>                fail_event = UsersOutboxORM(<br>                    aggregatetype = Outbox_instance.aggregatetype,<br>                    aggregateid = Outbox_instance.aggregateid,<br>                    eventtype = f&quot;{Outbox_instance.eventtype_prefix}_failed&quot;,<br>                    payload = Outbox_instance.payload.update({<br>                        &quot;exception&quot;: &quot;ResourceAlreadyExistsException&quot;<br>                    })<br>                )<br><br>                async with self.db.begin():<br>                    self.db.add(fail_event)<br><br>                raise ResourceAlreadyExistsException(f&quot;User with email {User_instance.email} already exists&quot;)<br>            else:<br>                # Some other kind of IntegrityError (e.g., null value, foreign key constraint, etc)<br>                logger.exception(f&quot;Error creating user: {str(e)}&quot;)<br><br>                fail_event = UsersOutboxORM(<br>                    aggregatetype = Outbox_instance.aggregatetype,<br>                    aggregateid = Outbox_instance.aggregateid,<br>                    eventtype = f&quot;{Outbox_instance.eventtype_prefix}_failed&quot;,<br>                    payload = Outbox_instance.payload.update({<br>                        &quot;exception&quot;: &quot;BaseAppException&quot;<br>                    })<br>                )<br><br>                async with self.db.begin():<br>                    self.db.add(fail_event)<br><br>                raise BaseAppException(f&quot;Database integrity error: {str(e)}&quot;) from e<br>            <br>        except ResourceAlreadyExistsException:<br>            raise<br><br>        except Exception as e:<br>            logger.exception(f&quot;Error creating user: {str(e)}&quot;)<br><br>            fail_event = UsersOutboxORM(<br>                aggregatetype = Outbox_instance.aggregatetype,<br>                aggregateid = Outbox_instance.aggregateid,<br>                eventtype = f&quot;{Outbox_instance.eventtype_prefix}_failed&quot;,<br>                payload = Outbox_instance.payload.update({<br>                    &quot;exception&quot;: &quot;BaseAppException&quot;<br>                })<br>            )<br><br>            async with self.db.begin():<br>                self.db.add(fail_event)<br><br>            raise BaseAppException(f&quot;Internal database error: {str(e)}&quot;) from e</pre><p>This is where we actually write to our auth.users and auth.users_outbox in a transactional manner. We also write to auth.users_outbox table in the case where we fail to write to these tables. When we fail to write to our database, we append the exception to the payload. This means consumers of this event are able to handle these failures based on not only the event type, but also the exception when a fail event occurs. For example if we try to create a User that already exists, when our User Service consumes a subcription_created_success event, we do not want our Subscription Service to perform a roll back and delete the subscription. But if something else goes wrong, e.g. a database connection error, we would want a roll back action.</p><p>We only need to care about writing to the Outbox table, because Debezium will take care of capturing the insertion and publish an event to Kafka.</p><p>Now we have set up auth.users_outboxtable. In the next section we are going to configure Debezium, so we are publishing messages to Kafka, when records are inserted in the auth.users_outbox table.</p><h3>Debezium Setup</h3><p><strong>Database Privileges</strong><br>To make Debezium work, we must grant a database user the correct privileges, following the principle of least privilege.</p><pre>-- Start transaction for atomicity<br>BEGIN;<br><br>-- Create roles<br>(...)<br>CREATE ROLE debezium_user WITH REPLICATION LOGIN PASSWORD &#39;secure_debezium_password&#39;;<br><br>-- Allow connection to the database<br>(...)<br>GRANT CONNECT ON DATABASE crypto_db TO debezium_user;<br><br>-- Allow usage of the &#39;auth&#39; schema<br>(...)<br>GRANT USAGE ON SCHEMA auth TO debezium_user;<br><br>(...)<br><br>-- Pre-create publication for Debezium (execute as superuser)<br>CREATE PUBLICATION dbz_publication_user FOR TABLE auth.users_outbox;<br>CREATE PUBLICATION dbz_publication_subscription FOR TABLE auth.subscriptions_outbox;<br><br>-- Grant debezium_user minimal privileges needed for CDC<br>GRANT SELECT ON TABLE auth.users_outbox TO debezium_user;<br>GRANT SELECT ON TABLE auth.subscriptions_outbox TO debezium_user;<br><br>(...)<br><br>-- Commit all changes<br>COMMIT;</pre><ol><li>We create the debezium_user with the REPLICATION attribute, allowing it to initiate logical replication connections and read the database’s Write-Ahead Log (WAL)-essential for Debezium’s CDC process.</li><li>We allow debezium_user to connect to our database, crypto_db , and usage of the auth schema, which includes our auth.users_outbox table.</li><li>We create a <em>publication</em> for each of the two outbox tables: auth.users_outbox and auth.subscriptions_outbox . PostgreSQL uses the concept of publications and subscriptions for logical replication. A publication is a set of tables whose changes are made available for replication. Debezium acts as a logical replication subscriber. It connects to the database, subscribes to these publications, and streams changes to downstream systems (like Kafka). By explicitly listing tables in the publication, we ensure only relevant data changes are exposed to Debezium, following the principle of least privilege.</li><li>We grant debezium_user the ability to read (SELECT) from the auth.users_outbox and auth.subscriptions_outbox tables. Debezium needs to read the contents of these tables to capture and stream changes. Restricting privileges to only SELECT on these specific tables follows the principle of least privilege, reducing security risks</li></ol><p>Now we have created debezium_user , which is now able to capture changes to our Outbox tables.</p><h4>Debezium Configuration</h4><p><strong>/src/db/settings.py</strong></p><pre>(...)<br><br>class DatabaseType(str, Enum):<br>    POSTGRES = &quot;postgres&quot;<br>    DYNAMODB = &quot;dynamodb&quot;<br><br>class Settings(BaseSettings):<br>    DATABASE_TYPE: DatabaseType = DatabaseType.POSTGRES # Default to postgres<br><br>    # --------------------------------------------------------------------<br>    # Postgres settings<br>    POSTGRES_DATABASE_URL: str = &quot;postgresql+asyncpg://user:password@postgresql:5432/crypto_db&quot;<br>    POSTGRES_DATABASE_URL_FOR_TESTING: str = &quot;postgresql+asyncpg://postgres:password@postgresql:5432/crypto_db&quot;<br>    <br>    # Debezium settings (only related to postgres)<br>    DB_HOST: str = &quot;postgresql&quot;<br>    DB_PORT: str = &quot;5432&quot;<br>    DB_USER: str = &quot;user_service_user&quot;<br>    DB_PASSWORD: str = &quot;super_secure_password&quot;<br>    DB_NAME: str = &quot;crypto_db&quot;<br>    DEBEZIUM_URL: str = &quot;http://debezium:8083/connectors&quot;<br>    # --------------------------------------------------------------------<br><br>    (...)<br>    <br>    model_config = SettingsConfigDict(<br>        env_file = &quot;.env&quot;,<br>        case_sensitive = True<br>    )<br><br><br>@lru_cache()<br>def get_settings() -&gt; Settings:<br>    return Settings()</pre><p>In our Settings we are now storing connection configuration for our Debezium connector. We need this to create a Debezium <em>connector</em> on start up. Let’s go through our <em>connector </em>configuration to understand it better.</p><p><strong>/src/repository/implementations/PostgreSQL/debezium_config.py</strong></p><pre>async def generate_config_dict(settings):<br>    return {<br>        &quot;name&quot;: &quot;users-outbox-connector&quot;,<br>        &quot;config&quot;: {<br>            &quot;connector.class&quot;: &quot;io.debezium.connector.postgresql.PostgresConnector&quot;,<br>            &quot;tasks.max&quot;: &quot;1&quot;,<br>            &quot;database.hostname&quot;: settings.DB_HOST,<br>            &quot;database.port&quot;: settings.DB_PORT,<br>            &quot;database.user&quot;: settings.DB_USER,<br>            &quot;database.password&quot;: settings.DB_PASSWORD,<br>            &quot;database.dbname&quot;: settings.DB_NAME,<br>            &quot;database.server.name&quot;: &quot;userservice&quot;,<br>            &quot;schema.include.list&quot;: &quot;auth&quot;,<br>            &quot;table.include.list&quot;: &quot;auth.users_outbox&quot;,<br>            &quot;plugin.name&quot;: &quot;pgoutput&quot;,<br>            &quot;publication.name&quot;: &quot;dbz_publication_user&quot;,<br>            &quot;tombstones.on.delete&quot;: &quot;false&quot;,<br>            &quot;topic.prefix&quot;: &quot;userservice&quot;,<br>            &quot;slot.name&quot;: &quot;debezium_user&quot;,<br><br>            &quot;transforms&quot;: &quot;outbox&quot;,<br>            &quot;transforms.outbox.type&quot;: &quot;io.debezium.transforms.outbox.EventRouter&quot;,<br>            &quot;transforms.outbox.route.by.field&quot;: &quot;aggregatetype&quot;,<br>            &quot;transforms.outbox.route.topic.replacement&quot;: &quot;userservice.${routedByValue}&quot;,<br><br>            &quot;transforms.outbox.field.event.id&quot;: &quot;id&quot;,<br>            &quot;transforms.outbox.field.event.key&quot;: &quot;aggregateid&quot;,<br>            &quot;transforms.outbox.field.event.type&quot;: &quot;eventtype&quot;,<br>            &quot;transforms.outbox.field.event.payload&quot;: &quot;payload&quot;,<br>            &quot;transforms.outbox.field.event.timestamp&quot;: &quot;created_at&quot;,<br>            &quot;transforms.outbox.field.event.timestamp.type&quot;: &quot;io.debezium.time.Timestamp&quot;,<br><br>            &quot;transforms.outbox.expand.json.payload&quot;: &quot;true&quot;,<br>            &quot;transforms.outbox.table.fields.additional.placement&quot;: &quot;eventtype:envelope:type&quot;<br>        }<br>    }</pre><p>This function generates, using our Settings , our <em>connector</em> configuration. In our connector we use our debezium_user to connect to the database, and we subscribe to the dbz_publication_user to track insertions in auth.users_outbox . We configure Debezium’s EventRouter SMT to dynamically route events to Kafka topics based on the aggregatetype field (e.g., userservice.user), and include eventtype in the payload for downstream consumers to distinguish event types.</p><p>For example, if we successfully insert the following record in auth.users :</p><pre>{<br>  email: &#39;test@example.com&#39;,<br>  hashed_password: None,<br>  is_active: False<br>}</pre><p>We would insert the following record in auth.users_outbox :</p><pre>{<br>  id: &#39;some_unique_id&#39;,<br>  aggregatetype: &#39;user&#39;,<br>  aggregateid: &#39;test@example.com&#39;,<br>  eventtype: &#39;user_created_from_new_subscription_success&#39;,<br>  payload: {<br>    email: &#39;test@example.com&#39;,<br>    is_active: False,<br>    subscription_id: &#39;unique_subscription_id&#39;<br>  },<br>  created_at = 14682387084298419 # dummy integer<br>}</pre><p>And, with our configuration, this is the event our consumers will consume:</p><pre>ConsumerRecord(<br>    topic=&#39;userservice.user&#39;,<br>    partition=0,<br>    offset=0,<br>    timestamp=1746240679362,<br>    timestamp_type=0,<br>    key=b&#39;{&quot;schema&quot;:{&quot;type&quot;:&quot;string&quot;,&quot;optional&quot;:false},&quot;payload&quot;:&quot;dummy@email.com&quot;}&#39;,<br>    value={<br>        &#39;schema&#39;: {<br>            &#39;type&#39;: &#39;struct&#39;,<br>            &#39;fields&#39;: [<br>                {&#39;type&#39;: &#39;string&#39;, &#39;optional&#39;: False, &#39;name&#39;: &#39;io.debezium.data.Json&#39;, &#39;version&#39;: 1, &#39;field&#39;: &#39;payload&#39;},<br>                {&#39;type&#39;: &#39;string&#39;, &#39;optional&#39;: False, &#39;field&#39;: &#39;type&#39;}<br>            ],<br>            &#39;optional&#39;: False,<br>            &#39;name&#39;: &#39;userservice.auth.users_outbox.user.Value&#39;<br>        },<br>        &#39;payload&#39;: {<br>            &#39;payload&#39;: &#39;{&quot;email&quot;: &quot;dummy@email.com&quot;, &quot;is_active&quot;: False, &quot;subscription_id&quot;: &quot;unique_subscription_id&quot;}&#39;,<br>            &#39;type&#39;: &#39;user_created_from_new_subscription_success&#39;<br>        }<br>    },<br>    checksum=None,<br>    serialized_key_size=73,<br>    serialized_value_size=360,<br>    headers=((&#39;id&#39;, b&#39;1f027c97-d169-63a8-9cbf-56b86948d5eb&#39;),)<br>)</pre><p>As shown, a consumer can use the eventtype and payload fields to determine the nature of the event (e.g., user_created_from_new_subcription_success) and access the relevant user data. For the purpose of this series, we are not going to dive deeper into specific Debezium configuration — we just need to be aware about the structure of the events we are going to consume to handle them properly.</p><p><strong>/src/main.py</strong></p><pre>(...)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    logger.info(&quot;Running startup tasks...&quot;)<br><br>    # Initialize settings<br>    settings = get_settings()<br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        # Create an async engine<br>        engine = create_async_engine(settings.POSTGRES_DATABASE_URL, echo=True, future=True)<br><br>        # Create an async sessionmaker factory<br>        app.state.postgres_session = async_sessionmaker(<br>            bind=engine,<br>            expire_on_commit=False,  # optional: objects stay active after commit<br>            class_=AsyncSession<br>        )<br><br>        # Create Debezium connector<br>        connector_config = await generate_config_dict(<br>            settings=settings<br>        )<br>        async with httpx.AsyncClient() as client:<br>            resp = await client.post(settings.DEBEZIUM_URL, json=connector_config)<br>            if resp.status_code not in {201, 409}: # Created (201) or Already Exists (409)<br>                raise RuntimeError(f&quot;Failed to register Debezium connector: {resp.status_code} {resp.text}&quot;)<br>        <br>    (...)<br><br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>(...)</pre><p>When we start our application we make a POST request to Debezium, to create the <em>connector</em> defined in our configuration. If the connector is successfully created, it means Debezium now “listens” to insertions in the auth.users_outbox tables and publishes events, in the way we configured, to Kafka when insertions occur.</p><p>This also means we have implemented the Outbox Table pattern using Debezium for CDC. Naturally, we will need to connect to a running instance of Debezium, which can be achieved using Docker. Check out my <a href="https://github.com/assibering/CryptoWebApp/blob/main/docker-compose.yaml">compose file</a> for reference.</p><p>This also wraps up how we can implement the Outbox Table Pattern when using PostgreSQL and Debezium for CDC. By using tools like Debezium, and not just developing our own CDC logic, we get a much more robust solution proven in other enterprise-grade applications.</p><p>A good rule of thumb in a modern enterprise-grade stack is to look for popular external tools, before developing them yourself. Even though it introduces added complexity and more maintenance, it will almost always be a more robust and more secure solution.</p><p>Now that we are able to publish messages to Kafka using the Outbox Table pattern, we are going to focus on consuming these messages in the next article. Additionally, we are going to use an example from my hobby project for reference, to see how our implementation of the SAGA Choreography- and Outbox Table patterns work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76b14710bd71" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-9-10-debezium-outbox-table-76b14710bd71">Building Enterprise Python Microservices with FastAPI in 2025 (9/10): Debezium &amp; Outbox Table…</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (8/10): SAGA- and Outbox Pattern]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-8-10-saga-and-outbox-pattern-0f4cae857fc3?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f4cae857fc3</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Tue, 20 May 2025 03:24:16 GMT</pubDate>
            <atom:updated>2025-05-21T15:43:19.310Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>Before we make the actual implementation, we are going to discuss how we can achieve loose coupling between microservices using event-driven architecture. First we are going to introduce some relevant topics to make a general understanding.</p><h3>Central Concepts in Event-Driven Architecture</h3><p><strong>Message broker<br></strong>A <em>message broker</em> is central to a pub/sub system: it receives incoming <em>events</em> published by <em>producers</em> and routes them to the appropriate <em>queues</em>, which <em>consumers</em> can subscribe to.</p><p><strong>Event<br></strong>An <em>event</em> typically consists of a <em>topic</em> and a <em>payload</em>. (Events can have additional metadata, but for simplicity, we’ll focus on these two parts.) The topic defines which <em>queue</em> the event goes into, i.e. a consumer will <em>subscribe</em> to specific topics and the <em>payload</em> is the data related to the event.</p><p>For example, when we create a subscription, we might publish an event with the topic created_subscription_success and the new subscription data (such as subscription_id and email) as the payload. Now other services <em>consuming</em> these created_subscription_success events, will now be able to react to the creation of a subscription. For example we our User microservice consumes this event, and creates a new user.</p><p><strong>Producer &amp; Consumer<br></strong>In an event-driven system, a <em>producer</em> (or <em>publisher</em>) is an entity that sends events to the broker. A <em>consumer</em> <em>subscribes</em> to a <em>queue</em> and processes incoming events.</p><p><strong>Queue<br></strong>A <em>queue</em> is defined by a <em>topic</em> and a <em>group ID</em>. When a broker receives an event, it places the event into all queues matching the event’s topic. This means we can have multiple queues representing the same topic, but with different group IDs. The purpose of the group ID is then to ensure:</p><ol><li>multiple instances of the same service are running, only one instance processes each event. This prevents duplicate processing and potential conflicts. The group ID ensures that all instances of the same service, subscribe to the same queue, making it impossible to pull the same event twice.</li><li>If multiple different services subscribe to the same topic, each service receives its own copy of the event. The group ID allows each service to have its own queue for the topic, and the broker ensures the event is delivered to all relevant queues.</li></ol><h3>Challenges</h3><p>In event-driven systems we face some common challenges: The dual write problem, and distributed transactions.</p><p><strong>Distributed Transactions<br></strong>The core idea of event-driven architecture is that a microservice performs an action and then informs other microservices about it, so they can react accordingly. A simple example is when our User Service create a user, our Subscription Service must create a subscription. The problem arises if our Subscription Service fails to create the subscription: we end up with an inconsistent state, where a user exists but their subscription does not. This is where the SAGA Choreography pattern becomes relevant: it enables us to roll back previous operations across microservices, helping to solve the distributed transaction problem.</p><p><strong>Dual Write Problem<br></strong>The dual write problem occurs when we want to update two external systems atomically. For example, we might want to write to the database and publish an event to the broker as a single, reliable operation. An example could be we want to create a subscription, and publish a created_subscription_success event. The problem occurs when one of the operations fail, i.e. we successfully create the subscription, but we are unable to publish created_subscription_success event — this could simply be because the broker is down. This is where the Outbox Table pattern becomes relevant, as it ensures this behaviour does not occur.</p><h3>SAGA Choreography Pattern</h3><p>Put simply, in the SAGA Choreography pattern each local transaction publishes domain events, which then trigger related transactions in other services. This means we can achieve rollbacks by publishing events that trigger compensating actions in other services. For example, if our User Service publishes a user_created_from_new_subscription_failed event, our Subscription Service can consume this event and delete the related subscription, effectively rolling back the previous insertion. This way we ensure database consistency across multiple services.</p><h3>Outbox Table Pattern</h3><p>To address the dual write problem, the Outbox Table pattern introduces a dedicated outbox table in the database to store events. This way we can make a database operation and write to the outbox table in atomic manner. We then use <em>CDC</em> (Change Data Capture) to read from the outbox table and ensure each event is published to the broker. This way we ensure that each database operation has a related event in the outbox table. If CDC fails to publish an event, we can implement retry logic or compensating actions to maintain consistency. Therefore we ensure consistency across our two systems by implementing the outbox pattern.</p><p>In this article, we explored how to achieve loose coupling between microservices using event-driven architecture. We covered common challenges and how patterns like SAGA Choreography and Outbox Table can address them. In the next article, we’ll dive into the actual implementation with Kafka and Debezium in our FastAPI microservice architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f4cae857fc3" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-8-10-saga-and-outbox-pattern-0f4cae857fc3">Building Enterprise Python Microservices with FastAPI in 2025 (8/10): SAGA- and Outbox Pattern</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (7/10): Unit- & Integration Testing]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-7-10-unit-integration-testing-c4a1d46594b6?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4a1d46594b6</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Tue, 20 May 2025 03:22:35 GMT</pubDate>
            <atom:updated>2025-05-21T15:45:20.113Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous articles we’ve focused on developing our microservice, following the Repository Pattern. We have achieved loose coupling wrt. database implementation, making it possible to easily swap out databases down the line. Another great thing about loose coupling, is when it comes to writing unit tests for our service, as you shall see in this article.</p><p>In addition to unit testing, we’ll also focus on integration testing our microservice in isolation — not distributed integration testing across multiple microservices. Since this series is about building an enterprise-grade microservice, we’ll focus on isolated tests; distributed tests are out of scope for now.</p><h3>Project Architecture</h3><p>We’re now adding our testsfolder. For reference, here’s the updated project structure. In this article we are going through everything that happens in our tests folder, and how we can extend our Dockerfile.dev to run tests automatically.</p><pre>├── Dockerfile.dev<br>├── Dockerfile.prod<br>├── README.md<br>├── docker-compose.yaml<br>├── pyproject.toml<br>├── src<br>│   ├── __init__.py<br>│   ├── db<br>│   │   ├── __init__.py<br>│   │   ├── db_context.py<br>│   │   ├── factory.py<br>│   │   └── settings.py<br>│   ├── dependencies.py<br>│   ├── exceptions.py<br>│   ├── logging_config.py<br>│   ├── main.py<br>│   ├── middleware<br>│   │   ├── __init__.py<br>│   │   ├── correlation_context.py<br>│   │   └── correlation_id_middleware.py<br>│   ├── repository<br>│   │   ├── implementations<br>│   │   │   ├── AWS_DynamoDB<br>│   │   │   │   ├── __init__.py<br>│   │   │   │   ├── awsdynamodb_UserRepository.py<br>│   │   │   │   └── utils.py<br>│   │   │   ├── AWS_RDS<br>│   │   │   └── PostgreSQL<br>│   │   │       ├── __init__.py<br>│   │   │       ├── debezium_config.py<br>│   │   │       ├── models<br>│   │   │       │   ├── ORM_User.py<br>│   │   │       │   └── __init__.py<br>│   │   │       └── postgres_UserRepository.py<br>│   │   └── interfaces<br>│   │       ├── __init__.py<br>│   │       └── interface_UserRepository.py<br>│   ├── routes<br>│   │   ├── UserController.py<br>│   │   └── __init__.py<br>│   ├── schemas<br>│   │   ├── UserSchemas.py<br>│   │   └── __init__.py<br>│   └── service<br>│       ├── UserService.py<br>│       ├── __init__.py<br>│       └── utils.py<br>├── tests<br>│   ├── __init__.py<br>│   ├── integration<br>│   │   ├── __init__.py<br>│   │   ├── conftest.py<br>│   │   └── test_api.py<br>│   └── unit<br>│       ├── __init__.py<br>│       ├── repository<br>│       │   ├── __init__.py<br>│       │   └── test_postgres_UserRepository.py<br>│       ├── test_user_controller.py<br>│       └── test_user_service.py<br>└── uv.lock</pre><h3>Unit tests</h3><p>First we are going to look at our unit tests, and we are going to start from the controller layer, then the service layer, and finally the repository layer.</p><p><strong>/tests/unit/test_user_controller.py</strong></p><pre>from src.dependencies import get_user_service<br>import pytest<br>from fastapi.testclient import TestClient<br>from unittest.mock import AsyncMock<br>from src.main import app<br><br><br>client = TestClient(app)<br><br>@pytest.fixture<br>def mock_user_service():<br>    &quot;&quot;&quot;Create a mock UserService for dependency injection.&quot;&quot;&quot;<br>    mock_service = AsyncMock()<br>    return mock_service<br><br>@pytest.fixture<br>def sample_user_response_inactive():<br>    &quot;&quot;&quot;Create a sample inactive UserResponse&quot;&quot;&quot;<br>    from src.schemas import UserSchemas<br>    return UserSchemas.UserResponse(<br>        email=&quot;test@example.com&quot;,<br>        is_active=False<br>    )<br><br>(...)<br><br># Tests for get_user method<br>@pytest.mark.asyncio<br>async def test_get_user_success(mock_user_service, sample_user_response_inactive):<br>    &quot;&quot;&quot;Test successful user retrieval.&quot;&quot;&quot;<br>    # Setup mock to return a user<br>    mock_user_service.get_user.return_value = sample_user_response_inactive<br>    <br>    # Override the dependency<br>    app.dependency_overrides[get_user_service] = lambda: mock_user_service<br>    <br>    # Make the request to the endpoint<br>    with TestClient(app) as client:<br>        response = client.get(&quot;/users&quot;, params={&quot;email&quot;: &quot;test@example.com&quot;})<br>    <br>    # Clean up the override<br>    app.dependency_overrides.clear()<br>    <br>    # Verify the service method was called correctly<br>    mock_user_service.get_user.assert_called_once_with(email=sample_user_response_inactive.email)<br>    <br>    # Assertions<br>    assert response.status_code == 200<br>    user_data = response.json()<br>    assert user_data[&quot;email&quot;] == sample_user_response_inactive.email<br>    assert user_data[&quot;is_active&quot;] == sample_user_response_inactive.is_active<br><br>(...)</pre><p>In this test we are using two <em>pytest fixtures</em> to:</p><ol><li>Mock a UserService object</li><li>Return a UserResponse model.</li></ol><p>These <em>fixtures</em> can be used throughout our tests. I will now go through what happens, step by step, in our first unit test.</p><ol><li>Our mock UserService instance mock_user_service is going to return our UserResponse model, when mock_user_service.get_user is called.</li><li>Since our controller <em>depends</em> on get_user_service , we must now override this dependency, such that it now depends on our mock_user_service .</li><li>We make the request to the endpoint, and since we are now using our mock_user_service , we return the UserResponse defined in our <em>fixture</em> when mock_user_service.get_user(email=&#39;test@example.com&#39;) is called.</li><li>We clear the <em>dependency override</em> so that after the test, the controller depends on the real UserService again, not our mock.</li><li>We assert that the get_user methods was called exactly once as expected.</li><li>We assert that the response is as expected.</li></ol><p>This example shows how we can test our controller, by mocking dependencies (get_user_service ), Objects (UserService ), and object methods ( UserService.get_user ).</p><p><strong>/tests/unit/test_user_service.py</strong></p><pre>import pytest<br>from unittest.mock import AsyncMock, patch<br><br><br>@pytest.fixture<br>def user_service():<br>    &quot;&quot;&quot;Create a UserService with a mocked repository.&quot;&quot;&quot;<br>    from src.service.UserService import UserService<br>    from src.repository.interfaces.interface_UserRepository import UserRepository<br>    <br>    # Create a mock repository directly<br>    mock_repo = AsyncMock(spec=UserRepository)<br>    <br>    # Return the service with the mock repo<br>    return UserService(user_repository=mock_repo)<br><br><br>@pytest.fixture<br>def sample_user_inactive_nopw():<br>    &quot;&quot;&quot;Create a sample User (result from user_repo.get_user) for testing.&quot;&quot;&quot;<br>    from src.schemas import UserSchemas<br>    return UserSchemas.User(<br>        email=&quot;test@example.com&quot;,<br>        is_active=False<br>    )<br><br>(...)<br><br># Tests for get_user method<br>@pytest.mark.asyncio<br>async def test_get_user_success(<br>    user_service,<br>    sample_user_inactive_nopw<br>    ):<br>    &quot;&quot;&quot;Test successful user retrieval.&quot;&quot;&quot;<br>    # Setup mock to return a User<br>    user_service.user_repository.get_user = AsyncMock(return_value=sample_user_inactive_nopw)<br>    <br>    # Call the method - returns UserResponse<br>    user = await user_service.get_user(sample_user_inactive_nopw.email)<br>    <br>    # Verify the repository method was called correctly<br>    user_service.user_repository.get_user.assert_called_once_with(sample_user_inactive_nopw.email)<br>    <br>    # Assertions<br>    assert user.email == sample_user_inactive_nopw.email<br>    assert user.is_active is False<br><br>(...)</pre><p>Now we are testing the service layer, and as you can see, we follow the same principles from before. The key point here is the effect of the Repository Pattern: we’re now mocking the interface, not the actual implementation. This means that we do not have to change our unit tests for our service layer, when we swap database implementations.</p><p><strong>/tests/unit/repository/test_postgres_repository.py</strong></p><pre>import pytest<br>from unittest.mock import AsyncMock, MagicMock, patch<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from sqlalchemy.exc import IntegrityError<br><br># Fixtures<br>@pytest.fixture<br>def mock_db():<br>    &quot;&quot;&quot;Create a mock AsyncSession for testing.&quot;&quot;&quot;<br>    mock = AsyncMock(spec=AsyncSession)<br>    # Setup the async context manager for begin()<br>    context_manager = AsyncMock()<br>    mock.begin.return_value = context_manager<br>    return mock<br><br>@pytest.fixture<br>def user_repo(mock_db):<br>    &quot;&quot;&quot;Create a UserRepository instance with a mock db.&quot;&quot;&quot;<br>    from src.repository.implementations.PostgreSQL.postgres_UserRepository import UserRepository<br>    return UserRepository(db=mock_db)<br><br>(...)<br><br>@pytest.fixture<br>def db_user():<br>    &quot;&quot;&quot;Create a sample UserORM object that would be returned from the database.&quot;&quot;&quot;<br>    from src.repository.implementations.PostgreSQL.models.ORM_User import UserORM<br>    user = MagicMock(spec=UserORM)<br>    user.email = &quot;test@example.com&quot;<br>    user.is_active = True<br>    user.hashed_password = &quot;hashed_password_value&quot;<br>    return user<br><br># Tests for get_user method<br>@pytest.mark.asyncio<br>async def test_get_user_success(user_repo, mock_db, db_user):<br>    &quot;&quot;&quot;Test successful user retrieval.&quot;&quot;&quot;<br>    # Import inside test function to avoid early import<br>    from src.schemas import UserSchemas<br>    <br>    # Setup mock to return a user<br>    mock_result = MagicMock()<br>    mock_result.scalar_one_or_none.return_value = db_user<br>    mock_db.execute.return_value = mock_result<br>    <br>    # Call the method<br>    user = await user_repo.get_user(&quot;test@example.com&quot;)<br>    <br>    # Assertions<br>    assert user.email == &quot;test@example.com&quot;<br>    assert user.is_active is True<br>    mock_db.execute.assert_called_once()</pre><p>Here we are testing our actual PostgreSQL implementation, by mocking the PostgreSQL database. The context_manager is used for testing database <em>transactions</em>, which we’ll use later on in this series. A database transaction is simply when multiple database operations are executed, and if one fails, all subsequent operations are cancelled, and all previous operations are rolled back. Also note that we are mocking the database response using UserORM to return our db_user as defined in the <em>fixtures</em>. This means when we call mock_db.execute we return our db_user . This in turn means that get_user translates this UserORM object into a User defined by our <em>schemas</em>.</p><p>In our unit tests it should now be clear that we are benefitting from the Repository Pattern, and that due to the loose coupling measures we’ve made earlier, we can very easily write unit tests for our service layer without worrying about the actual database implementation.</p><h3>Integration tests</h3><p>For integration tests, we necessarily depend on our actual database implementations, since we’re testing the system end-to-end. We are not going to set up a test database, which is obviously best practise, since we want to make real database operations. However, its been chosen to implement our tests in a way that would easily allows us to use a test database.</p><p>For this reason we are going to override our db_context that you might recall from our previous article. The db_context was our dependency function returning either an AsyncSession or a DynamoDBClient, dependent on whether our Settings were set to use either of those as its database backend. For the purpose of this article, we are simple going to focus on our PostgreSQL implementation.</p><p><strong>/tests/integration/conftest.py</strong></p><pre>import pytest<br>(...)<br><br># Load settings<br>settings = get_settings()<br><br># Define override_db_context at module level - will be set properly later<br>async def override_db_context():<br>    yield None<br><br>if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br><br>    test_engine = create_async_engine(settings.POSTGRES_DATABASE_URL_FOR_TESTING, echo=True)<br>    TestAsyncSessionLocal = async_sessionmaker(<br>        bind=test_engine,<br>        expire_on_commit=False,<br>        class_=AsyncSession<br>    )<br><br>    @pytest.fixture(scope=&quot;session&quot;)<br>    async def setup_database():<br>        &quot;&quot;&quot;Set up the test database.&quot;&quot;&quot;<br>        async with test_engine.begin() as conn:<br>            await conn.run_sync(Base.metadata.drop_all)<br>            await conn.run_sync(Base.metadata.create_all)<br>        yield<br>        # Clear tables instead of dropping them<br>        async with test_engine.begin() as conn:<br>            for table in reversed(Base.metadata.sorted_tables):<br>                await conn.execute(text(&#39;TRUNCATE TABLE auth.users CASCADE&#39;))<br><br>    @pytest.fixture<br>    async def db_session(setup_database):<br>        &quot;&quot;&quot;Create a fresh database session for each test.&quot;&quot;&quot;<br>        connection = await test_engine.connect()<br>        transaction = await connection.begin()<br>        <br>        session = AsyncSession(bind=connection, expire_on_commit=False)<br>        <br>        try:<br>            yield session<br>        finally:<br>            await session.close()<br>            await transaction.rollback()<br>            await connection.close()<br><br>    # Override the module-level function<br>    async def postgresql_context():<br>        async with TestAsyncSessionLocal() as session:<br>            yield session<br>    <br>    # Set the module-level function<br>    override_db_context = postgresql_context<br><br>elif settings.DATABASE_TYPE == DatabaseType.DYNAMODB:<br><br>    (...)<br><br>    # Set the module-level function<br>    override_db_context = dynamodb_context<br><br>else:<br>    raise ValueError(f&quot;Invalid DATABASE_TYPE: {settings.DATABASE_TYPE}. Must be one of {[e.value for e in DatabaseType]}&quot;)<br><br># Async client fixture<br>@pytest.fixture<br>async def async_client(setup_database):<br>    &quot;&quot;&quot;Create an async client for testing the API.&quot;&quot;&quot;<br><br>    # Override the db_context dependency<br>    app.dependency_overrides[db_context] = override_db_context<br>    <br>    transport = ASGITransport(app=app)<br>    async with AsyncClient(transport=transport, base_url=&quot;http://test&quot;, follow_redirects=True) as client:<br>        yield client<br><br>    # Clear all overrides<br>    app.dependency_overrides.clear()</pre><p>conftest.py configures our testing environment for our integration tests. So going through it step by step, if our database backend is PostgreSQL:</p><ol><li>We create an <em>Engine</em> and a <em>Session Factory</em>, just like we do in our start-up tasks. This is also where we allow to use another database, e.g. one specifically for testing.</li><li>The setup_database fixture clears auth.users table prior to running tests.</li><li>The db_session fixture creates a new <em>Session</em> per request.</li><li>The postgresql_context acts as our new db_context; we override the original dependency with this context for testing. . That means when we <em>inject</em> the postgresql_context into our get_user_repository , and thus we use the our test database when testing our service.</li><li>Finally, in async_client , we make the override and yield our client. This is AsyncClient is then the one we use to make test requests to our API, as you shall see in our actual integration tests. Finally, after each test, we remove our dependency override, so that we no longer depend on our testing context.</li></ol><p><strong>/tests/integration/test_api.py</strong></p><pre>import pytest<br>from httpx import AsyncClient<br><br>@pytest.mark.asyncio(loop_scope=&quot;session&quot;)<br>async def test_get_nonexistent_user(async_client: AsyncClient):<br>    &quot;&quot;&quot;Test retrieving a user that doesn&#39;t exist&quot;&quot;&quot;<br><br>    test_user_data = {<br>        &quot;email&quot;: &quot;nonexistent@example.com&quot;<br>    }<br><br>    response = await async_client.get(<br>        &quot;/users&quot;,<br>        params=test_user_data<br>    )<br>    <br>    assert response.status_code == 404<br>    data = response.json()<br>    assert &#39;error&#39; in data<br>    assert data[&quot;error&quot;] == f&quot;User with email {test_user_data[&#39;email&#39;]} not found&quot;</pre><p>Here we use the configurations made in conftest.py from before, to make actual API request, and thus test our service. In this example, we’re testing whether our service returns the expected error when a non-existent user is requested.</p><h3>Dockerfile</h3><p>Now, we want to run these files in our container. In order to do so we are going to build on top of our previous Dockerfile, where we were heavily inspired by uv’s multistage.Dockerfile . For simplicity, we are going to break the Dockerfile into logical bits.</p><p><strong>Builder Stage</strong></p><pre>###########################################<br># BUILDER stage<br>###########################################<br>FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS builder<br>ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy<br><br># Disable Python downloads, because we want to use the system interpreter<br># across both images. If using a managed Python version, it needs to be<br># copied from the build image into the final image; see `standalone.Dockerfile`<br># for an example.<br>ENV UV_PYTHON_DOWNLOADS=0<br><br>WORKDIR /app<br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    --mount=type=bind,source=uv.lock,target=uv.lock \<br>    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \<br>    uv sync --frozen --no-install-project --no-dev<br>ADD . /app<br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    uv sync --frozen --no-dev</pre><p><strong>Unit Test Stage</strong></p><pre>###########################################<br># UNIT TEST stage<br>###########################################<br>FROM python:3.12-slim-bookworm AS unit-test<br># It is important to use the image that matches the builder, as the path to the<br># Python executable must be the same, e.g., using `python:3.11-slim-bookworm`<br># will fail.<br><br># Copy the application from the builder<br>COPY --from=builder --chown=app:app /app /app<br><br># Place executables in the environment at the front of the path<br>ENV PATH=&quot;/app/.venv/bin:$PATH&quot;<br>ENV PYTHONPATH=&quot;/app&quot;<br><br># Run unit tests by default (using config from pyproject.toml)<br>CMD [&quot;python&quot;, &quot;-m&quot;, &quot;pytest&quot;, &quot;app/tests/unit&quot;]</pre><p>We simply run our unit tests with the command python -m pytest app/tests/unit. Obviously we do not have to have our service running to run unit tests, as we are not making actual requests.</p><p><strong>Integration Test Stage</strong></p><pre>###########################################<br># INTEGRATION TEST stage<br>###########################################<br>FROM python:3.12-slim-bookworm AS integration-test<br><br># Copy the application from the builder<br>COPY --from=builder --chown=app:app /app /app<br><br># Install bash for scripting<br>RUN apt-get update &amp;&amp; apt-get install -y bash<br><br># Test-specific env<br>ENV PATH=&quot;/app/.venv/bin:$PATH&quot;<br>ENV PYTHONPATH=&quot;/app&quot;<br><br># Create test runner script<br>RUN echo &#39;#!/bin/bash\n\<br>echo &quot;👉 Starting server...&quot;\n\<br>uvicorn src.main:app --host 0.0.0.0 --port 8000 &amp; \n\<br>SERVER_PID=$!\n\<br>\n\<br>echo &quot;👉 Waiting for server to start...&quot;\n\<br>sleep 5\n\<br>\n\<br>echo &quot;👉 Running integration tests...&quot;\n\<br>python -m pytest app/tests/integration\n\<br>TEST_EXIT_CODE=$?\n\<br>\n\<br>echo &quot;👉 Stopping server...&quot;\n\<br>kill $SERVER_PID\n\<br>\n\<br>exit $TEST_EXIT_CODE\n\<br>&#39; &gt; /app/run_integration_tests.sh &amp;&amp; chmod +x /app/run_integration_tests.sh<br><br># Run integration tests by default<br>CMD [&quot;/app/run_integration_tests.sh&quot;]</pre><p>To run our integration tests, we are first starting our server, and then running the tests. This is obviously necessary because our integration tests obviously need our server to be running to send requests. For this we are making a short script to be executed on container start:</p><ol><li>We start the server</li><li>We run the integration tests</li><li>We shut down the server</li></ol><p>You can find the actual Dockerfile <a href="https://github.com/assibering/CryptoWebApp/blob/main/UserService/Dockerfile.dev">here</a>, as it will also include a stage that runs both unit- and integration tests.</p><p>While this article focused more on making tests “work right” than on writing the “right” tests, it’s still important to consider which tests are most critical for your application. Ideally, we would have 100% code coverage, but oftentimes we want ship software before that happens — if it ever happens.</p><p>From our growing Project Architecture I hope you can see how important it is, to consider everything from using the right patterns to putting stuff in the right place — you can easily give yourself and others a hard time maintaining even a simple application. Also, our microservice’s core functionality has barely changed since the start of this series-which highlights the big gap between a school-project FastAPI app and an enterprise-grade FastAPI service.</p><p>In the next articles we are changing focus, as we will now focus on making our microservice work in a system of microservices. Up until now, we have made our service work as a single service, but in any microservice architecture, you’ll have multiple microservices <em>working together</em>. So in the following articles we shall discover how we can use event-driven architecture to make multiple loosely coupled microservices work together. Microservices <em>must</em> obviously be loosely coupled, because not being defeats the whole point — then we should have a <em>monolith</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4a1d46594b6" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-7-10-unit-integration-testing-c4a1d46594b6">Building Enterprise Python Microservices with FastAPI in 2025 (7/10): Unit- &amp; Integration Testing</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (6/10): DynamoDB]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-6-10-dynamodb-1653a0f2e22c?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/1653a0f2e22c</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sun, 18 May 2025 21:42:55 GMT</pubDate>
            <atom:updated>2025-05-19T15:28:18.567Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous article I implemented our PostgreSQL backend, and we saw our Repository Pattern, exception handling, and logging in action for the first time.</p><p>In this article we are going to really understand the Repository Pattern, as we will actively use it to implement a new database backend: DynamoDB. I chose DynamoDB primarily because it’s a NoSQL database, meaning we cannot reuse any logic from our PostgreSQL implementation. This highlights the power of the Repository Pattern and the loose coupling techniques we’ve implemented so far. By the end of this article I hope it is clear that changing to a completely different database is no problem, because of the architectural choices made up until this point.</p><h3>DynamoDB Table Definition</h3><p>First of all, I will quickly introduce our users table we are going to interact with from our User microservice.</p><p>Similar to the previous article, we are going to create a users table in DynamoDB. We are using <em>Localstack community edition </em>which is a localised AWS environment, making it possible to access AWS services, like DynamoDB, locally. To make it extra realistic, we’ve chosen to deploy to Localstack’s DynamoDB using Terraform. We’ve also created IAM roles to comply with the principle of least privilege. You can access all the terraform files <a href="https://github.com/assibering/CryptoWebApp/tree/main/Localstack/terraform">here</a>, but for this article we will just present the DynamoDB <em>resource</em>, i.e. the table definition we deploy to DynamoDB using terraform.</p><pre>resource &quot;aws_dynamodb_table&quot; &quot;users_table&quot; {<br>  name           = &quot;users&quot;<br>  billing_mode   = &quot;PAY_PER_REQUEST&quot;<br>  hash_key       = &quot;email&quot;<br><br>  attribute {<br>    name = &quot;email&quot;<br>    type = &quot;S&quot;<br>  }<br>}</pre><p>We’re creating a table named users with primary key email . In comparison to PostgreSQL, we have complete freedom in DynamoDB when it comes to inserting records with other fields — however, for consistency, we will stick to the same fields as before: email , hashed_password , and is_active .</p><h3>Session, Client and Client Factory</h3><p>Just like the previous article about PostgreSQL, we will quickly cover some of the concepts related to DynamoDB.</p><p><strong>Session</strong></p><p>The <em>Session</em> is created once on start-up of our application, and manages AWS credentials and configuration. This is equivalent in lifecycle (but not function) to the <em>Engine</em> described in the previous PostgreSQL article, as it is used to create <em>Clients</em> for each request.</p><p><strong>Client</strong></p><p>The <em>Client</em> is created per request and is used to interact with our DynamoDB database. Clients are lightweight and stateless, and are closed when the request is finished. This is analogous to the <em>Session</em> in the previous PostgreSQL article.</p><p><strong>Client Factory</strong></p><p>A <em>Client Factory</em> generates clients. Specifically we will create a Client Factory to create a Client per request, as described above. This is exactly the same behaviour as in the previous article, where we created a PostgreSQL <em>session</em> per request using a <em>Session Factory</em>.</p><p>Similar to the previous PostgreSQL implementation, we will also use the async database driver for DynamoDB: <em>aioboto3</em>.</p><h3>Settings</h3><p>We are now going to extend our /.env and /src/db/settings.py to also include DynamoDB.</p><p><strong>/.env</strong></p><pre>#Database<br># DATABASE_TYPE=postgres<br>DATABASE_TYPE=dynamodb<br><br># Postgres Configuration<br>POSTGRES_DATABASE_URL=postgresql+asyncpg://user_service_user:super_secure_password@postgresql:5432/crypto_db<br>POSTGRES_DATABASE_URL_FOR_TESTING=postgresql+asyncpg://postgres:your_secure_password@postgresql:5432/crypto_db<br><br># AWS Configuration<br>AWS_ACCESS_KEY_ID=some_key<br>AWS_SECRET_ACCESS_KEY=some_secret<br>AWS_REGION=us-east-1<br>AWS_ENDPOINT=http://localstack:4566</pre><p>As before, we’ve now added our AWS related environment variables. Note that we have commented out the database we are not using, in the above case you can se that DATABASE_TYPE=dynamodb, which will be read by our Settings .</p><p><strong>/src/db/settings.py</strong></p><pre>from pydantic_settings import BaseSettings, SettingsConfigDict<br>from functools import lru_cache<br>from enum import Enum<br><br>class DatabaseType(str, Enum):<br>    POSTGRES = &quot;postgres&quot;<br>    DYNAMODB = &quot;dynamodb&quot;<br><br>class Settings(BaseSettings):<br>    DATABASE_TYPE: DatabaseType = DatabaseType.POSTGRES # Default to postgres<br><br>    # --------------------------------------------------------------------<br>    # Postgres settings<br>    POSTGRES_DATABASE_URL: str = &quot;postgresql+asyncpg://user:password@postgresql:5432/crypto_db&quot;<br>    POSTGRES_DATABASE_URL_FOR_TESTING: str = &quot;postgresql+asyncpg://postgres:password@postgresql:5432/crypto_db&quot;<br>    # --------------------------------------------------------------------<br><br>    # --------------------------------------------------------------------<br>    # AWS settings<br>    AWS_ACCESS_KEY_ID: str = &quot;default_key&quot;<br>    AWS_SECRET_ACCESS_KEY: str = &quot;default_secret&quot;<br>    AWS_REGION: str = &quot;us-east-1&quot;<br>    AWS_ENDPOINT: str = &quot;http://localhost:4566&quot;<br><br>    AWS_ACCESS_KEY_ID_FOR_TESTING: str = &quot;default_key&quot;<br>    AWS_SECRET_ACCESS_KEY_FOR_TESTING: str = &quot;default_secret&quot;<br>    AWS_REGION_FOR_TESTING: str = &quot;us-east-1&quot;<br>    AWS_ENDPOINT_FOR_TESTING: str = &quot;http://localstack:4566&quot;<br>    # --------------------------------------------------------------------<br>    <br>    model_config = SettingsConfigDict(<br>        env_file = &quot;.env&quot;,<br>        case_sensitive = True<br>    )<br><br>@lru_cache()<br>def get_settings() -&gt; Settings:<br>    return Settings()</pre><p>Similar to before, we’ve simply added our AWS related settings.</p><h3>Dependency Injection</h3><p><strong>/src/main.py</strong></p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from .routes import UserController<br>from .exceptions import BaseAppException<br>import logging<br>from src.logging_config import setup_logging<br>from fastapi.responses import JSONResponse<br>from src.middleware.correlation_id_middleware import CorrelationIdMiddleware<br>from contextlib import asynccontextmanager<br>from src.db.settings import get_settings, DatabaseType<br>import aioboto3<br>from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker<br><br>setup_logging()<br>logger = logging.getLogger(__name__)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    # This is where you put code that was previously in @app.on_event(&quot;startup&quot;)<br>    logger.info(&quot;Running startup tasks...&quot;)<br><br>    # Initialize settings<br>    settings = get_settings()<br><br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        # Create an async engine<br>        engine = create_async_engine(settings.POSTGRES_DATABASE_URL, echo=True, future=True)<br><br>        # Create an async sessionmaker factory<br>        app.state.postgres_session = async_sessionmaker(<br>            bind=engine,<br>            expire_on_commit=False,  # optional: objects stay active after commit<br>            class_=AsyncSession<br>        )<br><br>    # Create the aioboto3 session at application startup if using DynamoDB<br>    elif settings.DATABASE_TYPE == DatabaseType.DYNAMODB:<br>        logger.info(&quot;Initializing DynamoDB session&quot;)<br>        app.state.dynamodb_session = aioboto3.Session(<br>            aws_access_key_id=settings.AWS_ACCESS_KEY_ID,<br>            aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,<br>            region_name=settings.AWS_REGION<br>        )<br><br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>origins = [<br>    &quot;http://localhost:8000&quot;,<br>]<br><br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;GET&quot;, &quot;PUT&quot;],<br>    allow_headers=[&quot;Authorization&quot;, &quot;Content-Type&quot;]<br>)<br><br># Enable Correlation ID middleware<br>app.add_middleware(CorrelationIdMiddleware)<br><br># Global exception handler<br>@app.exception_handler(BaseAppException)<br>async def app_exception_handler(request, exc):<br>    logger.error(&quot;Application error: %s&quot;, exc.message)<br>    return JSONResponse(<br>        status_code=exc.status_code,<br>        content={&quot;error&quot;: exc.message}<br>    )<br><br><br>app.include_router(UserController.router)</pre><p>On application start-up we are now creating an async <em>Session</em> when using DynamoDB as our database backend. We will then use this Session to create Clients on a per request basis.</p><p><strong>/src/db/db_context.py</strong></p><pre># src/db/context.py<br>from typing import Callable, AsyncGenerator, Any<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from .settings import get_settings, DatabaseType<br>from botocore.config import Config<br>from types_aiobotocore_dynamodb import DynamoDBClient<br>from fastapi import Request<br>from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker<br>import aioboto3<br><br><br># Type for a dependency that yields a value<br>ContextDependency = Callable[..., AsyncGenerator[Any, None]]<br><br>def get_db_context() -&gt; ContextDependency:<br>    &quot;&quot;&quot;<br>    Returns the appropriate database context dependency based on configuration.<br>    For PostgreSQL: Returns a dependency that yields a database session<br>    For DynamoDB: Returns a dependency that yields a DynamoDB client<br>    &quot;&quot;&quot;<br>    settings = get_settings()<br>    <br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        # Return the PostgreSQL session dependency<br>        async def get_postgres_context(request: Request) -&gt; AsyncGenerator[AsyncSession, None]:<br>            async_session_factory = request.app.state.postgres_session<br>            async with async_session_factory() as session:<br>                try:<br>                    yield session<br>                    await session.commit()  # Automatically commit if no exceptions<br>                except Exception:<br>                    await session.rollback()  # Rollback on exceptions<br>                    raise<br>        return get_postgres_context<br>    elif settings.DATABASE_TYPE == DatabaseType.DYNAMODB:<br>        # Return a DynamoDB context using the app-wide session<br>        async def get_dynamo_context(request: Request) -&gt; AsyncGenerator[DynamoDBClient, None]:<br>            # Get the session from app state<br>            session = request.app.state.dynamodb_session<br>            <br>            # Create a client for this request using the shared session<br>            async with session.client(<br>                &#39;dynamodb&#39;,<br>                endpoint_url=settings.AWS_ENDPOINT,<br>                config=Config(<br>                    connect_timeout=5.0,<br>                    read_timeout=10.0,<br>                    retries={&#39;max_attempts&#39;: 3}<br>                )<br>            ) as dynamodb_client:<br>                dynamodb_client: DynamoDBClient<br>                yield dynamodb_client<br>        <br>        return get_dynamo_context<br>    else:<br>        raise ValueError(&quot;Invalid DATABASE_TYPE&quot;)<br><br># Create the context dependency based on current configuration<br>db_context = get_db_context()</pre><p>Just like before, the returned dependency function is assigned to db_context. We can then use Depends(db_context) in our route handlers. Now that we are using DynamoDB, we will inject an async DynamoDB C<em>lient</em>, whereas with PostgreSQL we injected an async PostgreSQL <em>Session</em>.</p><p><strong>/src/db/factory.py</strong></p><pre># src/db/factory.py<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from types_aiobotocore_dynamodb import DynamoDBClient<br>from src.repository.interfaces.interface_SubscriptionRepository import SubscriptionRepository as SubscriptionRepositoryInterface<br>from src.repository.implementations.PostgreSQL.postgres_SubscriptionRepository import SubscriptionRepository as PostgresSubscriptionRepository<br>from src.repository.implementations.AWS_DynamoDB.awsdynamodb_SubscriptionRepository import SubscriptionRepository as DynamoSubscriptionRepository<br>from .settings import get_settings, DatabaseType<br>from typing import Union<br><br>def create_subscription_repository(db_context: Union[AsyncSession, DynamoDBClient]) -&gt; SubscriptionRepositoryInterface:<br>    &quot;&quot;&quot;<br>    Creates the appropriate repository based on configuration.<br>    For PostgreSQL: Uses the provided database session<br>    For DynamoDB: Uses the provided database client<br>    &quot;&quot;&quot;<br>    settings = get_settings()<br>    <br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        return PostgresSubscriptionRepository(db_context)<br>    elif settings.DATABASE_TYPE == DatabaseType.DYNAMODB:<br>        return DynamoSubscriptionRepository(db_context)<br>    else:<br>        raise ValueError(f&quot;Unsupported database type: {settings.DATABASE_TYPE}&quot;)</pre><p>Here, we use our db_context to create the correct repository implementation — either for DynamoDB or PostgreSQL. Note that the repository now also accepts a DynamoDBClient as input.</p><p><strong>/src/dependencies.py</strong></p><pre># src/dependencies.py<br>from fastapi import Depends<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from types_aiobotocore_dynamodb import DynamoDBClient<br>from src.repository.interfaces.interface_UserRepository import UserRepository as UserRepositoryInterface<br>from src.service.UserService import UserService<br>from src.db.factory import create_user_repository<br>from src.db.db_context import db_context<br>from typing import Union<br><br>async def get_user_repository(db: Union[AsyncSession, DynamoDBClient] = Depends(db_context)) -&gt; UserRepositoryInterface:<br>    &quot;&quot;&quot;<br>    Creates the appropriate repository based on configuration.<br>    The db parameter will be a database session for PostgreSQL or None for DynamoDB.<br>    &quot;&quot;&quot;<br>    return create_user_repository(db)<br><br>async def get_user_service(user_repository: UserRepositoryInterface = Depends(get_user_repository)) -&gt; UserService:<br>    return UserService(user_repository)</pre><p>Nothing changed here, except that we now accept DynamoDBClient as input here as well.</p><p>As our endpoints depend on get_user_service we have now achieved loose coupling, because our UserController do not “care” whether the user_service.repository is a DynamoDB or PostgreSQL implementation.</p><p>To summarise, we will quickly go through, step by step, what happens when an end-user sends a request:</p><ol><li>end-user sends GET /users?email=test@example.com</li><li>Our dependency injection Depends(get_user_service) is called and assigned to variable user_service .</li><li>get_user_service depends on get_user_repository, which depends on db_context.</li><li>From db_contextwe call our BaseSettingsto figure out, which dependency function to return. The settings are set to DynamoDB, and thus db = Depends(db_context)<em> </em>sets dbto an async DynamoDB Client.</li><li>get_user_repository<em> </em>now returns an instance of UserRepository, where the db of that instance is set to the async DynamoDB Client.</li><li>get_user_service now returns an instance of UserService, where the repository is the UserRepository returned from get_user_repository<em> </em>before.</li><li>user_service<em> </em>is finally set to an instance of UserService, where its user_repositoryis an actual DynamoDB implementation, with an actual async DynamoDB Client to use for interacting with the database.</li></ol><p>For reference, these are our UserService and UserRepository classes. Please note that UserRepository implements the interface_UserRepository.</p><p><strong>/src/service/UserService.py</strong></p><pre>(...)<br><br>class UserService:<br><br>    def __init__(self, user_repository: interface_UserRepository):<br>        self.user_repository = user_repository<br>    <br>    async def get_user(self, email: str) -&gt; UserSchemas.UserResponse:<br>        try:<br>            user = await self.user_repository.get_user(email)<br>            return UserSchemas.UserResponse(<br>                email=user.email,<br>                is_active=user.is_active<br>            )<br>        except ResourceNotFoundException:<br>            raise #Re-raise from repository layer<br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Error getting user: {str(e)}&quot;) from e<br><br>(...)</pre><p>Here, we see that user_repository must be an implementation of interface_UserRepository. The UserService is unchanged precisely because it depends on the interface, not the concrete implementation.</p><p><strong>/src/repository/implementations/AWS_DynamoDB/awsdynamodb_UserRepository.py</strong></p><pre>(...)<br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, client: DynamoDBClient):<br>        &quot;&quot;&quot;<br>        Initialize the DynamoDB repository.<br>        &quot;&quot;&quot;<br>        <br>        # Initialize DynamoDB client<br>        self.client = client<br>        <br>        # You could also use a table name prefix from settings<br>        self.table_name = &quot;users&quot;<br><br>    async def get_user(self, email: str) -&gt; UserSchemas.User:<br>        &#39;&#39;&#39;<br>        This function returns a User instance from the database.<br>        Or raises an exception if the user does not exist.<br>        &#39;&#39;&#39;<br>        try:<br>            response = await self.client.get_item(<br>                TableName=self.table_name,<br>                Key=await get_key(<br>                    pkey_name=&quot;email&quot;,<br>                    pkey_value=email<br>                )<br>            )<br><br>            if &#39;Item&#39; not in response:<br>                logger.warning(f&quot;User with email {email} not found&quot;)<br>                raise ResourceNotFoundException(f&quot;User with email {email} not found&quot;)<br>            <br>            return await dynamodb_to_basemodel(<br>                basemodel=UserSchemas.User,<br>                dynamodb_data=response[&#39;Item&#39;],<br>                include_empty_string_in_stringsets=False<br>            )<br>        <br>        except ResourceNotFoundException:<br>            raise<br><br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Internal database error: {str(e)}&quot;) from e<br><br>(...)</pre><p>Here we can see:</p><ol><li>That we need a DynamoDBClientto instantiate our DynamoDB implementation of UserRepository. This is what we get from our db_context. We are also using a table_name that defaults to users .</li><li>Using our client we are able to interact with our database in a pythonic manner. Specifically in the get_user method we are now making an actual async request to our database to retrieve a user.</li><li>Practical use of proper exception handling and logging. Just like we presented in the previous article, we now raise a ResourceNotFoundException when the requested user is not in our database, and if some other error occurs, we raise a BaseAppException with the actual error as message.</li></ol><h3>Containerization Strategy</h3><p>We can now use the official Localstack image to create a local instance of a DynamoDB database and test out our FastAPI microservice. However, we must first deploy to a running instance of DynamoDB using Terraform. Specifically, I use a deploy-local.shscript to do so. You can check out how <a href="https://github.com/assibering/CryptoWebApp/tree/main/Localstack">here</a>. For the purpose of this article, we will just show the docker compose file, to get everything running.</p><pre>services:<br>  localstack:<br>    container_name: &quot;crypto_localstack&quot;<br>    image: localstack/localstack:latest<br>    env_file:<br>      - ./localstack/environments/.env.local<br>    ports:<br>      - &quot;127.0.0.1:4566:4566&quot;            # LocalStack Gateway<br>      - &quot;127.0.0.1:4510-4559:4510-4559&quot;  # external services port range<br>    volumes:<br>      - localstack_data:/var/lib/localstack<br>      - &quot;/var/run/docker.sock:/var/run/docker.sock&quot;<br>    restart: unless-stopped<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;-f&quot;, &quot;http://localhost:4566/_localstack/health&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 5<br>  <br>  user_service_unit:<br>    # Build the image from the Dockerfile in the /UserService directory<br>    build:<br>      context: ./UserService<br>      dockerfile: Dockerfile.dev<br>      target: full-test #unit-test, integration-test, full-test, production (no tests are run)<br>    container_name: crypto_user_service<br>    env_file:<br>      - ./UserService/.env<br><br>    # Host the FastAPI application on port 8000<br>    ports:<br>      - &quot;8000:8000&quot;<br><br>    depends_on:<br>      localstack:<br>        condition: service_healthy</pre><p>Here we are doing things correctly compared to the previous article, as we are now storing our secrets in our/localstack/environments/.env.local file. Here we keep AWS credentials to deploy using terraform.</p><p>In this article we took advantage of our Repository Pattern to fairly easily swap out our PostgreSQL database from before, with a completely different NoSQL database: DynamoDB. I hope this clarifies <em>why</em> the Repository Pattern is so popular, and highlights the importance of making the right architectural choices before coding a solution that “just works.”</p><p>In the upcoming article we are going to look into unit- and integration testing using Pytest , and running our tests in a container.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1653a0f2e22c" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-6-10-dynamodb-1653a0f2e22c">Building Enterprise Python Microservices with FastAPI in 2025 (6/10): DynamoDB</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (5/10): PostgreSQL]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-5-10-postgresql-85b48497c324?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/85b48497c324</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sun, 18 May 2025 21:41:16 GMT</pubDate>
            <atom:updated>2025-05-19T15:28:26.539Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous article I implemented exception handling and logging following best-practises in an enterprise-grade microservice. In this article I will implement a real PostgreSQL database backend, to make the following clear through practical examples:</p><ol><li>The Repository Pattern, introduced in part 3, in action.</li><li>Realistic exception handling and logging.</li></ol><p>As a bonus you also get to see how one can integrate PostgreSQL in FastAPI applications, which is something you will realistically come across in a microservice stack in 2025.</p><h3>PostgreSQL Table Definition</h3><p>First of all, I will quickly introduce our auth.users table we are going to interact with from our User microservice.</p><pre>-- Create the schema if not exists<br>CREATE SCHEMA IF NOT EXISTS auth;<br><br>-- Create the users table<br>CREATE TABLE IF NOT EXISTS auth.users (<br>    email VARCHAR(50) PRIMARY KEY,<br>    hashed_password VARCHAR(200), -- hashed (including salt)<br>    is_active BOOLEAN<br>);</pre><p>We’re creating a table named auth.users with three fields: email, hashed_password, and is_active.</p><h3>ORM (Object-relational Mapper)</h3><p>First we need to understand the role of an <em>ORM</em>. The purpose of an ORM is to make a bridge between object-oriented code, in our case Python, and relational databases, in our case PostgreSQL. Instead of writing raw SQL, we can leverage ORM tools to interact with our relational database using familiar programming constructs. You define models as classes, and the ORM handles translating these objects to database tables and vice versa. Let’s get specific and create our UserORM .</p><p><strong>/src/repository/implementations/PostgreSQL/models/ORM_User.py</strong></p><pre>from datetime import datetime ,timezone<br>from sqlalchemy import Column, String, Boolean<br>from sqlalchemy.ext.declarative import declarative_base<br><br>Base = declarative_base()<br><br>class UserORM(Base):<br>    __tablename__ = &quot;users&quot;<br>    __table_args__ = {&quot;schema&quot;: &quot;auth&quot;}<br>    email = Column(String, primary_key=True, unique=True, index=True, nullable=False)<br>    hashed_password = Column(String, nullable=True)<br>    is_active = Column(Boolean, default=True)</pre><p>SQLAlchemy is probably the most widely used ORM library in Python, and the simplicity of using it makes it clear why. We simply make our ORM model, which matches our table definition from before, and state the name of the table and table schema, and we are pretty much good to go. We are now able to interact with our relational database, PostgreSQL, in a pythonic manner using this ORM object.</p><h3>Engine, Session and Session Factory</h3><p>Before we dive into the code, I will quickly describe some concepts related to SQLAlchemy to better understand the actual code.</p><p><strong>Engine</strong></p><p>The <em>Engine</em> is created once on start-up of our application, and manages the connection pool to our PostgreSQL database. This is only created once on start-up.</p><p><strong>Session</strong></p><p>The database <em>Session</em> is created on a per-request basis and is used to interact with our PostgreSQL database. Sessions are closed when the request is finished.</p><p><strong>Session Factory</strong></p><p>A <em>Session Factory</em> generates sessions. Specifically we will create a Session Factory to create a Session per request, as described above.</p><p>In our application we will use an asynchronous engine and asynchronous sessions. We use <em>asynchronous programming</em> throughout the FastAPI application as we want requests to be non-blocking — This is well-known best practise in FastAPI applications. When we use async behaviour in our engine and session, we simply ensure that database operations are also non-blocking.</p><h3>Settings</h3><p>Because we are now using an actual database, we want to keep stuff like URL, user, and password stored in some settings. Additionally, we want to keep these secrets in a separate file, to make sure it is never shared. For that purpose we will create 2 files:</p><ol><li>/.env — file to keep environment variables</li><li>/src/db/settings.py — using Pydantic’s BaseSetting, we configure settings for our application</li></ol><p><strong>/.env</strong></p><pre>#Database<br>DATABASE_TYPE=dynamodb<br><br># Postgres Configuration<br>POSTGRES_DATABASE_URL=postgresql+asyncpg://user_service_user:super_secure_password@postgresql:5432/crypto_db</pre><p>Note that the URL uses <em>postgresql+asyncpg</em>, which means we’re using the async PostgreSQL driver to achieve non-blocking database operations. We are using the <em>user_service_user</em> with password <em>super_secure_password</em> to connect to the <em>crypto_db</em> database running on <em>postgresql:5432</em>.</p><p><strong>/src/db/settings.py</strong></p><pre>from pydantic_settings import BaseSettings, SettingsConfigDict<br>from functools import lru_cache<br>from enum import Enum<br><br>class DatabaseType(str, Enum):<br>    POSTGRES = &quot;postgres&quot;<br>    DYNAMODB = &quot;dynamodb&quot;<br><br>class Settings(BaseSettings):<br>    DATABASE_TYPE: DatabaseType = DatabaseType.POSTGRES # Default to postgres<br><br>    # --------------------------------------------------------------------<br>    # Postgres settings<br>    POSTGRES_DATABASE_URL: str = &quot;postgresql+asyncpg://user:password@postgresql:5432/crypto_db&quot;<br>    POSTGRES_DATABASE_URL_FOR_TESTING: str = &quot;postgresql+asyncpg://postgres:password@postgresql:5432/crypto_db&quot;<br>    # --------------------------------------------------------------------<br>    model_config = SettingsConfigDict(<br>        env_file = &quot;.env&quot;,<br>        case_sensitive = True<br>    )<br><br>@lru_cache()<br>def get_settings() -&gt; Settings:<br>    return Settings()</pre><p>Our Settings that implements Pydantic&#39;s BaseSettingsreads from our .envfile as per defined in model_config. The get_settings method is used to return these settings, making them available in other parts of our application.</p><h3>Dependency Injection</h3><p>Now we will start putting it all together, and will demonstrate how we can inject our database into our FastAPI application, and thus achieve loose coupling.</p><p><strong>/src/main.py</strong></p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from .routes import UserController<br>from .exceptions import BaseAppException<br>import logging<br>from src.logging_config import setup_logging<br>from fastapi.responses import JSONResponse<br>from src.middleware.correlation_id_middleware import CorrelationIdMiddleware<br>from contextlib import asynccontextmanager<br>from src.db.settings import get_settings, DatabaseType<br>from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker<br><br>setup_logging()<br>logger = logging.getLogger(__name__)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    # This is where you put code that was previously in @app.on_event(&quot;startup&quot;)<br>    logger.info(&quot;Running startup tasks...&quot;)<br><br>    # Initialize settings<br>    settings = get_settings()<br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        # Create an async engine<br>        engine = create_async_engine(settings.POSTGRES_DATABASE_URL, echo=True, future=True)<br><br>        # Create an async sessionmaker factory<br>        app.state.postgres_session = async_sessionmaker(<br>            bind=engine,<br>            expire_on_commit=False,  # optional: objects stay active after commit<br>            class_=AsyncSession<br>        )<br><br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>(...)</pre><p>On application start-up we are now creating an async <em>Engine</em>, and an async <em>Session Factory</em> as described above. This means we can now request the app.state.postgres_session from other parts of our code to generate a new async <em>Session</em> to interact with our database.</p><p><strong>/src/db/db_context.py</strong></p><pre># src/db/context.py<br>from typing import Callable, AsyncGenerator, Any<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from .settings import get_settings, DatabaseType<br>from botocore.config import Config<br>from fastapi import Request<br>from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker<br><br><br># Type for a dependency that yields a value<br>ContextDependency = Callable[..., AsyncGenerator[Any, None]]<br><br>def get_db_context() -&gt; ContextDependency:<br>    &quot;&quot;&quot;<br>    Returns the appropriate database context dependency based on configuration.<br>    For PostgreSQL: Returns a dependency that yields a database session<br>    &quot;&quot;&quot;<br>    settings = get_settings()<br>    <br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        # Return the PostgreSQL session dependency<br>        async def get_postgres_context(request: Request) -&gt; AsyncGenerator[AsyncSession, None]:<br>            async_session_factory = request.app.state.postgres_session<br>            async with async_session_factory() as session:<br>                try:<br>                    yield session<br>                    await session.commit()  # Automatically commit if no exceptions<br>                except Exception:<br>                    await session.rollback()  # Rollback on exceptions<br>                    raise<br>        return get_postgres_context<br>    else:<br>        raise ValueError(&quot;Invalid DATABASE_TYPE&quot;)<br><br># Create the context dependency based on current configuration<br>db_context = get_db_context()</pre><p>The returned dependency function is assigned to db_context. We can then use Depends(db_context) in our route handlers, and FastAPI will inject an async PostgreSQL <em>Session</em>. This will become clear very shortly.</p><p><strong>/src/db/factory.py</strong></p><pre>from sqlalchemy.ext.asyncio import AsyncSession<br>from src.repository.interfaces.interface_UserRepository import UserRepository as UserRepositoryInterface<br>from src.repository.implementations.PostgreSQL.postgres_UserRepository import UserRepository as PostgresUserRepository<br>from .settings import get_settings, DatabaseType<br><br>def create_user_repository(db_context: AsyncSession) -&gt; UserRepositoryInterface:<br>    &quot;&quot;&quot;<br>    Creates the appropriate repository based on configuration.<br>    For PostgreSQL: Uses the provided database session<br>    &quot;&quot;&quot;<br>    settings = get_settings()<br>    <br>    if settings.DATABASE_TYPE == DatabaseType.POSTGRES:<br>        return PostgresUserRepository(db_context)<br>    else:<br>        raise ValueError(f&quot;Unsupported database type: {settings.DATABASE_TYPE}&quot;)</pre><p>Here, we use our db_context from before to create our actual PostgreSQL repository implementation.</p><p><strong>/src/dependencies.py</strong></p><pre># src/dependencies.py<br>from fastapi import Depends<br>from sqlalchemy.ext.asyncio import AsyncSession<br>from src.repository.interfaces.interface_UserRepository import UserRepository as UserRepositoryInterface<br>from src.service.UserService import UserService<br>from src.db.factory import create_user_repository<br>from src.db.db_context import db_context<br>from typing import Union<br><br>async def get_user_repository(db: AsyncSession = Depends(db_context)) -&gt; UserRepositoryInterface:<br>    &quot;&quot;&quot;<br>    Creates the appropriate repository based on configuration.<br>    The db parameter will be a database session for PostgreSQL or None for DynamoDB.<br>    &quot;&quot;&quot;<br>    return create_user_repository(db)<br><br>async def get_user_service(user_repository: UserRepositoryInterface = Depends(get_user_repository)) -&gt; UserService:<br>    return UserService(user_repository)</pre><p>These are the actual dependencies we are injecting into our endpoint — you might recall these from part 3. The difference now is that we have sorted our tight coupling from part 3, as our get_user_repository now depends on db_context. This means we can now change db_context to return another Session. Later we will implement DynamoDB and it will become clear that our endpoints will not be tightly coupled to the database we have chosen.</p><p><strong>/src/routes/UserController.py</strong></p><pre>from fastapi import APIRouter, Depends<br>from src.schemas import UserSchemas<br>from src.dependencies import get_user_service<br>from src.service.UserService import UserService<br><br>router = APIRouter(<br>    prefix=&quot;/users&quot;<br>)<br><br>@router.get(&quot;&quot;, status_code=200)<br>async def get_user(<br>    email: str,<br>    user_service: UserService = Depends(get_user_service)):<br>    return await user_service.get_user(email=email)</pre><p>Here we can see that our endpoint now uses dependency injection to get a user service instance, where user_service.repository will now be an actual database implementation of the interface_UserRepository, in our case the PostgreSQL implementation.</p><p>In this section, we’ve demonstrated how to inject a database using dependency injection and achieve loose coupling between our endpoints and our database of choice. To summarise, we will quickly go through, step by step, what happens when an end-user sends a request:</p><ol><li>end-user sends GET /users?email=test@example.com</li><li>Our dependency injection Depends(get_user_service) is called and assigned to variable user_service .</li><li>get_user_service depends on get_user_repository, which depends on db_context.</li><li>From db_contextwe call our BaseSettingsto figure out, which dependency function to return. The settings are set to PostgreSQL, and thus db = Depends(db_context)<em> </em>sets dbto an async PostgreSQL Session.</li><li>get_user_repository<em> </em>now returns an instance of UserRepository, where the db of that instance is set to the async PostgreSQL Session.</li><li>get_user_service now returns an instance of UserService, where the repository is the UserRepository returned from get_user_repository<em> </em>before.</li><li>user_service<em> </em>is finally set to an instance of UserService, where its user_repositoryis an actual PostgreSQL implementation, with an actual async PostgreSQL Session to use for interacting with the database.</li></ol><p>For reference, these are our UserService and UserRepository classes. Please note that UserRepository implements the interface_UserRepository.</p><p><strong>/src/service/UserService.py</strong></p><pre>(...)<br><br>class UserService:<br><br>    def __init__(self, user_repository: interface_UserRepository):<br>        self.user_repository = user_repository<br>    <br>    async def get_user(self, email: str) -&gt; UserSchemas.UserResponse:<br>        try:<br>            user = await self.user_repository.get_user(email)<br>            return UserSchemas.UserResponse(<br>                email=user.email,<br>                is_active=user.is_active<br>            )<br>        except ResourceNotFoundException:<br>            raise #Re-raise from repository layer<br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Error getting user: {str(e)}&quot;) from e<br><br>(...)</pre><p>Here, we see that user_repository must be an implementation of interface_UserRepository.</p><p><strong>/src/repository/implementations/PostgreSQL/postgres_UserRepository.py</strong></p><pre>(...)<br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, db: AsyncSession):<br>        self.db = db<br><br>    async def get_user(self, email: str) -&gt; UserSchemas.User:<br>        try:<br>            stmt = select(UserORM).where(UserORM.email == email)<br>            result = await self.db.execute(stmt)<br>            db_user = result.scalar_one_or_none()<br>            if db_user:    <br>                return UserSchemas.User(<br>                    email=db_user.email,<br>                    is_active=db_user.is_active<br>                )<br>            else:<br>                logger.warning(f&quot;User with email {email} not found&quot;)<br>                raise ResourceNotFoundException(f&quot;User with email {email} not found&quot;)<br>            <br>        except ResourceNotFoundException:<br>            raise<br><br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Internal database error: {str(e)}&quot;) from e<br><br>(...)</pre><p>Here we can see:</p><ol><li>That we need an AsyncSessionto instantiate our PostgreSQL implementation of UserRepository. This is what we get from our db_context.</li><li>Using our UserORMdescribed in the beginning of this article, we are able to interact with our database in a pythonic manner. Specifically in the get_user method we are now making an actual async request to our database to retrieve a user.</li><li>Practical use of proper exception handling and logging. We now raise a ResourceNotFoundException when the requested user is not in our database, and if some other error occurs, we raise a BaseAppException with the actual error as message.</li></ol><h3>Containerization Strategy</h3><p>We can now use the official PostgreSQL image to create a local instance of a PostgreSQL database and test out our FastAPI microservice. Luckily, you can read my article about doing exactly that <a href="https://medium.com/@asbjorn.bering_39499/automated-api-fetching-in-postgresql-5820775e00b1">right here</a> and follow my actual implementation for this project <a href="https://github.com/assibering/CryptoWebApp/tree/main/PostgresDB">here</a>. For the purpose of this article, we will just show the docker compose file, to get everything running.</p><pre>services:<br>  postgresql:<br>    build:<br>      context: ./PostgresDB<br>      dockerfile: Dockerfile<br>      target: no-crypto-fetch-setup<br>    container_name: crypto_postgres<br>    env_file:<br>      - ./PostgresDB/.env<br>    environment:<br>      POSTGRES_USER: postgres<br>      POSTGRES_PASSWORD: your_secure_password<br>    ports:<br>      - &quot;5432:5432&quot;<br>    volumes:<br>      - postgres_data:/var/lib/postgresql/data<br>    restart: unless-stopped<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U postgres&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 5<br>  <br>  user_service_unit:<br>    # Build the image from the Dockerfile in the /UserService directory<br>    build:<br>      context: ./UserService<br>      dockerfile: Dockerfile.dev<br>      target: full-test #unit-test, integration-test, full-test, production (no tests are run)<br>    container_name: crypto_user_service<br>    env_file:<br>      - ./UserService/.env<br><br>    # Host the FastAPI application on port 8000<br>    ports:<br>      - &quot;8000:8000&quot;<br><br>    depends_on:<br>      postgresql:<br>        condition: service_healthy</pre><p>Of course, we would never put our actual user and password directly in the compose file — instead, we would use a separate .envfile.</p><p>In this article we have now seen the Repository Pattern in action. We’ve achieved loose coupling wrt. choice of database throughout the application. It’s not fully powerful yet, since we only have a PostgreSQL implementation. In the next article, we’ll unlock the full potential of this architecture by adding a new database backend: DynamoDB.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85b48497c324" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-5-10-postgresql-85b48497c324">Building Enterprise Python Microservices with FastAPI in 2025 (5/10): PostgreSQL</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (4/10): Logging and Exception…]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-4-10-logging-and-exception-5b6f897d9dc2?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b6f897d9dc2</guid>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sun, 18 May 2025 21:39:48 GMT</pubDate>
            <atom:updated>2025-05-19T15:28:42.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Enterprise Python Microservices with FastAPI in 2025 (4/10): Logging and Exception Handling</h3><h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In the previous article, I built the initial structure for my FastAPI User Service microservice. In this article, we’ll focus on two essential aspects for making the service robust and observable: exception handling and logging.</p><p>The primary goal of proper exception handling is to ensure that end-users receive clear and meaningful responses when their requests fail. Effective logging, on the other hand, enables us to monitor the User Service, quickly detect unexpected behavior, and respond rapidly to issues as they arise. In an enterprise-grade microservice, we must therefore include exception handling and logging.</p><p>To begin with I introduce middleware as it becomes relevant to logging, then we dive into logging, and finally we will check out exception handling.</p><h3>Middleware</h3><p>Before we dive into logging, which is one of the main themes of this article, let’s quickly introduce middleware, as it plays an important role in how we log different events.</p><p>Middleware in FastAPI acts as a “gate” that sits between the end-user and our endpoints. It can inspect, modify, or even reject incoming requests before they reach our route handlers, and it can also process outgoing responses. In the previous article, you might recall that I created CORS middleware. The CORS middleware checks that the incoming request is made from an authorised origin. In a real-world scenario, this could be the URL of the frontend, which would naturally be authorised to interact with the API. In our current setup, we’ve just included localhost:8000 so we can interact with the service locally while it’s running in a Docker container.</p><p>Middleware isn’t limited to security checks. We can also use it to add properties to the incoming request, which is exactly what we’re going to do here. Specifically, we’ll include an X-Correlation-ID in the headers of each request for the purpose of logging. If a request doesn’t already have a correlation ID, our middleware will generate one (a UUID) and attach it to both the request and the response headers.</p><p>So, why do we use a correlation ID in logging? The correlation ID makes it possible to link a single request to all the downstream methods in our microservice that are executed as a result of that request. For instance, a user might send a GET /users?email=test@example.com request, which triggers the execution of several methods. With a correlation ID, we’re able to not only monitor the success or failure of these functions, but also relate them back to the original request. This is important because it makes it much easier to trace the flow of a request throughout our service and quickly find the root cause if something goes wrong.</p><p><strong>/src/middleware/correlation_context.py</strong></p><pre>import contextvars<br><br># Create a context var (thread-safe, async-safe)<br>_correlation_id_ctx_var = contextvars.ContextVar(&quot;correlation_id&quot;, default=None)<br><br>def set_correlation_id(correlation_id: str):<br>    _correlation_id_ctx_var.set(correlation_id)<br><br>def get_correlation_id() -&gt; str:<br>    return _correlation_id_ctx_var.get()</pre><ol><li>Creates an async-safe context variable using Python’s contextvars</li><li>set_correlation_id stores the ID in the current context</li><li>get_correlation_id retrieves it from any part of your code</li></ol><p>Why ContextVar?<br>Unlike thread-local storage, contextvars works with async/await code, making it safe for FastAPI&#39;s concurrent request handling. This ensures that each request’s correlation ID is isolated, even when many requests are handled concurrently.</p><p><strong>/src/middleware/correlation_id_middleware.py</strong></p><pre>import uuid<br>from starlette.middleware.base import BaseHTTPMiddleware<br>from starlette.requests import Request<br>from src.middleware.correlation_context import set_correlation_id<br><br>class CorrelationIdMiddleware(BaseHTTPMiddleware):<br>    async def dispatch(self, request: Request, call_next):<br>        correlation_id = request.headers.get(&quot;X-Correlation-ID&quot;) or str(uuid.uuid4())<br>        <br>        # Save in context var<br>        set_correlation_id(correlation_id)<br>        <br>        # Make it available to response too<br>        response = await call_next(request)<br>        response.headers[&quot;X-Correlation-ID&quot;] = correlation_id<br>        return response</pre><ol><li>Checks incoming headers → uses existing ID or generates new UUID</li><li>Persists ID to context variable (available during request lifecycle)</li><li>Adds ID to response headers (client can use for debugging)</li></ol><p>By adding the ID to response headers, we allow clients and downstream systems to trace requests end-to-end.</p><h3>Logging</h3><p>Now we have covered correlation IDs, what they do, why it’s important, and finally how we implement it. Now we are going to actually configure our logging.</p><p><strong>/src/logging_config.py</strong></p><pre>from logging.config import dictConfig<br><br>class CorrelationIdFilter:<br>    def __call__(self, record):<br>        from src.middleware.correlation_context import get_correlation_id<br>        record.correlation_id = get_correlation_id() or &quot;-&quot;<br>        return True<br><br>def setup_logging():<br>    dictConfig({<br>        &quot;version&quot;: 1,<br>        &quot;disable_existing_loggers&quot;: False,<br>        &quot;formatters&quot;: {<br>            &quot;json&quot;: {<br>                &quot;format&quot;: &#39;{&quot;timestamp&quot;: &quot;%(asctime)s&quot;, &quot;level&quot;: &quot;%(levelname)s&quot;, &quot;correlation_id&quot;: &quot;%(correlation_id)s&quot;, &quot;logger&quot;: &quot;%(name)s&quot;, &quot;message&quot;: &quot;%(message)s&quot;, &quot;pathname&quot;: &quot;%(pathname)s&quot;, &quot;lineno&quot;: %(lineno)d}&#39;,<br>            },<br>        },<br>        &quot;filters&quot;: {<br>            &quot;correlation_id_filter&quot;: {<br>                &quot;()&quot;: CorrelationIdFilter,<br>            },<br>        },<br>        &quot;handlers&quot;: {<br>            &quot;console&quot;: {<br>                &quot;class&quot;: &quot;logging.StreamHandler&quot;,<br>                &quot;formatter&quot;: &quot;json&quot;,<br>                &quot;filters&quot;: [&quot;correlation_id_filter&quot;],<br>            },<br>        },<br>        &quot;root&quot;: {<br>            &quot;level&quot;: &quot;INFO&quot;,<br>            &quot;handlers&quot;: [&quot;console&quot;],<br>        },<br>    })</pre><ol><li>Filter fetches correlation ID from context variable</li><li>JSON formatter includes correlation_id field</li><li>Filter attached to logging configuration</li><li>All console logs now include the correlation ID</li></ol><p>We choose to format our logs as JSON and log them to the console. This is best practice if we later want to implement external services like Grafana for monitoring — which is highly relevant in an enterprise-grade setup, where you typically use a service like Grafana. By logging in structured JSON, these tools can easily ingest, search, and visualise our logs, making it much easier to monitor service health, troubleshoot issues, and set up real-time alerts as our system scales.</p><p>We won’t be implementing these tools in this series, but as we are building an enterprise-grade microservice, we should make our logs compatible with them, since these tools are standard in enterprise-grade microservices.</p><p><strong>/src/main.py</strong></p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from .routes import UserController<br>import logging<br>from src.logging_config import setup_logging<br>from fastapi.responses import JSONResponse<br>from src.middleware.correlation_id_middleware import CorrelationIdMiddleware<br>from contextlib import asynccontextmanager<br><br>setup_logging()<br>logger = logging.getLogger(__name__)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    logger.info(&quot;Running startup tasks...&quot;)<br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>origins = [<br>    &quot;http://localhost:8000&quot;,<br>]<br><br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;GET&quot;, &quot;PUT&quot;],<br>    allow_headers=[&quot;Authorization&quot;, &quot;Content-Type&quot;]<br>)<br><br># Enable Correlation ID middleware<br>app.add_middleware(CorrelationIdMiddleware)<br>app.include_router(UserController.router)</pre><p>Finally, putting it all together, we are now setting up our logger, so our logs will be as configured as defined in /src/logger_config.py. We are also adding our CorrelationIdMiddleware so all requests are assigned a correlation ID.</p><h3>Logger in action</h3><p>Now we can start logging in our methods. We are going to use the PostgreSQL implementation of UserRepository as an example.</p><p><strong>/src/repository/implementations/PostgreSQL/postgres_UserRepository.py</strong></p><pre>from src.repository.interfaces import interface_UserRepository<br>from src.schemas import UserSchemas<br>import logging<br><br>logger = logging.getLogger(__name__)<br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, db = None):<br>        self.db = db<br><br>    async def get_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.User: <br><br>        # Dummy logging - log method was called<br>        logger.info(&quot;PostgreSQL get_user method called&quot;)<br><br>        return UserSchemas.User(<br>            email=email,<br>            is_active=is_active<br>        )<br>(...)</pre><p>This will result in the following logs written to console:</p><pre>{<br>  &quot;timestamp&quot;: &quot;2025-05-10 17:02:33&quot;,<br>  &quot;level&quot;: &quot;INFO&quot;,<br>  &quot;correlation_id&quot;: &quot;f47ac10b-58cc-4372-a567-0e02b2c3d479&quot;,<br>  &quot;logger&quot;: &quot;src.repository.implementations.postgres_UserRepository&quot;,<br>  &quot;message&quot;: &quot;PostgreSQL get_user method called&quot;,<br>  &quot;pathname&quot;: &quot;/app/src/repository/implementations/postgres_UserRepository.py&quot;,<br>  &quot;lineno&quot;: 12<br>}</pre><p>For now, our logs are simple, but as we implement proper exception handling, it will become clear how to use logging more effectively-especially for failures and critical events.</p><p>The purpose of logging is to monitor the health of our services. Therefore, we can log a multitude of things, and deciding what to log and what not to log is critical, as one should only monitor what is truly important to the health of the service. For instance the health of a service could be highly dependent of fast execution of one or more core functions. In this case it would be relevant to log execution times of these functions. Another obvious case is to log when something is failing, so it is possible to monitor repetitive failures, as this might indicate a flaw in the function itself, or some other recurring root cause that needs investigation. Setting up proper logging is therefore critical in an enterprise-grade microservice. It is also important to consider over-logging, as over-logging can create noise and make it harder to spot real issues.</p><h3>Exception Handling</h3><p>In this section I will introduce how to handle exceptions in a FastAPI application. This is done to make our API robust and return reasonable responses to the end-user when their request for some reason fails. To control exceptions, we are creating custom exceptions. This is critical as we want to have control over what is returned to the end-user. As services evolve we, in combination with our logs from before, will discover uncaught exceptions from time to time. But because of our logs, we are able to investigate these uncaught exceptions, and implement new custom exceptions to catch these. This is why proper logging and exception handling is really powerful.</p><p><strong>/src/exceptions.py</strong></p><pre>class BaseAppException(Exception):<br>    &quot;&quot;&quot;Base exception for our application&quot;&quot;&quot;<br>    def __init__(self, message: str, status_code: int = 500):<br>        self.message = message<br>        self.status_code = status_code<br>        super().__init__(self.message)<br><br>class ResourceNotFoundException(BaseAppException):<br>    &quot;&quot;&quot;Raised when a requested resource is not found&quot;&quot;&quot;<br>    def __init__(self, message: str):<br>        super().__init__(message, status_code=404)<br><br><br>(...)</pre><p>This is how we define our custom exceptions in our FastAPI application. It basically returns the message from the exception alongside an http status code. Additionally the exceptions <em>inherits</em> from the custom BaseAppException. The purpose is that every uncaught exception is returned to the end-user as a BaseAppException with http status code 500, which means <em>Internal Server Error</em>. Ideally, we will never return these, but as previously stated we can use our logs to track when these occur, and implement new custom exceptions to handle them reasonably. Sometimes we will want to return an <em>Internal Server Error</em>, but we don’t want it to then be the unhandled BaseAppException but rather something like a custom ConnectionException, which inherits from BaseAppException, if e.g. our service is unable connect to our database.</p><p><strong>/src/main.py</strong></p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from .routes import UserController<br>from .exceptions import BaseAppException<br>import logging<br>from src.logging_config import setup_logging<br>from fastapi.responses import JSONResponse<br>from src.middleware.correlation_id_middleware import CorrelationIdMiddleware<br>from contextlib import asynccontextmanager<br><br>setup_logging()<br>logger = logging.getLogger(__name__)<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    logger.info(&quot;Running startup tasks...&quot;)<br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>origins = [<br>    &quot;http://localhost:8000&quot;,<br>]<br><br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;GET&quot;, &quot;PUT&quot;],<br>    allow_headers=[&quot;Authorization&quot;, &quot;Content-Type&quot;]<br>)<br><br># Enable Correlation ID middleware<br>app.add_middleware(CorrelationIdMiddleware)<br><br># Global exception handler<br>@app.exception_handler(BaseAppException)<br>async def app_exception_handler(request, exc):<br>    logger.error(&quot;Application error: %s&quot;, exc.message)<br>    return JSONResponse(<br>        status_code=exc.status_code,<br>        content={&quot;error&quot;: exc.message}<br>    )<br><br><br>app.include_router(UserController.router)</pre><p>Here we implement a <em>Global exception handler</em>. The purpose is that whenever an exception is caught in our application, we raise it to the end-user. We aim to only raise BaseAppException or its subclasses, which is why we register a global handler for BaseAppException. The handler returns the exception caught in JSON format to the end-user, and also logs that an error occurred.</p><p><strong>/src/repository/implementations/PostgreSQL/postgres_UserRepository.py</strong></p><pre>from src.repository.interfaces import interface_UserRepository<br>from src.schemas import UserSchemas<br>from src.exceptions import ResourceNotFoundException, BaseAppException, ResourceAlreadyExistsException<br>import logging<br><br>logger = logging.getLogger(__name__)<br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, db = None):<br>        self.db = db<br><br>    async def get_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.User: <br><br>        # Dummy logging - log method was called<br>        logger.info(&quot;PostgreSQL get_user method called&quot;)<br>        <br>        try<br>            # Dummy exception<br>            if email == &quot;nonexistent@example.com&quot;:<br>                logger.warning(f&quot;User with email {email} not found&quot;)<br>                raise ResourceNotFoundException(f&quot;User with email {email} not found&quot;)<br>            <br>            return UserSchemas.User(<br>                email=email,<br>                is_active=is_active<br>            )<br>        <br>        # Catch the ResourceNotFoundException and raise to other layers<br>        except ResourceNotFoundException:<br>            raise<br><br>        # Catch the Uncaught exception and raise to other layers as BaseAppException<br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Internal database error: {str(e)}&quot;) from e<br><br>(...)</pre><p>Here, we demonstrate a common real-world scenario: an end-user requests information about a non-existent user record. In this case we want to return a 404 response (Not Found) and a message stating that the user was not found. Currently we are doing that correctly in our repository layer, but now we must also ensure our service layer catches this ResourceNotFoundException so it can “travel” all the way to the response we end up returning to the user.</p><p><strong>/src/service/UserService.py</strong></p><pre>from src.repository.interfaces import interface_UserRepository<br>from src.schemas import UserSchemas<br>from .utils import saltAndHashedPW<br>from src.exceptions import BaseAppException, ResourceNotFoundException, ResourceAlreadyExistsException<br>import logging<br><br>logger = logging.getLogger(__name__)<br><br>class UserService:<br><br>    def __init__(self, user_repository: interface_UserRepository):<br>        self.user_repository = user_repository<br>    <br>    async def get_user(self, email: str) -&gt; UserSchemas.UserResponse:<br>        try:<br><br>            user = await self.user_repository.get_user(email)<br>            return UserSchemas.UserResponse(<br>                email=user.email,<br>                is_active=user.is_active<br>            )<br><br>        except ResourceNotFoundException:<br>            raise #Re-raise from repository layer<br><br>        except Exception as e:<br>            logger.exception(f&quot;Error getting user: {str(e)}&quot;)<br>            raise BaseAppException(f&quot;Error getting user: {str(e)}&quot;) from e<br><br><br>(...)</pre><p>Here it becomes clear that we catch the exception from our repository layer in our service layer. Now this will finally be caught by our <em>Global exception handler</em> and returned to the user. This wraps up how I implement exception handling in a FastAPI application.</p><h3>Project Architecture</h3><p>We have added some new files, and this is how our current file and folder structure now looks with exception handling and logging.</p><pre>.<br>├── Dockerfile.dev<br>├── README.md<br>├── docker-compose.yaml<br>├── pyproject.toml<br>├── src<br>│   ├── dependencies.py<br>│   ├── exceptions.py<br>│   ├── logging_config.py<br>│   ├── main.py<br>│   ├── middleware<br>│   │   ├── __init__.py<br>│   │   ├── correlation_context.py<br>│   │   └── correlation_id_middleware.py<br>│   ├── repository<br>│   │   ├── implementations<br>│   │   │   ├── AWS_DynamoDB<br>│   │   │   │   ├── __init__.py<br>│   │   │   │   └── awsdynamodb_UserRepository.py<br>│   │   │   ├── AWS_RDS<br>│   │   │   └── PostgreSQL<br>│   │   │       ├── __init__.py<br>│   │   │       └── postgres_UserRepository.py<br>│   │   └── interfaces<br>│   │       ├── __init__.py<br>│   │       └── interface_UserRepository.py<br>│   ├── routes<br>│   │   ├── UserController.py<br>│   │   └── __init__.py<br>│   ├── schemas<br>│   │   ├── UserSchemas.py<br>│   │   └── __init__.py<br>│   └── service<br>│       ├── UserService.py<br>│       ├── __init__.py<br>│       └── utils.py<br>└── uv.lock</pre><p>In this article I have unlocked the power of proper exception handling and logging, and demonstrated how it can be implemented in a modern FastAPI application. In the next article, I’ll implement a real PostgreSQL backend so we can see logging and exception handling in action with real data in a real database.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b6f897d9dc2" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-4-10-logging-and-exception-5b6f897d9dc2">Building Enterprise Python Microservices with FastAPI in 2025 (4/10): Logging and Exception…</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (3/10): Project Setup]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-3-10-project-setup-1113658c9f0e?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/1113658c9f0e</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[repository-pattern]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[uv]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sat, 17 May 2025 00:47:02 GMT</pubDate>
            <atom:updated>2025-05-19T15:30:27.168Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github Repository</a></p><p>In this article we are going to initialise our FastAPI User Service microservice using uv. Recently uv has gained massive traction, and I won’t be spending time arguing why this is the best choice of package manager in 2025 — because an abundance of others have already done exactly that.</p><p>Afterwards, I am going to build our service following the Repository Pattern. The Repository Pattern is so popular by now that people are actually starting to hate it a little bit, but fact is that it works, it’s intuitive and there is a good chance your current and/or future co-developers will know it. Therefore it is still very relevant in 2025.</p><p>Finally, I am going to run our microservice in a container using Docker. Docker is still very much part of the landscape when it comes to containerisation in 2025, and it’s just as popular and relevant as always.</p><h3>Initializing our FastAPI application</h3><p>First off, I am going to create a new project using uv in my /UserService folder using:</p><pre>uv init</pre><p>This will create the following tree:</p><pre>.<br>├── .python-version<br>├── README.md<br>├── main.py<br>└── pyproject.toml</pre><p>The .python-version file holds information about, you guessed it, your python version, while your pyproject.toml file states your application dependencies. Initially, this is all you need. But as the application grows, some more dependencies will be added.</p><pre>[project]<br>name = &quot;userservice&quot;<br>version = &quot;0.1.0&quot;<br>description = &quot;Add your description here&quot;<br>readme = &quot;README.md&quot;<br>requires-python = &quot;&gt;=3.12&quot;<br>dependencies = [<br>    &quot;fastapi&gt;=0.115.12&quot;,<br>    &quot;uvicorn&gt;=0.34.1&quot;,<br>    &quot;pydantic&gt;=2.11.3&quot;<br>]</pre><p>Finally, you can run:</p><pre>uv sync</pre><p>or</p><pre>uv run main.py</pre><p>to create a .venv folder (a virtual environment) and a uv.lock file completing the initialisation of our User Service using uv. So why is uv so amazing? to summarise the abundance of articles it comes down to simplicity and efficiency. Written in Rust, uv is 10–100x faster than pip. And as I just demonstrated, it is incredibly easy to create virtual environments and handling dependencies. Many have already written articles praising uv, so I won’t bother you too much with it — just know it is amazing and if curious to why, take a look at their <a href="https://docs.astral.sh/uv/">official docs</a> and do a quick search here on Medium.</p><h3>Project Architecture: A Production-Ready Structure</h3><p>Oftentimes I read articles about structure, but too rarely they actually present actual file and folder structure. I believe consideration when it comes to file and folder structure is crucial, and also incredibly easy to present. So this is how our User Service is going to look initially:</p><pre>.<br>├── Dockerfile.dev<br>├── README.md<br>├── docker-compose.yaml<br>├── pyproject.toml<br>├── src<br>│   ├── dependencies.py<br>│   ├── main.py<br>│   ├── repository<br>│   │   ├── implementations<br>│   │   │   ├── AWS_DynamoDB<br>│   │   │   │   ├── __init__.py<br>│   │   │   │   └── awsdynamodb_UserRepository.py<br>│   │   │   ├── AWS_RDS<br>│   │   │   └── PostgreSQL<br>│   │   │       ├── __init__.py<br>│   │   │       └── postgres_UserRepository.py<br>│   │   └── interfaces<br>│   │       ├── __init__.py<br>│   │       └── interface_UserRepository.py<br>│   ├── routes<br>│   │   ├── UserController.py<br>│   │   └── __init__.py<br>│   ├── schemas<br>│   │   ├── UserSchemas.py<br>│   │   └── __init__.py<br>│   └── service<br>│       ├── UserService.py<br>│       ├── __init__.py<br>│       └── utils.py<br>└── uv.lock</pre><p>Please use this as reference when we dive into the code in the following sections. I won’t describe the reasoning behind this structure, which will hopefully be self-explanatory once we start implementing in the next sections.</p><p><strong>/src/schemas/UserSchemas.py</strong></p><pre>from pydantic import BaseModel, model_validator<br>from typing_extensions import Self<br>from typing import Optional<br><br>class ResetPassword(BaseModel):<br>    password: str<br>    password_repeat: str<br><br>    @model_validator(mode=&#39;after&#39;)<br>    def check_passwords_match(self) -&gt; Self:<br>        if self.password != self.password_repeat:<br>            raise ValueError(&#39;Passwords do not match&#39;)<br>        return self<br><br>class User(BaseModel):<br>    email: str<br>    hashed_password: Optional[str] = None<br>    is_active: Optional[bool] = None<br><br>class UserResponse(BaseModel):<br>    email: str<br>    is_active: bool</pre><p>First of all, these are the Pydantic schemas I am going to use. Pydantic is a popular data validation library, and it is widely used for FastAPI applications. It works very well in FastAPI applications, because request bodies are automatically validated when using Pydantic, e.g. are the required fields present with the correct data types in the request body. In the above you can also see a custom model_validator which checks whether the two passwords, password and password_repeat, are equal. This is relevant when a user sends a request to change their password, as they will only be able to do so, if the two passwords they provide are equal. You will see extensive use of these <em>User schemas </em>throughout the application.</p><p><strong>/src/repository/interfaces/interface_UserRepository.py</strong></p><pre>from abc import ABC, abstractmethod<br>from ...schemas import UserSchemas<br><br>class UserRepository(ABC):<br><br>    @abstractmethod<br>    async def get_user(self, email: str) -&gt; UserSchemas.User:<br>        pass<br><br>    @abstractmethod<br>    async def create_user(self, User_instance: UserSchemas.User):<br>        pass<br><br>    @abstractmethod<br>    async def update_user(self, User_instance: UserSchemas.User):<br>        pass</pre><p>To achieve loose coupling between the service layer and the actual database implementation, I create an interface. This means my actual implementation <em>implements</em> this interface, and it also means my service layer does not depend on the actual implementation, but on the interface. You can also see here that we have defined that the get_user<em> </em>method must return a User which we defined in our UserSchemas.pyabove.</p><p><strong>/src/repository/implementations/PostgreSQL/postgres_UserRepository.py</strong></p><pre>from src.repository.interfaces import interface_UserRepository<br>from src.schemas import UserSchemas<br><br><br>class UserRepository(interface_UserRepository.UserRepository):<br><br>    def __init__(self, db = None):<br>        self.db = db<br><br>    async def get_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.User: <br><br>        return UserSchemas.User(<br>            email=email,<br>            is_active=is_active<br>        )<br><br><br>    async def create_user(<br>        self,<br>        User_instance: UserSchemas.User<br>    ) -&gt; UserSchemas.User:<br><br>        return UserSchemas.User(<br>            email=User_instance.email,<br>            is_active=True if User_instance.is_active else False<br>        )<br>        <br><br>    async def update_user(<br>        self,<br>        User_instance: UserSchemas.User<br>    ) -&gt; UserSchemas.User:<br>        <br>        return UserSchemas.User(<br>            email=User_instance.email,<br>            is_active=User_instance.is_active<br>        )<br>            </pre><p>So here is our actual UserRepository implementation, which implements the aforementioned interface. Initially we will not bother with making an actual PostgreSQL implementation, but simply prepare for it. The actual PostgreSQL implementation will be presented later in this series.</p><p><strong>/src/service/UserService.py</strong></p><pre>from src.repository.interfaces import interface_UserRepository<br>from src.schemas import UserSchemas<br>from .utils import saltAndHashedPW<br><br>class UserService:<br><br>    def __init__(<br>        self,<br>        user_repository: interface_UserRepository<br>    ):<br>        self.user_repository = user_repository<br>    <br>    async def get_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.UserResponse:<br><br>        user = await self.user_repository.get_user(email)<br>        return UserSchemas.UserResponse(<br>            email=user.email,<br>            is_active=user.is_active<br>        )<br>        <br>    async def create_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.UserResponse:<br>        <br>        user = await self.user_repository.create_user(<br>            User_instance = UserSchemas.User(<br>                email=email<br>            )<br>        )<br>        return UserSchemas.UserResponse(<br>            email=user.email,<br>            is_active=user.is_active<br>        )<br><br>    <br>    async def reset_password(<br>        self,<br>        email: str,<br>        reset_password: UserSchemas.ResetPassword<br>    ) -&gt; UserSchemas.UserResponse:<br><br>        hashed_pw = await saltAndHashedPW(reset_password.password)<br>        user = await self.user_repository.update_user(<br>            UserSchemas.User(<br>                email=email,<br>                hashed_password=hashed_pw,<br>                is_active=True<br>            )<br>        )<br><br>        return UserSchemas.UserResponse(<br>            email=user.email,<br>            is_active=user.is_active<br>        )<br><br>    <br>    async def deactivate_user(<br>        self,<br>        email: str<br>    ) -&gt; UserSchemas.UserResponse:<br><br>        user = await self.user_repository.update_user(<br>            UserSchemas.User(<br>                email=email,<br>                is_active=False<br>            )<br>        )<br><br>        return UserSchemas.UserResponse(<br>            email=user.email,<br>            is_active=user.is_active<br>        )</pre><p>The Repository Pattern in action. Here you can see how my UserService depends on the interface, which is an abstraction, instead of the actual implementation. This means I can later make another implementation, e.g. DynamoDB, which <em>implements</em> the interface, and I would not need to change my service layer one bit. This is one of the major reasons it is incredibly popular, and later you shall see even more benefits when we start writing unit tests.</p><p><strong>/src/routes/UserController.py</strong></p><pre>from fastapi import APIRouter, Depends<br>from src.schemas import UserSchemas<br>from src.dependencies import get_user_service<br>from src.service.UserService import UserService<br><br>router = APIRouter(<br>    prefix=&quot;/users&quot;<br>)<br><br>@router.get(&quot;&quot;, status_code=200)<br>async def get_user(<br>    email: str,<br>    user_service: UserService = Depends(get_user_service)<br>):<br><br>    return await user_service.get_user(email=email)<br><br><br>@router.put(&quot;/reset-password&quot;, status_code=201)<br>async def reset_password(<br>    email: str,<br>    reset_password: UserSchemas.ResetPassword,<br>    user_service: UserService = Depends(get_user_service)<br>):<br><br>    return await user_service.reset_password(<br>        email=email,<br>        reset_password=reset_password<br>    )<br><br>@router.put(&quot;/deactivate-user&quot;, status_code=201)<br>async def deactivate_user(<br>    email: str,<br>    user_service: UserService = Depends(get_user_service)<br>):<br><br>    return await user_service.deactivate_user(<br>        email=email<br>    )</pre><p>Now, finally we are at our Controller layer, and as you can see we are using <em>Dependency Injection</em>, which is extremely popular in FastAPI. You see our endpoints <em>depend</em> on get_user_service<em> </em>and assigns whatever this function returns (Spoiler, it’s a UserService instance) to the user_servicevariable.</p><p><strong>/src/dependencies.py</strong></p><pre>from fastapi import Depends<br>from src.repository.interfaces.interface_UserRepository import UserRepository as UserRepositoryInterface<br>from src.service.UserService import UserService<br>from src.repository.implementations.PostgreSQL.postgres_UserRepository import UserRepository as PostgresUserRepository<br><br>async def get_user_repository() -&gt; UserRepositoryInterface:<br>    return PostgresUserRepository()<br><br>async def get_user_service(user_repository: UserRepositoryInterface = Depends(get_user_repository)) -&gt; UserService:<br>    return UserService(user_repository)</pre><p>Now it becomes clear what we are <em>injecting</em> into our endpoints. Right now we are not injecting real database sessions, but later in this series you will see how we extend this to actual database implementations with real PostgreSQL and DynamoDB async sessions. It also becomes clear that we achieve loose coupling wrt. our database when we inject an abstract repository interface (rather than a concrete implementation) into our service and controller layers. This means our business logic depends only on the interface, making it easy to swap out or extend database backends-such as replacing a PostgreSQL implementation with a DynamoDB implementation without changing the rest of our application code. For now, we are injecting our PostgresUserRepository, which obviously defeats the point. Later, when we actually implement our PostgreSQL and DynamoDB database backends, we will implement a similar dependency for get_user_repository as get_user_service and then actually achieve loose coupling.</p><p><strong>/src/main.py</strong></p><pre>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from .routes import UserController<br>from fastapi.responses import JSONResponse<br>from contextlib import asynccontextmanager<br><br>@asynccontextmanager<br>async def lifespan(app: FastAPI):<br>    # Startup code (runs before application startup)<br>    logger.info(&quot;Running startup tasks...&quot;)<br>    logger.info(&quot;Startup tasks completed&quot;)<br>    yield<br>    # Shutdown code (runs after application shutdown)<br>    logger.info(&quot;Running shutdown tasks...&quot;)<br>    logger.info(&quot;Shutdown tasks completed&quot;)<br>    # This is where you put code that was previously in @app.on_event(&quot;shutdown&quot;)<br><br>app = FastAPI(lifespan=lifespan)<br><br>origins = [<br>    &quot;http://localhost:8000&quot;,<br>]<br><br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;GET&quot;, &quot;PUT&quot;],<br>    allow_headers=[&quot;Authorization&quot;, &quot;Content-Type&quot;]<br>)<br><br>app.include_router(UserController.router)</pre><p>We include our UserController in our FastAPI app to make the endpoints accessible. Additionally, I make some simple CORS middleware, which allows us to access our endpoints from <em>localhost:8000</em>, which means I will later be able to interact with my container from outside the container. This is necessary if I want to test my endpoints from my local machine using Postman. Finally, I set up lifespan to prepare for start up and shutdown tasks.</p><h3>How is a request handled?</h3><p>In the previous section I presented my initial codebase for my FastAPI User Service. I will now go through, step by step, when my endpoint receives a request. I will use GET /users?email=test@example.comas my example:</p><ol><li>End-user makes a GET /users?email=test@example.com request.</li><li>UserController sets user_service as a new UserService instance by calling the get_user_service function which is injected into the endpoint. In this example the user_service.repository is the PostgreSQL implementation of the interface_UserRepository.</li><li>user_service.get_user(email=’test@example.com’) is awaited.</li><li>The user_service.get_user<em> </em>method awaits user_service.repository.get_user(email=’test@example.com’) and since our repository is the PostgreSQL implementation, that is the specific method that is called. This method in the PostgreSQL implementation simply returns a User<em>, </em>as defined in UserSchemas.py<em>,</em> with email set as the one used in the request, in this case <em>test@example.com.</em></li><li>user_service_get_user<em> </em>then transforms the Userinto a UserResponse, which is then returned to the UserController.</li><li>The UserController returns the UserResponse<em> </em>to the end-user.</li></ol><h3>Containerization Strategy</h3><p>In the previous section I built my initial simple User Service, and in this section we will run our application in a container using Docker and start interacting with it. This is our Dockerfile, heavily inspired by uv’s <a href="https://github.com/astral-sh/uv-docker-example">multistage Dockerfile</a>:</p><pre>###########################################<br># BUILDER stage<br>###########################################<br>FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim AS builder<br>ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy<br><br># Disable Python downloads, because we want to use the system interpreter<br># across both images. If using a managed Python version, it needs to be<br># copied from the build image into the final image; see `standalone.Dockerfile`<br># for an example.<br>ENV UV_PYTHON_DOWNLOADS=0<br><br>WORKDIR /app<br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    --mount=type=bind,source=uv.lock,target=uv.lock \<br>    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \<br>    uv sync --frozen --no-install-project --no-dev<br>ADD . /app<br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    uv sync --frozen --no-dev<br><br>###########################################<br># PRODUCTION stage<br>###########################################<br>FROM python:3.12-slim-bookworm AS production<br><br># Copy the application from the builder<br>COPY --from=builder --chown=app:app /app /app<br><br># Production-specific env<br>ENV PATH=&quot;/app/.venv/bin:$PATH&quot;<br>ENV PYTHONPATH=&quot;/app&quot;<br><br># Default prod command (no reload)<br>CMD [&quot;uvicorn&quot;, &quot;src.main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8000&quot;]</pre><p>We are using this multistage Dockerfile specifically because it is the most suitable for production. This is the my docker-compose.yamlfile:</p><pre>## From the example docker-compose.yaml file provided by uv<br>## https://github.com/astral-sh/uv-docker-example<br>## https://github.com/astral-sh/uv-docker-example/blob/main/compose.yml<br><br>services:<br>  web:<br>    # Build the image from the Dockerfile in the current directory<br>    build:<br>      context: .<br>      dockerfile: Dockerfile.dev<br><br>    # Host the FastAPI application on port 8000<br>    ports:<br>      - &quot;8000:8000&quot;<br><br>    develop:<br>      # Create a `watch` configuration to update the app<br>      # https://docs.docker.com/compose/file-watch/#compose-watch-versus-bind-mounts<br>      watch:<br>        # Sync the working directory with the `/app` directory in the container<br>        - action: rebuild<br>          path: .<br>          target: /app<br>          # Exclude the project virtual environment — it could be for a<br>          # different platform in the container<br>          ignore:<br>            - .venv/<br><br>        # Rebuild the image if dependencies change by checking uv.lock<br>        - action: rebuild<br>          path: ./uv.lock</pre><p>Now, we can simply run docker compose up — watchand our service is running. We are using--watch so that when we make code changes, the container is rebuilt. This makes life easier as we do not have to manually shut down and rebuild our container to test code changes.</p><p>This wraps up the first of the implementation articles in this series. It may seem like a lot of work to make a very simply FastAPI service, but down the line when we extend functionality, it is going to be so much easier to add new features because we made the right structure from the get go. In the next article we are going to dive into Exceptions and Logging, to make our API robust and easy to monitor.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1113658c9f0e" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-3-10-project-setup-1113658c9f0e">Building Enterprise Python Microservices with FastAPI in 2025 (3/10): Project Setup</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (2/10): Project Introduction]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-2-10-project-introduction-d078cf82918d?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/d078cf82918d</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sat, 17 May 2025 00:45:30 GMT</pubDate>
            <atom:updated>2025-05-19T15:28:52.619Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github repository</a></p><p>In the first part of this series, I am going to introduce my hobby project, and the part of my hobby project I am going to build during this series.</p><h3>The Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vwlCwTK7Rp-S6RUZm3XaCQ.png" /></figure><p>I am building an application, where users can start a subscription using Stripe, and once their subscription has started, they can do all sorts of things using protected endpoints, e.g. backtest automated trading strategies in the Analysis Service. For now, I am going to focus solely on the User Service and the Subscription Service. In the following sections I am going to introduce our user stories, i.e. what does each service need to do.</p><h3>User Service</h3><ol><li><strong>GET /users?email=$USER_EMAIL<br></strong><em>Query Parameter: email: str<br></em>We should be able to retrieve user information excl. password information. This might be irrelevant, but it is nice for testing in this development phase.</li><li><strong>PUT /users/reset-password?email=$USER_EMAIL<br></strong><em>Query Parameter: email: str<br>Request Body: {password: str, password_repeat: str}<br></em>We should be able to change the users password, and ensure that the 2 passwords (password and password_repeat) match. This is relevant because users will 1) initially need to set a password to activate their account, and 2) change a compromised password later on.</li><li><strong>PUT /users/deactivate?email=$USER_EMAIL</strong><br><em>Query Parameter: email: str</em><br>We should be able to deactivate a user, i.e. changing their activity status to inactive. This is relevant if we want to e.g. pause memberships.</li></ol><h3>Subscription Service</h3><ol><li><strong>GET /subscriptions?subscription_id=$SUBSCRIPTION_ID<br></strong><em>Query Parameter: subscription_id: str</em><br>We should be able to retrieve subscription information. Similar to the get user method introduced before, this might be irrelevant, but nice for testing.</li><li><strong>POST /subscriptions/create-subscription</strong><br><em>Request Body: {email: str, subscription_type: str}<br></em>We should be able to start a new subscription, e.g. of type “free_tier” or “paid_tier”. This is where the Stripe integration is going to help us later, but for now users can just freely start a subscription using this endpoint. This endpoint will not only create a subscription, but also a user. This is where Kafka becomes relevant later on.</li></ol><h3>Database considerations</h3><p>Since I am following a microservice architecture, each service has their own database. In my case it means they have their own tables in the same database. This means database users have been set up for each of these services with the minimum access following best practises. There is no need for e.g. the User Service to have access to the tables related to the Subscription Service, and therefore the User Service database user does not have access to these tables. This follows the principle of least privilege (PoLP). This is accomplished using roles in PostgreSQL and IAM policies in DynamoDB — please checkout the <a href="https://github.com/assibering/CryptoWebApp">Github Repository</a> to see how this is achieved as it won’t be covered in this series.</p><p>As you can see I am building very simple and small services, and the reason is that I am not focusing on making services with advanced functionality. I am focusing on making quality services that will work in a real-world enterprise-grade setup — and as you will see later on, we can easily extend functionality when our boilerplate code is done.</p><p>In this era of vibe coding it also becomes increasingly relevant to understand these concepts, as we can leverage it to make intelligent prompts to generate quality boilerplate code, but also so we can debug proposed boilerplate code. I also believe developing boilerplate code has a bad reputation for being boring, so hopefully I can make you feel the same by the end of this series.</p><p>In the next article we are going to start building and take the first step into the world of developing microservices in Python using uv, FastAPI and docker. By the end of the next series we will have a simple application in a running container that we can interact with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d078cf82918d" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-2-10-project-introduction-d078cf82918d">Building Enterprise Python Microservices with FastAPI in 2025 (2/10): Project Introduction</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Enterprise Python Microservices with FastAPI in 2025 (1/10): Introduction]]></title>
            <link>https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-1-10-introduction-c1f6bce81e36?source=rss-ba9403b68709------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1f6bce81e36</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Asbjorn Bering]]></dc:creator>
            <pubDate>Sat, 17 May 2025 00:42:52 GMT</pubDate>
            <atom:updated>2025-05-19T15:29:06.297Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building an enterprise-grade FastAPI microservice architecture — uv, Docker, Kafka, Debezium, PostgreSQL, Pytest, DynamoDB (AWS), IAM (AWS), Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*seEAZfmhDij1AzorqA7FMA.png" /></figure><p><a href="https://github.com/assibering/CryptoWebApp">Github repository</a></p><p>In this series, I’m going to build a microservice architecture following best practices in 2025 using modern tools. The selected technologies are based solely on their relevancy in 2025, and I admit that especially Kafka and Debezium might seem like overkill for a hobby project — but the idea is to simulate an enterprise-grade architecture, and for that, they become highly relevant.</p><p>In this first article, I’ll break down everything we’re going to cover as we build the application from scratch. I’ll also mention what we’re not going to cover, as many of those areas are already well-documented elsewhere. It’s important to note that this is based on an actual hobby project of mine, which is why I don’t use “hello-world” kind of examples.</p><h3>What am I going to build?</h3><p>To begin with, I’ll introduce my hobby project and the specific part of it this series will focus on.</p><h3>FastAPI Project Architecture</h3><p>I’ll cover how I aim to achieve loose coupling using the well-known Repository Pattern. The reasoning behind this will become clearer later, when I implement two very different databases: PostgreSQL and DynamoDB.</p><h3>Docker</h3><p>I’ll show how I run everything locally, and how you can interact with your API using Postman. Docker will be a recurring part of the series as our application evolves.</p><h3>Exceptions and Logging</h3><p>Any enterprise-grade application needs proper exception handling and logging for traceability. So, we’ll take a deep dive into how this should be set up in your FastAPI app.</p><h3>Connecting to PostgreSQL</h3><p>Here we’ll see the Repository Pattern in action for the first time. We’ll also use BeeKeeper to interact with our database and check that the API works as expected.</p><h3>Connecting to DynamoDB</h3><p>When we implement the second database, it should become obvious why the Repository Pattern is so popular. I’ll also show how to use NoSQL Workbench to interact with the database, and how to deploy it using Terraform — mimicking a real-world scenario.</p><h3>Testing</h3><p>We’ll write both unit and integration tests, and show how to run them inside a container. Here, you’ll also see how our Repository Pattern makes unit testing much easier.</p><h3>Event-driven Architecture</h3><p>I’ll introduce the SAGA Choreography Pattern for handling distributed transactions across microservices, and the Outbox Table Pattern for solving the dual-write problem. I’ll implement both and test them in practice.</p><h3>What am I NOT going to build?</h3><p>First of all — a lot! I’ll try to cover the most obvious exclusions here:</p><ol><li><strong>Authentication</strong> — There’s already an abundance of guides on how to add authentication to FastAPI and protect endpoints using dependency injection.</li><li><strong>DynamoDB Streams, Lambda, SQS, SNS</strong> — These are natural choices when using DynamoDB in event-driven systems. However, I’ve chosen to focus on the more widely adopted PostgreSQL, Debezium, and Kafka stack for now. One cool thing about the AWS services, though, is that they’re all free under the AWS Free Tier and supported by the Localstack community edition.</li><li><strong>Kafka and Debezium configuration</strong> — This series focuses on how to <em>use</em> these technologies in a FastAPI microservice setup, not how to configure them in production.</li><li><strong>Limited integration testing</strong> — I won’t be writing tests that span multiple services. This kind of integration testing is of course important, but it is also too elaborate to fit in the main scope of this project. We are going to focus on testing each service independently, and make a framework for doing so.</li><li><strong>Anything deployment</strong> — This is a Developer series, not a DevOps series. That said, I do keep deployment in mind when choosing tools like Docker and Terraform.</li></ol><p>I’m really looking forward to writing about all of this and sharing why it’s highly relevant for microservice architecture in 2025. Hopefully, you’ll find yourself understanding more of the concepts and the considerations behind — and feel motivated to try them out for yourself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1f6bce81e36" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-enterprise-python-microservices-with-fastapi-in-2025-1-10-introduction-c1f6bce81e36">Building Enterprise Python Microservices with FastAPI in 2025 (1/10): Introduction</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>