<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis ADAM on Medium]]></title>
        <description><![CDATA[Stories by Louis ADAM on Medium]]></description>
        <link>https://medium.com/@lou_adam?source=rss-64b79bb4cc53------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*E05LBzSz4i-wVSY0Z_BsLg.jpeg</url>
            <title>Stories by Louis ADAM on Medium</title>
            <link>https://medium.com/@lou_adam?source=rss-64b79bb4cc53------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:20:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lou_adam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Best practices for building & deploying REST API as data engineer (concrete example with FastAPI)]]></title>
            <link>https://python.plainenglish.io/best-practices-for-building-deploying-rest-api-as-data-engineer-concrete-example-with-fastapi-84522745a9f7?source=rss-64b79bb4cc53------2</link>
            <guid isPermaLink="false">https://medium.com/p/84522745a9f7</guid>
            <category><![CDATA[best-practice-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Louis ADAM]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 09:02:29 GMT</pubDate>
            <atom:updated>2024-11-30T19:57:52.868Z</atom:updated>
            <content:encoded><![CDATA[<h3>Best practices for building &amp; deploying REST API (concrete example with FastAPI)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ndFoz6Q2Aly3z_hNiYZucQ.png" /></figure><p>“You want to avoid the spaghetti plate”, was the advice I received when I implemented my first API a few years ago. Since then, I’ve had the chance to gain valuable experience and gather best practices from development to deployment. Here’s a recap of the best practices I’ve learned along the way.</p><p><a href="#7dfc">Part 1: Best Practices for FastAPI Development</a></p><ul><li><a href="#d593">Recommended directory structure</a></li><li><a href="#ef7a">Naming conventions for routes, models, and schemas</a></li><li><a href="#06bd">Leveraging OpenAPI for the documentation, best practices for error responses, and pydantic for effective data validation</a></li></ul><p><a href="#c451">Part 2: Writing Unit Tests and Linters setup</a></p><ul><li><a href="#dac8">Best practices regarding unit testing and test coverage</a></li><li><a href="#0e85">Setting up linters</a></li></ul><p><a href="#1b8f">Part 3: Implementing continuous integration &amp; deployment (CI/CD)</a></p><ul><li><a href="#c9b1">Create a clear pipeline structure</a></li><li><a href="#e1f5">Automate testing and linting</a></li><li><a href="#b6b2">Keep your pipeline fast</a></li><li><a href="#f484">Use environment variables for configuration</a></li></ul><p><a href="#d0ca">Part 4: Ongoing Maintenance and Monitoring</a></p><ul><li><a href="#9e81">API Versioning for evolving applications</a></li><li><a href="#2591">Best practices for effective logging &amp; Monitoring</a></li></ul><h4>Part 1: Best Practices for FastAPI Development</h4><h4><strong>Recommended directory structure</strong></h4><p>Goal here is simple: Modularity and maintanability! Once you have a system that gives you both, you should be able to handle the scalability easily.</p><p>In practice you can either divide a project by file type (routers, models ..) which is suitable for small applications with limited complexity, making navigation straightforward. Or, for larger applications with multiple domains and features, you can organize and divide your project by domains and functionalities. This approach can be very beneficial for managing the codebase effectively as the complexity and number of developers can grow through time.</p><p>This folder structure example follows a modular, domain-driven approach, which is beneficial for both maintainability and modularity thus scalability.</p><p>In this example each domain has:</p><p><strong>router.py</strong>: to handles endpoint routing specific to that domain</p><p><strong>schemas.py</strong>: which contains Pydantic models for data validation and serialization</p><p><strong>models.py</strong>: includes database models (might be ORM) which allow to have a clear structure of the database interactions</p><p>In addition, you can have<strong><em> </em></strong><strong>constants.py </strong>to store any constant values; <strong>exceptions.py</strong>to handle custom errors more consistantly; and finaly <strong>utils.py<em> </em></strong>where you have the helper functions (such as string manipulation, hashing …).</p><h4><strong>Naming conventions for routes, models, and schemas</strong></h4><p>Basically, keep it simple &amp; be consistent:</p><ul><li>Avoid overly complex names or jargon. Names should be easily understandable by other devs</li><li>Ensure that naming conventions are applied consistently throughout the project to improve readability and maintainability.</li></ul><p><strong>For routes:</strong></p><ul><li><strong>Use descriptive names + HTTP methods</strong>: Route names should clearly indicate the resource while the method shoud indicate the action being performed. For example: @app.get(“/items”)for retrieving items (@app.post for creation, @app.patch for update ..)</li><li><strong>Use plural nouns for resource collections</strong>: @app.get(“/products”) (not /product )</li><li><strong>Use snake case for route names</strong>: @app.get(“/user_profile”) instead of /userProfile</li><li><strong>Include Versioning</strong>: (if applicable) @app.get(“/v1/products”)</li></ul><p><strong>For models:</strong></p><ul><li><strong>Use singular nouns</strong>: model names should represent a single instance of the resource:User, Product, Order</li><li><strong>PascalCase for class names</strong>: for model class names to differentiate them from other variables: class User(BaseModel):</li><li><strong>Use descriptive attributes</strong> <strong>+ snake_case</strong>: first_name, last_name, created_at</li></ul><p><strong>For schemas:</strong></p><ul><li><strong>Follow a consistent pattern</strong>: that indicates the purpose of the schema, such as:ResourceTypeActionSchema (for instance UserCreateSchema)</li><li><strong>Use PascalCase for schema names </strong>(as for models) class UserCreateSchema(BaseModel):</li><li><strong>Attribute naming consistency</strong>: Match schema attribute names with model attributes (when possible) to avoid confusion</li></ul><h4>Leveraging OpenAPI for the documentation, best practices for error responses, and pydantic for effective data validation</h4><p>To help FastAPI generate complete documentation, you can utilize several features and best practices. Here are some of them:</p><ul><li><strong>Description and summary</strong></li></ul><p>These details provide more context about the endpoint purpose and functionality</p><pre>@app.get(<br>  &quot;/items/{item_id}&quot;, <br>  summary=&quot;Read a specific item&quot;,<br>  description=&quot;This endpoint retrieves a specific item by its ID.&quot;<br>)<br>async def read_item(item_id: int):<br>    return {&quot;item_id&quot;: item_id}</pre><ul><li><strong>Response model and status code</strong></li></ul><p>Use the response_model and status_code parameters in your route decorators to specify the expected response structure and HTTP status code:</p><pre>class Item(BaseModel):<br>    name: str<br>    price: float<br><br>@app.post(<br>  &quot;/items/&quot;, <br>  response_model=Item, <br>  status_code=status.HTTP_201_CREATED<br>)<br>async def create_item(item: Item):<br>    return item</pre><ul><li><strong>Responses for different scenarios</strong></li></ul><p>Use the responses parameter to document different possible responses:</p><pre>from fastapi import status<br><br>@app.get( <br>  &quot;/items/{item_id}&quot;, <br>  responses={ <br>    status.HTTP_200_OK: {<br>        &quot;description&quot;: &quot;Ok Response&quot;,<br>    },<br>    status.HTTP_404_NOT_FOUND {<br>        &quot;description&quot;: &quot;Item not found&quot;,<br>    },<br>  })<br>async def read_item(item_id: int): <br>  pass</pre><p>This approach helps users understand the possible outcomes of the API call by automaticaly generating associated documentation in the UI swagger.</p><ul><li><strong>Combined <em>responses</em> with custom pydantic model</strong></li></ul><p>The following approach ensures that responses are consistently formatted, making it easier for users to use the API.</p><pre>from fastapi import FastAPI, HTTPException, status<br>from pydantic import BaseModel<br><br>app = FastAPI()<br><br># Define the Pydantic model for a successful response<br>class OkResponse(BaseModel):<br>    item_id: int<br>    name: str<br>    description: str = None<br><br>@app.get(<br>    &quot;/items/{item_id}&quot;,<br>    responses={<br>        status.HTTP_200_OK: {<br>            &quot;model&quot;: OkResponse,  # Custom Pydantic model for 200 response<br>            &quot;description&quot;: &quot;Ok Response&quot;,<br>        },<br>        status.HTTP_404_NOT_FOUND: {<br>            &quot;description&quot;: &quot;Item not found&quot;,<br>        },<br>    }<br>)<br>async def read_item(item_id: int):<br>    item = items_db.get(item_id)<br>    if item is None:<br>        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=&quot;Item not found&quot;)<br>    return OkResponse(item_id=item_id, **item)</pre><p>Here the pydantic model (OkResponse) defines the structure of the successful response, including the fields that will be returned to the client.</p><ul><li><strong>Don’t show your doc in prod (unless it’s a public API)</strong></li></ul><p>You should avoid showing API documentation in production environments to prevent exposing sensitive information, endpoints, and potential security vulnerabilities to unauthorized users.</p><pre>from fastapi import FastAPI<br>from starlette.config import Config<br><br>config = Config(&quot;.env&quot;)  # parse .env file for env variables<br><br>ENVIRONMENT = config(&quot;ENVIRONMENT&quot;)  # get current env name<br>SHOW_DOCS_ENVIRONMENT = (&quot;local&quot;, &quot;staging&quot;)  # explicit list of allowed envs<br><br>app_configs = {&quot;title&quot;: &quot;My Cool API&quot;}<br>if ENVIRONMENT not in SHOW_DOCS_ENVIRONMENT:<br>   app_configs[&quot;openapi_url&quot;] = None  # set url for docs as null<br><br>app = FastAPI(**app_configs)</pre><h4>Part 2: Writing Unit Tests and Linters setup</h4><h4>Best practices regarding unit testing and test coverage</h4><ul><li><strong>Organize Tests Logically</strong>: Structure your tests by feature or domain to improve readability and maintainability. For instance, create subdirectories for tests related to users, products or orders (cf directory organization proposed previously).</li><li><strong>Isolate Tests</strong>: Ensure that each test is independent of others. Avoid shared state between tests to prevent side effects.</li><li><strong>Utilize Async Testing</strong>: If you’re using async endpoints, make sure to use @pytest.mark.asyncio to handle async test cases correctly.</li><li><strong>Use TestClient and fixtures</strong>: The TestClient functionality allows to test endpoints as if they were being called by a real client. In addition,pytest fixtures are great to get common test data or configurations and ensuring consistency of the tests.</li></ul><pre>import pytest<br>from fastapi.testclient import TestClient<br><br>client = TestClient(app)<br><br>@pytest.fixture<br>def sample_item():<br>    &quot;&quot;&quot;Fixture to create a sample item.&quot;&quot;&quot;<br>    return Item(id=1, name=&quot;Sample Item&quot;, description=&quot;A sample item for testing.&quot;)<br><br>@pytest.fixture<br>def create_item(sample_item):<br>    &quot;&quot;&quot;Fixture to create an item in the database for testing.&quot;&quot;&quot;<br>    response = client.post(&quot;/items/&quot;, json=sample_item.dict())<br>    assert response.status_code == 200<br>    return response.json()<br><br>def test_create_item(create_item):<br>    response = client.get(&quot;/items/1&quot;)<br>    assert response.status_code == 200<br>    assert response.json() == {&quot;id&quot;: 1, &quot;name&quot;: &quot;Sample Item&quot;, &quot;description&quot;: &quot;A sample item for testing.&quot;}</pre><p>In this example the fixture sample_item creates a reusable item, while create_item posts it to the API. Then the test:test_create_item checks if the created item can be retrieved successfully using the two fixtures.</p><ul><li><strong>Mock External Dependencies</strong>: Use libraries like unittest.mock to mock external services or databases during tests to isolate the application logic and avoid side effects.</li></ul><pre># main.py<br><br>from fastapi import FastAPI, HTTPException<br>import requests<br><br>app = FastAPI()<br><br>@app.get(&quot;/external-data/&quot;)<br>async def get_external_data():<br>    response = requests.get(&quot;https://example.com/data&quot;)<br>    if response.status_code != 200:<br>        raise HTTPException(status_code=500, detail=&quot;External service error&quot;)<br>    return response.json()</pre><p>To test this endpoint, we define the test_get_external_data_error. the mock simulates a server error from the external service, allowing to verify that your application correctly handles this case.</p><pre># Test for the get_external_data function <br><br>import pytest<br>from fastapi.testclient import TestClient<br>from unittest.mock import patch<br>from main import app<br><br>client = TestClient(app)<br><br>@patch(&quot;main.requests.get&quot;)<br>def test_get_external_data_error(mock_get):<br>    # Simulate an error from the external service<br>    mock_get.return_value.status_code = 500<br><br>    response = client.get(&quot;/external-data/&quot;)<br>    <br>    assert response.status_code == 500<br>    assert response.json() == {&quot;detail&quot;: &quot;External service error&quot;}</pre><ul><li><strong>Test Both Success and Failure Cases</strong>: tests should cover the full range of possible outcomes, both successful responses and error (invalid input, resource not found…). In addition, the tests should handle all the responses we’ve defined previously in the <em>Responses for different scenarios</em> section.</li><li><strong>Regularly Review Coverage Reports: </strong>Analyzing coverage reports (using pytest_covfor instance) helps to verify that all paths and functionalities are thoroughly tested. However, instead of focusing on high percentage, it’s more important to ensure that the business logic is fully tested.</li></ul><h4>Setting up linters</h4><p>As your project grows, more and more dev will contribute to it. In this context, linters are perfect to ensure code quality and consistency accross the team. To ensure those coding standard, my preferences goes toward the following packages:</p><ul><li><strong>Black</strong>: An opinionated code formatter that enforces a consistent style.</li><li><strong>isort</strong>: A tool to sort and format import.</li><li><strong>Pyflakes</strong>: A simple tool for detecting Python errors without executing the code (make sure there is no unused variables/imports etc..).</li><li><strong>pytest</strong>: Not only for testing but can also help ensure code quality through fixtures and assertions.</li><li><strong>Snyk</strong>: A security focused tool that checks for vulnerabilities in dependencies.</li></ul><p>And more recenlty I’v been using</p><ul><li><strong>Ruff: </strong>which can be used to replace previous packages while being pretty fast</li></ul><p>Once those are setup, you can incorporate them in the CI/CD to ensure that every code contribution respect quality standard (more details about that in the following part).</p><h4>Part 3: Implementing continuous integration &amp; deployment (CI/CD)</h4><p>For this part, we’ll use Gitlab to illustrate the different section example.</p><h4>Create a clear pipeline structure</h4><p>In order to ensure fast development cycles, to fix integration issues and maintain a high quality through automated tests, it’s important to make sure that the CI/CD pipeline is well organized with distinct stages. To enhance the readability and the maintainability, those stages should be named with descriptive &amp; simple names so that we can undestand the workflow at a glance.</p><pre># gitlab-ci.yml<br><br>stages:<br>  - &quot;📏 Lint&quot;<br>  - &quot;🏗️ Build&quot;<br>  - &quot;📝 Test&quot;<br>  - &quot;📦 Push&quot;<br><br>lint_job:<br>  stage: &quot;📏 Lint&quot;<br>  image: python:3.9<br>  script:<br>    - pip install flake8<br>    - flake8 your_fastapi_app/<br><br>build_job:<br>  stage: &quot;🏗️ Build&quot;<br>  image: docker:latest<br>  services:<br>    - docker:dind<br>  script:<br>    - docker build -t your_docker_image:latest .<br><br>test_job:<br>  stage: &quot;📝 Test&quot;<br>  image: python:3.9<br>  script:<br>    - pip install -r requirements.txt<br>    - pytest tests/<br><br>push_job:<br>  stage: &quot;📦 Push&quot;<br>  image: docker:latest<br>  services:<br>    - docker:dind<br>  script:<br>    - echo &quot;$DOCKER_PASSWORD&quot; | docker login -u &quot;$DOCKER_USERNAME&quot; --password-stdin<br>    - docker push your_docker_image:latest<br></pre><p>This simple example allows for a structured workflow where code quality is ensured before building, followed by testing to confirm functionality, and finally pushing the application for deployment.</p><h4>Automate testing and linting</h4><p>To ensure high code quality and a consistent code structure among developers working on the project, it’s essential to automate linting and unit testing within the CI/CD pipeline. This approach enforces coding standards and helps prevent issues or poorly formatted code from making their way into production.</p><pre># gitlab-ci.yml<br><br>stages:<br>  - &quot;📏 Lint&quot;<br>  - &quot;🏗️ Build&quot;<br>  - &quot;📝 Test&quot;<br>  - &quot;📦 Push&quot;<br><br>include:<br>  - local: jobs/test/python/black.gitlab-ci.yml<br>  - local: jobs/test/python/isort.gitlab-ci.yml<br>  ...<br>  - local: jobs/test/python/pytest.gitlab-ci.yml</pre><pre># black.gitlab-ci.yml<br><br>black:<br>  stage: &quot;📏 Lint&quot;<br>  rules:<br>    - when: always<br>  before_script:<br>    - pip install black<br>  script:<br>    - black --check --diff .</pre><pre># pytest.gitlab-ci.yml<br><br>pytest:<br>  stage: &quot;📝 Test&quot;<br>  variables:<br>    PYTHONPATH: &quot;${CI_PROJECT_DIR}&quot;<br>    TESTING: &quot;True&quot;<br>  before_script:<br>    - pip install pytest pytest-cov pytest-asyncio httpx fastapi<br>    - pip install -r requirements.txt<br>  script:<br>    - pytest tests/ </pre><p>This structure example ensures comprehensive testing and quality checks are automated within the CI/CD pipeline, enhancing code quality and security before deployment.</p><h4>Keep your pipeline fast</h4><p>One of the good practice regarding the implementation of the CI/CD is about reducing the time it takes to run. For that we can optimize it using caching strategies for dependencies and artifacts. For instance, wen can use the cache installed packages or docker layers cache, allowing following jobs to reuse them rather than reinstalling them every time.</p><pre># gitlab-ci.yml<br><br>...<br><br>cache:<br>  paths:<br>    - .docker-cache/<br><br>build_job:<br>  stage: &quot;🏗️ Build&quot;<br>  image: docker:latest<br>  services:<br>    - docker:dind<br>  variables:<br>    DOCKER_CACHE: .docker-cache<br>  script:<br>    - docker build --cache-from=your_docker_image:latest -t your_docker_image:latest .</pre><p>In this example, the cache path .docker-cache is defined to store cached Docker layers. This allows for faster subsequent builds by reusing previously built layers, thus providing faster feedback to developers.</p><h4><strong>Use environment variables for configuration</strong></h4><p>Manage environment-specific configurations (database creds, API keys …) using environment variables to avoid hardcoding sensitive information.</p><pre>push_job:<br>  stage: &quot;📦 Push&quot;<br>  image: docker:latest<br>  services:<br>    - docker:dind<br>  script:<br>    - echo &quot;$DOCKER_PASSWORD&quot; | docker login -u &quot;$DOCKER_USERNAME&quot; --password-stdin<br>    - docker push your_docker_image:latest</pre><p>The DOCKER_USERNAME and DOCKER_PASSWORD variables are used to store docker hub credentials securely. These should be set as protected environment variables in the CI/CD settings. To go further, external solutions can be use to manage secrets (Vault, AWS secret manager ..) while others tools can be used to scan the repository in search of exposed secrets (Gitleaks, TruffleHog …).</p><h4>Part 4: Ongoing Maintenance and Monitoring</h4><h4>API Versioning for evolving applications</h4><ul><li>Versioning the API application is essential for maintaining backward compatibility as the code evolves. A best practice is to include the version number in the URL path, allowing clients to specify which version they want to access. This ensure that existing clients continue to function without disruption.</li></ul><pre>from fastapi import FastAPI<br><br>app = FastAPI()<br><br>@app.get(&quot;/v1/items/&quot;)<br>async def read_items():<br>    return {&quot;version&quot;: &quot;v1&quot;, &quot;items&quot;: [&quot;item1&quot;, &quot;item2&quot;]}<br><br>@app.get(&quot;/v2/items/&quot;)<br>async def read_items():<br>    return {&quot;version&quot;: &quot;v2&quot;, &quot;items&quot;: [&quot;item1&quot;, &quot;item2&quot;, &quot;item3&quot;]}</pre><h4>Best practices for effective logging &amp; Monitoring</h4><ul><li><strong>Implement explicit logging</strong></li></ul><p>Logging is a good practice to helps track requests, errors, and overall what’s happening. you can use the middleware (a function that works with every request before it is processed by any specific path operation) to log incoming requests and responses.</p><pre>import logging<br><br>logging.basicConfig(level=logging.INFO)<br><br>@app.middleware(&quot;http&quot;)<br>async def log_requests(request: Request, call_next):<br>    logging.info(f&quot;Request: {request.method} {request.url}&quot;)<br>    response = await call_next(request)<br>    logging.info(f&quot;Response status: {response.status_code}&quot;)<br><br>    return response</pre><p>Additionaly, they are logging levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) to categorize the importance of logs. This helps in filtering logs based on severity.</p><pre>logging.debug(&quot;Debugging information&quot;)<br>logging.info(&quot;Informational message&quot;)<br>logging.warning(&quot;Warning: Potential issue detected&quot;)<br>logging.error(&quot;Error occurred during processing&quot;)<br>logging.critical(&quot;Critical error: system failure&quot;)</pre><ul><li><strong>Use Asynchronous Monitoring</strong></li></ul><p>Make sure your monitoring and logging don’t block the event loop. Making logging in a separate thread can help to ensure that monitoring doesn’t impact application performance</p><pre>import asyncio<br>import logging<br>from fastapi import FastAPI, Request<br><br>logger = logging.getLogger(__name__)<br><br>async def async_log(message: str):<br>    await asyncio.sleep(0)  # Simulate async logging (ex: async db)<br>    logger.info(message)<br><br>app = FastAPI()<br><br>@app.middleware(&quot;http&quot;)<br>async def log_requests(request: Request, call_next):<br>    await async_log(f&quot;Request path: {request.url.path}&quot;)<br>    response = await call_next(request)<br>    await async_log(f&quot;Response status: {response.status_code}&quot;)<br>    return response</pre><p>In addition, make sure to track general key metric such as the response time, the error rate, the latency … but also more specific data such as those related to endpoint errors or business logic.</p><p>Finaly, when it comes to logging and monitoring, several tools are commonly used to track those metrics, here are some of them:</p><ul><li>Sentry: Primarily used for error tracking and debugging. Excellent for monitoring and alerting on application errors.</li><li>Prometheus &amp; Grafana: Open-source and widely adopted for monitoring various applications, including FastAPI. Flexible and powerful, allowing custom metrics and dashboards (also might need more setup).</li><li>OpenTelemetry: Open-source observability framework that can be used with various backends. Provides instrumentation for metrics, traces, and logs.</li><li>DataDog: A cloud-based monitoring service that provides lots of monitoring features.</li></ul><p>Sources:</p><ul><li><a href="https://fastapi.tiangolo.com/tutorial/bigger-applications/#dependencies">Bigger Applications - Multiple Files - FastAPI</a></li><li><a href="https://fastapi.tiangolo.com/tutorial/path-params/?h=documentation#create-an-enum-class">Path Parameters - FastAPI</a></li><li><a href="https://technostacks.com/blog/mastering-fastapi-a-comprehensive-guide-and-best-practices/">Mastering FastAPI: A Comprehensive Guide and Best Practices</a></li><li><a href="https://dev.to/devasservice/fastapi-best-practices-a-condensed-guide-with-examples-3pa5?context=digest">FastAPI Best Practices: A Condensed Guide with Examples</a></li><li><a href="https://github.com/zhanymkanov/fastapi-best-practices?tab=readme-ov-file#project-structure">GitHub - zhanymkanov/fastapi-best-practices: FastAPI Best Practices and Conventions we used at our startup</a></li><li><a href="https://dev.to/ken_mwaura1/getting-started-monitoring-a-fastapi-app-with-grafana-and-prometheus-a-step-by-step-guide-3fbn">Getting Started: Monitoring a FastAPI App with Grafana and Prometheus - A Step-by-Step Guide</a></li></ul><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://differ.blog/"><strong>Create a free AI-powered blog on Differ.</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84522745a9f7" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/best-practices-for-building-deploying-rest-api-as-data-engineer-concrete-example-with-fastapi-84522745a9f7">Best practices for building &amp; deploying REST API as data engineer (concrete example with FastAPI)</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anatomy of a Data Platform — How to choose your data architecture]]></title>
            <link>https://medium.com/@lou_adam/anatomy-of-a-data-platform-how-to-choose-your-data-architecture-bc36472e7783?source=rss-64b79bb4cc53------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc36472e7783</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Louis ADAM]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 07:02:08 GMT</pubDate>
            <atom:updated>2024-04-04T07:02:08.165Z</atom:updated>
            <content:encoded><![CDATA[<h3>Anatomy of a Data Platform — How to choose your data architecture</h3><p>I’ve seen many data architectures during my assignments in different companies. Sometimes, these architectures are excellently customized to meet specific business requirements, while other times, they are not well-suited, resulting in costly refactoring projects and overhead in migrating existing systems to improved ones. Below is a brief guide to help you make informed decisions when implementing a data architecture.</p><ol><li><a href="#5924">Differences between data platform, data warehouse and data lake</a></li><li><a href="#aaf7">Common components of a data platform</a></li><li><a href="#6331">Understanding the different types of data platform architectures</a></li><li><a href="#2b08">How to choose and build the right architecture and component</a>s</li></ol><h4>1. Differences between data platform, data warehouse and data lake</h4><p>Before diving deeper in the data platform world, let’s differentiate between three often confused concepts: data platform, data warehouse, and data lake.</p><h4><strong>Data Lake</strong></h4><p>A data lake is a storage repository that holds vast amounts of raw, unstructured, semi-structured, and structured data <strong>in its native format</strong>. It is designed to store data at scale from diverse sources like IoT devices, socials, web logs, sensors, and more.</p><p>The simplest analogy to understand datalake is to compare is to your windows folder: you can store and organize in a flexible ways your different files regardless of their format. The data lake support a wide range of data types and formats, including text, images, videos etc. (JSON, XML, CSV, Parquet, Avro, …) while providing a foundation for usage such as advanced analytics, machine learning or more widely data science project. For instance, AWS S3 or Google cloud storage (GCS) are services that you can use for the creation of data lake.</p><h4><strong>Data Warehouses</strong></h4><p>A data warehouse is a centralized repository that stores structured (or semi-structured), cleaned, and processed data from various sources. Just as a library systematically stores and organizes books for easy access and retrieval, a data warehouse organizes structured data from various sources in a centralized repository.</p><p>The data is cleansed, transformed, and structured into predefined formats, making it optimized for analytical queries and reporting while providing high performance for business intelligence (BI) and decision-making processes. Snowflake, Bigquery (GCP) or Redshift (AWS) are services you could use to build your datawarehouse.</p><h4>Data Platform</h4><p>A data platform serves as a unified system for efficiently managing and analyzing large datasets. It integrates components like databases, data lakes, and data warehouses to handle structured and / or unstructured data depending on the use cases.</p><p>This infrastructure offers tools and services for various data-related tasks such as ingestion, integration, transformation, storage, processing, analysis, and visualization. In addition, by leveraging cloud-based technologies, data platforms ensure scalability, flexibility, and cost-effectiveness (if well defined) across the company data operations.</p><h4>2. Common components of a data platform</h4><p>While the architectural specifics of data platforms may vary, they typically share fundamental layers. Though these layers can be organized and nested differently to address specific use cases (different data flows), they generally have the same foundational components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPoDwcGTON_amJioiM6LSg.png" /></figure><ul><li><strong>Data sources:</strong> In that layer, you will find all of you data sources. They can be: structured such as data coming from your ERP, CRM, Customer database etc, semi structured for instance NoSQL, Json, XML etc or unstructured such as PDF, images, videos, social ..</li><li><strong>Ingestion</strong>: Data platforms need mechanisms for ingesting data from the various data sources into the storage layer. This can involve batch ingestion methods, streaming ingestion (real time), or a combination of both. Common tools and technologies used for data ingestion include Apache Kafka, AWS Kinesis or custom ETL processes.</li><li><strong>Storage</strong>: The goal of the storage layer, as it’s name suggest is to store the data, while handling the various type that arrived from the different sources. Storage can include traditional relational databases, distributed file systems, object storage or specialized databases like time series databases.</li><li><strong>Processing</strong>: Once data is stored (or interlocked with ingestion), it often needs to be processed and transformed before it can be analyzed or consumed by applications. The data processing can be done as batches or in real time, often through the creation of an ELT or ETL.</li><li><strong>Consumption</strong>: In this layer, mechanisms for querying, analyzing, and visualizing data to derive insights or to make data-driven decisions are provided. Technologies such as SQL engines, data visualization tools like Tableau or Power BI, and machine learning frameworks like TensorFlow or PyTorch are commonly used for analytics and BI.</li></ul><h4>3. Understanding the different types of data platform architectures</h4><h4><strong>The Lambda Architecture</strong></h4><p>This architecture is one of the most versatile architecture as it allow to perform both real-time and batch processing. Thus allowing the generation of diverse outputs, answering a broad spectrum of analytical requirements. It is typically structured into four layers: the ingestion, the batch, the speed and the serving layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sMDPXEkNFh6naKRaoRaBsA.png" /></figure><ul><li><strong>The ingestion layer</strong>: at its core, Lambda architecture operates on a streaming paradigm. Incoming data streams are initially captured and stored within a robust messaging engine such as Apache Kafka.</li><li><strong>The straming layer</strong> allow(near) real-time analytics by processing data as it enters the messaging engine. It takes charge of processing the the data into an analyzable format and ensures recent data is promptly available for querying or for time-sensitive analytical needs.</li><li><strong>The batch layer</strong> seamlessly accommodates historical analytics by batch processing stored streaming data from the messaging engine at a later stage.</li><li><strong>The serving layer</strong> acts as the interface for end consumers. It provides efficient access to processed data originating from both real-time streams and historical batch processing.</li></ul><h4>The Kappa Architecture</h4><p>The Kappa architecture is a data processing architecture that was introduced as an alternative to the Lambda architecture. It was designed to combine the real-time and batch layers streamlined approach to real-time data processing, making it well-suited for applications that require low-latency processing and simplified data management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aa-chWEQGjDbMA0ymSIPcg.png" /></figure><p>Data is ingested into the system once and then processed continuously in real-time without the need for storing intermediate results (immutable data streams). This approach eliminates the complexities associated with managing multiple storage layers and ensures consistency and simplicity in the data processing.</p><h4>The Data Mesh Architecture</h4><p>The Data Mesh architecture is a relatively new approach to building data platforms. It aims to address the challenges of scalability, flexibility, and data ownership in large organizations.</p><p>Instead of relying on a centralized data platform managed by a single team, the Data Mesh approach involves creating a decentralizing data infrastructure. In other words, each data domain is responsible for managing its own data product including data lifecycle, ingestion, storage, processing, and serving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqfVMO-q5qP4-Yzn_PT1Dw.png" /></figure><p>Rather than having a unified layer across the architecture, each of the layers (ingestion, processing, storing &amp; serving) are integrated within domain-driven data products, enabling the flexibility to build tailor-made solutions for business. For more details about the structure of the data product, you can read this article.</p><h4>4. How to choose the right architecture and components for your use case</h4><p>Many key questions need to be answered in order to know which approach would be the best for the business needs:</p><p><strong>What are the specific goals and objectives of the data strategy?</strong></p><p>By determining how the data platform will contribute to achieving these goals (and hopefully provide added value), you will be able to determine which architecture components are needed. For instance, goals such as enhancing product or service quality, expanding market reach and penetration, or driving innovation would necessitate different approaches, as the constraints for flexibility or processing timeline might differ.</p><p><strong>How important is flexibility and scalability for the data platform?</strong></p><p>Flexibility and scalability are central considerations for any data platform as it will influence its ability to adapt to evolving business needs and handle growing data volumes. This question will influence many aspect of the architecture and implementation choices such as:</p><ul><li>The data platform architecture. While the data platform architectures discussed earlier excel in scalability, Lambda and Kappa architectures may not always meet flexibility requirements due to challenges in reorganizing or migrating data within their structures. On the other hand, the data mesh paradigm offers flexibility by granting domains the autonomy to adapt their data pipelines according to their evolving business needs.</li><li>The choice between on-premise and cloud-based solutions: cloud-based platforms offer great scalability, allowing businesses to easily adjust resources based on demand, while on-premise solutions may provide more control but can be limited by hardware constraints.</li><li>Additionally, the decision between serverless and provisioned architectures. Serverless architectures abstract infrastructure management, allowing resources to scale up or down automatically based on demand (nice for unpredictable workloads or applications with varying usage patterns). In contrast, provisioned architectures require manual allocation of resources, which may lead to scalability challenges during low demand or peak loads periods. Therefore, organizations seeking flexibility and cost-efficiency may favor serverless architectures, while those with predictable workloads or specific performance requirements may opt for provisioned architectures to maintain more control over resource allocation and performance optimization.</li><li>The choice of database type is also important to consider, for instance the decision between NoSQL and SQL databases can significantly impact scalability and flexibility. NoSQL databases are great for their horizontal scalability and versatility in managing unstructured data. In contrast, SQL databases excel in structured data querying and maintaining transactional integrity. Therefore, selecting the appropriate database type is essential, as it directly influences the platform’s ability to ensure scalability and flexibility based on organizational needs.</li></ul><p><strong>What are the real-time processing and analytics needs?</strong></p><p>In certain scenarios, such as when implementing dynamic pricing or fraud detection systems, the necessity for real-time processing becomes a crucial requirement that dictates the architecture of your data platform. For instance, adjusting product prices based on various factors like demand and competitor pricing requires instantaneous decision-making to maximize revenue and competitiveness. In these situations, employing real-time data processing is an obvious choice, making the Kappa architecture particularly suitable.</p><p>However, it’s important to note that while the Kappa architecture excels in real-time processing and analytics, it might not be the optimal solution for tasks involving historical data analysis, such as long-term trend analysis or batch processing of historical datasets. In such cases, the lambda architecture could be a better option, as its batch layer is well-equipped to handle these specific needs.</p><p><strong>What are the resource constraints and budget considerations for building and maintaining the data platform?</strong></p><p>When building and sustaining a data platform, it’s crucial to carefully consider resource constraints and budgetary factors. This involves evaluating available resources, including budgetary allocations, the number of experts available, the recrutment cost if needed, and the capabilities of existing infrastructure.</p><p>In addition, to make informed decisions, it’s essential to assess the cost-effectiveness of various architectural options. This assessment should prioritize investments based on the anticipated return on investment (ROI) and how well they align with strategic objectives.</p><p>Finally, the choice between proprietary and open-source technologies plays a significant role in determining flexibility and scalability. Factors such as customization options, risks of vendor lock-in, and community support for scalability solutions should be weighed carefully. Emphasizing flexibility and scalability ensures that the data platform remains adaptable to evolving requirements and can efficiently accommodate future growth, regardless of the specific technologies selected.</p><h4>Sources and further reading</h4><p><a href="https://www.acceldata.io/article/what-is-a-data-platform-architecture">https://www.acceldata.io/article/what-is-a-data-platform-architecture</a></p><p><a href="https://www.youtube.com/watch?v=rvqCqK2Lpjg">https://www.youtube.com/watch?v=rvqCqK2Lpjg</a></p><p><a href="https://www.scitepress.org/papers/2018/68693/68693.pdf">https://www.scitepress.org/papers/2018/68693/68693.pdf</a></p><p><a href="https://medium.com/@mariusz_kujawski/from-database-to-ai-the-evolution-of-data-platforms-59f487e235df">https://medium.com/@mariusz_kujawski/from-database-to-ai-the-evolution-of-data-platforms-59f487e235df</a></p><p><a href="https://www.linkedin.com/pulse/data-platform-architectures-design-patterns-comparative-tfwoc/">https://www.linkedin.com/pulse/data-platform-architectures-design-patterns-comparative-tfwoc/</a></p><p><a href="https://www.splunk.com/en_us/blog/learn/data-platform.html">https://www.splunk.com/en_us/blog/learn/data-platform.html</a></p><p><a href="https://medium.com/towards-data-science/building-an-end-to-end-open-source-modern-data-platform-c906be2f31bd">https://medium.com/towards-data-science/building-an-end-to-end-open-source-modern-data-platform-c906be2f31bd</a></p><p><a href="https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/how-to-build-a-data-architecture-to-drive-innovation-today-and-tomorrow">https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/how-to-build-a-data-architecture-to-drive-innovation-today-and-tomorrow</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc36472e7783" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips and tricks to control and reduce your Biquery bill]]></title>
            <link>https://medium.com/@lou_adam/tips-and-tricks-to-control-and-reduce-your-biquery-bill-8806af6999c9?source=rss-64b79bb4cc53------2</link>
            <guid isPermaLink="false">https://medium.com/p/8806af6999c9</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Louis ADAM]]></dc:creator>
            <pubDate>Fri, 08 Mar 2024 18:37:12 GMT</pubDate>
            <atom:updated>2024-03-08T18:37:46.737Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*1goq4b1arW8otnO1wgxvJw.png" /></figure><h3>7 Tips and tricks to control and reduce your Biquery bill</h3><p>Ever had that heart-stopping moment when you check your BigQuery bill and it’s through the roof? I’ve seen it happen too many times, innocent queries turning into budget nightmares, or no controls leading to sky-high bills.</p><p>Here are my 7 tips from the lessons learned which hopefully will let you sleep well at night.</p><p><strong>Data organization and optimization<br></strong>1. <a href="#3336">Partition and Clustering Tables</a><br>2. <a href="#eeb3">Optimize your queries and good practices</a><br>3. <a href="#c420">Use cached results</a><br>4. <a href="#9285">Storage API to retreive data</a><br>5. <a href="#52f7">Data lifecycle management</a></p><p><strong>Cost management &amp; control<br></strong>6. <a href="#4d29">Cost controls</a><br>7. <a href="#fcd1">Choose the right pricing model</a></p><h3>Data Organization and Optimization</h3><h4>1. Partition and Clustering Tables</h4><p><strong>Partition</strong>: When using partitions, data are physically stored separately on the underlying storage. As each partition is stored as a separate unit, you will be able to query only the data associated with the specific partition you’re interested in, rather than scanning the entire table.</p><p>In addition you can use the partition_expiration_days option to automatically remove data older than a certain threshold and thus reducing the storage costs and the amount of scanned data over time.</p><p>The require_partition_filteroption is also relevant to reduce scanning costs as it forces users to query with filtering (WHERE clause) on the partition key preventing them from scanning the entire table.</p><pre>-- Google example<br>CREATE TABLE<br>  mydataset.newtable (transaction_id INT64)<br>PARTITION BY<br>  _PARTITIONDATE<br>  OPTIONS (<br>    partition_expiration_days = 3,<br>    require_partition_filter = TRUE);</pre><p><strong>Clustering</strong>: To help reduce query costs, clustered tables discard data so that it is not processed by queries. BigQuery sorts data in a clustered table based on the values of the clustering columns, then organizes it into blocks (related data are physically organised together).</p><p>When you run a query against a clustered table and it includes a filter on clustered columns, BigQuery uses the filter expression and block metadata to restrict the number of blocks scanned by the query. Thus allowing to scan only relevant blocks.</p><p><strong>Combining partitioning &amp; clustering to optimize query performance even further</strong></p><p>You see it coming, leveraging clusters and partitions in BigQuery combines the advantage of both worlds: faster query execution and enhanced performance thus reduced costs.</p><p>Let’s try it on a simple example:</p><pre>-- Create the table WITHOUT partition &amp; clustering<br>CREATE TABLE my_project.my_dataset.my_table (<br>    order_date DATE,<br>    country STRING,<br>    status STRING<br>);<br><br>-- Populate the table with sample data<br>INSERT INTO my_project.my_dataset.my_table(order_date, country, status)<br>VALUES<br>    (&#39;2024-01-01&#39;, &#39;USA&#39;, &#39;Shipped&#39;),<br>    (&#39;2024-01-02&#39;, &#39;USA&#39;, &#39;Processing&#39;),<br>    ...<br><br>SELECT country, status <br>FROM my_project.my_dataset.my_table<br>WHERE order_date = &#39;2024-01-04&#39;;<br><br>-----------------------<br>Duration: 0 sec<br>Bytes processed: 392 B<br>Bytes billed: 10 MB (minimum billing increment)<br>-----------------------</pre><p>In that first case, since there is no partitioning or clustering, the entire table had to be scanned.</p><pre>-- Create the table WITH partition &amp; clustering<br>CREATE TABLE my_project.my_dataset.my_partitioned_clustered_table (<br>    order_date DATE,<br>    country STRING,<br>    status STRING<br>)<br>PARTITION BY order_date<br>CLUSTER BY country;<br><br>-- Populate the table with sample data<br>INSERT INTO my_project.my_dataset.my_partitioned_clustered_table(order_date, country, status)<br>VALUES<br>    (&#39;2024-01-01&#39;, &#39;USA&#39;, &#39;Shipped&#39;),<br>    (&#39;2024-01-02&#39;, &#39;USA&#39;, &#39;Processing&#39;),<br>    ...<br><br>SELECT country, status <br>FROM my_project.my_dataset.my_partitioned_clustered_table<br>WHERE order_date = &#39;2024-01-04&#39;;<br><br>-----------------------<br>Duration: 0 sec<br>Bytes processed: 98 B<br>Bytes billed: 10 MB (minimum billing increment)<br>-----------------------</pre><p>In that second case, less data was processed thanks to partitioning which allows BigQuery to skip scanning unnecessary partitions based on the query filter.</p><p><em>Note: In general a </em><em>WHERE clause does not reduce the amount of data scanned unless the table is partitioned and/or clustered.</em></p><h4>2. Optimize your queries and good practices</h4><ul><li><strong>Select only necessary columns</strong>: Optimize query performance by selecting only the columns needed for your analysis, or use EXCEPT to exclude unnecessary columns and reduce query overhead.</li></ul><pre>SELECT * <br>FROM `bigquery-public-data.baseball.games_wide`<br><br>-----------------------<br>Bytes billed: 1.76 GB<br>-----------------------</pre><pre>SELECT gameId, startTime, duration<br>FROM `bigquery-public-data.baseball.games_wide`<br><br>-----------------------<br>Bytes billed: 38 MB<br>-----------------------</pre><ul><li><strong>Leverage schema preview: </strong>Rather than relying on <em>LIMIT</em>, which is doesn’t reduce cost (but might reduce duration), utilize schema preview, a free feature, to understand the structure of your data efficiently.</li></ul><pre>SELECT * <br>FROM `bigquery-public-data.baseball.games_wide`<br><br>-----------------------<br>Bytes billed: 1.76 GB<br>-----------------------</pre><pre>SELECT * <br>FROM `bigquery-public-data.baseball.games_wide`<br>LIMIT 100<br><br>-----------------------<br>Bytes billed: 1.76 GB<br>-----------------------</pre><ul><li><strong>Use window functions: </strong>Enhance query efficiency and readability by utilizing window functions instead of self-joins, especially in scenarios where you need to compute aggregated values over partitions of your data</li></ul><pre>-- Self-join query on data subset<br>SELECT<br>    gameId,<br>    homeTeamName,<br>    awayTeamName,<br>    (<br>      SELECT <br>        SUM(attendance) <br>      FROM data_subset b2 <br>      WHERE b2.homeTeamName = b1.homeTeamName <br>      AND b2.startTime &lt;= b1.startTime<br>    ) AS total_attendance_home_team,<br> FROM data_subset b1<br><br>-----------------------<br>Duration: 29 sec<br>Bytes billed: 53 MB<br>Slot milliseconds: 91726<br>-----------------------</pre><pre>-- window function<br>SELECT<br>    gameId,<br>    homeTeamName,<br>    awayTeamName,<br>    SUM(attendance) OVER (PARTITION BY homeTeamName ORDER BY startTime) AS total_attendance_home_team,<br>FROM data_subset<br>GROUP BY gameId, homeTeamName, awayTeamName, attendance, startTime<br><br>-----------------------<br>Duration: 0 sec<br>Bytes billed: 53 MB<br>Slot milliseconds: 228<br>-----------------------</pre><ul><li><strong>Avoid Cross Joins</strong>: Steer clear of using cross joins whenever possible to prevent generating unnecessarily large result sets, which can severely impact query performance.</li><li><strong>Minimize Data Selection in CTEs</strong>: To optimize query performance, select only the necessary data within your Common Table Expressions (CTEs), avoiding the retrieval of unnecessary information that could increase query execution time.</li></ul><h4>3. Use cached results</h4><p>Consider a scenario where you have a dashboard that displays daily sales metrics for an e-commerce platform. The dashboard includes several charts showing metrics such as total revenue, number of sales, and top-selling products. Different users access the dashboard frequently throughout the day to monitor sales performance and others KPIs.</p><p>In this scenario, utilizing cached queries creates temporary tables that persist for up to 24 hours. When users access the dashboard, these cached results are utilized, incurring no additional costs!</p><p>However, there are limitations to using the cache, for instance when using nondeterministic functions such as those involving dynamic time or date values like CURRENT_TIMESTAMP() or CURRENT_DATE</p><h4>4. Storage API to retreive data</h4><p>This one is a bit specific to certain use cases. Let’s suppose that you have to perform some data analysis in a jupyter notebook (Python) from large dataset stored in BigQuery.</p><p>Using the Storage Read API in BigQuery can potentially lead to cost reduction compared to directly querying the data using traditional BQ queries, especially in scenarios where you only need to process a subset of the data or perform complex transformations.</p><h4>5. Data lifecycle management</h4><p>First let’s note that after 90 consecutive days without editing a table the associated storage price automatically drop by 50%. The even better news here is that if you have a partitioned table, each partition’s eligibility for cost reduction is assessed independently. Meaning you can work with the most recent data while the oldest one benefits from storage reduction cost for instance.</p><p>As presented previously, the partition_expiration_days option can be a good tool to ensure lifecycle of partitions within your table. By setting an appropriate expiration period, you can automatically manage the retention of historical / deprecated data.</p><p>Last point concerns the long term storage cost of your data. In some case, it can be interesting to export data for archiving in GCS buckets. You will not be charged for the export (or load) operation, only for the bucket storage which has different options based on the frequency that you need to access the stored data (coldline, nearline etc.).</p><h3>Cost management &amp; control</h3><h4>6. Cost controls</h4><p><strong>Create a budget and alerts: </strong>Budgets and alerts can help reduce your costs by providing visibility into spending patterns, enabling proactive monitoring, and helping to identify and mitigate cost overruns. Here’s how they work:</p><ul><li><strong>Setting Budgets</strong>: You can set specific spending limits or budgets for your GCP projects or resources. By defining budgets based on your expected usage and costs, you establish a financial threshold that helps you stay within your allocated budget. In addition, you’ll have access to monitoring to identify ongoing and forecasted costs allowing you to anticipate that you do not exceed what you had planned.</li><li><strong>Alerting</strong>: Budgets can be configured to trigger alerts when spending approaches or exceeds the specified thresholds. For instance, you can setup email alert notifications that notify the billing admin, the project owner, users etc. if the actual or forecasted spend exceed a percent of your budget.</li><li>To put in place those tools, just go Budgets &amp; alerts page, specify the scope (time range, projects, services), your budget amount and the actions associated to the usage of that budget:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*325sk6nDfMFnHqVuWNywNg.png" /></figure><p><strong>Set custom quotas: </strong>Setting custom quotas is, from my point of view, one of the most important topic while configuring bigquery for your organization. They allow you to limit the amount of query data processed by day. They can be setup at project level (for all users in that project) or at user level.</p><p>Using this feature is quite simple, go to Quotas &amp; System Limitspage, select the type of quota you want to setup, for instance Query usage per day per userand click on edit quotas, you will then be able to update the value as you need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*4fjh6Su-rwfgaaHlS1uk7g.png" /></figure><p><em>Note: the reset of the quotas is done at midnight pacific time</em></p><h4><strong>7. Choose the right pricing model</strong></h4><p>Another attention point need to be raised regarding the pricing models that you are using as selecting the right pricing model that best fits your usage pattern might save you a lot of money.</p><p>Here you’ll have to distinguish two bigquery component that generate costs: the compute (cost associated to processed queries) and the storage (cost associated to data storage). Within each of those components there are different pricing options which depends on your situation and your use cases.</p><p><strong>Compute</strong>: for the compute, Bigquery has two main query pricing models</p><ul><li><strong>On-demand compute pricing</strong>: you are charged for the amount of bytes processed by a query, the first terabyte if free and if the query fails you are not charged. This model is great if you are not sure of how many queries or how many bytes you will process. Thus, it’s a good option for experimentation, buidling small analytics warehouse, proof of concept etc.</li><li><strong>Capacity compute pricing: </strong>is a flat rate option that allow you to pay a fixed amount each month / year (or even as you go) in exchange of processing capacity called slots (regardless of bytes processed by the queries). This pricing model is divided in three edition: standard, entreprise and entreprise plus which offer different benefits depending on your business needs. It might be interesting to choose this option if you have some visibility on future usage and can commit to 1 or 3 years in order to have access to preferential prices.</li></ul><p><strong>Storage</strong>: for the storage there are two options which should not lead to a dilemma as the transition from one to the other is automatic</p><ul><li><strong>Active storage</strong>: any tables or partition modified in the last 90 days</li><li><strong>Long-term storage: </strong>any tables or partition not updated in the last 90 days. In that case the storage price drops by 50%.</li></ul><p>Below a little sumup of all the pricing models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmjcO0qfl1e7KT7y5lpZsA.png" /></figure><h4>As a conclusion</h4><p>These are the BigQuery fundamental lessons learned to steer clear of those uncomfortable moments where you’re left explaining why the entire analytics budget was obliterated in just one day. May they guide you smoothly along your analytics journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8806af6999c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A starting point for data as product creation]]></title>
            <link>https://medium.com/@lou_adam/a-starting-point-for-data-as-product-creation-ec23892c8dee?source=rss-64b79bb4cc53------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec23892c8dee</guid>
            <category><![CDATA[data-product-owner]]></category>
            <category><![CDATA[data-product-platform]]></category>
            <category><![CDATA[data-product-creation]]></category>
            <dc:creator><![CDATA[Louis ADAM]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 15:57:12 GMT</pubDate>
            <atom:updated>2024-02-05T15:57:12.850Z</atom:updated>
            <content:encoded><![CDATA[<p>Data as a product refers to data that has been processed and packaged in a way that it can be directly used as a valuable asset. This involves transforming raw data into a refined format that provides actionable insights or information to its users or customers.</p><p>This article offers one perspective among many, aiming to shed light on structuring data products for effective implementation.</p><h4>I. Data product Infrastucture</h4><p>One of the comon infrastucture for data product is based on three layers in order to ingest, process and make the data available to consumer</p><figure><img alt="Schema of Data as product infrastructure" src="https://cdn-images-1.medium.com/max/913/1*9MqSWwsbILVcuyaOOI4Knw.png" /></figure><p>The three main layers of a data as product typically include:</p><p><strong>The Ingestion Layer:</strong></p><ul><li><strong>Purpose:</strong> This layer is responsible for collecting, importing, and storing raw data from various sources.</li><li><strong>Components:</strong> It includes data storage system such as a data warehouse or data lake (S3 or GCS bucket, bigquery …).</li></ul><p><strong>The Processing Layer:</strong></p><ul><li><strong>Purpose:</strong> This layer focuses on cleaning, transforming, and processing the raw data. It involves applying algorithms and computations either to get insights from the data or to have data ready for use (for instance, it could include the cleansing of customer data like email, phone …).</li><li><strong>Components:</strong> Data processing tools, algorithms, and frameworks are used in this layer. This may include data cleaning, feature engineering, aggregations, and other data manipulation tasks.</li></ul><p><strong>The Public Layer:</strong></p><ul><li><strong>Purpose:</strong> This layer is concerned with delivering the insights and information derived from the data to end-users or downstream systems.</li><li><strong>Components:</strong> Objects in this layer can be defined in some case as data product. They are for external consumption and can be accessed by consumers (other data product, dashboard, user application …)</li></ul><p>These layers collectively form the foundation of a data product, allowing organizations to structure and standardize the process of data as product creation.</p><h4>II. the different categories of data as product</h4><p><strong>Source-aligned (or raw)</strong></p><p>Source-aligned data products encapsulate raw data with minimal transformation from its original state within operational systems. These data products are uniquely tied to a source system and consolidates all source objects into a singular dataset.</p><p>This intricate architecture not only preserves the raw essence of the data but also ensures a structured and versatile framework for downstream analytical processes, providing both integrity and accessibility.</p><figure><img alt="Schema source aligned data product" src="https://cdn-images-1.medium.com/max/1024/1*Fsmirdk8nzDHhQ8a8YaRmA.png" /></figure><p>The output of a Source-Aligned Data Product is typically shared with and consumed by one or multiple consumer-aligned data product.</p><p><strong>Consumer-aligned</strong></p><p>A Consumer-Aligned Data Product is crafted to answer the specific needs of a particular consumer or a designated group of consumers within an organization. An exemple could be a dashboard or a tailored report finely tuned to fulfill the unique requirements of a specific department.</p><p>The creation of a consumer-aligned data product might involves a series of intricate steps such as unnesting, cleansing or transforming fields . These operations collectively mold the data into a consumer-aligned format.</p><p>Unlike source-aligned data products, consumer-aligned ones do not directly source data from the original systems. Instead, they consume data from one or more data products, leading to a dedicated dataset for each of them in the Landing Layer.</p><figure><img alt="Schema consumer aligned data product" src="https://cdn-images-1.medium.com/max/1024/1*B9fL7f6xDEIa0qORb31OHA.png" /></figure><p>The records from diverse sources are then reconciled, transformed if necessary, and persistently integrated into the processing Layer. This intricate process ensures that the data is not only personalized but also adheres to the specific standards and preferences of the intended consumers.</p><p><strong>Aggregate data as product</strong></p><p>The primary goal of the aggregate data product is to provide insights at a higher level by aggregating data based on certain criteria, such as averages, totals, or other statistical measures. These products involve the processing and transformation of raw or detailed data into a more condensed and higher-level form in order to extract meaningful patterns and trends. This could typically result in the creation of high-level KPIs to follow corporate objectives.</p><figure><img alt="Schema aggregate data product" src="https://cdn-images-1.medium.com/max/1024/1*UDksjkvEGN9xRwepVUPh2Q.png" /></figure><h4>III. Sum up with a concrete example</h4><p>Let’s consider a company that has a CRM system and a customer database. The leadership team has a pressing requirement for detailed reports on sales trends spanning over time. In addition, there exists a dedicated team focused on studying the contactability of clients, aiming to enhance their targeting strategy.</p><p>To cater to the specific needs of the contactability team, a consumer-aligned data product would be created. This approach would ensure access to the latest information regarding the evolution of client contactability through customer KPI creation. It would also empowers the team by enabling them to adapt and refine targeting strategies promptly.</p><p>Furthermore, an aggregated data product would be created to get insights about turnover and sales trends by distinct customer segments. This aggregated would allow the leadership team in making informed strategic decisions based on comprehensive trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1HkUo0T8rK5UbTtjMizVCw.png" /></figure><p>This approach facilitates distinct domains of responsibility for data management in both the operational and analytical planes by providing seamless integration of operational data into analytical processes, enabling in-depth analysis and strategic decision-making.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec23892c8dee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Impacts du Covid-19 sur le marché de l’immobilier français]]></title>
            <link>https://medium.com/@lou_adam/impacts-du-covid-19-sur-le-march%C3%A9-de-limmobilier-fran%C3%A7ais-ec97985ed2a?source=rss-64b79bb4cc53------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec97985ed2a</guid>
            <category><![CDATA[immobilier]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[france]]></category>
            <dc:creator><![CDATA[Louis ADAM]]></dc:creator>
            <pubDate>Wed, 24 Feb 2021 06:03:57 GMT</pubDate>
            <atom:updated>2021-02-24T06:03:57.403Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNLdNYCymHN0D2AzQ6RFLw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@matthieuoger?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Matthieu Oger</a></figcaption></figure><p>Depuis le début de la crise Covid, les analyses se multiplient dans de nombreux domaines pour anticiper à quoi ressemblera “le monde d’après”. Le marché de l’immobilier est lui aussi affecté par la crise, et son futur apparaît aujourd’hui incertain. Sans prétendre prédire l’évolution du cours de ce marché, cet article vise à en expliquer les déterminants essentiels, et les principaux enjeux.</p><h3>I. Les spécificités du marché de l’immobilier</h3><p>Le marché de l’immobilier est régi par les lois de tout marché économique : le prix des biens immobiliers résulte d’une rencontre entre l’offre et la demande de la part des ménages. Pour essayer d’anticiper les prix futurs de ce marché, il faut par conséquent analyser les différents déterminants de ces deux facteurs de formation des prix.</p><p><strong>La demande dépend à la fois de la capacité d’achat des agents, et de leur volonté d’achat.</strong></p><p>La capacité d’achat correspond au pouvoir d’achat immobilier individuel. Ce dernier est déterminé par le revenu des agents, mais également par leurs conditions de crédit. En effet, en raison du montant élevé des biens immobiliers, un tel achat requiert dans 80 % des cas un crédit bancaire. Par conséquent, la capacité d’achat des ménages peut être modifiée tant par une politique fiscale (par exemple une hausse des impôts pour rembourser la dette « Covid ») que par une politique monétaire (une baisse des taux d’intérêt par exemple).</p><p>Il ne faut pas négliger également la volonté d’achat des agents, qui va dépendre de leurs anticipations. En effet, un achat immobilier représente pour les ménages un investissement majeur, ils vont donc adapter leur stratégie d’achat selon leur anticipation du cours du marché, c’est-à-dire de la valeur des biens immobiliers. Dès lors, si les prix stagnent ou commencent à baisser, les agents vont retarder leur achat, anticipant un prix plus intéressant dans quelques mois. Peut ainsi apparaître une « prophétie auto-réalisatrice » : parce qu’ils anticipent que les prix vont baisser suite à la crise les agents retardent leur achat, cette baisse de la demande provoquant effectivement une baisse des prix immobiliers.</p><p><strong>L’offre obéit quant à elle à certaines spécificités du marché immobilier. Les biens immobiliers sont en effet soumis à une double contrainte : les délais de construction d’une part, la régulation afférente au marché d’autre part.</strong></p><p>Les délais de construction rendent l’offre de biens immobiliers rigide à court terme, puisque même si la demande augmente subitement les producteurs ne peuvent y répondre immédiatement.</p><p>La régulation immobilière tend également à rigidifier l’offre de logements, puisque les constructions doivent respecter les règles d’urbanisme, limitant parfois l’offre de constructions nouvelles. En ce sens, la loi ELAN adoptée en 2018 vise à simplifier les normes pour permettre de construire davantage de logements et plus vite.</p><p>Enfin, les offreurs de biens immobiliers sont également influencés par leurs anticipations, puisqu’ils peuvent réduire leurs projets de construction s’ils anticipent une baisse de la demande, due par exemple à une importante baisse des revenus.</p><p><strong>Commercialisation de logements neufs en France</strong></p><figure><img alt="Evolution de la population selon la catégorie socioprofessionnelle à Paris" src="https://cdn-images-1.medium.com/max/1024/1*Sx8bn9PivxyTLzzVaiFk4w.png" /><figcaption>Source : Insee, Paris Notaires Services</figcaption></figure><p>Cette rigidité de l’offre à court terme permet de parler d’un marché dépendant de la demande à court terme. Par exemple, le modèle économétrique mobilisé par la Caisse des dépôts montre que la hausse des prix depuis 2015 s’explique largement par de meilleures capacités d’emprunt.</p><h3>II. Les enjeux liés au marché de l’immobilier</h3><p>Les enjeux du marché spécifique qu’est l’immobilier sont souvent négligés dans le débat public. Pourtant, ces derniers sont multiples.</p><p>L’évolution des prix de l’immobilier affecte directement le pouvoir d’achat des ménages, principalement par le biais des évolutions des loyers qu’ils paient. Or, ce dernier a ensuite une influence sur la mise en place des politiques économiques. Ainsi, la faible progression des coûts liés à l’immobilier en Allemagne a d’autant facilité les réformes structurelles mises en place au début des années 2000 pour gagner en compétitivité, puisque la faible progression des salaires était rendue plus acceptable par un pouvoir d’achat moins limité par les dépenses immobilières.</p><p>Le marché immobilier est également un facteur clé dans la progression des inégalités sociales, entre ceux déjà détenteurs d’un bien immobilier et ceux qui n’en possèdent pas. En effet, les personnes disposant d’un logement dans une grande ville en France ont vu la valeur de leur bien augmenter au cours des dernières décennies, rendant d’autant plus difficile l’accession à la propriété pour les jeunes ménages primo-accédants. Un rapport du Sénat pointe ainsi que sont particulièrement concernés les “jeunes ménages primo-accédants, les familles, et bien sûr les catégories populaires, celles-ci étant contraintes de s’éloigner des centres-villes pour s’installer dans les périphéries parfois éloignées”. Alors même que les inégalités sociales sont de plus en plus étudiées et débattues dans l’espace public, le facteur des coûts immobiliers reste pourtant peu discuté.</p><p>Les prix de l’immobilier jouent également un rôle dans l’attractivité des villes françaises, entraînant des évolutions sociologiques profondes dans les plus grandes villes de France. Le politologue Jérôme Fourquet montre ainsi que Paris est par exemple devenue majoritairement habitée par des ménages de catégories socio-professionnelles supérieures, en raison de l’évolution des prix immobiliers ayant entraîné un éloignement des classes populaires vers les banlieues.</p><p><strong>Évolution de la population selon la catégorie socioprofessionnelle à Paris</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FCpDBYBni4ElnoqwKynASw.png" /><figcaption>Source : Insee, Paris Notaires Services</figcaption></figure><p>La mesure des prix immobiliers est également un sujet en soi, puisque des débats existent sur la manière de prendre en compte ce marché spécifique dans les mesures macro-économiques. Par exemple, l’économiste Philippe Herlin critique la méthodologie employée par l’INSEE pour mesurer l’évolution du pouvoir d’achat des ménages français. Selon lui, le coût du logement est largement minoré par cet institut. Ainsi, pour l’INSEE le coût des logements vaut pour 6% du budget total des ménages et le coût de remboursements bancaires est considéré comme nul. Dès lors, une prise en compte différente du coût du logement entraînerait une réévaluation du pouvoir d’achat des ménages à la baisse.</p><h3>III. Les différents scénarios envisageables d’évolution du marché immobilier</h3><p>Plusieurs scénarios d’évolutions des prix immobiliers peuvent être considérés : un effondrement des prix (1), une baisse des prix (2) et une hausse des prix (3).</p><ol><li><strong>Un effondrement des prix peu probable</strong></li></ol><p>Selon les économistes de la Caisse des dépôts, la répétition du scénario de la crise de 2008, où certains marchés immobiliers européens s’étaient effondrés, est peu crédible. En effet, du côté de l’offre on n’observe pas « d’emballement de la construction nationale », au sens où il n’existerait pas de bulle immobilière en France.</p><p>Du côté de la demande, on observe une certaine résilience qui s’explique par plusieurs facteurs. Une première explication, structurelle, tient en ce que depuis la crise de 2008 l’endettement privé est davantage surveillé par les autorités de régulation. Par exemple, ce dernier figure parmi les nouveaux indicateurs macroéconomiques proposés par la Banque de France. Le scénario espagnol de 2008, ou la crise financière avait révélé l’ampleur du niveau d’endettement privé, semble donc impossible.</p><p><strong>Dette immobilière des ménages rapportée à leur revenu disponible</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S2W7acNlRFrM1Lu78P-WEQ.png" /><figcaption>Source : OCDE</figcaption></figure><p>Une seconde raison, conjoncturelle, réside dans l’intervention massive de l’État français pour soutenir l’économie. Cette intervention, notamment avec le chômage partiel, a pu largement réduire l’impact de la crise sur la demande.</p><p><strong>2. Une baisse des prix possible</strong></p><p>La baisse des prix ne pourrait s’expliquer par une offre trop abondante par rapport à la demande, puisque rien n’indique que l’offre de logement soit trop élevée en France. Il faut donc se concentrer ici sur les déterminants de la demande.</p><p>Si l’on n’observe pas d’effondrement, une baisse de la demande reste possible. Elle s’expliquerait tout d’abord par une baisse de revenus, malgré l’intervention de l’État. De plus, une moindre consommation due à un effet « Ricardo-Barro » n’est pas à exclure : face à l’endettement massif de l’État, les agents anticipent que cette dette émise se traduira dans le futur par des hausses d’impôts, donc réduisent leur consommation présente. Enfin, le rôle des anticipations du cours du marché immobilier peut jouer, comme rappelé précédemment avec une anticipation auto-réalisatrice.</p><p><strong>3. Une hausse des prix envisageable</strong></p><p>Une hausse des prix pourrait résulter de deux facteurs : une forte réduction de l’offre de logement, ou une hausse de la demande de biens immobiliers.</p><p>Une hausse des prix pourrait s’expliquer par une pénurie d’offres, résultant d’une baisse des chantiers de construction suite à la Covid. Toutefois, malgré un faible nombre de permis de construire délivrés ces derniers mois, la majorité des chantiers ont été repoussés et non annulés. Ainsi, un ralentissement durable des chantiers de construction paraît peu concevable.</p><p>Une hausse de la demande est davantage probable. Selon les économistes Patrick Artus et Olivier Pastré, l’intervention massive des banques centrales pour faire face à la crise pourrait avoir comme conséquence une hausse des prix de l’immobilier. Ces dernières ont en effet réalisé une énorme création monétaire en monétisant les dettes émises par les États : la base monétaire de l’OCDE a ainsi augmenté de 70 % en 2020. Or, depuis trente ans la corrélation « classique » entre création monétaire et inflation des biens et services a disparu, au profit d’une nouvelle corrélation entre création monétaire et hausse du prix des actifs financiers et immobiliers.</p><p><strong>Une corrélation plus faible entre monnaie et inflation, mais renforcée entre monnaie et cours de l’immobilier</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0ZfafkaHv9p9Awz-Z9ifA.png" /><figcaption>Source : Insee, Banque de France</figcaption></figure><p>Ces économistes expliquent cette corrélation par le mécanisme de rééquilibrage de portefeuille : la création monétaire implique que les épargnants détiennent initialement trop de monnaie par rapport à leur souhait, ils essaient donc de se débarrasser de cette monnaie en achetant des actifs (actions, obligations, immobilier). A l’équilibre, la structure des portefeuilles est rééquilibrée par la hausse du prix des actifs due à ces achats. Ce mécanisme a pu être observé au Japon dans les années 1980, ou dans l’OCDE dans les années 1990.</p><p>Ainsi, la politique monétaire expansionniste menée pour faire face à la crise, en augmentant de manière très importante la quantité de monnaie en circulation, entraînerait à moyen terme une hausse des prix immobiliers.</p><p>Toutefois, face aux craintes d’une possible bulle immobilière, les banques restreignent les crédits en sélectionnant de manière plus exigeante les dossiers d’achats immobiliers. Le Haut conseil de stabilité financière demande désormais aux banques de ne pas prêter au-delà du taux d’endettement de 33%, et sur des durées maximales de 25 ans. Cette mesure aura donc potentiellement deux conséquences : approfondir la segmentation du marché bancaire immobilier entre les ménages disposant de revenus élevés et les autres, et transférer le surplus de monnaie émis par des achats d’actifs non pas immobiliers mais financiers. Ainsi selon l’Autorité des marchés financiers les achats d’actions par les particuliers ont été multipliés par trois au mois de mars 2020. Cette hausse pourrait toutefois s’expliquer également par les opportunités boursières permises par la crise sanitaire.</p><p>La plus grande sélectivité dans la sélection des dossiers immobiliers pourrait donc réduire l’ampleur de la hausse des prix immobiliers attendus par Artus et Pastré. Cette sélectivité pourrait également entraîner une évolution des villes concernées par les investissements immobiliers. En effet, les petites et moyennes villes étant plus accessibles financièrement, les dossiers d’investissement dans ces villes auraient plus de chance d’être acceptés.</p><p><strong>Conclusion</strong></p><p>Le marché de l’immobilier, contrairement au lieu commun véhiculé de “prix totalement irrationnels”, obéit en réalité à une logique propre. Cette dernière permet d’essayer d’anticiper les évolutions à venir du cours de ce marché, bien que la multiplicité des variables ne rende l’exercice difficile. En particulier, la confiance des ménages est imprévisible, ce qui explique la difficile modélisation des prévisions macro-économiques immobilières.</p><p>Cette difficulté à anticiper l’évolution de ce marché ne doit pourtant pas dissuader de s’intéresser davantage à l’évolution des prix immobiliers. En effet, ils ont des conséquences multiples dans nos sociétés, souvent méconnues et pourtant essentielles.</p><p><strong>Sources</strong></p><p>Conjoncture immobilière au 1er trimestre 2020</p><p>Rapport d’information du Sénat “<em>Les communes face à l’inflation des prix de l’immobilier : quels moyens d’action pour réguler le marché ?</em>”, mai 2020</p><p>Note : “<em>Quel impact attendre de la crise actuelle sur le marché immobilier résidentiel français ?”,</em> Caisse des dépôts, 7 juillet 2020</p><p><em>Pouvoir d’achat. Le grand mensonge</em>, Philippe Herlin, 2018</p><p><em>L’économie post covid</em>, Patrick Artus et Olivier Pastré, 2020</p><p><em>L’archipel français</em>, Jérome Fourquet, 2019</p><p>Article<em> “Marché immobilier : tendance et évolution des prix de l’immobilier”</em>, sur le site Notaire.fr, octobre 2020</p><p>Article “<em>Le marché du bâtiment : bilan et perspectives en 2020</em>”, sur le site batimat.com, 6 juillet 2020</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec97985ed2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>