<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eddy-OJB on Medium]]></title>
        <description><![CDATA[Stories by Eddy-OJB on Medium]]></description>
        <link>https://medium.com/@eddyojb?source=rss-271e93aae03------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*n41ZnDE67M8MaU5x</url>
            <title>Stories by Eddy-OJB on Medium</title>
            <link>https://medium.com/@eddyojb?source=rss-271e93aae03------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:15:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eddyojb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why I created Django Flow Forge for MLOps]]></title>
            <link>https://awstip.com/why-i-created-django-flow-forge-for-mlops-bfe446abc8a0?source=rss-271e93aae03------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfe446abc8a0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eddy-OJB]]></dc:creator>
            <pubDate>Thu, 07 Mar 2024 09:09:49 GMT</pubDate>
            <atom:updated>2024-06-09T13:43:09.638Z</atom:updated>
            <content:encoded><![CDATA[<h4>Eliminate Pitfalls in your AI App: Forge Scalable and Manageable Data Processing &amp; ML Work Flows</h4><p>The repository for this module can be found <a href="https://github.com/eddyojb88/django-flow-forge/"><strong>here on Github</strong></a><strong>,</strong> with <a href="https://eddyojb88.github.io/django-flow-forge/"><strong>documentation here.</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*4hJTkz78hlJeteqbOty9IA.jpeg" /><figcaption>Data Flows, Experiment Tracking and ML Ops doesnt have to be such a challenge</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MiStLlB7QF8MVVU8ns_O4g.gif" /><figcaption>Dashboard for visualising and monitoring your Flows as well as comparing experiments for Flows that capture the outputs of your data science experiments</figcaption></figure><h3>Introduction</h3><p>In a previous article, I explored various tools in the open source MLOps ecosystem, evaluating their pros and cons. I concluded that despite their advantages, most of these tools introduce unnecessary complexity without fully addressing essential requirements. An ideal solution should offer a standardized framework for:</p><ul><li>Managing and designing data science projects that enhances team communication</li><li>Allows for flexible scalability (can be used with technologies like Celery, Kubernetes, or Apache Beam)</li><li>Avoids reliance on notebooks (which pose technical risks)</li><li>Provides easy-to-use visual monitoring of project pipelines</li><li>Facilitates decent debugging</li><li>Enables straightforward model serving with strong security</li><li>Aids in comparing data science outcomes</li></ul><h3>The Pitch</h3><p>While all the media attention is on Gen AI, the silent killer of your dreams of any AI app is the lack of a good quality data ingestion process and an effective training process of your AI. This is the realm of ML Ops and although the market is projected to be worth <a href="https://www.alliedmarketresearch.com/mlops-market-A47295#:~:text=The%20global%20mlops%20market%20was,to%20machine%20learning%20development%20processes.">$37.4 billion by 2032</a>, the landscape is a mess, fraught with complexity, vendor lock-ins and high technical risks. Django Flow Forge aims to be a beacon of pragmatism when forging together your data science work flows, drastically reducing your technical complexity in the long run and simplifying your longer term path to success.</p><h3>Designed for Data Scientists and Data Engineers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*BxHnig4mO3QVjstd.gif" /></figure><p><strong>Example problem</strong>: The use of Jupyter Notebooks, while easy to learn to use and popular among Data Scientists for collaboration, often lead to <a href="https://www.youtube.com/watch?v=7jiPeIFXb6U">poor coding practices and reliability issues</a>, costing businesses a lot of time and money in the long run. For example, a company might invest $30k into a hopeful prototype that relied on the use of Notebooks, only to find the code and data ingestion process was flawed due to incorrect cell execution order and <strong>bad code, rendering the $30k investment wasted.</strong> Yes, this was a real example!</p><p>Some vendors also offer solutions to transition Notebooks to production settings but as implied above this is a house of cards.</p><p>Instead, leveraging <strong>Django</strong>, a mature web framework, and its <strong>Flow-Forge</strong> plugin can significantly reduce technical risks by streamlining and standardizing ML Ops workflows. This approach not only enhances project flexibility as it scales but also allows for a more impactful deployment without forcing the use of web server components from the start. You can be selective about the features you use from Django but still maintain an integrated solution that contrasts the disjointed and often fragile tech stacks common in growing data science projects, such as Langchain, Apache Beam, Kedro, Metaflow etc. promoting a more efficient and robust development environment.</p><h4>But Django is a Web Framework?</h4><p>If you take a<a href="https://medium.com/@eddyojb/thoughts-you-wont-get-from-chatgpt-do-you-need-ml-ops-2c954b9d47a6"> look at the other ML Ops technologies</a>, they are all fancy web frameworks with wrap around code to support e.g. storing and visualisation of ML results, pipelines or deploying async but each with signficant pitfalls. With the way the industry is moving towards Full Stack Data Science, Django Flow Forge contrasts existing tech by offering the following key features.</p><h3>Key Features</h3><p>- Easily define your data or ML work flows (aka pipelines)</p><p>- Visualize your flows for stakeholders and colleagues, both in planning and development phases (inspired by Kedro)</p><p>- Register your flow simply and execute either through a script, website or API</p><p>- Task monitoring and visualization of outputs and failures</p><p>- Visualise pipeline process outputs and failures for each task</p><p>- Compare data science experiment results easily (inspired by Kedro)</p><p>- Serve your models through regular Django or via API by plugging in Django Ninja</p><p>- A familiar environment for Django users, minimizing the learning curve. New users can learn only what they need as they go. Further engineering can be offloaded to Django devs if necessary</p><p>- Leverage Django’s robust security and admin interface</p><p>- Leverage Django’s existing scalability features, including Django Celery and flexbility with Kubernetes engines, without the worry of vendor lock-in</p><p>- Encourages standardisation of data science work flows</p><p>- Encourages teams to move away from Notebooks, which can easily cause more issues than they solve</p><h4>Capturing Results from Data Science Experiments. And what about MLFlow?</h4><p>MLFlow helps capture and visualise generic results from experiments. You can collect your MLFlow based metrics within django-flow-forge and store to a Task output. However, this isn’t advised. MLFlow appears rich in logging all sorts of metrics that you collect from all the big libraries, such as Tensorflow, HuggingFace, Scikit-learn etc.</p><p>However, this isn’t difficult to write this out yourself. If you are generically collecting metrics then you may not be thinking about the experiment effectively and, therefore, may not really be saving the business time. For example,</p><p>mlflow.autolog() # or mlflow.sklearn.autolog()</p><p>This gives the following output:</p><pre>&lt;Run: data=&lt;RunData:<br>metrics={‘RandomForestClassifier_score_X_val’: 0.72,<br> ‘training_accuracy_score’: 0.99625,<br> ‘training_f1_score’: 0.9962547564333545,<br> ‘training_log_loss’: 0.3354604497935824,<br> ‘training_precision_score’: 0.9962921348314606,<br> ‘training_recall_score’: 0.99625,<br> ‘training_roc_auc’: 0.9998943433719795,<br> ‘training_score’: 0.99625<br> },<br> params={‘bootstrap’: ‘True’,<br> ‘ccp_alpha’: ‘0.0’,<br> ‘class_weight’: ‘None’,<br> ‘criterion’: ‘gini’,<br> ‘max_depth’: ‘10’,<br> ‘max_features’: ‘sqrt’,<br> ‘max_leaf_nodes’: ‘None’,<br> […],<br> },<br>tags={‘estimator_class’: ‘sklearn.ensemble._forest.RandomForestClassifier’,<br> ‘estimator_name’: ‘RandomForestClassifier’,<br> ‘mlflow.autologging’: ‘sklearn’,<br> […]<br>}, […]&gt;&gt;</pre><p>However in classification problems, there is almost always a class imbalance problem, so I might be more interested in assigning equal weight to the classes using the f1_macro score. I might also only be interested in the following metrics:</p><ul><li>F1 Macro</li><li>Precision</li><li>Recall</li><li>ROC AUC</li></ul><p>This isn’t a lot of code to add and cuts the bloat that MLFlow records. Cut the unnecessary technical complexity but maintain flexibility — MLFlow is utterly wasteful since you have to maintain an entirely seperate server ecosystem to achieve something very simple in Django.</p><p><strong>Keep your system design simple, flexible and scalable!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfe446abc8a0" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/why-i-created-django-flow-forge-for-mlops-bfe446abc8a0">Why I created Django Flow Forge for MLOps</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thoughts you won’t get from ChatGPT: Do you need ML Ops?]]></title>
            <link>https://medium.com/@eddyojb/thoughts-you-wont-get-from-chatgpt-do-you-need-ml-ops-2c954b9d47a6?source=rss-271e93aae03------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c954b9d47a6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Eddy-OJB]]></dc:creator>
            <pubDate>Mon, 12 Feb 2024 23:13:56 GMT</pubDate>
            <atom:updated>2024-03-03T22:33:43.577Z</atom:updated>
            <content:encoded><![CDATA[<h3>Thoughts you won’t get from ChatGPT: Do you need ML Op Tools and which ones will minimise your headache?</h3><h4>Maybe you need ML Op tools to scale but there is no silver bullet. In many cases you may not need one</h4><p>This article is aimed at data scientists and architects who require Python in their projects but might not yet understand the requirement of MLOps nor understand which tools might be appropriate options. It is an enormous and overwhelming field and not always clear that extra tools are required. Ignore the pitches by the salesmen and let me try and help you out.</p><p><strong>Edit</strong>: Since writing this article, I have written a module called <a href="https://medium.com/p/b8843cee8e24/">Django Flor Forge</a>, which solves a lot of the issues mentioned in this article.</p><h4>Contents:</h4><ul><li>Too Long Didn’t Read: Summary and favourite frameworks</li><li>When Do We Care About Tooling Up for ML Ops?</li><li>What ML Ops Tools are available?</li><li>Closing thoughts</li></ul><h3>Summary (Too Long, Didn’t Read)</h3><p>MLOps feels like one of those made up terms but it can help in some use cases. The field has tools that can easily over complicate with all of them lacking key features, ranging from ease of use to task monitoring. Many data science projects don’t need large language models and require a focus on getting quality data and processing it well, meaning that they can manage with regular DevOps and basic project management for model tracking. However, when requiring scaling with task distribution and compute clusters with monitoring (e.g. LLMs), MLOps may come in to focus.</p><h4>Tell Me Your Favourite Open Source Frameworks</h4><p><strong>Edit</strong>: Since writing this article, I have written a module called <a href="https://medium.com/p/b8843cee8e24/">Django Flor Forge</a>, which solves a lot of the issues mentioned in this article.</p><p>My top 3 that reduce my ML Ops headache:</p><ul><li>Django Flow Forge — Yes, a mature web framework with some simple data modelling and flexibility to scale in the cloud can go a long way and keep your architecture simpler, flexible and impactful</li><li>Kedro — this is a fairly decent and professional looking solution (but still lacks some key features such as model serving and simple scaling</li><li>MLFlow for scaling deep learning logging in conjunction with Tensorflow / PyTorch on Kubernetes</li></ul><p>The rest I didn’t think had enough to make it to the top of the pile.</p><p>Having tried each open source tool in this article, I have created a summary table below of features that I think are important:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuLddvbE7sBGOLaAC7dfcw.png" /></figure><h3>The Context of ML Ops</h3><h4>When Do We Care About Tooling Up for ML Ops?</h4><p>Some say that <a href="https://becominghuman.ai/no-you-dont-need-mlops-5e1ce9fdaa4b">you don’t need tools for ML Ops </a>as it can add unnecessary complexity. I don’t disagree.</p><p>However, data and model solutions are almost always complex and I have not come across a scenario that simply and satisfactorily delivers in:</p><ul><li>Handling categorical data well (and when is this ever not relevant)</li><li>Offering clearly explainable results that are also consistent in practice</li><li>Excellent performance</li></ul><p>In order to combat this, you may find yourself in the scenario where you need to:</p><ul><li>Include more data, which may require scaling horizontally and distributing the training and selection process with your models</li><li>Improve handling of categorical data when making predictions (e.g. tabular data with xGBoost)</li><li>Work in a coordinated team in order to achieve results effectively</li><li>Monitor and maintain model training and serving</li></ul><p>This is where the lure of ML Ops tools comes in, promising to help you in the following:</p><ol><li>Solo or Team Efficiency &amp; Automation</li><li>Reproducibility</li><li>Scalability, including parallelisation</li><li>Error Handling &amp; Recovery</li><li>Scheduling</li><li>Maintenance (CI/CD), Monitoring &amp; Logging</li></ol><h4><strong>When Should We Not Employ MLOp tools:</strong></h4><ul><li>Simple Proof of Concepts</li><li>The model and / or data is relatively straight forward (usually data processing is the biggest and most time consuming part)</li></ul><h4>I Won’t Cover Closed Source Vendor Tools In this Article</h4><p>There are lots of solutions by vendors, including Vertex AI by Google, AWS Sagemaker, Azure Machine Learning, Databricks — the list goes on and on.</p><p>These all look decent and offer useful looking assistive tools and accelerators but I won’t be covering these solutions. In a lot of use cases, choosing a vendor is time consuming for a business and can be problematic from a financial and legal perspective or a data handling perspective. An example of the latter is that in the finance industry, keeping data in-house, on-prem and within the same country can be a non-negotiable.</p><p>In addition, most curious data scientists and engineers want to try the tech themselves!</p><h3>What ML Ops Tools are available?</h3><h4>Kubeflow</h4><p>Kubeflow comes near the top results when searching for ML Op tools. It aims to simplify the deployment, scaling, and management of machine learning workflows on Kubernetes, covering data prep, model training, serving and monitoring on different platforms.</p><p><strong>Tell me more:</strong></p><p>🟢 Great sounding principles and architecture</p><p>🟢 Appears feature reach for managing the entire ML lifecycle, including model serving</p><p><strong>Beware:</strong></p><p>🚩 Can be a massive headache to try out. e.g. it is not easy to run locally and may require getting Docker, Kubernetes, Virtualbox and Vagrant singing together, which I spent a few evenings achieving on Ubuntu</p><p>🚩 Complicated to try out in the cloud</p><p>🚩 Complicated architecture under the hood</p><p>🚩 Code examples don’t appear <a href="https://github.com/kubeflow/pipelines/blob/master/samples/core/xgboost_training_cm/xgboost_training_cm.py">readable</a> to me, so it is difficult to recommend to others</p><h4>Metaflow</h4><p>“Metaflow makes it quick and easy to build and manage real-life ML, AI, and data science projects.”</p><p>Tell me more:</p><p>🟢 Super easy to try out via <a href="https://outerbounds.com/metaflow/">Outerbounds</a> for free or locally</p><p>🟢 Very readable code</p><p>🟢 Great flexibility</p><p>Beware:</p><p>🚩 UI has fiddly to setup locally. When scanning the online community, it appears recommended to us a 3rd party to host the Meatflow Service and UI in order to help visualize your processes. This is not good from a flexibility perspective</p><h4>Kedro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*lxuZFQ--ZdrhZCdN.png" /></figure><p>Kedro is designed to enable the construction of reproducible, maintainable, and modular data science code.</p><p><strong>Tell me more:</strong></p><p>🟢 Great visualisation UI, which is engaging and stakeholder facing. <strong>Wow!</strong></p><p>🟢 Easy to try out and runs locally very easily</p><p>🟢 Code is clean and documentation assistance is brilliant</p><p>🟢 Seems like the most professional solution of the bunch</p><p>🚩 Unless I am missing something, it isn’t great for monitoring long running tasks unless you setup a custom dashboard for tracking</p><p>🚩 No API, which means no event driven running of tasks</p><h4>Apache Airflow</h4><p>This appears to be one of the most popular frameworks. Loading it up was relatively straightforward, though not as simple as Kedro or Metaflow.</p><p>Pros:</p><p>🟢 Easy to try out and runs locally easily</p><p>🟢 Rich features for running, scheduling and monitoring</p><p>Cons:</p><p>🚩 It does a lot, which means that understanding how it works is quite a lot</p><p>🚩 Doesn’t store ML results like Kedro or MLFlow</p><p>🚩 Integrating with existing code was a bit of a headache because of how everything is imported, which is in contrast to the simpler mechanisms offered by e.g. Metaflow</p><p>🚩 Running the server and the scheduler makes me realise that this is just another webserver with some clever logging and scheduling</p><h4>Luigi with GoKart</h4><p>Luigi is an impressive package aimed at handling complex data processing in parallel. GoKart is an extension of Luigi to handle extra tasks related to machine learning, such as saving model data. I enjoyed using GoKart.</p><p>Pros:</p><p>🟢 Easy to try out and runs locally easily</p><p>🟢 Decent features for running, scheduling and monitoring</p><p>🚩 Luigi UI is a bit ugly and not stakeholder facing</p><h4>MLFlow with Deep Learning Libraries</h4><p>This tool is focussed on model logging and model serving. It doesn’t including pipelineing. I see it as an appropriate tool in conjunction with deep learning modules like Tensorflow and PyTorch. For example, lets say you want to include much more data and distribute and scale training of your neural network, you could deploy your Tensorflow code on Kubernetes cluster by splitting up the data in to batches, copying the model and combining the gradients from the model at each stage:</p><pre>def main():<br>    # Use Tensorflows mirrored distrbuted training strategy for large datasets <br>    strategy = tf.distribute.MirroredStrategy()<br>    print(&#39;Number of devices: {}&#39;.format(strategy.num_replicas_in_sync))</pre><pre>    # Automatically log metrics, parameters, and models to MLflow<br>    mlflow.tensorflow.autolog()  </pre><pre>    with strategy.scope():<br>        model = build_model()</pre><p>🟢 Easy to try out and runs locally easily</p><p>🟢 Focussed library</p><p>🟢 Good compatiblity with deep learning frameworks like Tensorflow and PyTorch</p><h4>Whats wrong with using vanilla Django with Celery to solve the issues that MLOps tools present?</h4><p>After trying all of the tools above, I realised that they<strong> all feel like fancy web frameworks with an extra sprinkling</strong> of managing parallelisation for large data sets, a UI and some connectors to cloud computing at scale.</p><p>There is no reason why you can’t do this with a web framework, some task queing and data modelling at a rapid pace.<br><strong>Edit</strong>: I have since made <a href="https://medium.com/p/b8843cee8e24">Django Flow Forge to help</a> with this.</p><p>All frameworks described above require learning, yet with Django you can do this plus much more, splitting out and scaling your solutions as you require. <strong>Has the community needlessly missed a trick here and overcomplicated the process? </strong>It seems likely to me.</p><p>Some pros of this approach:</p><p>🟢 Easy to try out and runs locally easily</p><p>🟢 With some quick ChatGPT, you can create a nice looking stakeholder facing page to summarise your data and tasks. With Django-HTMX the frontend could have a SPA feel like with React</p><p>🟢 Distributable, scalable processing locally or in the cloud with Celery, which has native Django support</p><p>🟢 ML models can be versioned with some simple data modelling logic</p><p>🟢 If necessary, ML models can be served at scale using Kubernetes, or the project could be duplicated and a Django-Ninja API (a copy of FastAPI) used to serve in a detached location</p><p>🟢 Mature and secure</p><p>🟢 Django Channels with websockets is an ideal solution for LLM chat apps</p><p>🚩 The community has shunted Django for machine learning training and serving, probably because it isn’t shiny new like FastAPI — new frameworks can be riddled with security issues though, so why take the risk if its not any faster?</p><p>🚩 Django can be slow to write data at scale (but there are workarounds)</p><h4>What about tools to convert Notebooks in to code?</h4><p>If you are doing this, then you are probably approaching any Ops incorrectly in general. Notebooks are great for quick exploration of machine learning models but they are <a href="https://www.youtube.com/watch?v=7jiPeIFXb6U">generally terrible for anything complex</a> and only <a href="https://towardsdatascience.com/5-reasons-why-jupyter-notebooks-suck-4dc201e27086">terrible for production</a> worthy code (debugging functions anyone?). This is why features such as Kubeflow Notebooks makes no sense to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*_rV0MhXF4r6OWTcebTb7Lg.png" /></figure><h3>Closing thoughts</h3><p>What is really clear is that different tools excel at different tasks. The frustrating reality at the moment is that no tool offers it all and most have some sort of headache inducing pain points about them.</p><p>However, if there was one tool that I thought could offer it all and ease the pain, it would be Kedro — all I feel it needs to add is task monitoring and extra compatibility with deep learning frameworks like how MLFlow works. What is interesting is that someone has clearly thought the same and <a href="https://github.com/Galileo-Galilei/kedro-mlflow">thought to combine the two</a>. Haven’t tried it yet but sounds promising!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c954b9d47a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django with HTMX — Economical Modern App Development At Last]]></title>
            <link>https://medium.com/@eddyojb/django-with-htmx-economical-modern-app-development-at-last-066b53825e1e?source=rss-271e93aae03------2</link>
            <guid isPermaLink="false">https://medium.com/p/066b53825e1e</guid>
            <category><![CDATA[htmx]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Eddy-OJB]]></dc:creator>
            <pubDate>Wed, 17 Jan 2024 21:19:41 GMT</pubDate>
            <atom:updated>2024-08-21T15:33:41.335Z</atom:updated>
            <content:encoded><![CDATA[<h3>Django with HTMX — Economical Modern App Development At Last</h3><h4>If commercially viable software is about developing a flexible and modern app as economical and quickly as possible, should you ditch React for your next software project?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jk9DXm1hAo7wUJDPlUd9Nw.jpeg" /><figcaption><a href="https://images.unsplash.com/photo-1461023058943-07fcbe16d735?ixlib=rb-4.0.3&amp;q=85&amp;fm=jpg&amp;crop=entropy&amp;cs=srgb&amp;dl=demi-deherrera-L-sm1B4L1Ns-unsplash.jpg">Delicious Coffee </a>— Required for Combating Javascript Fatigue.</figcaption></figure><p><strong>Audience</strong>: This blog post is aimed at web developers and technical data product developers.</p><h4>Contents:</h4><ul><li>Shaddap and Give Me the Code</li><li>Motivation</li><li>Commercial Justifications for Django with HTMX</li><li>Limitations of HTMX</li><li>Resources</li></ul><h4><strong>Shaddap and Give Me the Code</strong></h4><p>For an example Django project with HTMX, <a href="https://github.com/eddyojb88/docker-django-htmx">head over to my Github</a>, which contains examples that I have cobbled together from the community in order to quickly assess the capabilities of HTMX. It is not an optimised repo (I am a recent father).</p><p>I have a particular interest in data analytics use cases, so there are examples in this project of swapping out charts from libraries like Plotly without a backend API, which was easy. It doesn’t look super sexy on my app but if you spend a little time you can make it look super nice. I have implemented an app service at my company using Django+HTMX and am pleased with how easy this is in comparison to having a complicated backend and frontend stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdzlbhGIMbAJFzv0boiRkg.gif" /><figcaption>User clicks the button to fetch data and display. Clicking the button again replaces the graph with new data</figcaption></figure><p>There are also examples from Django-htmx and echarts in the repo. Echarts looks great and the transitions are slick but the code provided by advantch seemed overly complex. Best just to go with straight up echarts js library and chatgpt your way through it, works and looks like the best charting library to me.</p><p><strong>The rest of this article will now focus on justifications for ditching a Javascript framework frontend as well as pitfalls of HTMX which I didn’t feel were covered by others yet.</strong></p><h3><strong>Motivation</strong></h3><p><strong>Python and React</strong></p><p>React is an incredibly rich ecosystem and there are lots of Django and React <a href="https://github.com/topics/django-react">starter setups</a>, like this one <a href="https://github.com/topics/django-react">by marcelovicentegc</a>, which worked for my use case recently as it allowed me the flexibility of using Python and Django for real world problems in the backend and React for the frontend. However, the complexity quickly explodes for this setup, caused by the use of an API to link the two systems. This made me re-evaluate my tech stack.</p><p><strong>What about low-code data solutions from Azure, Tableau, Alteryx etc.</strong></p><p>There are tons of low-code solutions out there in order to help remove development for my use cases (data analytics) and general app development and I have spent weeks trying to make some work. The reality is that they never do what you want them to without you needing to go far off piste. Most of the time, my clients demands have required greater flexibility and at this point, I end up with a stack of compromises that no one wants and in my experience the app engagement typically falls off a cliff.</p><p>For me, <strong>‘low-code’ is now about working our way through a series of prompts and checking the suitability of AI generated code. </strong>This is perfect for Python developers and extremely flexible over some of the bad point and click solutions that I witnessed became prevalent at major organisations I have worked with, such as the use of combining Alteryx and Tableau. I felt like a salmon in a river when I saw this trend happen, but with the latest developments in Gen AI I have a feeling this will come full circle with product development feeling more flexible, fast moving and exciting again.</p><p><strong>What about Dash, Streamlit etc. to code Python Apps with Frontends?</strong></p><p>These app building technologies are tailored towards small analytical or data science related apps. The Having used solutions like Dash and Streamlit for over 7 years, I find them impressively easy to pick up. However, the flexibility, performance and deployment options become limited without going off piste and producing something clunky in the long run. Snowflake have also recently acquired Streamlit and, in my view, they should stick to what they are great at, which is data warehousing. I have spent years using their Python based ‘Snowpark’ and libraries, which I didn’t enjoy so I don’t have high hopes for their Streamlit integrations.</p><h3>Commercial Justifications for Django with HTMX</h3><h4><strong>Hiring Complexity when Splitting the Backend and Frontend</strong></h4><p>In order to develop modern SPA apps with separated frontends and backends, you have to have engineers familiar in the associated tech stack, which could include multiple languages like Python and Javascript. In my case, I was also using the Django and React frameworks. The more you mix tech, the smaller the pool of developers available to you and it is harder hiring for this cost effectively than one might think, so you may end up hiring a backend team and a frontend team. You could ditch the Python backend altogether but in my view this language is so great for data applications that it doesn’t make sense to me most of the time.</p><h4><strong>Staying Fast and Economical by Increasing Team Availability</strong></h4><p>If using HTMX in conjunction with a Python framework like Django, you can have just one team that can all work on the same task sets. The overall dev team could be smaller and quicker. This strategy eliminates bottlenecks that <strong>will occur </strong>if backend and UX teams are wastefully waiting for a frontend team (sometimes a solo javascript developer) to finish their app improvements. HTMX means mixing up the work for all developers, something that I personally value highly in my day to day.</p><h4><strong>Reduce the lines of code and codebase complexity</strong></h4><p>The more code you have, the harder it is maintain. When I had an app with a backend and frontend split out, the duplication was a slog when I had to:</p><ul><li>Define my data models in the backend</li><li>Perform a secondary layer of logic in order to serialize the data in a specific way and feed this through an API</li><li>Define the same set of models but this time in the frontend stack in order to handle the data being received through the API</li><li>Define form data validation logic on the frontend and backend instead of using only the prebuilt form functions in the backend</li></ul><p>There is probably more to this list but the point is that I only have to define my models once in Django, a well documented and mature framework for over 15+ years and fresh async capabilities. With HTMX, I can ditch the duplicative efforts, which should save me enough time that I can chill out and reflect on what has been built so far and what the client actually wants.</p><h4>Web Performance</h4><p>Thank you <a href="https://www.youtube.com/watch?v=3GObi93tjZI&amp;t=1063s">David Guillot for this table</a> from a real SAAS conversion of a React app to Django with HTMX:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*ApaZqh5AtVRy3hDNpJ3o5A.png" /></figure><h4>But what about selling our data through the API at a future date?</h4><p>I would worry about it when the time comes, which I doubt it will. I say: if it can be ditched, ditch it.</p><h4>Maturity of Django vs Frontend Technologies</h4><p>Django is probably older than you or I have been coding and it is super mature but great.</p><p>I have also been using Django for 8 years now and although it isn’t as simple as FastAPI or Flask at first, these microframeworks tend to inadvertantly stay hands off with security, whereas Django’s mature and opinonated approach helps novices steer past them.</p><h3><strong>Limitations of HTMX</strong></h3><p>From my experience of using HTMX with Django, there are defeinitely draw backs but this may be owing to the age of HTMX and a growing user base.</p><h4>Replacing only what you need</h4><p>An example of an annoying feature is that when I went to display a graph and change the data dynamically via a selection dropdown, I had to replace the graph entirely instead of only updating the data only. <a href="https://htmx.org/examples/animations/">HTMX comes with transition styling</a> that brushes over the aesthetics of this problem but the transitions appear to be relevant for newer browsers only. Perhaps I did something incorrect. If someone is using an old browser anyway, do they really care about this? Probably not. Therefore, this is a minor problem in my view, which bugs me as a styling imperfection but given the growth trajectory of HTMX I am sure that a better solution will pop up soon.</p><h4>Statefulness</h4><p>I read on Reddit a comment by a developer stating that ‘yeah but you should stick to React for highly stateful apps’. This is fair enough if you are chasing a perfect app. HTMX won’t allow you to hyper optimise everything about your app, so I think its best suited for simpler apps. That said, I created a fairly complex and good looking data entry, processing and visualisation app at my company.</p><p>So was this comment about sticking to React wishful thinking from a sunk cost fallacy? State is handled via the backend with HTMX and lets face it, that is the easiest place to manage it, which is probably fine for 95% of use cases in my view. You can create multi chained events with HTMX so the only use cases that I imagine it causes an issue for is where your app needs to be an aesthetically perfect app. If you are making a perfect app then I would argue that you are doing software development wrong and in a few months the time wasted on perfecting an app won’t matter because your clients demands have probably moved on, especially if you are in a startup or a highly agile team.</p><p><strong>Learning Curve and Going Off-Piste</strong></p><p>Sometimes I found myself thinking that HTMX was too good to be true as I’d find some loading or data issue, but then I’d find that either my code was bad or I was using HTMX incorrectly. My point is that there is still a learning curve and you go slightly off piste of the standard Django way but its a hell of a lot easier than learning an entire frontend tech stack.</p><p><strong>Debugging</strong></p><p>Although HTMX provides debugging helpers, I didn’t feel like they really tied in with a Django backend. Perhaps I wasn’t using it properly but the experience wasn’t as smooth as, for example, Plotlys Dash.</p><h3><strong>Resources</strong></h3><p>I heavily relied on the great work of others who I owe credit to:</p><ul><li><a href="https://www.youtube.com/watch?v=3GObi93tjZI&amp;t=1063s">DjangoCon — Real SAAS product coverted from React to Django HTMX</a></li><li><a href="https://www.youtube.com/watch?v=Ula0c_rZ6gk&amp;list=PL-2EBeDYMIbRByZ8GXhcnQSuv2dog4JxY">BugBytes —</a> this course on HTMX is handy</li><li><a href="https://www.youtube.com/watch?v=RzpgzbKVKqA">Plotly charts in Python with HTMX</a></li><li><a href="https://htmx.org/examples">Official HTMX docs</a> incl. animation tactics</li><li>U<a href="https://www.advantch.com/blog/how-to-add-charts-to-your-python-application/?utm_campaign=blog">sing echarts via Advantech</a></li><li><a href="https://medium.com/u/e9b739a82cb7">Duy Tr</a> for an excellent tutorial on <a href="https://medium.com/@duytran2310/single-page-crud-with-htmx-and-django-part-2-d6b577aee33f">Data Tables with HTMX</a></li><li>Advantch for another excellent tutorial on Large <a href="https://www.advantch.com/blog/django-htmx-server-side-datatables/">Data Tables with HTMX</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=066b53825e1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gen AI in Software Development]]></title>
            <link>https://medium.com/@eddyojb/gen-ai-in-software-development-3196f74fb52d?source=rss-271e93aae03------2</link>
            <guid isPermaLink="false">https://medium.com/p/3196f74fb52d</guid>
            <dc:creator><![CDATA[Eddy-OJB]]></dc:creator>
            <pubDate>Mon, 15 Jan 2024 22:43:08 GMT</pubDate>
            <atom:updated>2025-07-28T10:12:29.788Z</atom:updated>
            <content:encoded><![CDATA[<h4>Beyond the hype: Considerations that you won’t get from ChatGPT</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*ha97_qTqiBA4ZPK8JhaMCw.png" /><figcaption>This is becoming a more frequent problem. Surely this is something we should be more aware of when building new products?</figcaption></figure><p>*I first wrote this article in 2023 and in 2025 it hasn’t aged.</p><p>This article started out originally as some simple notes on page, intended to organise lessons learned after I had developed a few apps using Gen AI as my coding assistant since 2021, which was when we had GPT2.<br>However, I felt that my notes also provide <strong>practical considerations to those unsure about the valuable use cases and pitfalls of Gen AI, such as Copilot, in order to code scalable software beyond the hype. </strong>I felt that this research was unavailable when asking ChatGPT,<strong> </strong>or it was buried too deeply within a mountain of mis-comprehension about AI when searching via Google.</p><p>This article assumes that you are interested in helping your business <strong>code</strong> <strong>apps efficiently with Gen AI as an assistive tool.</strong></p><h3>Too Long, Didn’t Read</h3><ul><li><strong>Yes, Gen AI is brilliant in software development but it’s all about use case. If you are in the same shoes as me, you may be reliant on Gen AI day to day but how much code is being generated that you don’t understand? What is the damage if the work is wrong? If the work is wrong, what is your insurance policy?</strong></li><li>Be cautious of claims about Gen AI (or any other AI) that sounds too good to be true, such as ‘<a href="https://sifted.eu/articles/diffblue-ai-code"><em>we will use AI to automatically code an entire financial data app with no human in the loop’</em></a><em>. </em>Even if this could be done, it could cause major issues down the line in my view. Luckily, the public are well aware of the shoddy workmanship at Fujitsu that lead to the <a href="https://www.bbc.co.uk/news/business-66843548">Post Office Horizon software scandal</a>, which should serve as a reality check for when software development goes unchecked</li><li><strong>All AI solutions have no concept of the risk of time wasted</strong>, meaning that it can never be better than a junior developer. Yes, it can quickly cobble together lots of untested research but it cannot be credible beyond that. However, a junior developer can test the credibility of others code and tweak with originality. This means it is extremely important to understand the importance of being correct in your use case. In my view, this places a higher emphasis on mid level to senior developers in the landscape of today</li><li><strong>Gen AI is like a junior developer (but less brilliant in the long term)</strong><br>Using Gen AI to help code is brilliant and like having a junior developer perform a lot of research and options quickly. When factoring in the errors made by Gen AI engines, my view is that the efficiencies are maybe in the region of 10% to 30% for general coding but ≤ 10% for coding data products. I am bias as I am from the data world but errors can be more fatal in developing data products, which are <strong>almost always complex and come with non-trivial issues.</strong> This makes predictive AI products like <a href="https://www.pecan.ai/">Pecan.ai</a> highly questionable. This is in contrast to business processes, such as creating user interfaces for apps, where the error can be visible in the app and easily fixed.</li></ul><h3>Considerations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0y6LJxmr9MjQb-DB35N2Aw.jpeg" /><figcaption>Still true 4 years on and will be for a lot longer.</figcaption></figure><h4>What are Appropriate Use Cases for Gen AI in Coding Software and Data Apps</h4><p>Every developer is coding with Gen AI and if they aren’t then they are living under a rock. However, in my experience, only mid to senior level developers are going to more robustly understand whether the code generated by Gen AI is going to be:</p><ul><li>Practical enough to troubleshoot down the line for the business</li><li>Efficient enough for the business</li><li>Providing the expected output for the business</li></ul><p>AI coding engines are appropriate for sourcing <strong>small blocks of code, followed by the testing of each block. </strong>This is why I feel that Gen AI is usually not as rapid as we may want, given that the quality control process takes time. This is why I strongly disapprove of technologies such as <a href="https://www.pecan.ai/">Pecan.ai</a>, which supposedly allow anyone to make predictions with business data that they won’t understand. If your business aim is to gain a competitive edge using data, using a Gen AI like this has a decent chance of being counter productive as it has no sense of risk of being wrong (and it <strong>will</strong> be wrong) and potentially damage your business prospects by confusing it with dodgy insights. What Gen AI is really doing is <strong>speeding up the process of searching Stackoverflow and creatively applying its best guess with your own problems.</strong></p><p>It is my view that you could create small applications with AI generated code but the true efficiencies aren’t the 55% time saved touted by tech companies in 2023 but more like 10% to 20% when time taken to correct errors have been factored in. The gains appear much less with complex data tasks. However, the irony is that<strong> all data tasks in business are complex.</strong> I am yet to come across a single valuable solution that isn’t complex. For instance, what do you do when the dataset you are using to make predictions about business sales smells wrong? What if, for example, someone massaged a metric on purpose for political reasons (I have seen this as a regular issue in every industry I have worked in) — what do you do? An AI won’t know — a human has to <strong>look at and explore</strong> the data first, ask questions, ring some people up, have a laugh about it and get on with it.</p><p><strong>High volume, low value tasks</strong><br>Larger, unchecked swathes of AI generated code blocks are generally a bad idea but could be considered for high volume, low value tasks. You could use it for high value tasks but you would have to expect failures and factor this in to a quality control process and an insurance policy. Other applicable use cases might include where the answer is never going to be clear cut, so being incorrect may not matter as much. This could be in:</p><ul><li>Prototyping</li><li>Creating web app UI’s and associated rudimentary code</li><li>Content generation in social media</li><li>Routine customer chatbots</li><li>Rudimentary data analysis or idea generation</li></ul><h4><strong>When Should Your Team Not be Coding with Gen AI</strong></h4><p>Be extremely cautious of teams using it to write code that extracts and transforms business data from a database in order to answer a business query, such as ‘how much money has this Post Office made in the last year vs how much was received’. Be even more cautious if this is being developed with complex data sets by junior developers in isolation, managed by non-technical execs (I have seen this occur a lot over 13+ years).</p><p>The <a href="https://www.bbc.co.uk/news/business-66843548">Horizon software scandal at the Post Office</a> resulted in many innocent post masters going to prison. The<a href="https://www.computerweekly.com/news/252526586/Horizon-system-EPOSS-code-writers-lacked-basic-programming-skills-public-inquiry-hears"> programmers behind this were inexperienced</a>, which is like having a team of junior developers paving the way during your development. This isn’t dissimilar to how most AI code generators function.</p><p>This is why Matt Welsh’s diagram for a ‘future’ app development team is way off:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAguoIJHpjci18Fi0di5Lg.png" /><figcaption>Taken from <a href="https://www.youtube.com/watch?v=JhCl-GeT4jw&amp;t=2672s">Dr Matt Welsh</a>’s talk. Although Matt says he codes, he clearly does not</figcaption></figure><p>I understand Matt’s view in the diagram above. Admittedly, a year ago I had the same view. However, my experience tells me that a human code ‘reviewer’ is still required to properly look at code and stick it all together carefully like Lego i.e. they are a developer that can get a bit more done in a day, depending on the complexity of the task. Depending on the ambition of a product, the team size really depends on the use case.</p><p><strong>Smelly AI code — Where does it matter most?</strong><br>I have personally wasted afternoons backtracking to solve issues that were created relying on Gen AI in order to write complex code that extracted and transformed data using a relatively simple language called SQL. It all looked plausible but the resultant code and numbers I received were so far off reality. With no one in the team familiar with the AI code, it took a while to amend. I may as well have thought about it from scratch!</p><p>Ironically, I have found myself with someone elses code wondering whether ChatGPT can help me understand it. I pray to the GPT gods and throw the die with it, praying it can provide some illumination . I think this is an appropriate use case, getting the brain going on it, even if it’s wrong.</p><p>Therefore, we can say that applications where an individuals health or money is directly at risk from unattended code is probably a bad idea, unless blocks of code have been created in small chunks and validated properly, or a quality control process with acceptable risk limits and waivers posed. You can guess my thoughts at <a href="https://sifted.eu/articles/diffblue-ai-code">Diffblue who are seeking to write entirely automatically generated apps using AI for JPMorgan</a>. <br><strong>It just smells wrong.</strong></p><h4>Worth Talking to Your Senior Tech Team</h4><p>I have now heard multiple senior leaders question whether experience matters as much anymore — good luck with that one. I have also heard multiple times now that junior developers can do everything now with that the cost of intelligence falling. No, not really. Yes, they could write a lot of code but what good is that if, as I outlined above, the cost down the line is high? For example, if your aim is to use data in order to increase efficiency in the business and gain a compeitive edge,<strong> you want correct insights to move the needle</strong>,<strong> not some made up one.</strong></p><h3><strong>One last thought</strong></h3><p>An insight that Dr Matt Hart recently blew my mind with was the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*lTqkQNwxv6mY8oc1Zi3f4w.png" /></figure><p>I think that he has a point with the level of disruption coming in the tech industry. However, how much is the opportunity cost of developing an original and compelling new product? It is probably much higher. I am both optimistic and pessimistic about AI, and I think its borderline psychopathic not to have any pessimism (surely Sam Altman and Yann Le Cunn must feel some uncertainty?). However, I have a suspicion that the tech sector will quickly get much more imaginative, creative and potentially more exciting in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3196f74fb52d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do Interest Rates Tackle Inflation?]]></title>
            <link>https://medium.com/@eddyojb/do-interest-rates-tackle-inflation-af4bfce45b88?source=rss-271e93aae03------2</link>
            <guid isPermaLink="false">https://medium.com/p/af4bfce45b88</guid>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cost-of-living]]></category>
            <category><![CDATA[interest-rates]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Eddy-OJB]]></dc:creator>
            <pubDate>Wed, 22 Nov 2023 10:06:55 GMT</pubDate>
            <atom:updated>2024-01-17T10:18:09.197Z</atom:updated>
            <content:encoded><![CDATA[<h4>If correlation means causality then possibly not. It may have an effect, but the effect might be weak on inflation and brutal on society</h4><h3>Contents</h3><p><strong><em>Too Long Didn’t Read<br>Motivation<br>The Test Question<br>Method — What’s wrong with VAR models?<br>Results &amp; Discussion<br>Closing Thoughts</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*Xzod4NDmnYi9dfR6DrKG7w.jpeg" /><figcaption>Image from the <a href="https://www.bankofengland.co.uk/explainers/how-do-higher-interest-rates-help-to-lower-inflation">Bank of England</a>. Wow, somebody sure loves mustard yellow over there. Setting aside the questionable colour scheme, I hold up to the light the Bank of England’s rationale for raising interest rates</figcaption></figure><h3>Too long, didn’t read</h3><p>As a data scientist and an outsider to the field of Economics, I explored the link between UK interest rates and inflation.</p><p>I identified <strong>c</strong>orrelations that broadly tie in <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiTm-7N3LSCAxW5QUEAHZBvCIUQFnoECDMQAQ&amp;url=https%3A%2F%2Fwww.elibrary.imf.org%2Fdownloadpdf%2Fbook%2F9781484339749%2Fch015.pdf&amp;usg=AOvVaw0iWWBLsP6zEwyBjn1G92wO&amp;opi=89978449">with models from the International Monetary Fund (IMF)</a> but identified issues that led me to question conventional wisdom in the field:</p><ul><li><strong>Asking a different question:</strong> Mainstream economists use ‘VAR models’ in order to reason about the economy whilst making a big assumption that interest rates do indeed tame inflation. This is why I took a step back and<strong> </strong>correlated interest rate hikes with<strong> </strong>real-world <strong>inflation outcomes </strong>for<strong> </strong>each consecutive financial quarter instead</li><li><strong>Counter-intuitive outcomes: </strong>It is plausible that interest rate hikes stoke inflation higher, especially in the earlier stages of interest rate hikes. If it does eventually bring down inflation, the overall effect might be weak</li><li><strong>Major changes to inflation:</strong> I cannot see how interest rate changes explain major changes to inflation rate outcomes <strong>— </strong>I think that other factors probably cause <a href="https://www.wsj.com/economy/central-banking/uk-inflation-fell-to-two-year-low-in-october-358c55c3">big drops in inflation</a></li><li><strong>Self-critical feedback:</strong> At the end of this article, I list what I think could improve my own work as well as ideas to mix up the field of Economics</li></ul><p>The Bank of England head Andrew Bailey recently said that <a href="http://The statistics clearly show that there are other factors beyond interest rates are much much more important when it comes to inflation. Former Shadow Chancellor Ed Balls was asked during the Political Currency podcast  about Andrew Baileys statement that interest rates are working to bring down inflation. In short, Ed Balls called bullshit on Andrew Bailey (as I am) that much bigger factors are at play than interest rates to bring down inflation, such as the state of the US economy, which is much more influential on inflation than interest rates. Thanks for backing me up Ed! Ed Balls in Strictly Come Dancing. He sure knows how to boost his career!">interest rates were working</a> to bring down inflation, a statement that former shadow chancellor <a href="https://open.spotify.com/episode/1H1FTBFgS6RAJ1QW6Z82fd?si=870834543a83400d&amp;nd=1">Ed Balls showed skepticism on</a>, stating that the US economy is much more influential on our inflation than UK interest rates. I can believe this, so is it necessary to drag down a perilous UK economy with interest rate hikes?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*PCSM4XmFxdtmpkHa2v_P6A.gif" /><figcaption>Using the wrong type of fire extinguisher can result in unintended results</figcaption></figure><h3>I am a programmer too, just give me the code</h3><p>For any fellow Python programmers out there, my research code is written in Python and open and available on <a href="https://github.com/eddyojb88/interest_rates_analysis/tree/main">Github</a> with data sources and statistical results inline.</p><h3>Motivation</h3><p>Feeling inspired by another great <a href="https://open.spotify.com/show/3a7KblTE1mS3Vz7Fo1WE9Jhttps://open.spotify.com/show/3a7KblTE1mS3Vz7Fo1WE9J">Bubble Trouble podcast</a> by the brilliant Will Page and Richard Kramer, I decided to take a look behind a sceptical dig at interest rate policies by Will, who is former chief economist at Spotify. He was making the point that a correlation of interest rates and inflation does not mean that they are causally linked.</p><p>However, there are ways to help tease out causality in statistics and I wanted to explore this. I grew a big itch, so big that it managed to pull me away from watching every Harry Potter back to back on Netflix and in to the misty world of wand waving Economists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*rWJyGX5D80UwAexmttTf6g.png" /><figcaption>The Bank of England Chief Andrew Bailey as a wizard. Thank you Generative AI. Apparently Andrew needs 6 or 7 fingers to make up policies for the Bank of England.</figcaption></figure><h3>The Test Question</h3><blockquote><em>Central banks claim that raising interest rates helps tackle inflation. If true, then I would expect yearly hikes to the interest rate to consistently correlate with falls in the inflation rate at later points in time.</em></blockquote><h4>Rationale</h4><p>When the Bank of England raised interest rates by 3% last year, it corresponded to an increase in <a href="https://www.finder.com/uk/mortgage-statistics">mortgage payments by £500 per year for an average UK household. </a>This means millions of us have £500 less to spend on goods — for most, that’s not something you find behind the back of a sofa. This is one supportive theory behind reducing inflation.</p><p>From the description above, I think it should be clear not to look at the interest rate and inflation rate itself, but the<strong> difference in interest rate</strong> corresponding to a <strong>difference in spare household income</strong> and<strong>, </strong>therefore, <strong>the</strong> <strong>difference to inflation rate</strong> after a period has passed.</p><h3>Method (and what’s wrong with VAR Models that Economist use?)</h3><p>I used lending rates data for central banks and the Consumer Price Index (CPI) data as a proxy for inflation from the <a href="https://data.bis.org/">Bank of International Settlements </a>(BIS). I used linear regression to test <strong>yearly changes to interest rates vs differences to the quarterly inflation rate at each consecutive financial quarter.</strong></p><p>I used all UK data from 1960 but<strong> </strong>excluded the period after 2008 because in my view that period already undermines the use of interest rates.</p><h4>Why didn’t you use a VAR model like other Economists?</h4><p>Yes, I am an imposter with less domain expertise. I am only observing what I see from the outside, which is that I don’t view VAR models as an appropriate choice. I really like the concept of them as they allow for interest rates to affect inflation and, in reverse, for inflation to affect interest rates. However, they are designed to help reason about a dynamic relationship between GDP, inflation, interest rates and employment, heavily relying on the huge assumption that interest rates really do affect inflation. Just because we can reason about a system, it doesn’t make the reasoning correct or mean that policy decision making is having the intended effect.</p><p>This is why I felt like I needed to step back and correlate interest rates with <strong>inflation</strong> <strong>outcomes </strong>i.e. for each consecutive financial quarter after an inflation hike, do we generally see the desired outcome of inflation coming down in the historical data? Is this statistically and practically significant? I don’t think VAR models allow for this reasoning.</p><h3>Results &amp; Discussion</h3><h4>What does the Inflation Rate and Interest Rate Look Like in the UK?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*aPldoWMiX9hGk5xr72iiAA.png" /><figcaption>Quarterly CPI (Pink) vs Central Bank Rates (Blue) for the UK</figcaption></figure><p>Interest rates and inflation appear to move approximately in tandem and inflation can rise or fall dramatically in just 1 financial quarter! However, the pink line and elephant in the room is the period following the 2008 financial crisis showing that inflation can do its owns thing.</p><h4>Correlating Interest Rates and Inflation (Taking A Lag Period in to Account)</h4><blockquote>When plotting the data we can visually see other factors beyond interest rates that are much more important when it comes to major hikes or drops in inflation</blockquote><p>Since a change in interest rates could take time to come in to effect, I used the yearly change in interest rates (adding a lag period in order for it to take effect) and correlated this with UK inflation rates for each consecutive financial quarter.</p><p>Here is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*NczY3mO_mb3hVDgeu9CnKA.png" /><figcaption>Using linear regression to model yearly interest rate vs inflation rate changes after an additional lag of 2 financial quarters has passed. Counter-intuitively, the relationship is positive here</figcaption></figure><p>In the graph above, 1 year of interest rate changes has passed, such as a 3% rise in the last year, and we have waited a further 2 financial quarters to pass in order to see if it generally affects the quarterly inflation rate. We can see a positive correlation.</p><p>Below is another plot but this time after 2 years has passed. Even though the inflation trend is going down, there are more extreme data points scattered either side of the line - the line of best fit fails to explain these major changes to inflation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*3eu4UKj1Y70jnhoC2cG6hQ.png" /><figcaption>Measuring yearly interest rate hikes vs inflation after a lag of 2 years using linear regression. Quite a spread of data towards the end of the graph!</figcaption></figure><p>If you break up the data and measure the correlation for each consecutive financial quarter after a yearly interest rate hike, you get the correlation curve below. After waiting for 12 financial quarters, the correlation is at its trough:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIWjpNUlb86KIEb5JKl1Ww.png" /></figure><p>The graph above has a remarkably similar shape like to the one IMF present in their paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*JSgQg_SaxE1FxKkyJTMCoA.png" /><figcaption>Taken<a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiTm-7N3LSCAxW5QUEAHZBvCIUQFnoECDMQAQ&amp;url=https%3A%2F%2Fwww.elibrary.imf.org%2Fdownloadpdf%2Fbook%2F9781484339749%2Fch015.pdf&amp;usg=AOvVaw0iWWBLsP6zEwyBjn1G92wO&amp;opi=89978449"> from the IMF</a>. They have set ‘0’ as the point where interest rates ‘kick in’</figcaption></figure><p>However, there are issues that I outline below. Results for correlations are presented in the table below with lag periods up to 24 quarters after yearly changes to interest rates. I have presented correlation values, p-values and R² values as a supplementary (and controversial) measure of model quality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*E40GklqNZM-rve8BpfQiwg.png" /><figcaption>Highlighted in red are the p-values that I’m not happy with in an assumed linear relationship between interest rate hikes and inflation outcomes</figcaption></figure><p>Remember that we are looking at outcomes of inflation after interest rate hikes for each consecutive financial quarter, so if we are to take correlations as causal, then this suggests that the inflation outcome for a time lag of 1 and 2 quarters stokes inflation higher, indicated by regression coefficients above 0 and low p-values.</p><p>If we look at a further 6 to 14 quarters later, we can observe intermittent periods with negative correlations and acceptable p-values i.e. inflation coming down with associated interest rate hikes. However, there are periods where there are no decent p-values, such as 7, 10 and 12 financial quarters after. This signals that<strong> this model isn’t consistent over time</strong>. I think that in order to take interest rate changes seriously in the linear models that economists use, it should be consistent. However, we know that the world is non-linear and world events can dwarf any modelled effect.</p><p>The quality of the model isn’t great but if we take this model at face value and estimate the inflation outcome of a yearly interest rate hike of 3% running its course to 18 financial quarters after, we get a roughly 1% drop in the inflation rate.</p><p>I have also included controversial R² values here, which quantify what we can see in the plots i.e. that <strong>interest rate hikes don’t explain a lot when it comes to major changes to inflation.</strong></p><h4>What do the VAR models say?</h4><p>I have applied a basic VAR model, which can be found appended to my code but as I mentioned earlier, I decided against the use of a VAR model.</p><h4>Economists talk about supply driven vs demand driven inflation events — can a pattern be teased out by separating these events?</h4><p>Climate change, wars and technological advancements all affect inflation, so I tried to assign these to each period but realised that it was both an enormous effort and very subjective.</p><h4>What about interest rates for the US and the Eurozone?</h4><p>For these economic zones, the results were still not convincing to me.</p><h4>Is there a rationale to explain how inflation would increase after an interest rate hike?</h4><p>If businesses face higher interest rates on their loan repayments then I think that they simply pass the costs on to consumer in anticipation. It is much easier than implementing business efficiencies, which can take years. Unsurprisingly, many employees succeed in getting a pay rise in return,<a href="https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/bulletins/averageweeklyearningsingreatbritain/august2023"> as we have seen over the last year in the UK.</a></p><h4><strong>BoE’s Interest rate target is 2%. Why?</strong></h4><p>Why not 5%? What not a range like 2% to 5%? I haven’t seen a decent explanation here. It sounds like a nonsense policy to me.</p><h3>Closing Thoughts</h3><p>Central banks are important, despite what the crypto bros tell me everyday, and I understand that central banks feel the need to be seen doing <em>something </em>about<em> </em>inflation. However, I am not convinced that using interest rates in order to tackle inflation works. I am firmly in the indeterminate side of the fence, leaning towards ineffective.</p><p>I read from the former Fed head, Bill Dudley, that interest rates are different now that we are now in a new age of an ‘<a href="https://www.ft.com/content/079e1706-5118-4e7f-a0dd-e182b520deb6">excess reserves </a>regime’. In my view, more data needs to roll in before asserting something like that.</p><h4>Improvements to my work and the Economics Field</h4><ul><li>I don’t like using the CPI in order to measure inflation — we should have a much more far fetching data gathering procedure for it. The ROI on that could be huge. The CPI doesn’t include train fares (thank you for that suggestion Will Page) and given that the micro makes up the macro, this is a big deal when we think about how our daily work habits change over time</li><li>I think a robust analysis of the extreme changes in inflation might improve my work i.e. how sensitive to change are the models. However, given that extreme changes appear to be king, I am not sure this matters</li><li>Fact: the world and the economy is highly non-linear. Non-linear models would be more appropriate than the linear regression models that I have built</li><li>I feel that an engineering approach would help the fields of Economics i.e. redeploy efforts on empirical data gathering at scale over theoretical modelling and the endless theoretical ‘proofs’; an interdisciplinary approach that includes science, psychology, data engineering, finance and members of the public that have actually worked in a business — so not Andrew Bailey)</li></ul><p>When looking at the impending doom of my mortgage rate hikes, I found my analysis upsetting — I hope that somewhere in my code I made a big mistake and that I am wrong about interest rates</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af4bfce45b88" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>