In the last 2 years, coding agents like Claude Code and Codex, to name a few, have become an integral part of the software engineering process. They are used by solo freelancers all the way to the largest corporations and I, as a senior software engineer, use them on a daily basis on certain aspects of my work. Generally I find them extremely useful to work with and I can say for sure they measurably increase my code writing throughput.

These coding agents really shine when it comes to creating proof-of-concepts or one-time use programs for personal use and I can surely say they’re pretty fun to work with in scenarios like these. But when it comes to using them in actual products that run in a production environment serving hundreds, thousands or millions of users, not everything is nice and dandy.

In this article my goal is to provide a high level overview of how using coding agents in production environments affects the digital products themselves, outline the problem we’re currently facing and share some thoughts on how we can address it going forward.

Disclaimer: This article was written in August 2026. If you’re reading this from the future, the points made below might not be relevant. Or they can still be. Who knows…

The Noise Problem

As of August 2026 all coding agents share a core defect that originates from how Large Language Models work and how the agents themselves reason through their loops. If you’ve used coding agents you’ve probably already witnessed that in many cases the generated output contains noise. By the term “noise” I mean something that doesn’t really fit into the codebase we’re working on, such as:

• An extremely optimized search function that can run in parallel (or even distributed), while our product needs a basic search function to run on a few hundred items
• Functions handling errors that don’t make sense for the product and cannot occur at all
• Complicated abstractions in absurd cases that end up making the codebase more rigid
• Exceedingly verbose technical documentation and readme files including lots of unnecessary information while missing important points

It really makes sense to think about this noise entering our codebases through entropy. It’s a term borrowed from physics, but the idea is simple: it’s a measure of how disordered a system is, similar to why a desk left alone tends to get messier over time rather than tidier on its own. Entropy in the world tends to increase, and we put active effort into fighting it whenever we want things to stay organized and controllable.

In software, this matters for two reasons. First, there’s complexity. How easy is it to understand the intention behind a line of code, a function, or a whole system? Every feature you add makes that a little harder, and as engineers we try to keep that increase as small as we can. Second, software runs on real hardware with finite resources. A feature that quietly costs 20% more RAM, CPU, or network calls is entropy too. It’s just measured in resource demand instead of readability.

Why This is a Coding Agent Problem

So coding agents are prone to increasing the entropy through the changes they apply into the codebase. But humans make changes too and yes we increase the entropy as well. But why is this considered a coding agent problem and not a human problem too?

Short answer: The amount of changes a human can do in the codebase in a certain time is incomparable to the amount of changes an agent can make. Bundled with the fact that all lines of code that will theoretically go live to a production environment and serve real people, must be reviewed by humans, already creates an imbalance between composing and reviewing software.

It’s Already Happening in Open Source

A great example of this imbalance is currently taking place in the open source world. Many open source projects that we use every day are flooded with PRs containing changes written by coding agents that haven’t been properly reviewed by a human, trying to solve irrelevant problems or unnecessarily refactoring parts of the codebase. All these accumulate and in some way pollute these projects as they demand attention from the human maintainers. And there are practical examples of this, like this post by the creator/maintainer of curl, Daniel Stenberg or this post by Anthony Fu who is an engineer of Vue’s core team. They both describe how the time spent on really low quality AI-generated contributions has increased dramatically over the past years and how this “noise” makes it really hard to find the actually meaningful contributions in the PR list. And the situation escalated further some weeks ago when Codeberg announced on their blog that they are banning projects that “mostly consist of code written by generative AI” as they think that coding agents pose a threat to the open source community.

What the Data Says

And this is not strictly related to open source. In the most recent analysis conducted by Faros.ai (titled “More Code. Not Production-Ready Code”) based on over two years of data from 22,000 engineers and 4,000 teams across their platform, it found:

• +33.7% task throughput per developer
• +66.2% epics completed per developer
• +16.2% PR merge rate per developer
• +210% tasks completed per team with an associated PR

But all these appear to come with a huge cost as the analysis also found:

• +861% code churn. This indicates massive code re-writes and code instability which can be a great indicator of the entropy increase
• +51.3% average PR size
• +59.7% average files edited per PR
• +25% average review comments per PR
• +22.7% average length of PR review comments
• +31.3% PRs merged without any review

And what are the consequences of this increase in entropy in the actual product? The same analysis states:

• +242.7% incidents per PR
• +57.9% monthly incidents
• +54% bugs per developer
• +28.7% bugs per PR

Even though there has been a significant acceleration in the software writing part (upstream), as we’ve all experienced by using coding agents, at the same time quality has declined at the review stage. And some of that low quality code reaches the production environment as the review pipeline (downstream) struggles to cope with the increased throughput. Of course, there are scenarios where coding agents are utilized during the review process which often helps (especially using agents of rival companies reviewing one another) but still all code changes must be checked by a human.

Tame Them. Don’t Let Them Tame You

Undeniably this is the core issue the industry is facing right now, and while there can be many half-measures to be taken, my personal opinion is that resolving this issue means attacking it at its root. Faros frames this as an authoring problem, not a review problem, and says the fix belongs upstream: better context, guardrails, agent instructions. I agree it’s an authoring problem, but authoring isn’t just the agent generating text. The engineer directing it, and shipping it under their name, is an author too. We, the engineers, are the architects of a bug fix or a feature implementation and, thus, we must fully understand each line, yes each line, the coding agents write for us. We must have full ownership of our work and be able to justify why we chose this approach over something else. If we find ourselves against an unknown concept, library or piece of tech in general, we must take a step back to familiarize with it, learn its inner workings, its quirks, etc before accepting or rejecting its integration into our system. Our job is not to accept or reject proposed changes. Our job is to understand our product, its needs, the technology we use and how we can make it better so that it serves real users.

What’s the point in making open source contributions if you can’t wrap your head around the repository you’re trying to contribute to? What’s the point in proposing a solution to a problem that you can’t explain in simple words?

Coding agents gave us a tremendous advantage. But they’re prone to increasing the entropy of our system, so we must tame them. Not the other way around. I know this will slow things down during the code writing part, but we’re not paid to write code. We’re not type writers. We are paid to solve problems and provide solutions. I really don’t care if we closed the ticket on time. What I do care about is that we’ve managed to provide a solution to the problem afflicting our product. I know it’s easy to click on the accept option a couple of times and call it a day, but that defeats our goal. We mustn’t miss the forest for the trees.

This mentality shift can start with, even, the basics, for example writing documentation. This was the most mundane of all tasks as everybody hates to do this but everybody is extremely happy when they read a well-documented piece of code or product. Missing documentation or even worse lying documentation can make a 15-minute task take five times longer. Again LLMs are pretty good at writing documentation but they might miss certain aspects of the use case that aren’t part of the codebase, so before creating one make sure it contains only the needed information without the extra fluff LLMs tend to add. Don’t send a five-page PDF when the documentation could be a five-bullet-point list. Most probably you will be the one who will need to read it in a couple of months.

Another example is writing unit tests. Coding agents do know how to write tests but they don’t know what makes sense to test for your product simply because they do not have the entire picture. Of course, in some scenarios what to test is straightforward but in others it’s really not, because your team might have limited effort so choosing what to test is a really big deal. Don’t just say “Oh and don’t forget to add unit tests” before sending it for review and state explicitly what should be tested. You must know your system better than your coding agent.

Also coding agents tend to write what I call “smart snippets”. Your application needs some basic data transformation and the coding agent generates a very complex, hyper-optimized function, as if we would use it in the Linux kernel or for low-latency trading. Does your CRUD application need a smart function or a clear and readable one? Choose readability and transparency over premature and, in most cases, unnecessary optimizations.

Final Thoughts

To sum things up, the way we write software has indeed changed. I don’t know if coding agents will stay forever but for now they are the primary author as the Faros.ai analysis confirmed. However, this doesn’t mean we should use them in an autopilot mode when we’re dealing with production systems. At least not for now. When they act on their own, without the necessary oversight, they tend to increase the entropy and add noise to your codebase, making it look more generic (a loss of indentity), less readable and, as the analysis showed, prone to bugs and incidents.

For the sake of keeping the software quality of the products we love and use on a daily basis as high as possible, tame your coding agents.

It’s pretty safe to say that all software engineers are familiar with the concept of code testing. In fact, many tutorials include dedicated content on how to build testable applications as well as how to write tests. At the company level, from startups to enterprise software businesses, most claim, to some degree, that they write unit tests and often showcase their coverage numbers. “Our product has X% coverage” or “We don’t ship if the coverage is less than Y%.”

Even though all of this sounds great and aligns with widely accepted best practices, unit testing goes far beyond just measuring code coverage, regardless of the method used. In fact, it’s often misrepresented in the industry. In this article, we aim to explore what it truly means to write unit tests. Why do we do it, after all? Is it just to boast about our line coverage and make our clients (technical or not) feel reassured about using our product? The answer, as it turns out, is not that simple.

Unit Testing

Before diving into the specifics, let’s take a moment to introduce what unit testing fundamentally is. Simply put, unit testing is the practice of ensuring that the individual “units” of a codebase function as intended. By “units,” we typically refer to the fundamental building blocks of a software application, most often, functions. Each unit test is designed to verify that a specific aspect of a given function behaves in accordance with the product’s requirements. By thoroughly testing all, or nearly all, functions in an application, we aim to safeguard the codebase against bugs and regressions that may arise when new features are added or existing issues are addressed.

When unit testing a function, we isolate it from its environment, that is the rest of the codebase, to test it thoroughly without interference from external factors. To achieve this isolation, we mock and/or stub all external dependencies, allowing us to focus solely on the function’s internal workflow. Each unit test typically consists of three stages:

1. Setting up the experiment: We define the initial conditions under which the test will run. This typically involves preparing the input data for the function and configuring how the stubbed or mocked dependencies should behave.
2. Acting: This is the simplest step, where we invoke the function under test using the input prepared in the previous stage.
3. Asserting: After executing the function, we verify its output to ensure it behaves as expected, both by asserting what it should do and what it shouldn’t do.

Finally, the most common approach to evaluating how well-tested a codebase is relies on test coverage metrics. These metrics help quantify the extent to which the code is exercised by tests. The most popular ones include:

• Line Coverage: Every line of code is executed.
• Branch Coverage: Every possible if/else branch or conditional path is tested.
• Function Coverage: Each function is called in tests.
• Condition Coverage: All boolean sub-expressions in conditionals are evaluated both true and false.

Example Setup

To demonstrate our point, let’s define a very simple coding example. Although the example is written in TypeScript and uses a specific set of libraries to assist with testing, keep in mind that all the concepts discussed can be applied to any programming language or testing framework.

Our goal here is to work with an imaginary codebase where we want to ensure that our processOrder function behaves as expected:

The function is straightforward, so we’ll keep the explanation brief. At a high level, it receives an order as input, performs some calculations (such as total cost and discount), processes the charge via an external service, and finally returns the order results. To keep things simple, we haven’t covered edge cases, such as price or quantity being zero or negative, or missing/null properties.

For the sake of completeness, we’re including the contents of the type.ts and paymentGateway.ts files.

Writing Tests

To unit test the processOrder.ts function effectively, we use a set of auxiliary libraries:

• Mocha: A flexible test framework providing structure for writing and running unit tests
• Sinon: A library for creating stubs and mocks to control and inspect function behavior
• Proxyquire: A library allowing to override dependencies for better isolation.

Again, these libraries are used for demonstration purposes only. Any other set could be used without affecting the essence of the examples.

Bad Mentality

First, let’s address the “bad mentality” when it comes to writing unit tests. This mindset often begins with someone blindly enforcing testing requirements by insisting, for example, “we need to hit x% test coverage.” Technically, achieving high test coverage is fairly straightforward. In our example, we can reach 100% coverage with just four individual tests:

So, mission accomplished. Our codebase is 100% tested, every line, branch, and function is covered. The product team is happy, the client is happy, and we, the engineering team, should be happy too. Right?

No. We can’t be happy.

Because in reality, our codebase isn’t truly 100% tested. We wrote tests that achieve full coverage, but we haven’t actually verified any of the core functionality. To illustrate the problem, imagine that in a future commit, an engineer accidentally removes the discount calculation (check line 19):

If we run the unit tests, all of them pass, even though our function is seriously flawed. The user ends up paying the full amount without receiving the discount they’re entitled to. Of course, this is a simple example; hopefully, no engineer would make such a mistake in a small application like this. But in real-world scenarios, regression issues happen more often than you might expect.

Good Mentality

Aiming for an x% test coverage is actually a very good goal, and there’s nothing wrong with it. The problem discussed in the previous section didn’t arise from the goal itself, but from how superficially we approached it. We simply wrote basic tests without asking the most important question: Why do we test?

That’s a question every engineer should ask themselves before writing any unit test. What exactly are we trying to verify by testing the processOrder function? What must this function do, and what must it never do? If we had asked these questions upfront, our unit tests would have been much stronger.

Let’s get practical. At minimum, the function should:

• Apply the discount properly
• Call PaymentGateway.charge
• Return an invoice with the appropriate data
• Throw an error when the list of items is empty
• Not call PaymentGateway.charge when the list of items is empty

These points can be directly translated into the following test cases:

Re-running the unit tests will now reveal the regression problem introduced by the fictional engineer:

Conclusion

How can you write unit tests that truly matter for your digital product? It’s actually quite simple: always ask why. Why do I need to test this function? Testing shouldn’t be done just to increase coverage numbers. Coverage metrics are not the ultimate goal, but they’re an oversimplified way to measure the “extent” of our tests. That said, coverage metrics can often be misleading. This means we need to shift our mindset away from focusing solely on these numbers and instead concentrate on what really needs to be tested. When we ensure the core functionalities are thoroughly battle-tested, we can be confident that our test coverage will naturally be as high as possible.

We, at fromScratch Studio, strongly advise you to:

Test what matters most for your product. Don’t test just for the numbers, as they can be misleading.

TypeScript Unit Tests That Matter: Go Beyond Code Coverage was originally published in fromScratch Studio on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every AI engineering team loves building and evaluating machine learning models. It’s undoubtedly the most exciting part of the pipeline, where engineers come together to see how their creation performs in action. It’s also the stage where they might realize they’ve developed something groundbreaking, pushing the boundaries of innovation.

To reach this point, many steps must be completed first. Of course, data is essential. In today’s world, data is everywhere, and with platforms like Kaggle and Hugging Face, accessing it has never been easier. However, finding annotated data — that is, data with corresponding class labels — can still be a challenge.

This often occurs when we gather data ourselves from various sources. The data might come from web crawlers or, more frequently, from other projects or products within the company. For example, we may want to build a machine learning model that determines whether a review for a product on our e-shop is “Positive,” “Negative,” or “Neutral.” While this scenario is highly simplified and may be considered “solved” with pre-trained models, even these models can produce low-confidence predictions, making it a valid and relevant use case.

At fromScratch Studio, we frequently encounter such challenges, and depending on the scale of the scenario, we employ different techniques to solve the problem. With that in mind, we’ve decided to demonstrate how easy it is to create such a pipeline using straightforward tools.

In this article, we will explain how to create an annotated dataset using Python scripts, pre-trained models, GitLab CI, and Label Studio.

If you want to see a somewhat similar pipeline, check this article: ETL Pipelines With Python Azure Functions

Problem Statement

We will work on the aforementioned sentiment analysis task, where we aim to build a dataset consisting of user texts and labels indicating “Positive,” “Negative,” or “Neutral” sentiment. The user texts will be provided by the data engineering team from one of our products. We have chosen to use the twitter-roberta-base-sentiment model to generate the sentiment labels, but we’ve noticed instances where the prediction confidence is low.

High Level Solution

A popular method for resolving such issues is to bring a human into the loop. All low-confidence entries will be reviewed by a human to determine the final sentiment. While the pre-trained model’s prediction may be correct, there’s also a chance it may not be entirely accurate.

We will use a straightforward architecture for this pipeline:

1. The data engineering team will be providing data in an Azure Blob Storage on a daily basis. You can use any kind of storage or service you want (even Google Drive) with minor adjustments.
2. Our Gitlab CI pipeline will be scheduled to run once a day to:
   - Collect the data.
   - Perform sentiment analysis.
   - Identify low-confidence cases
   - Import the data as pre-annotated entries into Label Studio
3. Our Labeling team will check the cases and provide the final verdict

Solution

We will build the pipeline step by step. We recommend using a private Google Colab notebook for experimentation before pushing your script to GitLab. For obvious reasons, we won’t use a private dataset; instead, we will use the multiclass-sentiment-analysis-dataset. This dataset includes sentiment labels, so let’s prepare a sample to simulate the data that the data engineering team will be providing.

Getting Data

First, we need to create an Azure Blob Storage account, add a new container named ‘sentiment’, and upload the CSV file we previously generated. You can name the container whatever you like. Next, retrieve the ‘Connection String’ from the ‘Security -> Access keys’ tab. To read the data from the container, we will install the azure-storage-blob dependency:

Then run the following code:

The `csv_data` will look like this:

Sentiment Analysis

In the second step we need to perform sentiment analysis:

Next, we need to identify the low-confidence predictions. For the purposes of this article, we will classify all predictions with 75% confidence or lower as low-confidence predictions.

Prepare Pre-Annotated Data

Before importing the data into Label Studio, we want to include the model’s annotations as well. Label Studio expects the pre-annotated data to adhere to a specific format. To create this format, run:

We can save the pre-annotations in a JSON file.

Import

Finally, we have to import the data into Label Studio.

In Label Studio, ensure that you create a new project using the Text Classification template. In the script, replace {LABEL_STUDIO_DOMAIN} with the actual domain where you host Label Studio and {PROJECT_ID} with the ID of the project you just created (this will be `1` for new Label Studio instances). Finally, you can find the Label Studio API token in the settings.

And that’s it! If everything runs smoothly, you will be able to see the pre-annotated data in your Label Studio instance:

Gitlab CI

The final step before concluding this article is to create the scheduled pipeline that will run on our Gitlab instance. First of all, we make sure to create the following Gitlab CI variables which are self explanatory:

• `AZURE_CONNECTION_STRING`
• `CONTAINER_NAME`
• `LABEL_STUDIO_API_TOKEN`
• `LABEL_STUDIO_IMPORT_ENDPOINT`

Then, we push this script into our Gitlab repository:

Also, we create a `requirements.txt` file with all of our dependencies:

Finally, we add the following CI yaml file:

At this point, we can run the pipeline and view our annotations in Label Studio!

To schedule this pipeline to run once a day, navigate to the Pipeline schedules page in GitLab. Select your branch and set the cron expression to 0 0 * * * to run the pipeline at 00:00 every day. And that’s it!

Conclusion

In this article, we showcased a simple pipeline that reads data from Azure Blob Storage, provides sentiment analysis predictions, and imports the low-confidence results into Label Studio for further investigation. This architecture is highly convenient for scenarios where the data flows are not complex, and the demands on storage and memory are minimal. However, keep in mind that more complex scenarios may require more sophisticated infrastructures and tools, such as Airflow and Kafka. For many everyday cases, this solution will do the trick!

Efficient Data Labeling: Leveraging Label Studio, GitLab CI, and Python in Your Pipeline was originally published in fromScratch Studio on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to the third article on predicting energy consumption demands for the city of London. In the first article, we explored the London Energy Dataset and the London Weather Dataset, where we observed that incorporating auxiliary features — such as weather data, temperature, cloud coverage, and precipitation — enhanced the model’s performance by 44%. In the second article, we investigated what are known as serially dependent features, or lag features. Lag features capture information related to cycles, where changes in the target value are not directly correlated with time but rather with previous target values. After conducting several experiments, we concluded that using nine lag features yields the best results, resulting in a 35% reduction in MAE compared to the best model from the first article.

In this endeavor, we will explore additional tools frequently used in time series problems that can enhance model performance.

This article will build on the foundations laid in the previous two. To fully understand the content and run the baseline models effectively, please review the previous articles carefully before proceeding.

1. Time Series Forecasting with XGBoost and LightGBM: Predicting Energy Consumption
2. Time Series Forecasting with XGBoost and LightGBM: Predicting Energy Consumption with Lag Features

Rolling Statistics and Moving Averages

Rolling statistics and moving averages are statistical methods used to summarize information over a specific period of time. One such method is the 7-day rolling mean. For example, let’s consider the following energy consumption values in KWH:

To calculate the 7-day rolling mean as of today, we need to follow these steps:

Essentially, the 7-day rolling mean captures information about consumption over the past week. In contrast to lag features, which enable the model to learn the direct influence of previous consumption values on the current value, rolling statistics provide a broader perspective on the overall behavior of the time series. Such aggregations can often identify trends and variability that lag features might overlook. Additionally, rolling statistics can highlight recurring patterns over specific intervals and can, to some extent, mitigate the impact of random noise due to their smoothing effect.

However, it’s important to note that not everything is straightforward. Choosing an inappropriate window size can be quite challenging. If you select a small window, you might miss essential patterns; conversely, if you use a larger window, you may inadvertently include too much noise.

When using the 7-day rolling mean, we treat each of the seven days equally, meaning that Day 6 contributes the same amount to the calculated mean as Today. While this approach may be suitable in certain scenarios, it might not be ideal for our purposes. For instance, if we aim to use the 7-day rolling mean to predict tomorrow’s consumption demands and wish to place more emphasis on the more recent days, we can employ the Exponential Moving Average (EMA). This statistical method builds on the classic 7-day rolling mean but assigns higher weights to recent observations. By doing so, the model theoretically becomes more responsive to recent changes.

There are many implementations of the Exponential Moving Average (EMA). One of the most basic formulas is as follows:

Where:

• EMAₜ​ is the Exponential Moving Average at time t.
• Xₜ​ is the value of the time series at time t.
• EMAₜ₋₁​ is the Exponential Moving Average at the previous time step.

α is the smoothing factor, which determines the weight given to the most recent observation.

The α factor can take values in the range of [0, 1]. More commonly, we use the following formula for α, which takes into account the chosen window N.

To compare the basic 3-day rolling mean with the Exponential Moving Average (EMA), let’s use the consumption data from Table X to calculate both the 3-day rolling mean and the 3-day EMA.

It is evident that the EMA reacts more quickly to changes, making it more responsive to the actual consumption of that day.

Experiments

Before we delve into the new experiments, let’s revisit the results from the previous article, where the baseline model was identified as the best model from the first article:

To prepare the DataFrame for training, we run the following:

First, let’s conduct some experiments using the classic rolling mean in conjunction with XGBoost. To do this, we will use the following lines of code:

Please note the shift function at the end. Omitting this will lead to data leakage since the mean will include today’s consumption too.

Special thanks to charuni poornima for pointing this out and improving this article!

Next, we initiate the training process and await the results:

The 2-day run produced results close to the best run that utilized lag features. However, the lag features appear to outperform the moving averages in terms of MAE, MSE, and MAPE. This indicates that the model does not benefit significantly from using the rolling mean.

What about the EMA? Assigning different weights to the consumption values from previous days may impact the results. Let’s replace the rolling mean with the EMA:

Once again, we will execute the experiment snippet.

The 2-day EMA results are even closer to those of the run with 9 lag features. This demonstrates that the EMA may capture more details of the time series compared to a simple moving average. Nevertheless, even the EMA does not provide any advantage over the use of lag features.

Conclusion

While the benefits of using rolling statistics and moving averages are evident in many cases, it’s important to note that they may not always outperform lag features; in fact, their performance can be relatively similar. There are scenarios where these methods may not only fail to enhance the model but could also lead to performance degradation. The effectiveness of these techniques largely depends on the nature of the data and the algorithm employed. I encourage you to experiment with incorporating the n-day rolling standard deviation into your model, but be prepared for the possibility that the results may not meet your expectations. Ultimately, successful modeling requires continuous experimentation and adaptation to the specific characteristics of your data. It’s also worth emphasizing that this outcome is not universal, and further experiments can help identify the most effective approach.

Over the past five years, advancements in artificial intelligence have caught the world by surprise. This surge began in 2019 and 2020 when large language models (LLMs), such as GPT-2 and GPT-3, captured public attention. These models demonstrated near-human performance on a range of language tasks, including generating recipes and cooking instructions. LLMs are trained on vast datasets, primarily sourced from the internet, using state-of-the-art and costly hardware.

Building such a model requires expertise across multiple domains, including data collection, data preprocessing, artificial intelligence algorithms, and machine learning. Even with this expertise, a massive computing infrastructure is necessary to carry out the training, along with a substantial amount of electricity to power it.

Creating a model of this scale is certainly expensive, but also highly valuable. It’s understandable that companies maintain secrecy about the specifics of these LLMs, as they are considered valuable assets. For example, ChatGPT, one of the most well-known LLMs, offers free access to its basic models through a chat interface, with paid access available via API.

Llama 3.1

Before building something using ChatGPT, one must consider the cost implications, which can be restrictive in application development. However, a significant development has occurred with Meta’s open-sourcing of its new language model, Llama 3.1, a fully open-sourced model that boasts performance capabilities comparable to GPT-4. Notably, Meta released three different model versions that are computationally feasible for inference on mid-range computers. According to Mark Zuckerberg, the reasons behind this decision have been clearly articulated in his recent post, providing insight into the thinking behind this move.

When I first heard about automation capabilities like those offered by ChatGPT, one idea immediately came to mind: I could streamline many of my day-to-day tasks that previously required manual interaction or conventional conversations with AI chatbots. For example, I’ve been using ChatGPT to check and refine the spelling and grammar in many of my written texts. Not only do I use it for casual notes, but also for more formal purposes like writing official research reports. Having a second pair of eyes review your content is always beneficial, as it helps catch errors and improve overall clarity.

That’s why I created LexiCore in just a few hours! LexiCore is an AI-powered text enhancement tool in the form of a Chrome extension, utilizing the powerful LLaMA 3.1 model (specifically the smaller version with 8 billion parameters). It reads any text selected by the user on a webpage and suggests an improved version. The extension makes an HTTP call to an API running LLaMA 3.1 and displays the response in a stylish tooltip.

Disclaimer: LexiCore isn’t a new concept in today’s AI landscape, nor do I claim it outperforms ChatGPT — which in fact it doesn’t. What I’m highlighting is the accessibility of building AI-powered tools without restrictions, and I’m encouraging everyone to automate tasks that were previously done manually, leveraging opensource AI resources to drive community progress and innovation!

LexiCore

As I’ve mentioned, LexiCore is a straightforward Chrome extension that retrieves a user’s selection from a webpage and communicates with a Llama 3.1 instance. The codebase is remarkably concise, consisting of just 2 file scripts that achieves this functionality in just 100 lines of code — making it easily comprehensible even for novice software engineers.

The place where actual magic happens is the Llama model. One of the easiest ways to run the model locally is by using the Ollama service. After installation and setup, you can interact with it via a REST API listening on http://localhost:11434. Responses come in chunks of information, mirroring the same visual experience as ChatGPT’s web interface, where a stream of knowledge flows in to facilitate engaging conversations.

Currently, both extension and model must be run locally. To install the extension, follow these steps to successfully import the GitHub project. Once imported correctly, you will see the following popup when you click on the extension icon, verifying that everything is set up properly.

Then, run `ollama run llama3.1`. To verify that the API works as expected, make a basic request:

curl http://localhost:11434/api/generate -d ‘{“model”: “llama3.1”, “prompt”:”What’s the meaning of life?”}’

I’m pretty sure the answer will be kind of useless. Nonetheless, if you get one you are ready to use the extension end-to-end.

Select a text on the webpage and hit Ctrl + Q. A tooltip should be displayed with a loading text just like the following:

Be patient! The first couple of requests are generally much slower than the next ones.

Voila! The suggestion is for you to use! To close the tooltip just hit Ctrl + Q.

Conclusion

Here’s an improved version of the text: In this post, I aim to show just how simple it is to build tools powered by robust, free, and open-source models. To illustrate this point, I’ve created a tool that I now use daily, which automates the manual requests I used to make in the ChatGPT interface. While the Chrome extension has already been useful, there’s certainly room for enhancement in various aspects, and refining the prompt engineering process could lead to significant improvements in its overall quality of results.

The repository is available for you to explore and use, so please don’t hesitate to submit issues or pull requests. If you’re looking for a more customized experience, consider forking the project to tailor its features to meet your specific needs. This is exactly what open-source software is all about — empowering users to take ownership of their tools and adapt them to fit seamlessly into their daily workflows.

Full Github Repository: https://github.com/gkamtzir/LexiCore

If you enjoyed this article and want to see more content on machine learning, AI, and software engineering, feel free to follow me. I regularly share insights, tutorials, and experiences on the latest trends and tools in the tech world. Stay tuned for more!

Svelte and SvelteKit are relatively new frameworks that have surprised the web development community with their innovative approach. They offer a fun and efficient way of writing code, attracting immediate interest from software engineers. Key advantages include small build sizes after compilation and efficient use of the real DOM instead of a virtual DOM (more details here). These benefits convinced the engineering team at fromScratch Studio to adopt Svelte for both small-scale and enterprise digital products.

Svelte is fast and performant out-of-the-box. However, we can further optimize our code to achieve even higher execution speeds. One can move towards this direction through memoization.

Memoization

Memoization is a performance optimization technique used in computing to speed up programs by storing the results of expensive function calls and reusing the cached result when the same inputs occur again. Essentially, it involves caching the results of function executions based on the input arguments so that the function doesn’t need to recompute the result if it’s called again with the same inputs.

To employ memoization, a function must be pure. Pure functions always return the same output given a specific set of arguments, regardless of how many times they are invoked. For example, a function that calculates Lucas Numbers is pure, as the result remains consistent for the same input.

That might be too theoretical. Let’s ground it with a practical example.

Object of Key-Value Pairs

Suppose we want, indeed, to memoize the function that calculates the Lucas Numbers. A version of the base function would be:

Applying memoization can be done with various approaches. The most basic one is using a plain Javascript object that caches previous calculations. The arguments are stored as keys, while the outputs as values.

To get the memoized version of the `lucas` function we do:

I’ve added two console logs to check when we use the cache or when we actually calculate the value.

Memoization works, almost, beautifully, right? I say almost because we can implement a little enhancement capable of improving the performance even further. The lucas function is recursive, meaning it calls itself multiple times. However, the calls in the body of the function are not memoized. This means that all the subsequent calls will not have access to the cache. Let’s fix it right away!

Indeed, we now have a common cache that is accessed by all subsequent calls.

We must remember that performance gains come at the expense of memory. While more caches result in faster calculations, they also increase memory consumption.

What about memoization on more real-world applications?

Memoization with plain JavaScript objects is helpful for performance optimization. However, in web development, calling a function with the same arguments multiple times doesn’t always guarantee the same output. The reason is simple: the world we live in changes.

For instance, consider building a weather app that shows the current temperature based on the user’s location. We make an HTTP call to an API, providing the latitude and longitude coordinates, and receive a number in response indicating the temperature in Celsius.

It’s clear that calling the API for the same location can return different results, as the temperature changes over time.

So, can the function that fetches temperatures be memoized? In many cases it surely can!

We can memoize the API calls by considering how frequently the data changes and how often we want to reflect updates. For example, the API documentation might specify the intervals at which temperature data is updated. Alternatively, as system architects, we might set a manual update interval (e.g., every minute or every 5 minutes) to prevent excessive API calls from users. The appropriate approach depends on the specific requirements of the system.

Memoization with Time-to-Live Features

To implement time-dependent memoization, we need to enhance the key-value pair caching with a time-to-live (TTL) factor. For example, if the app data is updated every minute, the TTL should be set to 60 seconds. This means that cached results will be recalculated after 60 seconds.

I’ve added the TTL factor to the previous implementation of memoization:

Now that we have the actual memoization function, we can use it in our Svelte application. In the `lib` folder of our application I’ve created a `memoize.ts` file containing the TTL version.

Suppose we need to fetch the temperature data when loading the `/` route. Then, we need to do the following:

For the sake of simplicity, the `getCurrentTemperature` function returns dummy temperature data and the TTL was set to 10 seconds.

To better understand the improvement, note the 2-second delay when navigating to the / route. The temperature response takes 2 seconds to load initially, but subsequent loads are immediate due to memoization. After 10 seconds, the response will be reloaded again.

The full project can be found here.

Keep in mind that this example involves a minimal response payload. In a more practical scenario, where an endpoint returns extensive data — such as historical or forecasting information for a large dashboard with numerous charts — memoizing the API call function can significantly reduce both computation and client-to-server traffic. Additionally, if the API incurs costs, memoization can help decrease expenses by minimizing usage and staying within the quota.

Conclusion

In this article, we’ve explored how memoization combined with Time-to-Live (TTL) caching can significantly enhance the performance of a Svelte application. By caching the results of costly function calls and managing their lifespan, we can reduce redundant computations and network requests, leading to a more efficient and responsive application.

Memoization not only optimizes performance but also simplifies data management by keeping frequently accessed data readily available. Implementing memoization with TTL helps strike a balance between performance gains and memory usage, ensuring that stale data is refreshed appropriately.

Through practical examples, we’ve illustrated how to apply these concepts in a Svelte app, highlighting the tangible benefits of this approach. As you build and scale your Svelte applications, consider incorporating memoization with TTL to achieve faster load times and a smoother user experience, ultimately enhancing the overall quality of your software.

Boost Svelte/SvelteKit Performance with Memoization Techniques was originally published in fromScratch Studio on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a previous article, we’ve gone through the process of creating a model capable of predicting the demand of energy consumption for the city of London. Essentially, it was a time series forecasting problem where we had utilized the London Energy Dataset and the London Weather Dataset to build Ensemble models, such as XGBoost and LGBM, in order to accurately estimate the future needs of electric power. At first, we approached the task by modeling its time-dependent properties. Then, we added auxiliary information by incorporating weather data which improved the results by a significant margin. In this article, we will use the so-called lag features to boost the model performance even further.

The current article will use the previous one as a basis from which we will improve our models. I strongly advise you to meticulously read it before delving into this one, since we will be using its entire codebase.

Part 1: Time Series Forecasting with XGBoost and LightGBM: Predicting Energy Consumption

Part 3: Enhancing Time Series Forecasting with XGBoost: Incorporating Rolling Statistics and Moving Averages

Lag Features

As it is already mentioned, we’ve tackled the problem by exploring its time-dependent features. Even though these are generally considered to be the most influential in a time series scenario, there are additional properties that can be used to better model the task, like its serially dependent properties. To make use of these, one has to integrate past values of the target variable as input features. These past values are what we call lag features. For instance, in our problem we can use the demand of the previous day as an input feature while estimating the demand of the current day. Of course, we can include as many past days as we want where each will be considered as a separate lag feature.

Lag features are extremely useful for capturing cycles. In a time series, as cycles we define growths and decays in the target value that are not related to time, but mainly to previous target values. These fluctuations are not seasonal and their frequencies vary.

To take into account such cycles we need to use the lag features. To visualize the serial dependence we can use lag plots. One of the most popular lag plots is the autocorrelation plot, which showcases the correlation between the target and one specific lag.

To plot these we would need the following function:

So, let’s use the `df_avg_consumption` dataframe from the previous article to create the lag plots for 12 lags:

It is more than clear that there is a relationship, mostly linear, between the target and its lags, where the correlation drops by approximately 1–2% when moving from one lag to another. These findings indicate that we would have some gains in the model performance if we use some lag features.

Choosing Lag Features

So, the question is how do we know which lag features to include? In our scenario, should we choose 1 lag, 2 or 100? Most probably the previous day will play some role in today’s forecast, but what about the results of a week ago? Also, our intuition says that most likely lag 1 will contain some of the information contained in lag 2, while lag 2 will contain some of the information contained in lag 3 and so on. This means that we can end up having redundant lags even though their correlation with the target value might be high enough.

To properly choose our lags we need to check the partial autocorrelation plot. In a sense, partial autocorrelation plots outline the amount of “new” information coming from a lag by taking into account all the previous ones too. By doing so, we are able to determine which features are indeed useful and which hold information we already have.

We can plot these with the help of the `plot_pacf` function of the statsmodels Python package:

We can clearly see that the first 9 lags possibly contain valuable information since they’re out of the bluish area. This area is related to the confidence interval that was chosen in the `plot_pacf` function, with the default value being 95%.

Experiments

Before moving on to the experiments, let’s quickly remember what’s our task. As described in the previous article, we want to forecast the energy consumption from August of 2013 to March of 2014 by training on data from November of 2011 to July of 2013.

With that being said, we’re going to run some experiments to check if the lag features actually improve our models, as well as if having more than 9 lags can lead to even more performant models.

We have run the following 3 experiments for each model:

• Using 4 lag features
• Using 9 lag features (partial autocorrelation plot marks this one as the best choice)
• Using 12 lag features

Of course, our baseline models will be the best models of the previous article. Their results are shown below:

To train the models we will use the codebase of the previous article with the only enhancement being the insertion of the lag features:

It should be noted that after adding the new lag features, the initial samples will contain NaN values. Generally speaking, the missing values should be filled-in somehow, but since we use XGBoost and LGBM we can leave them as is and our algorithms will take care of them.

The results of the 3 new experiments with XGBoost are:

It is fairly obvious that even the first experiment with only 4 lags significantly improves the model performance. The MAE is reduced by almost 35%, the MSE by 30.5% while the MAPE dropped from 16% to 13.4%.

In the second experiment, the results are even better. Incorporating 9 lags decreases the MAE by 35.7%, the MSE by 32% and, finally, the MAPE falls to 13.2%.

When utilizing 12 lags the model still performs better compared to the baseline, but the performance is worse than the previous 2 experiments. This is an indicator that extra lags not only fail to improve our results, but they might “confuse” the model leading to poorer performance.

Looking at the LGBM results, lags do improve the model too. However, the model reacts a bit differently when we add new lags compared to XGBoost. More specifically, it performs better when 4 lags are included with the performance dropping slightly when adding 9 or 12 lags.

In general terms, even though the partial autocorrelation plots provide decent information, one should play around with the different lag configurations before arriving to any concrete conclusion regarding which lags to choose.

Conclusion

Time series can have serially dependent properties too, such as cycles. One way of exposing these to our model is through the lag features. To properly choose the number of lag features one has to utilize lag plots, like autocorrelation and partial autocorrelation plots, as well as conducting extensive experiments. In our energy consumption forecasting scenario, using lag features improved our models remarkably, reducing MAE by 35% and MSE by 32%.

WRITER at MLearning.ai // Control AI Video 🗿/imagine AI 3D Models

Mlearning.ai Submission Suggestions

Managing data workflows isn’t something new. We’ve been doing it for years by utilizing various tools and platforms. There are lots of different approaches to follow depending on the scenario you’re working on. One of them is Azure functions. In this article we’re going to check what is an Azure function and how we can employ it to create a basic extract, transform and load (ETL) pipeline with minimal code.

If you want to check another approach on ETL, make sure to read this: Efficient Data Labeling: Leveraging Label Studio, GitLab CI, and Python in Your Pipeline

Extract, transform and Load

Before we begin, let’s shed some light on what an ETL pipeline essentially is. As it has previously been pointed out, ETL stands for extract, transform and load, and refers to the process that extracts data from a source point, transforms the data in some way and loads the output data to a target point. The source and target points can be of any storage service, for instance an Azure Blob Storage container, an AWS S3 bucket or a database system to name a few.

There are mainly 2 different types of ETL: batch and streaming. A batch ETL works under a predefined schedule in which the data are processed at specific points in time. On the other hand, a streaming ETL is executed quite frequently as new data arrives. For example, Internet of Things (IoT) devices broadcast data in a continuous manner, so in order to be able to monitor them we would need a streaming ETL.

Except ETL, there are other similar processes such as EL and ELT. EL stands for extract and load, and its primary goal is to just move the data from one place to another where the destination is usually a Data Warehouse or a Data Lake. ELT stands for extract, load and transform. The most fundamental difference between ELT and ETL is that the former first loads the data into the target storage and, then, processes them. We won’t delve into more details on the different pipelines in this article. However, if you want to learn more let me know in the comments.

Azure Functions

As Microsoft states, Azure functions are serverless compute services allowing users to run code without the need of managing your own infrastructure. An Azure function contains code written in a programming language, for instance Python, which is triggered on demand. Azure supports multiple triggers such as HTTP requests and data storage changes. You can think of them as relatively small units of work, where each does something specific. Their cost depends on the resources consumed during execution following the pay-as-you go pricing model.

Task

Our task here is to create an ETL pipeline that will be triggered each time a new JSON file appears. The file will contain user transactions consisting of 3 columns: UserId, Amount and Date. The pipeline must read the data, aggregate the amounts per user and, finally, load the output data to another storage unit. The initial file will be provided by, let’s say, the web development team of our company in an Azure Blob storage container. We, the data engineering team, will have to extract the data, preprocess them and, finally, load the data into some other Blob storage container.

Project Setup

The first step to setting up our project is to have a Microsoft Azure account. If you don’t have one, by creating a new account you’ll get $200 in credits for free, which are more than enough for the purposes of our pipeline. Then, we will need to create a new resource group, where all of our resources will reside. After this, create a new storage account with 2 containers. I’ve named the first container `webdev` and the second one `dataengineering`. In the `webdev` container the web development team will upload the JSON files, while in the `dataengineering` container we will store the preprocessed files.

Now that we have a basic infrastructure, let’s create our Azure function workspace. The easiest way to do this is to use the Azure extension in VSCode. You’ll need to login to your Azure account, hit the `Create function` button and follow the wizard. In this article we are going to use the Python v2 programming model with a blob storage trigger.

The project structure has 3 primary files:

• function_app.py: Contains the core implementation of the Azure function
• local.settings.json: Contains environment variables such as the Azure Blob Storage connection string
• requirements.txt: Contains the dependencies of our application

For our scenario, we need the azure-functions package in order to build our function, the azure-storage-blob to communicate with our blob storage and the pandas package to manipulate our data.

My local.settings.json has a property called AzureWebJobsStorage which is essentially the connection string to my storage account. You should put there your own connection string:

Implementation

Azure functions make it very easy for us to implement relatively complex data orchestrations with just a few lines of code, without the overhead of managing our own infrastructure. And just to prove you how easy and clean it is, take a look at the function_app.py file:

The blob_trigger decorator indicates that our function will be triggered by our webdev container which is located in the storage account that uses the AzureWebJobsStorage connection string. Our function receives the blob file and the very first operation we have to do is to convert it to a pandas dataframe in line 15. Then we preprocess our data in line 19 and create a unique file for our new dataset in lines 22–23. Eventually, we save our dataframe to the dataengineering container of our storage service. To keep our function clean and tidy, we’ve moved the preprocess flow and the instructions that save our dataframe, into a separate file called utilities.py.

The preprocess function is self-explanatory. It utilizes the UserId column to group the rows and then calculates the total amount of each user’s transactions. It is important to reset the index in order to include the UserId as a column in the output dataframe. The save_dataframe_to_blob converts the dataframe to a CSV file (line 21), creates a connection to our dataengineering container (lines 24–30) and, lastly, uploads the file (line 31).

We can run the entire flow in debug mode in VSCode just like we would do in any environment without needing to register an actual Azure function in our Azure account. After you hit the Run button, try to manually upload the following file into the webdev container via the Azure portal:

If everything runs smoothly, you’ll see the logging of the initial dataframe size in the VSCode terminal and, of course, a new file in the dataengineering container with a name similar to the one I got: transactions_1688807481_preprocessed.csv.

Conclusion

And that’s basically it! These minimal functions will not only get our job done, but additionally they can support more sophisticated data operations. Furthermore, Azure functions can be used in an Azure Data Factory pipeline which extends by much what can be accomplished in terms of data workflows.

Feel free to play around with the current implementation and let me know in the comments if you’d like to see a more advanced pipeline built in an Azure Data Factory.

BECOME a WRITER at MLearning.ai // invisible ML // 800+ AI tools

Mlearning.ai Submission Suggestions

Decoding the Value of Noise: Integrating Noise Pollution Data into Real Estate Pricing with Openhouse

Openhouse (openhouse.gr) is a real estate platform which is currently under release by fromScratch Studio. Its distinguishing feature is based on a very intuitive, yet usually forgotten, fact that buying or renting a property essentially means buying or renting its neighborhood too. In today’s urban environment, properties, either commercial or residential, are far from being isolated. In a general sense, the surrounding environment affects the value of its properties directly or indirectly. For instance, you can think of a case where one house with X characteristics is located in the city center while another one with the same X characteristics is located somewhere in the countryside. Even though they might be structurally identical, their price evaluation would be radically different. This might seem as an extreme example, but we observe pretty similar cases in properties within the same city.

It should be clear by now how crucial it is not only to pinpoint the factors that influence the value of a property, but additionally quantify somehow the extent of that influence. Of course to do so, we need human expertise in areas such as urban planning, engineering, and software development. However, without the needed data there is not much to accomplish. That is why at fromScratch Studio when we say that we love data, we mean it. We think of data as a very valuable asset that if treated properly, it can have a tremendous impact on the end users. And in our case, treating data properly can potentially help someone somewhere find the house of his life.

In September of 2022, we’ve decided to put some effort into investigating how noise pollution acts on property prices. We already had a fairly large database of properties in the city of Thessaloniki Greece, where the platform is mainly active, and we needed the actual noise data. Needless to say that open datasets were nowhere to be found. We weren’t able to find even incomplete or partial datasets. If we think about it, this should be expected to some extent. Measuring noise pollution is far from trivial, since it requires specialized equipment and trained personnel. We can’t just scrap the public web for this.

Eventually, after some days of relentlessly searching the web, we were fortunate enough to find some official studies conducted by the Hellenic Ministry of Environment and Energy. These studies were focused in three different municipalities of Thessaloniki providing averages for both day and night noise pollution in highly detailed heatmaps. In some municipalities, they went a bit further by distinguishing the noise based on its source (road network, flying paths, etc.). The excitement didn’t last too long. It gradually started falling since we couldn’t find the actual noise measurements which was essentially what we needed. What can we do now?

After a brainstorming session, the engineering team changed the question from “what can we do” to “what can we do with these heatmaps?”. And that’s when we decided to embark on a difficult journey of extracting the noise from those heatmaps. It was an ambitious plan, but the idea of having such data at last, made it look beautiful. The most fundamental point we had to realize was the fact that we simply cannot extract the noise without losing information. Luckily, in our use case we don’t require extreme precision when it comes to noise. We care about how noise is actually perceived by humans, because that’s what matters when someone evaluates a neighborhood. So, our goal narrowed down to extracting the sense-of-noise which was fully plausible by our technical approach. As might be expected, things didn’t go entirely as planned. However, with some additional configuration and tuning of the extraction method, we ended up with a brand new sense-of-noise dataset for the area of Thessaloniki. Finally, we could start working on the fun part.

The concept was to create a machine learning model capable of predicting housing prices. In the first case, the model will be trained with the features we already had. In the second case, the model will be trained with the same features plus the new noise index. By doing so, we will be able to practically compare the two models in terms of accuracy and investigate whether noise enhances or undermines the overall performance. On top of that, we wanted to shed some light into the impact of noise, as a feature, on prices. Technically, this is known as explainability and interpretability. In plain English, explainability can tell us which features (for instance the property size) are considered to be important in the process of predicting the target value (that is the price), while interpretability tries to measure the extent and the direction by which a feature affects the target value.

Which model should we train? We spend quite some time (aka weeks) researching the literature and the latest trends. Many methods and techniques were evaluated and discussed thoroughly. Long story short, we’ve decided to use Ensemble models because they’ve proved to work really well with real estate data. The next step was to build the desired MLOps infrastructure that will host our experiments. In order to be able to fully monitor and effectively oversee the entire training process, there must be strict flows, metrics and evaluation standards. The approach we’ve adopted is outlined in this Medium article.

On the very first run of experiments, we trained different kinds of Ensemble models with different configurations on the entire region of the 3 municipalities where we had noise data. Unfortunately, the results were quite bad. The model that incorporated the noise index was performing slightly worse than the baseline. The presence of noise as a feature not only didn’t help improve the predictions, but it made things even worse! We’ve started looking into the correlation between the price per square meter and the sense-of-noise to find that, actually, there is no correlation when we take into consideration all 3 municipalities as one undivided area. And, if we think about it, this makes total sense. Each region of a city has its own values, needs, goals and purposes.

Let’s suppose we have 2 properties where one is located in the city center and the other is located in the suburbs. We expect that noise will influence the 2 properties differently, since someone who wants to buy a house in the center will be more tolerant to noise pollution than someone who’s searching for houses in the quiet suburbs. Basically, one buyer sacrifices the quietness of the suburbs to have access to all the goodies a city center provides (public services, entertainment, commerciality, etc.), while the other does the exact opposite.

While having this in mind, we have selected 3 different regions based on their distance to the city center and the number of properties we had in each. The correlation between price per square meter and noise values was much clearer. We chose to train different models for each area and the results were far better when compared to the initial runs. In all areas, the presence of the noise feature was improving the performance of the models significantly. However, the most interesting finding was not the performance gain, but the way noise acts on prices in each region. More specifically, there were regions with diametrically opposite correlations. For instance, in the city center of Thessaloniki, regions with increased noise throughout the day have more expensive properties. Again, most probably this is due to the commerciality that describes the city center. In contrast, in the suburbs as we move to areas with increased noise pollution, the prices drop dramatically.

These correlations can be depicted by the following 2 partial dependence plots. The plots outline the dependence between the target value (price) and the average day noise in decibels.

So, noise pollution is a pretty peculiar feature. Contrary to, let’s say, the size of a property which almost always causes prices to rise as it increases and prices drop when it decreases, noise is not amenable to any general interpretation. It can have fundamentally different effects on properties in different areas in the same city. The essence of the word different encapsulates characteristics that outline the actual profile of the area. Usually, these are directly correlated to its commerciality, the services it provides (public or private) and its urban planning in general. The distinctive nature of noise makes it a bit difficult to work with. To be able to fully utilize its potential in a machine learning model, urban planning experts should also participate in the process by providing useful insights.

And this concludes the beginning of a long journey. We’ve started with an ambitious plan for incorporating noise pollution data into our platform, while having no such data. After some digging and extensive research, our engineering team managed to extract the data and build machine learning models capable of not only predicting housing prices, but also explaining the role of noise in the process. This research voyage has opened up new horizons on how noise pollution can be used to create more accurate and robust machine learning models, as well as how noise affects real estate prices.

And the best part is that all this knowledge will not just stay in a paper somewhere online, but it will be used practically to help users find the property they truly desire.

Special thanks to Prof. Grigorios Tsoumakas for providing guidance throughout the entire process.

Resources:

• https://arxiv.org/abs/2302.13034

Decoding the Value of Noise: Integrating Noise Pollution Data into Real Estate Pricing with… was originally published in fromScratch Studio on Medium, where people are continuing the conversation by highlighting and responding to this story.

Developing machine learning applications is a task that encapsulates many distinct steps. It starts by framing the problem, collecting, cleaning and analyzing the required data and then we move to the part where we build and deploy the model. From experience, the latter steps account for approximately 10–20% of the overall time and effort we put into the process. Of course, these actions are not strictly sequential, because usually, if not always, after we run some experiments we tend to play around with the data in order to achieve better performance. Additionally, we may alter the experiments themselves, for example try different model parameters or even other methods for tuning the hyperparameters. And this cycle goes on and on until we reach the desired results.

Tracking the experiments does not only urge from the need for structurally better model building, but it potentially affects the entire product where the machine learning application lives. For instance, think of a scenario where the CMO of your company for the period of summer 2023 wants to use the exact same model that we’ve used during summer 2022. And by “same” we mean the same model in terms of parameters and the exact same training data. This is a business decision that we, as engineers, must pull it off. If the experiments were tracked appropriately, re-launching the model should be a matter of minutes. In the worst case, we would have to re-train the model, but we would know for sure that we have the correct settings. If the experiments were not tracked, then… good luck.

It should be clear by now that this cycle can end up being quite messy. We train a bunch of machine learning models with dissimilar configurations and data that we compare through error metrics to find out the one that suits us the best. The question is: how can we efficiently manage this flow? Is there a way or a method or a tool that can help bring order to this chaotic situation? Luckily, the answer is “Yes” and the methodology is called Machine Learning Operations or MLOps in short. You have probably heard the term DevOps in conventional software development. Data Scientists and Machine Learning engineers in an act of “jealousy” adopted the concept and changed the term to MLOps.

The main goal of MLOps is to meticulously track the process of training and deploying a model into production. In this article, I’m going to focus on how to track models during training. A lot of people start their experiments in a Jupyter notebook which is really helpful when it comes to testing some configurations or confirming some assumptions. However, when it comes to running full scale experiments the need for a more structured approach arises. While working on different ML research projects and ideas I realized that in every project I pretty much reinvented the wheel in regards to MLOps. Time after time, I was creating a similar boilerplate in order to properly track my experiments. That being said, during my last project I decided to isolate this boilerplate and change it a bit to make it more generic. My goal was to construct a reusable and extensible scaffold that could be used by anyone that wants to monitor the data and the models during training.

Data Version Control and MLFlow are the two main pillars of the solution. DVC is a very powerful and practical open-source tool which helps with versioning huge datasets. It is built on top of git, and it works in a very similar way. DVC uses external storage such as Azure blob storage, Amazon’s S3, Google cloud storage or even a basic google drive folder in order to store the version history of large data. MLFlow is, also, an open-source platform that manages entire machine learning workflows, from tracking experiments and code packaging with reproducible runs, all the way to sharing and deploying models.

Essentially, I’ve grouped these two alongside other useful libraries such as Sklearn in order to build a really flexible workbench capable of hosting fully trackable experiments. The entire solution is available on Github where I outline its structure and use as well as how it can be extended. For the rest of this article, I’ll go through a slightly more comprehensive analysis of the scaffold.

To better understand the flow let’s say that we’re working with the well-known diabetes toy dataset where we want to predict how the disease progresses. We are dealing with a regression problem, since our target is a continuous variable. For the sake of this walkthrough, we will choose to use a decision tree which is a pretty basic regressor. It goes without saying that you’re free to choose any model from the Sklearn library or, in general, any model that can be utilized by the Sklearn API. So, for our decision tree we will need to create a very primitive script:

The script consists of 3 distinct phases: the initialization of the model, the parameters’ setting and the `run_experiment` call. The initialization should be straightforward to all Sklearn users. The parameters are set in a way to be fed into a grid search process that will handle the hyperparameter tuning. Besides grid search, we support random search and Bayesian optimization. For random search the parameters are set in exactly the same way. However, for Bayesian optimization they are configured a bit differently since we use the scikit-optimize library:

Congratulations! You are now ready to run any experiment you like. Before doing so, make sure to save your dataset in the `data` folder which is fully tracked by DVC and run `pip install .` to install the required dependencies (I strongly recommend creating a virtual environment). The experiments can be started via command line. For instance, to run an experiment named “Test” with grid search optimization we can do:

py .\decision_trees_experiment.py --data-file "./data/diabetes.csv" --mode "grid_search"  –target “Y” --experiment-name "Test"

When the experiment is complete, you will find in the `mlruns` folder a new folder named after the id of the experiment that was conducted. Each experiment will be stored in the `mlruns` folder. To visualize the results, you will have to run `mlflow ui` and then open `http://localhost:5000` in your browser. There you will be able to find every little detail about all the experiments and effectively compare them. For more details on this, make sure to check the official documentation. Additionally, in the `results` folder we store specific details for each experiment, such as error metrics, the model itself and some informative plots, like feature importance, partial dependence and permutation importance plots. Both `mlruns` and `results` folders are tracked by DVC. The results can be enhanced at will by adding the corresponding evaluation metrics in the `core/evaluation.py` script. Also, since this is a regression task all the error metrics are regression oriented, meaning that for a classification task we will need to adjust them.

To test how Bayesian optimization works we will need to create a new run in the context of the previous experiment. To do so, run:

py .\decision_trees_experiment.py --data-file "./data/diabetes.csv" --mode "bayesian"  –target “Y” --experiment-id {add_experiment_id_here}

The entire argument list is explained in the readme file. New arguments can be added through the `core/utilities.py` file. Another useful parameter is ` — scale` which dictates which scaling should be applied to the data. In the `core/scale.py` file we have predefined two different scalers: standard and min-max. To test how standard scaler affects the results run:

py .\decision_trees_experiment.py --data-file "./data/diabetes.csv" --mode "bayesian"  –target “Y” –scale “standard” --experiment-id {add_experiment_id_here}

New scalers or new ways of applying scaling can be added in the `scale.py` script.

Last but not least, it’s really important to push the results on your repository. To do so, make sure you have installed DVC on your machine and then follow the instructions to set up DVC remote storage. Then you will be able to use `dvc add` and `dvc push` to push the results and allow your collaborators to see your progress.

And that’s pretty much it! Feel free to fork the repository to use it as is or change any part you like to make it fit your needs better. If you think that I’ve missed something in the flow, please open a PR and propose your solution/fix/enhancement. I would be glad to merge it.

Github repository: https://github.com/gkamtzir/mlops-scaffold

BECOME a WRITER at MLearning.ai

Mlearning.ai Submission Suggestions