As Trendyol’s Platform — Extensibility team, we set out to solve a clear problem:

Give every team at Trendyol the ability to automate with AI, without needing to be experts or have a technical background.

Like many teams, we embraced the rise of Large Language Models (LLMs) and, a year ago, built a Retrieval-Augmented Generation (RAG)-based chatbot. By leveraging company document sources, our chatbot provided answers on Slack, minimizing context switching.

However, its success led to an influx of feature requests, overwhelming our ability to keep up.

Recognizing the need for a more scalable solution, we envisioned a platform where anyone, regardless of technical expertise, could build applications using LLMs or by integrating internal Trendyol tools.

This vision led to the birth of Trendyol AI Workflows 🚀

A low-code platform that democratizes automations with AI

Enabling teams across the company to create powerful, context-aware tools tailored to their needs.

Let’s take a closer look at Trendyol AI Workflows:

1. Getting Started: A simple path for users to jump in and create their first workflow
2. Model Garden and Activities: How we offer flexibility with multiple LLMs and internal tools
3. Architecture Deep Dive: The engine that powers Trendyol AI Workflows
4. How Teams Are Using It? Use cases and the value it’s delivering across Trendyol

Getting Started

Jump in and create the first workflow 🏃‍♀

Trendyol AI Workflows makes building LLM-powered apps as easy as dragging and dropping. With its intuitive, no-code UI, anyone — from engineers to business teams — can spin up a custom automation tool in minutes.

The core building blocks of a workflow are called activities. Think of them as modular tasks that you can connect to shape your app’s logic. Unsurprisingly, over 90% of our workflows are anchored around the LLM activity, which serves as the brain behind the operation.

As a result of a workflow, you get an API endpoint where you can freely integrate further into your automations, businesses, and apps.

Next, let’s explore how we expanded the building blocks and LLM capabilities to meet automation requests and growing demands.

Activities and Model Garden

Flexibility at your fingertips 👆

We started small — with just the essentials: Input, Output, and the AI activity. But as usage grew and teams got more creative, so did our toolbox. We introduced powerful additions like HTTP Request, Slack, BigQuery, and conditional logic with the If activity.

Trendyol has its own technology ecosystem, so we built activities tailored to our internal needs. For example, we integrated our Stack Overflow-like Q&A platform into the Knowledge Base activity, making domain-specific answers instantly accessible.

Security was non-negotiable from day one. Almost every activity, from Slack to HTTP Request, needs secrets to execute. So we created a dedicated Secret activity. It securely pulls tokens from a backend Vault, abstracting away the risk and hassle for users.

On the LLM front, we embraced the best of both worlds. The AI activity supports models from OpenAI, Google, and Anthropic, while also powering up workflows with Trendyol’s self-hosted models — because sometimes, you want full control at lower cost.

But we know, choosing the right model can be overwhelming, especially for non-AI folks. That’s why we built the Model Garden — a curated space where users can browse models, understand their strengths, and pick the best fit for their use case.

The real power of Trendyol AI Workflows lies in the model portfolio and flexibility to utilize them 💪 You can mix and match any combination of activities, LLMs, and automate with workflows to build scalable, production-grade services.

Curious how it scales under the hood? Hold that thought — we’ll break it down in the next section.

Architecture Deep Dive

The engine that powers AI Workflows 🤖

Trendyol AI Workflows is built on Temporal, an open-source orchestration engine that ensures reliable, durable, and scalable execution.

When a workflow is triggered from the TY AI Workflows UI, the TY AI API acts as the starter. It generates a DSL and sends a gRPC call to Temporal to initiate the workflow.

Temporal places the request into a task queue, where dedicated workers pick it up and execute the defined activities — from calling LLMs and posting to Slack, to querying BigQuery or internal tools like our knowledge base.

Durability and observability are built-in: Temporal handles retries, state management, and offers a UI to trace workflow execution.

The system integrates with both internal and open-source components — LiteLLM for LLM proxying, Onyx for document indexing, and Vespa as Vector DB for fast RAG-style retrieval.

How Teams Are Using It?

Use cases and the value it’s delivering across Trendyol ⭐️

Product managers are spinning up AI-based tools without writing a line of code. Developers are diagnosing production issues by analyzing pod logs, Prometheus metrics, and Cloudflare traces — all in seconds, thanks to automated workflows.

Our international sellers are generating thousands of product entries at scale, complete with attributes, tags, and descriptions, slashing manual effort to near zero.

We’ve automated meal claim processing by combining image analysis with claim reasoning, reducing call center workload by 80%.

More than 100 teams have already been onboarded, and we’re just getting started. With new capabilities like cron jobs and loop support on the way, the possibilities keep expanding.

The future of AI-driven automation is here — and it’s being built, used, and scaled daily at Trendyol.

Join Us

Do you want to be a part of our growing company? We’re hiring! Check out our open positions from the links below.

Home - Trendyol Careers

Trendyol AI Workflows 🚀 was originally published in Trendyol Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Trendyol, we operate as one team. This is only possible through open and direct communication in a hybrid working setup. So, Slack is our indispensable tool for day-to-day work. We receive and resolve technical issues, clarify HR-related questions, and onboard new joiners through Slack.

We also strive to be better every day. Not to brag but we achieve better constantly with our Trendyol Ways of Working (WoW) 😎

Adhering to certain standards in our communication and daily tools improves our efficiency. Moreover, it helps us to reduce the cognitive load and context switching during the day so that everyone can focus on the important — creating the business impact and shipping the features.

Bringing those together at the rise of the Large Language Model (LLM) era, Marvin was born:

Our AI Assistant to automate internal know-how search and processes

Let’s dive deep into Marvin’s capabilities and underlying architecture. I promise to keep it not too technical 🧡

1. Capabilities
2. Architecture and LLM models

Capabilities

Marvin is an LLM-based tool, aiming to automate Trendyol teams’ daily lives. The core capabilities involve replying to technical and HR-related questions.

We have an internal technology portal called Pandora. We use Pandora to accumulate technical know-how, measure developer productivity, and provide visibility of developer teams’ work.

We brought Pandora's know-how one step closer to the developers with Marvin. Now, they can find answers to their technical questions and resolve bugs with Marvin on Slack channels. We used Mixtral 8x7b model and Pandora documents as a source here.

Moreover, if the developer is unsatisfied with the answer, Marvin supports the user with links, allowing developers to search further in Pandora. All in all, our developer teams can find solutions to bugs and technical answers with less effort, with Marvin.

After observing the positive feedback from developers, Marvin started to serve HR teams.

We are growing fast, and welcoming new team members every month. As expected, they are warming up with the team and processes by asking questions during onboarding. Our HR teams are supporting them so that they adapt Trendyol WoW, culture, and internal processes faster. On average, HR teams spend around 35% of their time each month supporting such questions.

We automated this process with Marvin where anyone can find answers to their HR-related questions through a Slack bot, faster. We used our internal HR documents as a source here.

Now, let’s look into the inner workings of Marvin 🧐

Architecture and LLM models

Retrieval Augmented Generation (RAG), Prompt Engineering, and LLMs are at the core of Marvin. We evaluate and store our RAG metrics at Langfuse.

These components enabled us to use a pre-trained LLM with our internal documents. With prompts, we enabled Marvin to know the context it is in. A detailed architecture with the complete list of components is here:

We used Chroma DB as our vector DB. Every night, we ingest documents and store them as embeddings. You can think of embedding as a mathematical representation of the words as a vector and a Vector DB solution designed to efficiently store embeddings.

When LLM receives a question, it provides the answer by finding and retrieving the most similar embedding from Vector DB.

We selected the open-source Mixtral-8x7B model because it outperformed other LLMs in the global benchmarks. This is not set in stone though, LLM space is advancing wildly, and we are following up with the recent models and advancements.

Also, we track cost-related and conversational metrics. We carefully monitor answer relevancy, context precision, context relevancy, and faithfulness at Langfuse.

As the last component, we used Slack adapters for the human-Marvin interaction. This helped us to onboard the team easily to Marvin because Slack is our best friend at Trendyol 👩🏻‍🤝‍👨🏼

Marvin is growing everyday and 33 teams are using it. We saved 57 hours of our HR teams per month and are receiving positive feedback from our users.

Our user base is growing every day! So we are currently adding more features. Marvin is improving to support more teams at Trendyol, at scale.

Stay tuned to learn more about Marvin, and how we automated the technical assignment review within the recruitment process.

Join Us

Do you want to be a part of our growing company? We’re hiring! Check out our open positions from the links below.

https://careers.trendyol.com/

Meet Marvin: Trendyol’s Helpful AI Assistant was originally published in Trendyol Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

To navigate challenges, with tools

I was back in the job search market in August 2023. Despite the challenging status quo and massive layoffs from big tech companies, I took a leap of faith and left my company to dedicate full-time to finding my next role in the Machine Learning/Data Science area.

Naïvely, I estimated that I would be landing a dream role in a few months. The reality turned out to be longer than this. So, this process has been a marathon rather than a sprint for me. Luckily, I finished the marathon this month 🎉

Now, I want to share the precious rewards — my learnings — to motivate job seekers and let them know they are not alone in this marathon full of ups and downs.

Here are the tips I followed and learned while running this marathon:

1. Define your criteria for your next role
2. Rejections will be more than successful responses
3. Don’t spray and pray
4. Use AI-based recruiting tools

P.S. There is a bonus tip at the end 😉

Disclaimer: Dedicating full-time to a job search requires careful budget planning and having sufficient financial resources. It is perhaps the most significant consideration I experienced during my marathon. However, I designed this blog post to focus on the navigating the challenges and providing the tools to support you in your interview, CV and Cover Letter preparation.

Define your criteria for your next role

As you gather more work experience, you become more conscious of which work environments you thrive in. In addition to the role description, the company’s culture and values, the work setup (remote/on-site/hybrid), and the impact/results you create are important factors that add to your success.

So ask yourself those key questions:

• Are you more productive in a remote/hybrid/in-office setup?
• What kind of leadership and management style is for you?
• What are the cultural elements that you look for? What values resonate with you the most?
• What motivates you day-to-day? Do you want your work to make a positive impact every day?

These questions will help you solidify your ideal role and company. You can get a gist of these topics by asking the right questions in the interviews, especially for the cultural values.

I was preparing my interview questions with the help of keyvalues.com. It is a great website for designing the right questions for the interviews to get a gist of the company's cultural values, leadership style, and team’s daily routines.

Rejections will be more than successful responses

Prepare yourself to receive more rejections than successful responses. This diagram that went viral on LinkedIn is proof of this statement. It shows the application statistics of a job seeker in marketing tech. From my experience, I can tell it is a valid statistic for the data science/machine learning domain.

I got 23 successful responses and 5 offers out of 100 applications. The rest was either ghosting or rejections. At first, I was demotivated and felt down with every rejection I received. I even cried after some rejections where I felt a high culture fit with the company and a strong match with my skills 🥲. After several downs, I learned to go through the disappointment of the rejection and not to couple it with my self-worth.

Also, a rejection applies to a particular role and time. The same position might open again in the future and requirements might change. You likely improved yourself in the meantime.

For example, I was interviewing with two companies in November 2023 for the MLOps Engineer and Data Scientist roles. I was eliminated from the process after the first interview. Fast forward to February 2024, I got offers from both companies for a Machine Learning Engineer role. So never give up and keep trying ✌🏼

Don’t spray and pray

“I was applying for as much as 50 jobs a day in two different career tracks”.

…said the creator of the above image.

It is almost impossible to tailor a résumé and make 50 applications in a day. This is an example of the spray-and-pray approach. It is the traditional way of submitting a generic résumé for every job posting through the company’s careers page.

In today’s competitive job search market, only quality applications pass the first screening. To make a high-quality application, you have to tailor your résumé and cover letter for each application.

In your CV, order your skills so that the matching skills with the job description appears first, mention your qualities that the job description lists, and make sure you use active verbs.

A customized cover letter is also necessary to stand out in the application process. Most companies document their recruitment process, values, and qualities they look for in candidates. Also, learn about the products & problems they are working on before crafting your cover letter. Then embed those learnings and values in your cover letter and ensure to highlight the technical skills listed in the role requirements.

Most importantly, use your network to pass the initial screening, and don’t hesitate to reach out to your network (ex-colleagues, college friends, etc.). Ask if they want to be a reference for you. You will be surprised to see that people are there to support you. I believe this support is coming from an empathy space where every one of us has been a job seeker in life once 😌

Moreover, most companies have an employee referral system so it is a win-win situation for both of you if the process ends successfully.

Use AI-based recruiting tools

Around September, I realized spraying and praying was not working for me. That is where I learned about AI-based tools to boost my CV and cover letter preparation process.

I chose Teal from various tools listed in this post on LinkedIn. I used the Resume Builder feature to build a tailored resume for each application.

Also, the Matching section helped me to quantify the match between the job posting and my CV. It calculates a percentage based on the keywords in your CV and job posting. While tailoring your CV, you can aim for a good match score to help you pass the initial screening. Moreover, Teal suggests improvements for your CV in terms of wording and framing your achievements.

AI-based tools are still our assistants, they are still not there to replace humans in creative writing. I see cover letters and CVs as a form of creative writing in a professional manner. So don’t solely rely on a cover letter or CV prepared by an AI-based tool. Always evaluate their output, and use their suggestions as feedback.

Bonus Tip: The job market is more vibrant in the first quarter of the year because of new budgets and people’s tendency to change jobs due to salary adjustments made at the beginning of the year. This is a personal observation and a hint from many years of HR-experienced friend working at Amazon 😉

Thank you for reading!

After six months, I completed this marathon with a role that satisfies most of my criteria. It was quite a challenging journey for me with ups and downs.

I benefitted so much from the tips above so I cannot keep them to myself. I hope my learnings will light your path and support your job search 🤗

For comments or constructive feedback, you can reach out to me on responses, X (Twitter), or LinkedIn!

4 Tips From My Job Search Marathon was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

I changed my career from SAP Consultant to Data Scientist roughly 4 years ago. After following a curriculum I designed for myself, I secured my Machine Learning Engineer role within a startup ecosystem in a year.

It isn’t easy to condense all that I’ve learned, the tools I’ve used, and the experiences I’ve had during these past four years into a single article. Nonetheless, I’ll highlight the ones that have particularly benefited me.

As I progressed in my role, I cultivated software development skills by using a variety of tools and following certain best practices on the job:

1. Git and Version Control
2. Writing Readable and Clean Code
3. Exploring Different Development Tools

I will not only explain them in this article but also mention how they helped me improve my soft skills and productivity. Let’s get started 🚀

Git and Version Control

Git is an open-source version control system widely used in software development. It organizes projects and manages collaboration between developers working on the same project. I was not using Git when I was working solo; I was manually versioning my code and notebooks 🙃

When collaboration comes into play, Git becomes a necessity. It helps to track the progress of the project and fosters collaboration.

It is a vast topic to learn from, and great sources are out there (like this one). Today, I will focus on the “commit” term and how it helped me organize my thought process.

Git commit is like taking a snapshot of your code in time.

One of my first learnings in my early days was to have organized Git commits and concise commit messages.

Later on, I realized that thinking beforehand about your commits and how to structure them also helps you to organize your work and design your code with a better logical pattern.

Here is an example of how Git commits can be organized in the context of Data Science from one of my recent projects:

From the collaboration perspective, breaking down commits — so that each code change should have one purpose at a time — will also help your colleagues review your code faster.

Writing Readable and Clean Code

My previous mentor once told me, “Even your grandma should be able to read your code” — a disclaimer: Not from an ageist perspective but rather to imply that everyone should be able to read and understand your code easily.

Jokes aside, reflecting your thought process in your code and writing a self-explanatory code will help anyone review your work and understand it faster.

I learned how to craft readable and clean code both on the job and by reading several industry-wide books:

• Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin
• Philosophy of Software Design by John Ousterhout.

“So if you want to go fast, if you want to get done quickly, if you want your code to be easy to write, make it easy to read.”

Robert C. Martin.

and an implementation of this practice from my recent work:

Believe me, your future self will thank you for writing readable and clean code, not only your team!

Exploring Different Development Tools

When it comes to experimenting with machine learning models or prototyping solutions to test their viability, using notebooks is often the first choice.

And Jupyter notebooks are a great medium for that.

Before landing in the Machine Learning Engineer role, I primarily worked on my projects using Jupyter notebooks. As my previous team enforced using PyCharm, I met with my first integrated development environment (IDE).

I felt a bit overwhelmed by the numerous functionalities and the user interface of PyCharm, as a Data Scientist mostly using one-time notebook solutions.

In time, PyCharm has become my second-nature tool.

Its code completion and error-highlighting features have become my indispensable helpers and contributed to my productivity, a lot.

Mastering Git within IDEs has significantly improved my ability to organize work, boosted my coding speed, and enabled me to focus on the task at hand more efficiently. Also, I’m always grateful to myself for writing clean code when I revisit it months later 😉

Thank you for reading! This blog post holds a special place for me. It signifies my return to blogging and technical writing 🤗

For comments or constructive feedback, you can reach out to me on responses, Twitter, or Linkedin!

A Machine Learning Engineer’s Must-Have Tools was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Automate your pipeline with Docker and Luigi

• Part 1: Understand, clean, explore, process data
• Part 2: Set metric and baseline, select and tune model
• Part 3: Train, evaluate and interpret model
• Part 4: Automate your pipeline using Docker and Luigi (you are reading now)

Disclaimer: This article series is not a tutorial about Docker and Luigi. It is the last article of an article series “Building an Automated Machine Learning Pipeline” that focuses on building end-to-end ML pipeline and showing how to automate it using certain elements of both tools. This article will make more sense to you if you read the previous ones from the links above.

In this article series, we set our course to build a 9-step machine learning (ML) pipeline and automate it using Docker and Luigi.

1. Understand & Clean & Format Data
2. Exploratory Data Analysis
3. Feature Engineering & Pre-processing
4. Set Evaluation Metric & Establish Baseline
5. Select an ML Model based on the Evaluation Metric
6. Perform Hyperparameter Tuning on the Selected Model
7. Train and Evaluate the Model
8. Interpret Model Predictions
9. Draw Conclusions & Document Work

As a result of this pipeline, we built our ML solution and called it, the wine rating predictor because we are trying to infer the quality of wine represented with the points using a sample dataset. We defined the requirements for our wine rating predictor in the first article as:

- understandable because our audience will likely have limited knowledge of statistics and ML.

- performant because the complete production dataset could have millions of rows.

- automated in a way that it can run on any production system without requiring specialized configurations and setups.

We have so far satisfied the understandability and addressed performance — to some extent:

• We have selected an understandable model, random forest regressor, as the underlying ML algorithm of the wine rating predictor. Moreover, we have explained how random forest regressor works as well as its hyperparameters in the second and third article.
• We have built a performant model by performing hyperparameter tuning on the random forest regressor. We still have room for it though.

Today, we will wear a software engineer’s hat and address the last requirement — automation. We will take the steps below from the ML pipeline, run them on Docker Containers and connect them with Luigi Tasks. (Don’t worry we will elaborate on them throughout the article 🙂)

If you have noticed we have two additional steps because a typical real-world ML pipeline starts with getting the data from a source. Also, we included the training-test dataset split and the rest is the known steps of the ML pipeline.

To run the flow above, we will first look at Docker and Luigi and understand their use in the context of this article series. Therefore, we will start by explaining these tools and their necessary elements.

Then, we will connect the dots using these tools. Eventually, we will put the pieces together and run the flow which will accomplish our mission of automation!

1. What Is Docker and How It Is Used for the Wine Rating Predictor?
2. What Is Luigi and Why It Is Selected as an Orchestration Tool?
3. Put the Pieces of Docker and Luigi Together
4. Run and Automate the Pipeline!

You can find the GitHub Repository here:

cereniyim/Wine-Rating-Predictor-ML-Model

One more disclaimer before we start:

Disclaimer: Structure of the repository and download_data.py , util.py , docker-clean.sh and docker-compose.yml files are provided to me as the basis of the code challenge. Rest of the code is written by me.

1. What Is Docker and How It Is Used for the Wine Rating Predictor?

What is Docker?

The official definition of Docker from Wikipedia is as follows:

Docker is a set of platform as a service products that uses operating system-level virtualization to deliver software in packages called containers.

I would define it as creating virtual environments for your application/software/data science/ML projects so that it can run seamlessly on any system that has Docker.

How do we create virtual environments with Docker?

Via containers. They are isolated virtual environments to run your applications, for our case, ML solutions.

How do we create Docker Containers?

From images. They are templates that are used to create containers. They contain information about your underlying operating system, environment and libraries — with versions.

How do we create Docker Images?

By reading the instructions from a Dockerfile. It is a set of instructions and commands that a user could call on the command line to create an image. You can create a local image in your system with the docker build Dockerfile -t <image_name> command.

How images and containers are connected?

Containers are the running instances of images. When you run an image you create a container that is an isolated environment.

How did we use Docker to build Wine Rating Predictor?

We used Docker to build virtual environments and manage the dependencies of the libraries so that we can run the flow seamlessly. The automation flow below is designed so that each box runs on a separate Docker container.

I am sure you have noticed some directories and files in the repository that we haven’t mentioned yet. You can think of download_data, make_dataset, clean_data, extract_features, transform_data, impute_data, train_model and evaluate_model directories as the boundaries of the boxes in the automation flow that contains a Dockerfile and the source code as a Python file.

Before looking at how to create a Dockerfile, let’s understand the use of base_docker here.

Each step of the automation flow runs on a separate container. However, they are modules written in Python and the only difference between them are the libraries used. So, base_docker is used to specify the common environment variables and install the required Python version and the libraries. It also contributed to the performance of the wine rating predictor by installing required packages once rather than installing them every time on a separate container.

How to write a Dockerfile

When creating a Dockerfile start by the steps as if you are creating a local environment on your machine. Considering your aim is to deploy an ML solution with Python, you can build your image on top of an existing Python Image in the Docker Hub. An important point here is to select an existing Python Image that is suitable for your case since there are several versions and sizes are available.

FROM python:3.7-slim
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
ENV PYTHONPATH=/opt/orchestrator:$PYTHONPATH

COPY requirements.txt /opt/base_docker/

RUN pip install -r /opt/base_docker/requirements.txt

WORKDIR /opt/base_docker/

slim variant is chosen and Python version 3.7 is specified, same Python version as in my local system.

ENV command is used to update the PATH environment variable for the underlying operating system (Python 3.7-slim for our case) that your container installs.

Libraries with the versions mentioned in the requirements.txt are copied and installed with the pip_install. In addition to the aforementioned libraries of Python, click is also installed — more on the use of that later.

As the last instruction, the working directory of a Docker Container is defined with the WORKDIR command.

Now, our base_docker is ready, exploring any Dockerfile from the automation flow will be sufficient because all other Dockerfiles have been written with the same logic. Here is an example from clean_data:

FROM code-challenge/base-docker

COPY . /opt/clean_data/

WORKDIR /opt/clean_data/

Now our base image become the base_docker. Every file in the clean_data directory is copied and a working directory of a Docker container is created.

It is time to have a look at Spotify’s beloved Luigi!

2. What Is Luigi and Why It Is Selected as an Orchestration Tool?

What is Luigi?

The official definition from the docs is as follows:

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

It is basically an orchestration tool to stitch many tasks together in a directed acyclic graph (DAG) structure:

Why It Is Selected as an Orchestration Tool?

When you build an ML solution and as the scale grows, this can easily become complex and messy. Luigi is a robust workflow management tool that can prevent chaos and complexity.

How Luigi Works as an Orchestration Tool?

There are two fundamental building blocks of Luigi: Task and Target

What is a Luigi Task?

The steps of a workflow are tasks, usually a single unit of work, where computations are done. A Luigi workflow is built on top of tasks.

How Luigi Tasks are connected?

Each task is connected with a target. A target can be a file, a checkpoint in the workflow or any kind of output generated by the task.

We are dumping outputs generated by each task in the data_root:

In the flow, each task is dependent on the output of the previous task except the initial task download_data. Each file is passed as a Parameter so that it can be consumed by the latter step in the workflow.

When designing a workflow, Luigi recommends the atomic structure: Each task should have a single file as output like in the download_data and train_model tasks.

For the rest of the tasks, we imitate atomicity with the SUCCESS flag. When we perform computations on both training and test datasets, we create a SUCCESS flag as an output of the task, after dumping the output files in the directory. This way, the next task only checks whether a SUCCESS flag exists.

How do we define task dependencies?

Task dependencies are defined with the requires() method. Here is a task outline from Luigi documentation:

For our case, task dependencies are obvious since we defined a linear automation flow.

How do we run Luigi Workflows?

Unfortunately, Luigi does not come with a triggering mechanism. If you want to run a Luigi workflow you use the command line specifying the module name and the last task in the project directory:

luigi --module <modul_name> <task_name> 

When a workflow is triggered, Luigi checks if the output of the last step exists. If not, then it checks backwards if the outputs of the predecessor steps exist. For our case, this would be train_model, impute_data, transform_data, extract_features, clean_data, make_dataset and download_data .

If an output of a task from any step of the flow exists Luigi resumes the flow where it has left of. This is a very useful and important trait to prevent your ML pipeline from crashing when it contains partial data.

Now we know the essential elements of Luigi and Docker we can move forward with completing the picture!

3. Put the Pieces of Docker and Luigi Together

We will put the pieces of Docker and Luigi in the docker-compose.yml and orchestrator directory.

The utility functions provided by the awesome engineers of Data Revenue has enabled me to connect the pieces of Docker and Luigi. Functions and classes available in the util.py played an important role that connected Docker and Luigi for the wine rating predictor. For example, DockerTask — the object that we pass as a parameter between the tasks — is not available as a parameter in the Luigi normally.

First, let’s have a look at task.py in the orchestrator directory where I built the automation flow using utilities. Then, let’s understand the purpose of docker-compose.yml.

Define Tasks at Task.py

• Luigi Parameter: Each task starts with a LuigiParameter object where we specify the input/output files and directories of the task. For example, the last task EvaluateModel where we evaluate the ML model with the test set takes the test_features,test_target and the trained model file (model.sav) as an input LuigiParameter. Since this task creates multiple files as output (PredictionsVSActuals.png and FeatureImportances.png) we also define a SUCCESS flag to mimic atomicity.
• Image: We define which Docker Image to use to create the container.
• Dependent Task: If there are any dependent tasks, define them here with the requires() method. The dependent task of the EvaluateModel is TrainModel.
• Command: The command to be executed by the container. Here command calls the source code available in the evaluate_model.py with the relevant parameters.
• Output: The output() method returns one or more Target objects (either the output file or a SUCCESS flag)

Recall that source codes for each step of the flow is available in the respective directory as Python file. Let’s have a look at the source code of train_model :

After defining the helper functions convert_features_to_array and convert_target_to_array, we defined necessary click commands.

Click, as defined in the official documentation:

It is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.

We used Click to be able to pass the arguments to the python scripts from the command line. Finally, we train the model with the fine-tuned parameters that we have decided in the Perform Hyperparameter Tuning on the Selected Model step on the training dataset. Finally, we save the trained model in model.sav file as an output.

Use of Docker Compose

The official definition for Docker Compose from the Docker Docs is as follows:

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

This definition suits to our case since we run the automation flow on 9 containers including base_docker. You can read more about the use and commands of docker-compose here.

The important part for us here the command to run Luigi tasks is written here, so when we run containers through docker-compose, it will trigger the automation flow with the luigi — module task EvaluateModel-scheduler-host luigiid command.

4. Run and Automate the Pipeline!

This is the moment of truth! We are going to run the automation flow and automate the 9-Step ML Pipeline. It is the last of the last step, stick with it for one more minute 😉

1. Open your terminal and navigate to the project directory

For me this is:

cd GitHub/Wine-Rating-Predictor-ML-Model

2. Build Docker Containers

Instead of building multiple Docker Containers one by one, we are going to use a shell script to build Docker Containers: build-task-images.sh

./build-task-images.sh 0.1

The below message shows that Containers are built successfully. So we are good to go for triggering automation flow.

Successfully tagged code-challenge/evaluate-model:0.1

4. Trigger Automation Flow from Docker Compose:

We will trigger the workflow with

docker-compose up orchestrator

Upon writing this command, Luigi checks whether each task is complete and outputs the following information:

Checking if EvaluateModel(
    no_remove_finished=False,
    in_test_features_csv=/usr/share/data/interim/test_features.csv,
    in_test_target_csv=/usr/share/data/interim/test_target.csv,
    in_trained_model=/usr/share/data/output/model.sav,
    out_dir=/usr/share/data/output/, 
    flag=.SUCCESS_EvaluateModel) is complete

Checking if TrainModel(
   no_remove_finished=False,
   in_train_features_csv=/usr/share/data/interim/train_features.csv,
   in_train_target_csv=/usr/share/data/interim/train_target.csv,
   out_dir=/usr/share/data/output/) is complete

Checking if ImputeData(
    no_remove_finished=False,
    in_train_csv=/usr/share/data/interim/train_transformed.csv,
    in_test_csv=/usr/share/data/interim/test_transformed.csv,
    out_dir=/usr/share/data/interim/, 
    flag=.SUCCESS_ImputeData) is complete

Checking if TransformData( 
  no_remove_finished=False,
  in_train_csv=/usr/share/data/interim/train_features_extracted.csv,
  in_test_csv=/usr/share/data/interim/test_features_extracted.csv,
  out_dir=/usr/share/data/interim/, 
  flag=.SUCCESS_TransformData) is complete

Checking if ExtractFeatures(
    no_remove_finished=False,
    in_train_csv=/usr/share/data/interim/train_cleaned.csv,
    in_test_csv=/usr/share/data/interim/test_cleaned.csv,
    out_dir=/usr/share/data/interim/, 
    flag=.SUCCESS_ExtractFeatures) is complete

Checking if CleanData(
    no_remove_finished=False,
    in_train_csv=/usr/share/data/interim/train.csv,
    in_test_csv=/usr/share/data/interim/test.csv,
    out_dir=/usr/share/data/interim/, 
    flag=.SUCCESS_CleanData) is complete

Checking if MakeDatasets(
    no_remove_finished=False,
    in_csv=/usr/share/data/raw/wine_dataset.csv,
    out_dir=/usr/share/data/interim/, 
    flag=.SUCCESS_MakeDatasets) is complete

Checking if DownloadData(
    no_remove_finished=False, 
    fname=wine_dataset, 
    out_dir=/usr/share/data/raw/,
    url=https://github.com/datarevenue-berlin/code-challenge-2019/releases/download/0.1.0/dataset_sampled.csv) is complete

Then runs the task in one by one. Another important information for us is:

INFO:evaluate-model:Mean square error of the model is: 4.95

After a successful run you see a 🙂:

Bonus Points: Let’s see how Luigi resumes the flow where it has left of
Let’s say you prepared training datasets and run the automation flow, then you will build the next steps afterwards starting with the training the model. Luigi won’t repeat successfully run previous steps and only runs TrainModel and EvaluateModel tasks in that case.

Conclusions

I would say Wow!

It was quite a journey spanning over 4 articles and a month ⛵️

We started from the very beginning and analyzed every step of the 9-step ML pipeline through the first, second and third article.

• Building an Automated Machine Learning Pipeline: Part One
• Building an Automated Machine Learning Pipeline: Part Two
• Building an Automated Machine Learning Pipeline

In this last article, we introduced the relevant elements of Docker and Luigi and explained their importance and use for the wine rating predictor. We completed the last missing piece of our comprehensive pipeline and did a dry run of an ML solution — just as if it runs on production systems.

Special Thanks

I think it is time to thank my significant other Serkan Durusoy. He has been the most precious mentor, editor and study buddy I had in my data science expedition!

He helped me to overcome the ups and downs of changing a career. Also, I want to acknowledge his support to complete this article series as well as the grit he provided for me. Thank you, my dear ❤️

Thanks for reading 🙂 Please feel free to use this pipeline, the code and the repository for your own projects.

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!

Before we go, I also want to share some of the useful resources that I used to prepare for this project. See you in the next article 👋

My Suggested Resources to Learn More About Docker and Luigi:

• Scale Your Machine Learning Pipeline by Data Revenue
• Docker for Beginners: Full Course by KodeKloud
• Build a Docker Container with Your Machine Learning Model by Tina Bu
• Luigi Documentation
• Docker Documentation
• Building Data Science Pipelines With Luigi and Jupyter Notebooks by Mattia Ciollaro

Building an Automated Machine Learning Pipeline: Part Four was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building an Automated Machine Learning Pipeline: Part Three

Training and evaluating the model, interpretation of model results and final conclusions

• Part 1: Understand, clean, explore, process data
• Part 2: Set metric and baseline, select and tune model
• Part 3: Train, evaluate and interpret model (you are reading now)
• Part 4: Automate your pipeline using Docker and Luigi (live!)

In this article series, we set our course to build a 9-step machine learning (ML) pipeline and automate it using Docker and Luigi — just one step left to that article 😉. We will assemble the steps eventually so that it can run on any production system without requiring specialized configurations and setups.

We are following an intuitive ML pipeline to build wine rating predictor:

1. Understand & Clean & Format Data
2. Exploratory Data Analysis
3. Feature Engineering & Pre-processing
4. Set Evaluation Metric & Establish Baseline
5. Select an ML Model based on the Evaluation Metric
6. Perform Hyperparameter Tuning on the Selected Model
7. Train and Evaluate the Model
8. Interpret Model Predictions
9. Draw Conclusions & Document Work

The problem itself is a supervised regression problem because we are trying to determine points (target variable) using the defined feature set. Points is a number between 80 and 100 indicating the quality of the wine.

Our raw data looked like this:

We started by formulating the business problem and explaining our objective:

The goal of the project is to build a wine rating predictor using a sample dataset to show good prediction is possible, perhaps as a proof of concept for a larger project.

Through the first three steps of the pipeline covered in the first notebook, we cleaned, understand and formatted the dataset. Moreover, we added and transformed features which prepared the training and test datasets to be used in any ML model ultimately:

In the second notebook, we selected the evaluation metric as the mean square error (MSE) which is an appropriate one for our problem. Then, we calculated the baseline (resulted in 9.01) using the mean square error computing logic in the Set Evaluation Metric & Establish Baseline.

Then, we tested several different ML algorithms and decided to use random forest regressor form scikit-learn for building the wine rating predictor in the Select an ML Model based on the Evaluation Metric step. We performed hyperparameter tuning on the selected algorithm and improved our model performance further in the Perform Hyperparameter Tuning on the Selected Model step.

We have significantly lowered the baseline MSE, from 9.01 to 4.99 which resulted in 45% improvement!

In this article, we are going to complete the remaining last three steps: We are going to evaluate our model on the test set and interpret the predictions. We will be concluding the overall pipeline and reporting the highlights in the last step.

The code behind this article can be found in this notebook. The complete project is available on GitHub:

cereniyim/Wine-Rating-Predictor-ML-Model

P.S: I encourage you to read previous articles if you haven’t already before we start!

• Part 1: Understand, clean, explore, process data
• Part 2: Set metric and baseline, select and tune model

Before We Begin:

The datasets that we are going to use is available in notebooks/transformed. The model that we saved in the previous notebook is available in the same directory as this notebook.

We are going to need the following libraries of Python in this notebook:

Let’s load the datasets and the model:

X_train = pd.read_csv("transformed/X_train.csv")
y_train = pd.read_csv("transformed/y_train.csv")

X_valid = pd.read_csv("transformed/X_valid.csv")
y_valid = pd.read_csv("transformed/y_valid.csv")

test_features = pd.read_csv("transformed/test_features.csv")
test_target = pd.read_csv("transformed/test_target.csv")

random_forest_model = (pickle
                       .load(
                           open('random_forest_model.sav', 
                                'rb')))

Convert them to arrays using the same functions that we used in the second article:

X = pd.concat([X_train, X_valid]) 
y = pd.concat([y_train, y_valid])

X_array = convert_features_to_array(X) 
y_array = convert_target_to_array(y)

X_test_array = convert_features_to_array(test_features)
y_test_array = convert_target_to_array(test_target)

7. Train and Evaluate the Model

The training set​ will be used to map patterns between the features (X_array) and target (y_array) and to build the wine rating predictor. This process is called ​training the model​ or building the model​. The mapping will be done with the fit method called on our random forest model.

random_forest_model.fit(X_array, y_array)

Before evaluating the model, the actual target values (y_test_array) are held-out from the test set (X_test_array) to utilize as a comparison to the predicted values. Both will be used to generate points/ratings of wine using the patterns found during the model fitting.

Feeding the test set into the model and generating predictions is called ​evaluating the model. When we call the predict method on our random forest model, the array of predictions are generated.

predictions = random_forest_model.predict(X_test_array)

Let’s see how our model performed on the test set (part of the training dataset that our model has never seen before):

8. Interpret Model Predictions

As data scientists, we are the interpreters of machine learning technologies. We should be able to explain how a solution works and what are the further improvement areas. This happens at the Interpret Model Predictions step.

We are going to interpret the model predictions from below perspectives:

1. Visualize a single decision tree
2. Compare actual values with the predicted values of points
3. Look at feature importances
4. Examine an individual prediction of the model

8.1. Visualize a Single Decision Tree

We had defined how the random forest algorithm works in the previous notebook:

Random forest regressor ​is an ensemble algorithm that builds multiple decision trees at once and trains them on various sub-samples and various subsets of the features of the dataset.

Our random forest algorithm consists of 200 decision trees. It may not provide us much information if we plot all trees at once. However, visualizing a single decision tree is sufficient to understand how wine rating predictor generates predictions.

Let’s visualize the first 3 nodes of a decision tree from our random forest regressor algorithm to understand further how our algorithm works:

Looking at this picture, you can think of a decision tree as a series of yes/no questions asked and answered like a flowchart. This specific tree is built using 5645 data points out of a total of 8948 data points — a random subsample of the complete dataset.

The root node is initiated by asking if the region_1 is less than 320.5. It can be easily realized that the initial MSE is the same as the baseline estimate of MSE. Predicted points is 88.425.

The root node is split into two as true and false child nodes consisting of 5018 (88% of the initial subsample) and 627 data points (12% of the initial subsample) respectively.

The false child node of the root node asks if the price is less than 30.5 and answers this question using the 12% of the initial subsample. The MSE improved slightly compared to the previous node, as an indicator of a more coherent node. Predicted points is 87.818.

This process continues iteratively until a leaf node is reached. When a node’s MSE does not improve further or when the specified criteria for reaching a leaf node (having a minimum of 2 samples in a leaf node for our model) is met, the process stops.

8.2. Compare Actual Values with the Predicted Values of Points

To understand how actual points and predicted points differ let’s visualize them side by side with a histogram:

PlotPredictedVSActual(predictions, y_test_array)

Recall that points show a normal distribution and 95% of the points lie in the 82.5–94.5 range. Our model successfully predicted the points within this range but failed to predict the points below 84 and points above 94 which are less-frequent values of the distribution. This is also the reason for the narrower distribution of the predictions.

This is a further and interesting improvement area for our wine rating predictor, but we will not touch it for now.

8.3. Look at Feature Importances

Another way to evaluate the model is to look at which features that the model considers most important. Following is the number for the relative contribution of each feature in determining a point:

PlotFeatureImportances(random_forest_model, test_features.columns)

According to the model, the strongest predictor is price. It is followed by the region_1, year, variety and the taster_name for predicting the points of wine.

These are in line with our preliminary observations developed during the Exploratory Data Analysis step.

Price is an indicator of wine’s quality and age. Therefore, we expect it to be an essential feature of the model.

Top 5 features form 80% of the total importances. Isn’t it another beautiful example of Pareto Princinciple?

From the features extracted, year is the most important among them and whether a wine is sparkling or not is the least important predictor of points.

Another significant observation is the note and taste-related features extracted from variety and description became moderately-important features for our predictor. A comprehensive use of NLP and sentiment analysis may help to find more valuable features from the description and variety features.

So far, we have interpreted the predictions of our wine rating predictor from an aggregate perspective. Now, let’s examine an individual prediction from our model.

8.4. Examine an Individual Prediction

We are going to get some help from an explainer library called LIME (locally interpretable model-agnostic explanations) to observe a prediction. This library creates an approximate local model in a place where data points are condensed and uses this model to explain individual predictions of any ML algorithm.

We will observe the most incorrect prediction detected from the maximum of the residuals. Then, we are going to create LimeTabularExplainer object.

# find the residuals and wrong prediction
residuals = abs(predictions - y_test_array)
wrong = X_test_array[np.argmax(residuals), :]

# Create a lime explainer object
explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data = X_array, 
    mode = 'regression',
    training_labels = y_array,
    feature_names = test_features.columns)

Following code block will plot how the wine rating predictor concluded a wrong prediction. It predicted 90.3 points whereas the actual point was 82. This is the most incorrect prediction of the model:

# Display the predicted and true value for the wrong instance
print('Prediction: {}'
      .format(random_forest_model.predict(wrong.reshape(1, -1))))
print('Actual Value: {}' 
      .format(y_test_array[np.argmax(residuals)]))

# Explanation for wrong prediction
wrong_exp = explainer.explain_instance(
    data_row = wrong, 
    predict_fn = random_forest_model.predict)

# Plot the prediction explaination
figsize(20, 10)
plt.rcParams['font.size'] = 24

wrong_exp.as_pyplot_figure()
plt.title('Explanation of a Wrong Prediction', size = 24)
plt.xlabel('Effect on Prediction', size = 24)

This plot shows the contribution of each feature to the wrong prediction example. Positive contributions are colored in green and negative contributions are colored in red.

As the price is the strongest predictor and this example has a price above 40, those significantly increased the model’s prediction of points. On the other hand, country, is_red and is_white balanced the predictions by decreasing it. This is a great example of how machine learning algorithms may also fall into the common pitfalls as we humans do. Don’t you ever buy an expensive wine to guarantee the high quality 🙂

9. Draw Conclusions & Document Work

As we are the interpreters of machine learning technologies, it is important to tailor our message and deliverables according to our audience. Especially, if you are presenting your results to your manager or client you must be very concise and relevant. You might expect that they will be more interested in your reasoning and outcome, so focus on them while presenting your findings 😉

If you remember from the first article, I built this project as a take-home assignment of the ML Engineer interview process at Data Revenue. The objective of the assignment is to decide whether to build a full-production solution by looking at the results of the wine rating predictor. So, I prepared a report to present my decision with the most crucial points of the 9-step ML pipeline and draw conclusions.

I proposed building the full-production solution counting on

Given the sample dataset, determined set of features and the tuned random forest regressor, the answer is ​yes, a​ ​​full-production solution is applicable with those.

The model significantly lowered (45%) the baseline MSE and resulted in 4.93 MSE on the test set. This serves as evidence to build the full-production solution.

.. and highlighted the most two most important findings:

- The wine rating predictor can infer the points of wine with a reasonable variance (MSE) of 4.9.

- The most useful predictors of points are determined as price, region_1, year, variety and the taster_name.

.. and I pointed out the further improvement areas:

Extending prediction range: so that the less-frequent target values can be predicted by the model. The current sample dataset is dominated by wines from the most dominant countries (US, Italy, France, Spain), feeding more data from non-present countries might help the model to learn better the determiners of the less-frequent points.

Using NLP to extract stronger predictors: currently, the strongest predictor is “price”. Extracting more features from the description related to wine taste and notes can help create strong predictors like price, decreasing prediction error of the model.

You can find the complete report here.

It ended up as a thorough (a habit I carry from consultancy) but succinct report. Besides, I kept the code aside because the report was prepared for a hypothetical client whose statistics and ML knowledge is limited.

You can prepare a similar summary report (I prepared in a PDF format) for your own projects or learning purpose by converting your Jupyter notebooks into documents. You can use various download options available in the File > Download As. HTML or LaTeX formats are pretty straightforward to use. (if you have the LaTeX Studio installed in your machine, then you can convert it to pdf format later on). You can read more on that here.

We should always keep in mind that the “science” in data science serves a purpose, ultimately for a non-scientific purpose. Our work is mostly used to support decision-making processes in businesses or solving real-world problems. Presenting the results (verbally or written) and adjusting the language according to your audience will be a valuable skill you have.

Conclusions for the Complete 9-Step ML Pipeline

We came a long way from structuring the raw data to making the meaning out of them. We started by cleaning and formatting the data, then we trained and tuned the selected model (random forest regressor). We concluded the 9-step pipeline by evaluating the model and interpreting its results. Finally, we have highlighted the most important outcomes of the project in a report.

1. Understand & Clean & Format Data
2. Exploratory Data Analysis
3. Feature Engineering & Pre-processing
4. Set Evaluation Metric & Establish Baseline
5. Select an ML Model based on the Evaluation Metric
6. Perform Hyperparameter Tuning on the Selected Model
7. Train and Evaluate the Model
8. Interpret Model Predictions
9. Draw Conclusions & Document Work

We have followed these steps sequentially in the article series. However, I went back and forth between these steps 20% of the time when I was working on this project. All in all, I believe building a machine learning solution is an iterative and experimental process, so never hesitate to improve and revisit the previous steps!

Special Thanks

I want to thank Will Koehrsen here. He is one of the inspirers for me to write about my data science/machine learning projects.

I first learned this pipeline from one of his articles and adjusted it for my own objective and problem. Now, I am using this pipeline as a guide when building any kind of machine learning/data science project. It helps me to organize my thought process and see how individual elements of machine learning is connected!

We have one last article to go which will automate this pipeline with Docker and Luigi. Stay tuned in the next week!

Building an Automated Machine Learning Pipeline: Part Four

Thanks for reading 🙂 Please feel free to use this pipeline, the code and the repository for your own projects.

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!

Stay safe and healthy until then 👋

Building an Automated Machine Learning Pipeline was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Setting the Evaluation Metric & Establishing a Baseline, Selecting the Algorithm and Performing Hyperparameter Tuning steps

• Part 1: Understand, clean, explore, process data
• Part 2: Set metric and baseline, select and tune model (you are reading now)
• Part 3: Train, evaluate and interpret model (live!)
• Part 4: Automate your pipeline using Docker and Luigi (live!)

In this article series, we set our course to build a 9-step machine learning (ML) pipeline (we are calling it the wine rating predictor) and automate it. Eventually, we will observe how each step congregates and runs in production systems.

We are working on a supervised regression problem. We want to develop a performant, understandable and good wine rating predictor that can predict the points, a quality measure of wine.

In the first article, we defined the problem and our motivation behind building the wine rating predictor. Then, we had a detailed look at the data by visualizing the relationships between the features and the target along with the Understand & Clean & Format Data and Exploratory Data Analysis steps.

In the Feature Engineering & Pre-processing step, we added new and more useful features. Besides, we prepared the training and test datasets to use during training and evaluation of the model. As the final step of the first article, we created validation dataset out of the training dataset for model selection.

In this article, we are going to complete the following steps:

4. Set Evaluation Metric & Establish Baseline

5. Select an ML Model based on the Evaluation Metric

6. Perform Hyperparameter Tuning on the Selected Model

The code behind this article can be found in this notebook. The complete project is available on GitHub:

cereniyim/Wine-Rating-Predictor-ML-Model

Feel free to share, fork, and utilize this repo for your projects!

Before We Begin:

The datasets that we are going to use is available in notebooks/transformed

• X_train consisting of features (country, province, region_1, variety, price, year, taster_name, is_red, is_white, is_rose, is_dry, is_sweet, is_sparkling, is_blend) and y_train consisting of the target (points) to train the model
• X_valid consisting of features and y_valid consisting of target to validate the model

We are going to need the following libraries of Python in this notebook:

Let’s load the datasets into dataframes and convert to an array using below functions:

X_train = pd.read_csv("transformed/X_train.csv")
X_train_array = convert_features_to_array(X_train)

X_valid = pd.read_csv("transformed/X_valid.csv")
X_valid_array = convert_features_to_array(X_valid)

y_train = pd.read_csv("transformed/y_train.csv")
y_train_array = convert_target_to_array(y_train)

y_valid = pd.read_csv("transformed/y_valid.csv")
y_valid_array = convert_target_to_array(y_valid)

4. Set Evaluation Metric & Establish Baseline

If this was a Kaggle competition, we would skip this step of the pipeline because we would be given with the evaluation metric.

However, in real-world applications of data science/machine learning, the evaluation metric is set by data scientists in line with the stakeholder’s expectations from the ML model. That is why this is an important step.

After we decide on our evaluation metric, to quantify our initial motive — building a good wine rating predictor and to compare our model’s performance against, we are going to form a baseline.

4.1. Set Evaluation Metric

The mean square error (MSE) is set as the evaluation metric. It is the average of the sum of squared residuals where a ​residual is the subtraction of the actual values from the predicted values of the target variable.

In other words, evaluation of the model is done by looking at the measure of how large the squared errors (residuals) are spread out.

We selected MSE because it is interpretable and analogous to variance, and it is a widely-used optimization criterion among ML models. (E.g. linear regression, random forest)

4.2. Establish a Baseline

A ​baseline​ can be explained as generating a naive guess of the target value by using expert knowledge or a few lines of code. It also helps to measure the performance of the ML model.

If the built-model (wine rating predictor) cannot beat this baseline, then the selected ML algorithm may not be the best approach to solve this problem or we might want to revisit the previous steps of the pipeline.

Recall that the points (target) has a normal distribution between 80 and 100. The mean is 88.45 and the variance is 9.1.

We are going to form the baseline by:

• using these statistics and
• calculating differences between each point in the validation set and the mean of the training dataset, then taking the average of the sum of the squared differences (similar approach of computing MSE):

# set baseline as mean of training set's target value
baseline = (np
            .mean(
                y_train_array))

# calculate MSE baseline
mse_baseline = (np
                .mean(
                    np.square(
                        baseline - y_valid_array)))

It is not a coincidence that the variance of the points and baseline error is almost equal. You can think of this baseline MSE as the manually calculated variance with a smaller set from our dataset.

This number (9.01) will accompany us in the next step — Select an ML Model based on the Evaluation Metric while we are testing different ML algorithms.

5. Select an ML Model Based on the Evaluation Metric

I always find useful trying out several algorithms with different rationales behind, because I believe this whole process involves experiment as well! (“scientist” part of being a data scientist makes sense now 🙃)

While searching for the best algorithm, we are going to observe both the improvement of the MSE and the run time of the algorithms (with %%time magic at the beginning of the cells) and compare the algorithm’s MSE to our baseline MSE. At the same time, we will keep in mind the understandable and performant requirements of the wine rating predictor.

We are going to experiment with one linear algorithm, two distance-based algorithm and two tree-based algorithms, ordered from the simplest to most complex:

• Linear regression
• K-nearest neighbors regressor
• Support vector regressor
• Random forest regressor
• Light gradient boosting regressor

We are going to train them with the training set, and compare their generalization performances with the validation set. Below function will do the work for us.

At the end of this step, we are going to elaborate on how the selected algorithm works.

5.1. Linear Regression:

Linear regression slightly decreased the baseline metric, showing that it is not a candidate to be a good predictor.

5.2. K-Nearest Neighbors Regressor

Distance-based models use the euclidian distance (or other distance measures) for training, thus varying ranges causes distance-based models to generate inaccurate predictions. To apply distance-based algorithms, we scaled the datasets with normalization before here in the notebook.

K-nearest neighbors regressor performed better than the linear regression. However, MSE is still high, showing that this algorithm is not a good predictor as well.

5.3. Support Vector Regressor

Support vector regressor performed better than the k-nearest neighbors regressor at a higher run-time. All in all, MSE decreased 35% showing that this algorithm might be a candidate for building a good predictor.

5.4. Random Forest Regressor

Random forest regressor performed better than the support vector regressor at lesser run-times. It decreased the MSE 44% and replaced support vector regressor in the good-predictor list.

5.5. Light Gradient Boosting Regressor

Light gradient boosting regressor (Light GBM) showed the best performance of all tried models. It also lowered the baseline MSE 45% and showed that it is a potential candidate for a good predictor at a lower run time.

As a summary:

It might not be a fair selection of algorithms since we only train them with the default hyperparameters. Nevertheless, this is the experimental step of the pipeline, that is why both light GBM and random forest algorithm has been given a chance for further improvements in the Perform Hyperparameter Tuning on the Selected Model step. (You can find it here in the notebook) In this article, we will use random forest regressor due to the understandable and performant requirements and we will only elaborate on it.

Before moving on to the next step, let’s understand how random forest regressor works:

Random forest regressor ​is an ensemble algorithm that builds multiple decision trees once and trains them on various sub-samples and various subsets of the features of the dataset. Its random selection of the dataset and feature subsamples makes this algorithm more robust.

A Decision tree​ uses a tree-like structure to make predictions. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. A decision node has two or more branches, each representing values for the feature tested. Leaf node represents a final decision on the target.

6. Perform Hyperparameter Tuning on the Selected Model

A hyperparameter is the set of defined parameter(s) by the data scientist or ML engineer whose values are not affected by the training process of the model. On the other hand, a parameter of a model is searched and optimized by the model during the training process and affected by the dataset.

I loved this quotation from Jason Brownlee on Machine Learning Mastery to prevent confusion of the two:

“If you have to specify a model parameter manually then, it is probably a model hyperparameter.”

An example of a parameter from our simplest model: Coefficients of the linear regression model, which are optimized through the model training.

An example of a hyperparameter from our selected model: The number of trees constructed in a random forest model, which is specified by us or by scikit-learn.

Hyperparameter tuning is the process of defining, searching and perhaps improving the performance of the model further. We are going to search for the best set of parameters with the random search and k-fold cross-validation.

6.1. Random Search

Random search is the process of searching for the combination of the defined parameters randomly and comparing the defined score (mean squared error, for this problem) in each iteration. It is fast and run-time-efficient, but you may not always find the most optimal set of hyperparameters due to the searching of a random combination of defined hyperparameters.

We are going to search for the below hyperparameters of random forest regressor with the hyperparameter_grid dictionary:

• n_estimators: number of trees to be used in the model, default is 100.
• min_samples_split: minimum number of samples required to split an internal node, default value is 2.
• min_samples_leaf: minimum number of samples required to be at a leaf node, default value is 1.
• max_features: number of features to consider when looking for the best split, default value is auto.

# define search parameters 
n_estimators = [100, 200, 300, 500, 1000] 
min_samples_split = [2, 4, 6, 10] 
min_samples_leaf = [1, 2, 4, 6, 8] 
max_features = ['auto', 'sqrt', 'log2', None]

# Define the grid of hyperparameters to search 
hyperparameter_grid = {
    "n_estimators": n_estimators,               
    "min_samples_split": min_samples_split,                       
    "min_samples_leaf": min_samples_leaf,                         
    "max_features": max_features}

6.2. K-Fold Cross-Validation

K-fold cross-validation is the method used to assess the performance of the model on the complete training dataset. Rather than splitting the dataset into two static subsets of training and validation set, the dataset is divided equally for the given K. Then the model is trained with K-1 subsets and tested on Kth subset iteratively. This process makes the model more robust to overfitting — more on that at the end of the article.

To perform hyperparameter tuning with random search and k-fold cross-validation we are going to add training and validation datasets and continue with one training set from now on.

# add dataframes back for to perform random search and cross-validation
X = pd.concat([X_train, X_valid])
y = pd.concat([y_train, y_valid])

X_array = convert_features_to_array(X)
y_array = convert_target_to_array(y)

Since we have less than 10.000 rows in the training dataset, we are going to perform 4 fold cross-validation to have sufficient number of data points in each fold. We are going to consolidate random search and k-fold cross-validation in the RandomizedSearchCV object:

rf_random_cv = RandomizedSearchCV(
    estimator=rf,
    param_distributions=hyperparameter_grid,
    cv=4, 
    n_iter=25,
    scoring='neg_mean_squared_error',
    n_jobs=-1, 
    verbose=1,
    return_train_score=True,
    random_state=42)

rf_random_cv.fit(X_array, y_array)

With the fit method, we initiate the search for the random combination of the values defined for each hyperparameter in the hyperparameter_grid. At the same time, "neg_mean_squared_error" is calculated for each fold in each iteration. We can observe the determined best set of parameters by calling the best_estimator_ method:

After hyperparameter tuning, the best set of hyperparameters are determined as:

• n_estimators: 200
• min_samples_split: 4
• min_samples_leaf: 2
• max_features: ‘sqrt’

Let’s see if those hyperparameters will help us to improve the MSE of the random forest regressor further.

6.3. Revisit MSE of Random Forest Regressor After Hyperparameter Tuning

rf_random_cv_model = rf_random_cv.best_estimator_

The MSE is decreased from 5.41 to 4.99 and tuned model’s run-time resulted in 1.12 seconds which is lower than the run-time of the initial random forest model (1.89 seconds). Hyperparameter tuning has not only improved the evaluation metric but also lowered the run-time in our case.

Given the sample dataset, the determined set of features and the tuned random forest regressor we have successfully built a good wine rating predictor without falling into the underfitting and overfitting areas!

Some Notes on Overfitting and Underfitting

Although this topic is another article by itself, I think it is important to address here since we mentioned these concepts.

Overfitting happens when the ML model perfectly fits (or memorizes) the training dataset rather than grasping the common patterns between the features and the target. An overfit model will have a high variance, I find a resemblance between this situation and a fragile glass home. It is perfectly built for its current conditions, but it is less likely to survive if the conditions are changed.

Did you remember that cross-validation makes our model more robust to overfitting?

The reason behind is that our model sees a different dataset in each fold during training — like a house already encounters different weather conditions. This improves the generalization performance which makes the model more resistant to overfitting.

Underfitting happens when the ML model fails to grasp the relationships between the features and the target due to not having enough data points or features. It may perform as bad as a random guess about the predictions. An underfit model will have a high bias, I think of this situation as a half-built shed. It should have to be undergone some construction to serve its purpose.

Did you remember the linear regression model that only showed a slight improvement on the baseline MSE?

That was an example of an underfitting. Linear regression model resulted in almost the same MSE as the baseline MSE and failed to understand the relationship between the features and the target.

All in all, both overfitting and underfitting decrease the generalization performance of the machine learning models, and results in unsatisfactory levels of evaluation metric. Although we did not test the tuned random forest regressor specifically the overfitting case, the 45% improvement shows that we are at the fine line between underfitting and overfitting.

Before moving on to the conclusions let’s save our model in the directory so that we can pick up where we left off in the last article and notebook.

filename = 'random_forests_model.sav'
pickle.dump(rf_random_cv_model, 
           open(filename, 'wb'))

Conclusions

In this article, we completed the intermediate steps of the machine learning pipeline. After some quick recap on the first article and objectives, we

1. Set the evaluation metric as mean square error due to its wide-use in ML algorithms and explainability.
2. Established a baseline aligning with the mean square error calculation which resulted in 9.01.
3. Tried several different ML algorithms, and selected random forest regressor which reported MSE of 5.4 at 1.93 seconds run time.
4. Fine-tuned hyperparameters of the random forest regressor and achieved 8% performance improvement compared to the initial random forest model (MSE: 4.99 at 1.12 seconds run time).

If we consider our initial motivation (building a good wine predictor) and the baseline MSE (9.01), we are going on the right path as we approach the end of the pipeline.

We have significantly lowered the baseline MSE, from 9.01 to 4.99 which results in 45% improvement!

The third article will start by loading the fine-tuned random forest regressor model and will focus on the evaluation of the model with the test set, as well as the results of it. (steps 7, 8, and 9).

Building an Automated Machine Learning Pipeline

The last article will automate this pipeline with Docker and Luigi.

Building an Automated Machine Learning Pipeline: Part Four

Thanks for reading 😊

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!

Stay safe and healthy until then 👋

Building an Automated Machine Learning Pipeline: Part Two was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Cleaning, Exploratory Data Analysis and Feature Engineering steps

• Part 1: Understand, clean, explore, process data (you are reading now)
• Part 2: Set metric and baseline, select and tune model (live!)
• Part 3: Train, evaluate and interpret model (live!)
• Part 4: Automate your pipeline using Docker and Luigi (live!)

During my data science learning journey, I learned through reading articles on Towards Data Science as much as I learned from online courses and various coding challenges. Following such diverse sources of learning that come with their own interpretations of machine learning (ML) helped me a lot. Today, I will bring a part of my perspective to the ML applications that I distilled throughout this journey and start an article series. It will hopefully show how to build an automated ML pipeline with a beginner-friendly language and structure.

We are going to follow an intuitive ML pipeline to build a predictor to rate wine quality:

1. Understand & Clean & Format Data
2. Exploratory Data Analysis
3. Feature Engineering & Pre-processing
4. Set Evaluation Metric & Establish Baseline
5. Select an ML Model based on the Evaluation Metric
6. Perform Hyperparameter Tuning on the Selected Model
7. Train and Evaluate the Model
8. Interpret Model Predictions
9. Draw Conclusions & Document Work

This first article covers steps 1, 2 and 3. The code behind this article can be found in this notebook. The second and third articles in the series will have covered the entire pipeline.

After we build the pipeline, the fourth article will focus on how we can automate the wine rating predictor with Docker and Luigi. This part is especially exciting for me because it introduced me to concepts and tooling that let me understand how an ML solution runs on production systems.

Disclaimer: The core structure and utilities in this automation are taken from a coding challenge that I took on recently as part of an ML Engineer interview process at an amazing company called Data Revenue. I did not get the job because they were looking for someone with more emphasis and experience in engineering whereas I’m more on the science part. Ultimately, I wore a software engineer’s hat and learned a lot during the interview process.

The complete project is available on GitHub for the impatient:

cereniyim/Wine-Rating-Predictor-ML-Model

Problem Definition & Objective:

In this project, we are provided with a sample dataset. In a real-world scenario, one can possibly obtain (perhaps scrape) this data from online sources like winemag.

The goal of the project is to build a wine rating predictor using a sample dataset to show good prediction is possible, perhaps as a proof of concept for a larger project.

We are going to use the results of the wine rating predictor to confirm that ML applications are effective solutions to business problems such as optimizing a wine portfolio for a seller.

We should always keep in mind that the “science” in data science serves a purpose, ultimately for a non-scientist. A typical project has some implied base requirements.

Our wine rating predictor should be understandable because our audience will likely have limited knowledge of statistics and ML. Furthermore, it should be performant because the complete production dataset could have millions of rows. Finally, it should be automated in a way that it can run on any production system without requiring specialized configurations and setups.

Before We Begin

The data for this project is available in data_root/raw/wine_dataset in the repository. For now, we will use 90% of it as the training set and the remaining 10% as the test set. They are both available in data_root/interim as train.csv and test.csv .

We are going to use Python 3.7 and will need the following libraries in this notebook:

1. Understand & Clean & Format Data

In this step, we will get ourselves familiar with the dataset, understand what our dataset conveys and clean it by removing redundant rows and columns.

Let’s load the training and test datasets into data frames and have a look at some rows of the training dataset:

train = pd.read_csv("../data_root/interim/train.csv") 
test = pd.read_csv("../data_root/interim/test.csv")
train.sample(5)

There are some empty rows in the dataset and we have only 2 numeric variables: points and the price.

Points is the dependent variable that we are trying to predict, and will be referred to as the target. The rest are the independent variables that can be used in determining the target and will be referred to as features.

We are trying to grasp the relationships between independent variables and a dependent variable and use this relationship to generate predictions, which makes the wine rating predictor a supervised regression machine learning model:

• Supervised: Dataset includes the wine ratings (labelled as points) and independent variables which can be used in determining the points.
• Regression: Our target variable — points consist of continuous integers ranging from 80 to 100.

Most of the column names are self-explanatory if you ever thought about which wine to buy at a supermarket. However, I think it is important to understand how a particular wine receives points and domain-specific features.

Points are assigned to a wine by a taster. A wine may receive different points from the same taster. When a taster is assigning points to a wine, s/he also provides a description of the notes and taste about the wine.

The designation is the name of the vineyard and region_1 & region_2 features describe the places where the grapes of the wine are grown. Here is how a Barlow 2005 Barrouge Red looks like:

Now, let’s have a look at the data types and number of non-empty (non-null) of the training and test dataset:

train.info()

test.info()

This is an interesting dataset because 90% of the features are categorical — variables that are in object Dtype. ML models can only work with numerical data and non-null values. Thus, we are going to develop some strategies in the Feature Engineering & Pre-processing to handle the categorical data and missing values in our model.

1.1. Missing values of the datasets:

The function below will enable us to observe the missing values as a percentage per feature.

Features that have missing values of more than 50% are not likely to provide significant information to our model. Therefore, region_2 will be dropped from both datasets.

In this sample dataset, target value has no missing values. If we would have missing values in the target, we would have dropped all the missing values not to distort the existing distribution of the target.

1.2. Cardinality of the non-numeric features:

Cardinality is the number of unique values that a categorical feature has. Let’s have a look at it for our training dataset:

object_columns = (train
                  .select_dtypes(include="object")
                  .columns)

for column in object_columns:
    print("{} has {} unique values."
          .format(column, 
                  train[column]
                  .nunique()))

We can infer that a feature has a

• high cardinality if it has more than 1000 rows out of 9000: description, designation, title and winery.
• moderate cardinality if it has more than 100 rows: province, region_1 and variety.
• low cardinality if it has less than 100 rows: country, region_2, taster_name and taster_twitter_handle.

High cardinality features may cause problems when we are training our model. Even though we transform each unique value of a categorical feature to a new feature, they may bring the curse of dimensionality. So, we are going to use the high cardinality features for adding new features, then we are going to remove them from the datasets.

Remember that this is a sample dataset. So we cannot guarantee that we have seen all the possible values of each categorical feature when we are training the model. If our wine rating predictor encounters with a new value in the test set, it will throw errors. That is why we are going to revisit this in the Feature Engineering & Pre-processing.

1.3. Removing collinear features:

Two variables are called collinear if there exists a linear association between them. In case of categorical features, they are called collinear if they are providing the same information to predict a target.

Collinear features decrease the generalization performance on the test set and interpretability of the model. Recall that one of the requirements for our wine rating predictor was interpretability.

In our sample dataset, taster_name and taster_twitter_handle signals for the collinearity. After making sure that all values of the taster_twitter_handle are covered by the taster_name (not shown here but checked here in the notebook), it is removed to build a good and explicable wine rating predictor.

1.4. Findings from Understand & Clean & Format Data:

Up until now, we had the first glance of the dataset and detected the features to remove:

• designation and winery due to the high cardinality
• region_2 due to having missing values more than 50%
• taster_twitter_handle due to collinearity

Moreover, the training set contains some duplicate rows. In addition to the list above, we are going to drop them and clean the data with the below function:

drop_columns = ["designation", "winery", "taster_twitter_handle", "region_2"]

train_cleaned = CleanDatat(train, drop_columns, "points")
test_cleaned = CleanData(test, drop_columns, "points"

Now, we are going to use the train_cleaned.csv and test_cleaned.csv to visualize and explore the features and target further in the Exploratory Data Analysis.

2. Exploratory Data Analysis

Exploratory data analysis (EDA) is the step where we have a detailed look at the variables and visualize the relationships between features and target. In this step, we are going to seek and follow the clues of strong determiners of the target. At the end of this step, we will decide which features to process in the Feature Engineering & Pre-processing.

First, we will examine the points (target), numeric features and link between them. Then, we will investigate the low-cardinality features and their relationship to the target. As we find valuable insights, we will elaborate on them through the steps of EDA.

2.1. Distribution of points:

Distribution is a description of a variable’s range and how data is spread in that range. We can easily observe a distribution by plotting a histogram:

figsize(10, 8)
plt.rcParams['font.size'] = 14
plot_histogram(train_cleaned, "points")

Points show a normal distribution (a bell-shaped curve is obvious in the distribution) as expected from a random variable. Range of the points is distributed between 80 and 100 with a mean of 88.45 and a median of 88. Moreover, standard deviation — a measure of the spread of a data range is 3.03 resulting in a variance of 9.1 as the squared deviation. Based on the normal distribution, we can confidently say that 95% of the points lie in the 82.5–94.5 range.

2.2. Distribution of price:

When we plot the histogram of the price it showed a vague picture because it was highly right-skewed (available here in the notebook). To get a reasonable one, we will observe it in the 0–200 range:

figsize(10, 8) 
plt.rcParams['font.size'] = 14
plot_histogram(train_cleaned, "price", 200) 
plt.xlim(0,200)

This is still a right-skewed or positively skewed distribution where the right tail is longer and most of the values are concentrated on the left-hand side. Median of the price is 25 and the mean price is 35.5.

In order to use price as a predictor, we need to assume that the points received do not cause any price change. Otherwise, this will cause data leakage since we’ll be including a predictor at a later point in time. This could potentially spoil our future predictions of points.

Nonetheless, assuming that price is not affected by the points assigned by a taster, it is an indicator of wine’s quality and age. Therefore, we expect it to be an essential feature of the model.

2.3. Points and country:

To visualize the relationships of several variables, we will use a density plot where we plot the distribution of target per unique values of a categorical feature. A density plot can be interpreted as the continuous and smoothed version of a histogram which conveys the shape of the distribution easily.

To keep the plots interpretable, we will plot the distribution of points for the countries with more than 100 occurrences.

# make a list of countries that has most occurences
countries = (train_cleaned["country"]
             .value_counts())

freq_countries = list(
    countries[
        countries.values > 300]
    .index)

freq2_countries = list(
    countries[
        (countries.values <= 300) &
        (countries.values >= 100)]
    .index)

figsize(20, 10)
plt.rcParams['font.size'] = 14

# plot points distribution for most frequent countries
plt.subplot(1, 2, 1)
plot_distribution(train_cleaned, "points",
                  freq_countries, "country")

plt.subplot(1, 2, 2)
plot_distribution(train_cleaned, "points",
                  freq2_countries, "country")

Wine producing market is dominated by the US, France, Italy and Spain which is easily available from the plots. Country is a differentiating feature to determine points of wine. However, there are some exceptions: USA & Portugal or Austria & Germany.

2.4. Points and price with country

Assuming that the price is an essential determiner for the points, we will explore points and price relationship with country.

# create dataframe of most frequent countries
points_freq_countries = train_cleaned[
    train_cleaned
    .country
    .isin(freq_countries)]

figsize(20, 10)
plt.rcParams['font.size'] = 14

# plot a scatterplot of Points and Price
plt.subplot(1,2,1)
sns.scatterplot(x='price', y='points',
                hue='country', 
                data=points_freq_countries,
                alpha=0.7)
plt.xlabel("Price", size=14)
plt.ylabel("Points", size=14)
plt.title("Points vs Price for Most Frequent Countries", 
          size=16)

# plot a scatterplot of Points and Price
plt.subplot(1,2,2)
sns.scatterplot(x='price', y='points', 
                data=train_cleaned,
                alpha=0.7)
plt.xlabel("Price", size=14)
plt.ylabel("Points", size=14)
plt.title("Points vs Price for all Countries with price range 0-200", size=16)
plt.xlim(0,200)

The plot on the left-hand side shows the points and price relationship for the wines produced by the dominant countries in the complete price range. The plot on the right-hand side shows the points and price relationship for all countries zoomed in 0 and 200 price range.

Both plots show that there is a strong positive correlation exists between the points and price, which serves as proof that price is an important predictor. Another valuable insight is, there are two wines from Italy and one from France that received full points from tasters, all of their prices are greater than 200.

2.5. Points and taster name

Points, represents the score of a particular wine received from a taster. So, a different taster might imply different interpretations of a wine. To investigate this we are going to plot a violin plot.

Violin plot ​is my favourite plot that conveys the distribution of the data shape and summary statistics with the violins outside and slim rectangles inside respectively.

figsize(14, 8)
plt.rcParams['font.size'] = 14

f = sns.violinplot(data=train_cleaned,
                   x="taster_name",
                   y="points")

f.set_xticklabels(f.get_xticklabels(),
                  rotation=90)

plt.title("Points from Different Tasters",
          size=16)

Each taster’s summary statistics and distribution of points are unique resulting in different violins above. Looking at this picture, we expect it to be the second most important predictor of the model.

2.6. Findings from Exploratory Data Analysis

We explored points, price, country and the taster_name and relationships between them. In addition to them, variety, region_1 and province will remain in the final feature list since:

• variety and region_1 are major elements in differentiating the grapes of the wine so as the wine’s taste and quality
• province provides location information when combined with country

Moreover, we observed a normal distribution in the points and identified its mean and variance, which will be useful for us in the Set Evaluation Metric & Establish Baseline. We expect price and taster_name to be the two most important predictors in determining points.

Now, we will move forward with the long-waited and mentioned Feature Engineering & Pre-processing part.

3. Feature Engineering & Pre-processing

Feature engineering is the process of extracting and transforming information from raw features and potentially create a lot more useful features. We are going to take what we’ve learned about our features in the previous parts and encode that information into new ones. Since our features are mostly categorical, we are going to search for numerical or binary features in the description, variety and title.

Then, we are going to prepare both datasets to be used as an input for our wine rating predictor. This process is called feature pre-processing. This is a necessary step because ML models can only work with numerical data and non-missing values.

Building a good wine rating predictor starts with identifying the most useful features to determine points. At the end of this step, we will identify the most effective feature set, which will provide a good and understandable wine rating predictor in return. We are going to follow feature extraction, categorical feature transformation and missing value filling respectively.

3.1. Feature extraction

• description: It contains information about the wine’s colour, taste and notes (like sweet, dry). Taste and colour-related words are searched and extracted from the description with the function below. If a taste or colour-related word exists in the description, the corresponding feature to that word is assigned with a value of 1, otherwise 0.

• title: It contains the production year of the wine. If year information is not available in the title year is assigned to 0.

• variety: It contains information about whether different types of grapes are blended. is_blend is assigned to 1 if several grape varieties present, otherwise 0.

Now, using those functions above and search keywords below, we are going to add new features.

# create search terms for new features 
# to be extracted from description
is_red_list = ["red", "Red", "RED",
               "noir", "NOIR", "Noir",
               "black", "BLACK", "Black"]

is_white_list = ["white", "WHITE", "White",
                 "blanc", "Blanc", "BLANC",
                 "bianco", "Bianco", "BIANCO",
                 "blanco", "Blanco", "BLANCO",
                 "blanca", "Blanca", "BLANCA"]

is_rose_list = ["rose", "ROSE", "Rose",
                "rosé", "Rosé", "ROSÉ"]

is_sparkling_list = ["sparkling", "SPARKLING", "Sparkling"]

is_dry_list = ["dry", "Dry", "DRY",
               "dried", "Dried", "DRIED"]

is_sweet_list = ["sweet", "Sweet", "SWEET"]

desc_extracting_dict = {
    "is_red": is_red_list,
    "is_white": is_white_list,
    "is_rose": is_rose_list,
    "is_sparkling": is_sparkling_list,
    "is_dry": is_dry_list,
    "is_sweet": is_sweet_list
}

train_features_added = ExtractFeatures(train_cleaned) test_features_added = ExtractFeatures(test_cleaned)

After the feature engineering, we have 14 features in the train_features_added and test_features_added data frames: country, province, region_1, variety, price, year, taster_name, is_red, is_white, is_rose, is_dry, is_sweet, is_sparkling, is_blend.

3.2. Feature pre-processing

As the first part of feature pre-processing steps, categorical features (country, province, region_1, taster_name and variety) will be turned into the numerical features. We selected the ​ordinal encoding ​approach for the transformation, which is the process of assigning positive integers consecutively starting from 1 to each unique value of a particular feature in the order of appearance.

Remember that this is a sample dataset, so we have to consider the possible values that are not present in this sample dataset but may show up in the test dataset. To manage them we are going to use ordinal encoder from the category_encoders library.

When we set handle_unknown parameter to the "value", the ordinal_encoder object will assign the first-time seen values of the test set to -1. Then, we run fit method on the training set, ordinal_encoder object maps the values of the given features to the integers. Then, we run transform method on both datasets, the transformation process is completed according to mapping done during fitting and we get full-numeric datasets in return.

ordinal_encode_cols = ["country", "province", "region_1", "variety",  "taster_name"]

data_encoded = EncodeCategoricalData(train_features_added,
                                     test_features_added)
train_encoded = data_encoded[0]
test_encoded = data_encoded[1]

As the second part of the feature pre-processing, we will handle missing values. An easy option is to drop them. However, we already have only 8500 data points to train the model. Therefore, we are going to proceed by filling them. ​Process of filling missing values of features is called imputation. We are going to use the following strategies for different features:

• taster_name: Since we expect taster_name to be one of the most important features, it is imputed with a constant value (0 stands for “Unknown taster”). That way, we can maintain each taster’s effect in determining points.

• price and year: Imputed with median value since they are numeric columns and the median is not affected by the extreme values.

• country, province, region_1 and variety: Imputed with most frequent value since they are categorical and the most frequent values of each feature align. Those are US, California, Napa Valley and Pinot Noir respectively.

One important consideration during imputing is, whatever imputing strategy is selected, both training and test sets are imputed with the same values. For instance, the median of the price in the training dataset is 25 and this value is assigned to missing values of the training dataset and the test dataset to prevent data leakage into the test set.

transformed_data = ImputeMissingValues(train_encoded, test_encoded)

train_features = transformed_data[0]
train_target = transformed_data[1]
test_features = transformed_data[2]
test_target = transformed_data[3]

Before moving on to the next article and the section, we are going to reserve some part of the training set as a validation set to provide an unbiased evaluation of a model.

Training dataset will be used for training the machine learning model. Training a machine learning model is the process of the grasping the patterns between the features and the target by our model, and using them for further predictions.

Validation dataset — the held-out part of the training dataset will be used as a measure of how well our model will generalize to a first-time-seen dataset. This measure will be one of the objective selection criteria of the Select an ML Model based on the Evaluation Metric part.

I can hear you thinking why don’t we use the test set for model selection then? Because, if we used the test set for this purpose, we would be selecting a biased model that is trained perfectly fit to our test set. Moreover, the model performance would not be representative.

The test set will come into the play when we are evaluating our fully-specified model at the Train and Evaluate the Model.

In light of the information above, let’s create the validation set by separating 25% of the training set randomly:

X_train, X_valid, y_train, y_valid = train_test_split(
    train_features, 
    train_target, 
    test_size=0.25, 
    random_state=42)

Training, validation and test datasets are available under the notebooks/transformed in the repository. They are all ready to be fed into the wine rating predictor!

Conclusions

In this article, we completed the first three steps of a machine learning pipeline. After defining the problem and objectives, we

1. Cleaned data by removing redundant and collinear features; and duplicate rows
2. Explored data to have a detailed understanding of the target, features and their relationships
3. Added more useful features (hopefully!), prepared and transformed both datasets

We also completed the crucial step of creating a validation set from the training set.

The second article will show how to select an evaluation metric, establish a baseline, select an ML model and tune its hyperparameters by optimizing the selected model’s evaluation metric (steps 4, 5 and 6).

Building an Automated Machine Learning Pipeline: Part Two

The third article will train and evaluate the model with the training and test dataset respectively. It will show how to interpret an ML model’s predictions and draw conclusions (steps 7, 8, and 9).

Building an Automated Machine Learning Pipeline

The last article will automate this pipeline with Docker and Luigi.

Building an Automated Machine Learning Pipeline: Part Four

Thanks for reading 😊 now the article series is complete and you can read the next ones!

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!

Stay safe and healthy until then 👋

Building an Automated Machine Learning Pipeline: Part One was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

And examples from its practical applications

Previously on In Action Blog Posts: We looked at the Central Limit Theorem to understand better underlying theorem behind the normal distribution. I encourage you to read if you haven’t already!

Central Limit Theorem In Action

Today, we will observe another important statistical concept in action and perform a hypothesis test. I will explain the statistical significance not only with a concrete and catchy example but also in simple terms. I will use the same dataset I used to explain the Central Limit Theorem — the life expectancy in the world of 186 countries as of 2018.

Let’s revisit the most important plot and calculation from the Central Limit Theorem In Action:

The mean (µ) of average life expectancy in the world is 73 with a standard deviation (σ) 7 and it is an almost normal distribution. Using the Central Limit theorem, the standard error was calculated as 0.57 by dividing the population’s standard deviation to the square root of sample size.

Imagine a wanderer statistician who discovered an island where only 50 people live. All inhabitants are perfectly healthy and over 73 years-old. She conducts a small survey and learns that the average life expectancy is 76 on the island. Moreover, she finds out that everybody on the island has been doing intermittent fasting. Considering the average life expectancy of 73, this is an intriguing case!

Now, our curious statistician wants to know if the intermittent fasting could be the path to live a longer and healthier life putting all other possible determinants aside. So, she decides to perform an experimental study where she seeks the answer to the question:

Does intermittent fasting increase human lifespan?

To understand the wanderer statistician’s study and its results’ significance, we need to understand the terms below:

1. Null and Alternative Hypothesis
2. Standard Normal Distribution and Z-score
3. P-values
4. Significance level

Don’t feel intimidated by them, I will keep my word for simplicity. 😉

1. Null and Alternative Hypothesis

A hypothesis is a statement about the relationship between variables. The hypothesis test is a way to test the results of a hypothesis, an observation, a claim, or an experiment to see if your results are statistically significant. The hypothesis test is constructed around two contrasting claims: Null hypothesis and alternative hypothesis.

The null hypothesis either describes the current status of the population/sample or states that there is no difference between the sample and population parameters. The alternative hypothesis, the opposite of the null hypothesis, either describes the future or desired state of the population/sample or states that there is a difference between the sample and population parameters.

Our statistician sets the null and alternative hypothesis for her study as follows:

• Null hypothesis: Intermittent fasting does not increase the average life expectancy. (µ ≤ 73)
• Alternative hypothesis: Intermittent fasting increases the average life expectancy. (µ >73)

Her goal is to observe if the intermittent fasting leads to an increase (from 73 to 76.) of the average human lifetime, which makes the hypothesis test one-tailed directional test.

2. Standard Normal Distribution and Z-score

Many theorems assume a population or a sample follows a normal distribution. The widespread use of normal distribution derives from the easiness of assessing data points in terms of the standard deviation. Let’s remember the important properties of the normal distribution:

• It is a continuous and symmetrical distribution.
• 68% of data lies within ± 1, 95% of data lies within ± 2 and 99.7% of data lies within ± 3 standard deviations from the mean.

The standard normal distribution is a special normal distribution where the mean is 0 and the standard deviation is 1. Any normal distribution can be transformed into a standard normal distribution by subtracting the mean from each data point and dividing the subtraction by the standard deviation. This process is called standardization.

Z-score is the result of a calculation applied to a data point during the standardization. In the standard normal distribution above, z-scores are on the x-axis (positive z-score indicates the value is above the mean and negative z-score indicates the value is below the mean), whereas cumulative probability values are on the Cumulative % bar and under the distribution.

Z-score defines the data point’s location in terms of its distance to the mean which is measured by the standard deviation. For example, by looking at the z-score and Cumulative % above:

• The probability of a random variable being less than or equal to 1 is 84.1% (P(x ≤ 1) = 0.841).
• The probability of a random variable being greater than or equal to 2 is 2.3% (P(x ≥ 2) = 0.023)

We can calculate the probability value, abbreviated as p-value, from a z-score or vice-versa.

3. P-values

P-value is the probability of obtaining test results at least as extreme as the results actually observed during the test, assuming that the null hypothesis is correct. [Source: Wikipedia]. I put it in more simple terms:

In a world where the null hypothesis is true, p-value indicates the percentage of random chance or noise involved in getting the test results at least as extreme as actual results.

Let’s analyze p-value further in our context:

• France is a sample with the mean (x̄) 83 from a normal-like distribution with a mean (µ) 73 and the standard deviation (σ) 7.
• When we calculate the z-score (83–73)÷7, it is 1.42. This places France 1.42 points above the mean.
• Our world, thus the null hypothesis is: “The average life expectancy in France is higher than the global average by 1.42 points.”

Say, we took a sample from France and found out that average life expectancy is actually lower than the global average by 1 point with a p-value of 0.16. This means, in a world where the null hypothesis is true, there is a 16% random chance that we would measure the average life expectancy in France at least 1 point lower than the global average.

In other words, if the average life expectancy in France is actually higher than the global average by 1.42 points, we would still measure it is lower by at least 1 point about 1/6 of the time due to noise. Thus, the lower the p-value the more significant the results are since it is less likely to be caused by random chance or noise.

P-value can be calculated with the z-table or a library from R or Python. Our curious statistician will take some help from the scipy stats package of Python and the norm object from the same library to denote the standard normal distribution. Furthermore, to calculate the p-value she will use the cumulative density function (cdf) method from the norm object. This method will yield the same probability values for a given z-score as in the Cumulative % in the standard normal distribution above.

To complete her experimental study the only missing piece left is the significance level — denoted with α.

4. Significance level

The significance level (α) is the threshold p-value set to decide if the test results are statistically significant. It also represents how much of randomness we can tolerate in our test/study. The significance level is usually set to 0.05, 0.01 or 0.001.

If the test result’s p-value is less than the significance level (α), then we can conclude that the obtained test results are statistically significant and they are not due to a random chance or noise.

It is important to set the significance level before starting the experiment. Otherwise, it won’t be an objective test. Our curious statistician also knows that and sets the significance level to 0.05. Finally, she is going to complete the study and observe the statistical significance in action!

Steps performed so far by our wanderer statistician:

1. She decided to perform an experimental study to answer the question:

• Does intermittent fasting increase human lifespan?

2. She determined the null and alternative hypothesis:

• Null hypothesis: Intermittent fasting does not increase the average life expectancy. (µ ≤ 73)
• Alternative hypothesis: Intermittent fasting increases the average life expectancy. (µ >73)

3. She set the test statistics and calculated the p-value:

• The discovery of 50 healthily-aged people is just another sample from the average life expectancy distribution in the world. So, she determined the test statistics as the average life expectancy on the island where everyone is doing intermittent fasting and calculated the p-value as:

4. She determined the significance level at 0.05:

• The region on the right-hand side of the α is called the critical region. It denotes the area of p-values smaller than the significance level. If the test statistics fall in this region, she will conclude that the test results are statistically significant. (Z-score of p-value 0.05 for a one-tailed directional test is 1.64)

When she puts all the pieces together to uncover the path of a healthier and longer life:

Test statistics fell into the critical region, it means the test results are statistically significant and she rejects the null hypothesis!

There is statistically significant evidence that the intermittent fasting increases the average life expectancy, at significance level 0.05. Obtained test results are not due to a random chance.

In a world where the average life expectancy is 73, the probability (p-value) of getting 50 sample with mean 76 due to random noise is 0.0015 which is less than the 0.05.

Whether or not the test results are called statistically significant depends on the significance level. With the p-value set to 0.001, the test statistics would no longer fall into the critical region. (Z-score of p-value 0.001 for a one-tailed directional test is 3.09)

In that case, we would have concluded that test results are not statistically significant and accepted the null hypothesis. (Intermittent fasting does not increase the average life expectancy)

Practical Applications of the Statistical Significance

The test of our curious statistician is called a z-test. If our sample size had been less than 30 or there had been two samples to compare, then this would be a t-test. If we had compared two or more samples this would be an ANOVA (analysis of variance) test.

Hypothesis tests and statistical significance are used in almost every discipline from social sciences to physics. We can use them to compare the performances of different models in the context of machine learning. We can confidently perform a hypothesis test and determine which one performs better using statistical significance. Here is an article that I like about the use of statistical significance in machine learning:

Compare which Machine Learning Model performs Better

Thanks for reading! If you want to have a closer look at the “action” of the statistical significance, you can check out this Deepnote Notebook. Source of my catchy example is from the Gapminder’s Open Numbers which can be found here.

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin! Stay safe and healthy 👋

Statistical Significance In Action was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

And examples from its practical applications

Statistics is a must-have knowledge for a data scientist. Central Limit Theorem is the cornerstone of it.

I learn better when I see any theoretical concept in action. I believe there are more people like me out there, so I will explain Central Limit Theorem with a concrete and catchy example today — hoping to make it permanent in your mind for your use.

Do you remember what is the average life expectancy in the world today? (Hint: check out here if you don’t)

It is 73.

This is a number summarizing the life expectancy data all over the world, to be precise 186 countries. This number will tell us more if we put it into the full context.

The context is comprised of statistical terms, so let’s first understand them:

Population: The set that contains all data of elements, individuals or measurements from your experimenting space. Our experimenting space is world and population is the life expectancy data from all countries — population size is denoted by N.

Sample: It is a randomly selected subset from the population — sample size is denoted by n.

Distribution: It describes the data/population/sample range and how data is spread in that range. Distribution of life expectancy in the world as of 2018, is as follows:

Mean: Average value of all data from your population or sample. For this example, population mean is 73 — denoted by µ (Mu) for populations and x̄ (x bar) for samples.

Standard Deviation: Not all countries’ average life expectancy is 73. They are distributed in a range as we see above. Standard deviation is a measure of how spread your population is — denoted by σ (Sigma).

Normal Distribution — Bell-Shaped Curve — Gaussian Distribution: When your population is spread perfectly symmetrical with σ standard deviations around the mean value, you get the following bell-shaped curve:

We scratched the surface of the necessary statistical terms to be used in explaining Central Limit Theorem, now let’s see the theorem in action!

What kind of distribution is average life expectancy in the world as of 2018? Is it normally distributed or is it some other distribution?

Well, it is not a perfectly normal distribution, since it seems to be pulled to the left side, known as left-skewed distribution.

Now let’s take some samples from the population. In other words, let’s take 1000 subsets of size 150 from the population and plot the distribution of each sample’s mean (x̄). This process is called the sampling distribution:

This is a perfect normal distribution! Actually, any sampling distribution from any population shows normal distribution because Central Limit Theorem states that:

Regardless of the initial shape of the population distribution, sampling distribution will approximate to a normal distribution. As the sample size increases, sampling distribution will get narrower and more normal.

This video explains the theorem unexpectedly interesting way using rabbits and dragon wings 🙂

Coming back to our explanation of the Central Limit Theorem, it allows us to calculate the standard deviation — known as the standard error — of the sampling distribution with the following formula:

Let’s look if this calculation also holds for our population and sampling distribution created:

As expected it holds! The intuitive explanation of this formula is; as sample size increases, the sampling distribution will become a more accurate representation of the population parameters (mean and standard deviation).

Bonus Theorem: Did you recognize where the center of the sampling distribution is? Do you think that it is a coincidence that both the sampling distribution and population mean is 73?

Of course not! This is explained by another fundamental theorem in statistics: Law of large numbers. As the more samples are drawn from a population and as the sample size increases, the mean of the sampling distribution will get closer to the population mean.

Practical Applications of Central Limit Theorem

In any machine learning problem, the given dataset represents a sample from the whole population. Using this sample, we try to catch the main patterns in the data. Then, we try to generalize the patterns in the sample to the population while making the predictions. Central limit theorem helps us to make inferences about the sample and population parameters and construct better machine learning models using them.

Moreover, the theorem can tell us whether a sample possibly belongs to a population by looking at the sampling distribution. The power of the theorem lies here in too.

When I typed the first words for this blog post, I intended to write about statistical significance and hypothesis testing but I realized that Central Limit Theorem is at the heart of explaining them. So, I will leave them to the next article. Follow my Medium account to learn about them 😉

Thanks for reading! If you want to have a closer look at the “action” of the theorem, you can check out this Deepnote Notebook. Source of my catchy example is from the Gapminder’s Open Numbers which can be found here.

If you enjoyed this one, you can view some of my other articles here:

• Why I Decided to Become a Data Scientist
• Recipes for the Visualizations of Data Distributions
• Predicting Forest Cover Types with the Machine Learning Workflow
• Customer Segmentation with Machine Learning

For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!

Sources:

A Gentle Introduction to the Central Limit Theorem for Machine Learning - Machine Learning Mastery

Central Limit Theorem In Action was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.