Step 1: Set Up the Environment

1.1 Create a New Project in CML

1. Log into Cloudera Machine Learning (CML):

• Access the CML dashboard through your CDP account.

1. Create a New Project:

• On the CML dashboard, click on the “Create Project” button.
• Name the project “Customer Churn Prediction”.
• Select the appropriate workspace or environment.

1.2 Set Up the Workbench

1. Start a New Session:

• In your newly created project, click on “Start Session.”
• Choose Python 3 as the language for the session.
• Configure the session with the necessary resources (CPU, memory).

1. Install Required Python Packages:

• Once the session is running, open a terminal or create a new notebook in the session.
• Install the following packages by running the command:

!pip install pandas scikit-learn mlflow matplotlib

Step 2: Data Preparation

2.1 Upload Dataset

1. Obtain Dataset:

• Download a customer churn dataset, such as the Telco Customer Churn dataset from Kaggle. https://www.kaggle.com/datasets/blastchar/telco-customer-churn

1. Upload Dataset to CML:

• Go to your project, click on “Data,” and upload the dataset file (e.g., Telco-Customer-Churn.csv).

2.2 Data Preprocessing

1. Load the Dataset:

• In a new notebook, load the dataset using pandas:

import pandas as pd
df = pd.read_csv("Telco-Customer-Churn.csv")

Explore the Data:

• Display the first few rows to understand the structure:

df.head()

Handle Missing Values:

• Identify and handle any missing values in the dataset:

df.isnull().sum()
df.dropna(inplace=True)

Encode Categorical Variables:

• Convert categorical variables into numerical values using one-hot encoding:

df = pd.get_dummies(df, drop_first=True)

Split the Data:

• Split the data into features (X) and target (y) variables, and then into training and testing sets:

from sklearn.model_selection import train_test_split

X = df.drop('Churn_Yes', axis=1)
y = df['Churn_Yes']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 3: Model Training

3.1 Train the Model

1. Train a Logistic Regression Model:

• Train a simple logistic regression model:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score

model = LogisticRegression()
model.fit(X_train, y_train)

Evaluate the Model:

• Evaluate the model on the test data:

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)

print(f"Accuracy: {accuracy}")
print(f"Precision: {precision}")
print(f"Recall: {recall}")

Log Model with MLFlow

1. Set Up MLFlow Tracking:

• Initialize MLFlow in your notebook:

import mlflow
import mlflow.sklearn

mlflow.set_experiment("Customer_Churn_Prediction")

Log the Model and Metrics:

• Log the model, metrics, and parameters to MLFlow:

with mlflow.start_run():
    mlflow.sklearn.log_model(model, "model")
    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_metric("precision", precision)
    mlflow.log_metric("recall", recall)
    mlflow.log_param("model_type", "Logistic Regression")

Step 4: Model Deployment

4.1 Deploy the Model in CML

1. Deploy as a REST API:

• In CML, navigate to “Models” and click on “Deploy New Model.”
• Select the model logged in MLFlow from the “Customer_Churn_Prediction” experiment.
• Name the model deployment, and select the appropriate Python function (e.g., predict) to deploy the model as an API endpoint.

1. Test the API Endpoint:

• Once deployed, test the API by sending a sample input to the endpoint and verifying the output:

import requests
import json

url = "https://<CML_model_endpoint>/predict"
payload = json.dumps({"data": X_test.iloc[0].tolist()})
headers = {'Content-Type': 'application/json'}

response = requests.post(url, data=payload, headers=headers)
print(response.json())

Step 5: Model Monitoring

5.1 Set Up Monitoring

1. Monitor Performance in CML:

• Use CML’s model monitoring features to track key metrics like accuracy, precision, and recall over time.
• Set up alerts to notify you if the model performance degrades.

1. Retraining Pipeline:

• Set up a CI/CD pipeline in GitLab integrated with CML to automate model retraining and deployment when new data is available.

Step 5: Model Monitoring and CI/CD Pipeline Setup Using GitHub Actions

5.2 Setting Up a CI/CD Pipeline in GitHub

Step 5.2.1: Setting Up GitHub Repository

1. Create a New GitHub Repository:

• Log in to GitHub and create a new repository for the project. Name it something relevant, like customer-churn-mlops.

1. Push Your Project Files:

• Clone the repository to your local machine or directly push the project files from your local environment to the GitHub repository:

git clone https://gitlab.com/yourusername/customer-churn-mlops.git
cd customer-churn-mlops
git add .
git commit -m "Initial commit"
git push origin main

Organize Project Structure:

• Structure your repository with the following files and directories:

customer-churn-mlops/

customer-churn-mlops/
├── data/
│   └── Telco-Customer-Churn.csv
├── models/
├── notebooks/
│   └── churn_model_training.ipynb
├── scripts/
│   └── train_model.py
├── .gitlab-ci.yml
├── requirements.txt
└── README.md

Step 5.2.2: Create Python Scripts for Training and Deployment

1. Create train_model.py Script:

• Create a Python script under the scripts/ directory to handle model training and logging with MLFlow:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score
import mlflow
import mlflow.sklearn

def load_data():
    df = pd.read_csv("../data/Telco-Customer-Churn.csv")
    df = pd.get_dummies(df, drop_first=True)
    return df

def train_and_log_model():
    df = load_data()
    X = df.drop('Churn_Yes', axis=1)
    y = df['Churn_Yes']
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    model = LogisticRegression()
    model.fit(X_train, y_train)

    accuracy = accuracy_score(y_test, model.predict(X_test))
    precision = precision_score(y_test, model.predict(X_test))
    recall = recall_score(y_test, model.predict(X_test))

    mlflow.set_experiment("Customer_Churn_Prediction")

    with mlflow.start_run():
        mlflow.sklearn.log_model(model, "model")
        mlflow.log_metric("accuracy", accuracy)
        mlflow.log_metric("precision", precision)
        mlflow.log_metric("recall", recall)

if __name__ == "__main__":
    train_and_log_model()

Create requirements.txt:

• List all the dependencies required for the project:

pandas
scikit-learn
mlflow

Step 5.2.3: Configure GitHub Actions Workflow

1. Create GitHub Actions Workflow:

• Create a .github/workflows/ci-cd.yml file in your repository. This file will define the CI/CD pipeline:

image: python:3.9

stages:
  - test
  - deploy

before_script:
  - pip install -r requirements.txt

test_model:
  stage: test
  script:
    - python scripts/train_model.py

deploy_model:
  stage: deploy
  script:
    - |
      mlflow models serve -m "models:/Customer_Churn_Prediction/Production" -h 0.0.0.0 -p 5000

Explanation:

• name: Names the workflow.
• on: Specifies the trigger events (push, pull requests, or a scheduled run).
• jobs: Defines the tasks to be executed.
• build-and-deploy: The main job in the workflow.
• steps: Lists each step in the pipeline:
• Checkout code: Checks out the repository code.
• Set up Python: Sets up the Python environment.
• Install dependencies: Installs the required Python packages.
• Train and log model: Runs the training script to log the model in MLFlow.
• Deploy model: Deploys the model using MLFlow’s model serving functionality.

· Configure Secrets in GitHub:

• Go to your GitHub repository, navigate to Settings > Secrets and variables > Actions, and add any required secrets:
• MLFLOW_TRACKING_URI: The tracking URI for your MLFlow server.
• MLFLOW_S3_ENDPOINT_URL, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY if you’re using S3 as a backend for MLFlow.

· Trigger the Workflow:

• Push the changes to GitHub, and the pipeline will automatically trigger based on the defined events:

git add .
git commit -m "Added CI/CD pipeline configuration"
git push origin main

1.

• Monitor the workflow’s progress under the Actions tab in your GitHub repository.

Step 5.2.4: Automate Model Retraining and Deployment

1. Schedule Regular Retraining:

• The cron syntax in the workflow file schedules the pipeline to run at regular intervals. Adjust it as needed for your project (e.g., daily, weekly).

1. Trigger Retraining on Data Updates:

• Optionally, set up a process where pushing new data to the repository triggers the retraining pipeline automatically.

Final Outcome:

With this GitHub Actions CI/CD pipeline, your MLOps process is automated. The pipeline will:

1. Automatically train the model whenever there’s a change in the code or data.
2. Log the trained model and metrics to MLFlow.
3. Deploy the latest model using MLFlow’s serving feature.

This setup ensures that your machine learning model is continuously updated and deployed, keeping your production environment aligned with the latest data and code.