Introduction to Amazon SageMaker

This article summarises what I learnt during a one-day Amazon SageMaker workshop in May 2022. We explored the machine learning lifecycle, from preparing datasets to deploying and monitoring models.

The examples reflect the workshop’s environment and software versions. Some APIs, instance types and dependencies may have changed since then. The code snippets also assume that the shared variables and imports in the workshop notebooks have already been configured.

Introduction

Building a machine learning application involves much more than training a model. Data preparation, feature engineering, deployment and monitoring can all take significant time.

Amazon SageMaker brings these activities together through managed services and a development interface called SageMaker Studio.

The workshop covered four main stages:

• Prepare data.
• Build models.
• Train and tune models.
• Deploy models.

We also explored tools for automating parts of this workflow, including SageMaker Canvas, Autopilot, Debugger and Model Monitor.

One of the organisers demonstrated SageMaker Canvas using the Titanic dataset. The goal was to predict whether a passenger would survive the disaster. Canvas provides a visual interface for building models without writing the full training pipeline yourself.

Data preparation

Before training a model, we need to collect, inspect, clean and transform its input data. This often accounts for a substantial part of a machine learning project.

Typical preparation tasks include:

• Converting data into the format expected by the algorithm.
• Transforming features into useful representations.
• Identifying missing values, outliers and inconsistencies.
• Splitting the dataset into training, validation and test sets.
• Exporting the prepared data for training.

Many algorithms require numerical inputs, so categorical features may need encoding. One-hot encoding is one common approach, although the appropriate method depends on the algorithm and dataset.

SageMaker training jobs can receive separate input channels, such as training and validation channels. These channels can point to datasets stored in Amazon S3.

SageMaker Data Wrangler

SageMaker Data Wrangler provides a visual workflow for importing, exploring and transforming data.

For example, we can import a CSV file, inspect feature distributions, identify outliers and apply transformations. A quick model assessment can help us understand whether the prepared features contain useful predictive information.

The resulting preparation workflow can then be exported or incorporated into a repeatable pipeline.

Jupyter notebooks remain useful when we need more control over the analysis. Other data types, such as audio files and images, can also be stored in S3 and processed through suitable workflows.

SageMaker Feature Store

Without a feature store, teams may repeat similar feature engineering work for every new model.

SageMaker Feature Store provides a central place to publish and reuse features across teams and projects. It can also help maintain consistency between the features used during training and those used during inference.

It supports two storage modes:

• Online storage: designed for low-latency feature retrieval during real-time inference.
• Offline storage: designed for historical analysis, model training and batch workflows.

The main benefit is reuse: build a feature once, then make it available to multiple models.

Workshop setup

The workshop used AWS Event Engine to provide temporary access to an AWS environment:

https://dashboard.eventengine.run/login

Participants entered an event hash supplied by the organisers. The workshop materials were shared at:

https://tinyurl.com/ntucontlab

The sample repository can be cloned with:

git clone https://github.com/aws-samples/amazon-sagemaker-immersion-day.git

For the first lab, we selected “Option 2: NumPy and Pandas”.

Preparing the bank marketing dataset

The first dataset concerned a bank’s direct marketing campaign.

The objective was to predict whether a customer would subscribe to a term deposit. This could help the bank prioritise potential customers and use its marketing resources more effectively.

The features included:

• Demographic information, such as age and occupation.
• Customer information, such as housing loans and personal loans.
• Previous campaign interactions.
• Information about the latest marketing contact.

The target variable was:

“Has the client subscribed to a term deposit?”

The possible values were “yes” and “no”.

Exploring the data

We started by inspecting categorical feature frequencies and numerical feature distributions:

# Frequency tables for categorical features
for column in data.select_dtypes(include=['object']).columns:
    display(
        pd.crosstab(
            index=data[column],
            columns='% observations',
            normalize='columns'
        )
    )

# Summary statistics and histograms for numerical features
display(data.describe())
%matplotlib inline
hist = data.hist(bins=30, sharey=True, figsize=(10, 10))

One important observation was that most customers did not subscribe to a term deposit.

This class imbalance matters when evaluating the model. A model that always predicts “no” could achieve a superficially high accuracy while offering little practical value. Precision, recall, F1 score and precision–recall AUC may therefore be more informative than accuracy alone.

Transforming the features

Next, we prepared the features.

The dataset used a special value of 999 in the “pdays” column to indicate that a customer had not previously been contacted. We created a separate indicator for this condition.

We also created a feature identifying customers who were students, retired or unemployed, then converted categorical features into numerical columns:

data['no_previous_contact'] = np.where(
    data['pdays'] == 999,
    1,
    0
)

data['not_working'] = np.where(
    data['job'].isin(['student', 'retired', 'unemployed']),
    1,
    0
)
model_data = pd.get_dummies(data, dtype=int)

We also needed to consider whether each feature would be available when making a real prediction.

For example, call duration is only known after the call has finished. If our goal is to decide whom to contact beforehand, including it would introduce information that is unavailable at prediction time.

The workshop removed call duration and several economic variables:

model_data = model_data.drop(
    [
        'duration',
        'emp.var.rate',
        'cons.price.idx',
        'cons.conf.idx',
        'euribor3m',
        'nr.employed'
    ],
    axis=1
)

The decision to remove the economic variables was specific to the workshop’s use case. In a real project, we would assess whether those values were available and reliable at the intended prediction time.

Splitting the dataset

We randomly divided the dataset into three groups:

• 70% for training.
• 20% for validation.
• 10% for testing.

train_data, validation_data, test_data = np.split(
    model_data.sample(frac=1, random_state=1729),
    [
        int(0.7 * len(model_data)),
        int(0.9 * len(model_data))
    ]
)

The training set is used to fit the model. The validation set supports model selection and tuning. The test set should remain untouched until the final evaluation.

For an imbalanced classification problem, a stratified split is often preferable because it preserves the class proportions across the datasets. If the prediction problem is time-dependent, a chronological split may be more appropriate.

Exporting the data to S3

The SageMaker XGBoost container used in the workshop expected headerless CSV files, with the target value in the first column.

We prepared and uploaded the training and validation datasets:

pd.concat(
    [
        train_data['y_yes'],
        train_data.drop(['y_no', 'y_yes'], axis=1)
    ],
    axis=1
).to_csv('train.csv', index=False, header=False)

pd.concat(
    [
        validation_data['y_yes'],
        validation_data.drop(['y_no', 'y_yes'], axis=1)
    ],
    axis=1
).to_csv('validation.csv', index=False, header=False)
s3 = boto3.Session().resource('s3')
s3.Bucket(bucket).Object(
    os.path.join(prefix, 'train/train.csv')
).upload_file('train.csv')
s3.Bucket(bucket).Object(
    os.path.join(prefix, 'validation/validation.csv')
).upload_file('validation.csv')

Building, training and deploying models

The workshop introduced three ways to train a model in SageMaker:

1. Use a built-in algorithm.
2. Bring your own training script.
3. Bring your own container.

These options provide increasing flexibility.

With a built-in algorithm, we configure the training job without implementing the algorithm ourselves. With our own script, we control the training logic while using a supported framework container. With our own container, we control the runtime environment as well.

A typical training workflow looks like this:

• Store training data in S3.
• Select a training container image.
• Configure and launch a training job.
• Save the trained model artefacts to S3.
• Deploy the model using an inference container.

Amazon Elastic Container Registry, or ECR, stores container images. The trained model artefacts are normally stored in S3, not ECR.

Using a built-in algorithm

For the bank marketing dataset, we used gradient-boosted trees.

Gradient boosting combines a sequence of relatively simple models. Each new model attempts to improve on the errors made by the existing ensemble.

XGBoost is a widely used implementation of this approach.

To configure a SageMaker estimator, we specified:

• The XGBoost container image.
• An IAM execution role.
• The training instance type and count.
• The S3 location for model output.
• The algorithm’s hyperparameters.

sess = sagemaker.Session()

xgb = sagemaker.estimator.Estimator(
    image_uri=container,
    role=role,
    instance_count=1,
    instance_type='ml.m4.xlarge',
    output_path='s3://{}/{}/output'.format(bucket, prefix),
    sagemaker_session=sess
)
xgb.set_hyperparameters(
    max_depth=5,
    eta=0.2,
    gamma=4,
    min_child_weight=6,
    subsample=0.8,
    silent=0,
    objective='binary:logistic',
    num_round=100
)
xgb.fit({
    'train': s3_input_train,
    'validation': s3_input_validation
})

The fit() call supplies the input data channels. The output location is configured separately through output_path.

After training, we deployed the model behind a real-time inference endpoint:

xgb_predictor = xgb.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge'
)

Deployment makes the model available for predictions. It does not replace evaluation: we should still assess its performance on the held-out test set before relying on it.

Bringing your own script

If we already have training code written in a framework such as TensorFlow or PyTorch, we can adapt that script to run in SageMaker.

AWS provides framework containers that include supported dependencies. Depending on the selected image, these may include GPU drivers, CUDA libraries and other performance-related components.

Training and inference may use different images because their runtime requirements differ.

Supported training containers use tooling such as the SageMaker Training Toolkit to pass configuration to the script, including hyperparameters and data locations.

This approach gives us control over the model code without requiring us to build the entire container environment ourselves.

Bringing your own container

For greater control, we can build a custom Docker image.

The workshop used a decision-tree example with the following files:

• nginx.conf: configures the Nginx front end.
• predictor.py: implements the Flask application and prediction logic.
• serve: starts the inference-serving processes, including Gunicorn.
• train: runs the training algorithm.
• wsgi.py: exposes the Flask application to the WSGI server.

For this example, the two files most likely to require changes were train and predictor.py.

Building the Docker image

The workshop’s Dockerfile installed Python dependencies and the serving stack, then copied the application into the image:

FROM ubuntu:18.04

LABEL maintainer="Amazon AI <sage-learner@amazon.com>"
RUN apt-get -y update && apt-get install -y --no-install-recommends \
    wget \
    python3-pip \
    python3-setuptools \
    nginx \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN ln -s /usr/bin/pip3 /usr/bin/pip
RUN pip --no-cache-dir install \
    numpy==1.16.2 \
    scipy==1.2.1 \
    scikit-learn==0.20.2 \
    pandas \
    flask \
    gunicorn
ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PATH="/opt/program:${PATH}"
COPY decision_trees /opt/program
WORKDIR /opt/program

PYTHONUNBUFFERED allows logs to appear promptly, while PYTHONDONTWRITEBYTECODE prevents Python from generating unnecessary bytecode files.

The final lines copy the application into the image and set its working directory.

This Dockerfile is a historical workshop example. Its base image and dependencies are old and should be replaced with supported, security-reviewed versions before production use.

We then made the training and serving scripts executable and built the image:

cd lab03_container

chmod +x decision_trees/train
chmod +x decision_trees/serve
sm-docker build . --repository sagemaker-decision-trees:latest

The workshop’s build tool pushed the resulting image to ECR and returned an image URI in this format:

<account-id>.dkr.ecr.ap-southeast-1.amazonaws.com/sagemaker-decision-trees:latest

Uploading training data

We used the SageMaker Python SDK to upload the local dataset directory to S3:

WORK_DIRECTORY = 'lab03_data'

data_location = sess.upload_data(
    WORK_DIRECTORY,
    key_prefix=S3_prefix
)

Training with the custom container

Next, we created an estimator using our ECR image.

The main configuration values were:

• image_uri: the container image in ECR.
• role: the SageMaker execution role.
• instance_count: the number of training instances.
• instance_type: the training instance type.
• output_path: the S3 location for the trained model artefacts.
• sagemaker_session: the SageMaker session.

account = sess.boto_session.client(
    'sts'
).get_caller_identity()['Account']

region = sess.boto_session.region_name
image_uri = (
    '{}.dkr.ecr.{}.amazonaws.com/'
    'sagemaker-decision-trees:latest'
).format(account, region)
tree = sagemaker.estimator.Estimator(
    image_uri=image_uri,
    role=role,
    instance_count=1,
    instance_type='ml.c4.2xlarge',
    output_path='s3://{}/output'.format(
        sess.default_bucket()
    ),
    sagemaker_session=sess
)
file_location = data_location + '/iris.csv'
tree.fit(file_location)

The fit() call submits a training job through SageMaker’s CreateTrainingJob API.

Deploying the custom model

After training completed, we deployed the model and configured a CSV serializer:

from sagemaker.serializers import CSVSerializer

predictor = tree.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge',
    serializer=CSVSerializer()
)

The deploy() method creates the model and endpoint configuration, then launches the endpoint.

Get Timothy Shan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Endpoints can be viewed in the SageMaker console or Studio. Their invocation URLs follow this pattern:

https://runtime.sagemaker.<region>.amazonaws.com/endpoints/<endpoint-name>/invocations

This is not an anonymous public web endpoint. Invocation requires appropriate AWS authentication and permissions. The SDK handles request signing when configured with valid credentials.

Cleaning up

Real-time endpoints can continue incurring charges while they remain provisioned, even when they are idle. Free-tier eligibility depends on the account, resource and applicable offer, so it should not be assumed.

When the endpoint is no longer needed, delete it:

predictor.delete_endpoint()

Deleting an endpoint does not necessarily remove its model, endpoint configuration, ECR image or S3 artefacts. Review and remove those resources separately if they are no longer required.

The following notebook command removes only the local workshop directories:

!rm -rf lab03_container lab03_data

AutoML with SageMaker Autopilot

SageMaker Autopilot automates several stages of model development.

Given a supported dataset and a target column, it can perform tasks such as:

• Data analysis.
• Training and validation splitting.
• Feature preprocessing.
• Model selection.
• Hyperparameter tuning.

The workshop’s configuration could take several hours to complete, depending on the dataset and search settings.

Canvas provides a visual interface for automated model-building workflows, while Autopilot can be accessed programmatically.

Preparing the dataset

For supported tabular workflows, Autopilot handles many common preprocessing tasks, including missing-value treatment and categorical encoding.

However, automated preprocessing does not remove the need to understand the data, identify leakage or choose meaningful evaluation criteria.

It is also useful to reserve an independent test set.

Continuing with the bank marketing dataset:

train_data = data.sample(
    frac=0.8,
    random_state=200
)

test_data = data.drop(train_data.index)
test_data_no_target = test_data.drop(
    columns=['y']
)

Autopilot can manage the training and validation split within the supplied training data. We retain the test set for a separate evaluation.

Launching an Autopilot job

The workshop’s tabular example used CSV data stored in S3.

The key inputs were:

• The S3 location of the input dataset.
• The S3 location for output artefacts.
• The target column name.
• An IAM execution role.

In this bank marketing example, the target column was “y”. For the Titanic demonstration, the target would instead be the dataset’s survival label.

We could also specify the problem type, such as regression, multiclass classification or binary classification.

from time import gmtime, strftime, sleep

timestamp_suffix = strftime(
    '%d-%H-%M-%S',
    gmtime()
)
auto_ml_job_name = (
    'automl-banking-' + timestamp_suffix
)
print('AutoMLJobName: ' + auto_ml_job_name)
sm.create_auto_ml_job(
    AutoMLJobName=auto_ml_job_name,
    InputDataConfig=input_data_config,
    OutputDataConfig=output_data_config,
    AutoMLJobConfig=autoMLJobConfig,
    AutoMLJobObjective=autoMLJobObjective,
    ProblemType='BinaryClassification',
    RoleArn=role
)

Supported formats, dataset limits and APIs depend on the chosen Autopilot workflow and version. Check the relevant documentation before reusing this historical example.

Understanding the job stages

The workshop described three high-level stages:

1. Data analysis: inspect the dataset and identify candidate machine learning pipelines.
2. Feature engineering: transform individual features and create suitable model inputs.
3. Model tuning: evaluate candidate pipelines and tune their hyperparameters.

We polled the job status to track progress:

while True:
    describe_response = sm.describe_auto_ml_job(
        AutoMLJobName=auto_ml_job_name
    )

    job_run_status = describe_response[
        'AutoMLJobStatus'
    ]
    print(
        job_run_status
        + ' - '
        + describe_response['AutoMLJobSecondaryStatus']
    )
    if job_run_status in (
        'Failed',
        'Completed',
        'Stopped'
    ):
        break
    sleep(30)
if job_run_status != 'Completed':
    raise RuntimeError(
        describe_response.get(
            'FailureReason',
            'Autopilot job ended with status: '
            + job_run_status
        )
    )

Inspecting the best candidate

After the job completed, we retrieved the best candidate:

best_candidate = sm.describe_auto_ml_job(
    AutoMLJobName=auto_ml_job_name
)['BestCandidate']

best_candidate_name = best_candidate[
    'CandidateName'
]
objective_metric = best_candidate[
    'FinalAutoMLJobObjectiveMetric'
]
print(best_candidate)
print(
    'CandidateName: '
    + best_candidate_name
)
print(
    'FinalAutoMLJobObjectiveMetricName: '
    + objective_metric['MetricName']
)
print(
    'FinalAutoMLJobObjectiveMetricValue: '
    + str(objective_metric['Value'])
)

“Best” means best according to the configured objective metric. Before deployment, we should still evaluate the candidate against the business requirements and independent test data.

Debugging training jobs

SageMaker Debugger helps inspect supported training jobs by collecting tensors and metrics.

This can help diagnose issues such as loss failing to improve or unexpected training behaviour.

Configuring data collection

We enabled collection by adding a DebuggerHookConfig to an estimator:

from sagemaker.debugger import (
    DebuggerHookConfig,
    CollectionConfig
)

debugger_hook_config = DebuggerHookConfig(
    s3_output_path='s3://{}/debugger'.format(
        bucket
    ),
    collection_configs=[
        CollectionConfig(
            name='metrics',
            parameters={
                'save_interval': '10'
            }
        )
    ]
)

The S3 output path determines where debugging data is stored.

The collection configuration identifies the data to collect. In this example, we requested the “metrics” collection at a configured interval of 10 steps.

The amount of stored data depends on the model, dataset, selected collections and collection frequency.

Adding debugging rules

Collecting tensors does not automatically configure every diagnostic rule.

Rules must also be supplied when we want SageMaker to evaluate particular conditions.

Debugger supports built-in rules and custom rules. Built-in rules address common training problems; custom rules allow application-specific checks.

The workshop used the LossNotDecreasing rule:

from sagemaker.debugger import (
    rule_configs,
    Rule
)

rules = [
    Rule.sagemaker(
        rule_configs.loss_not_decreasing(),
        rule_parameters={
            'collection_names': 'metrics',
            'num_steps': '10'
        }
    )
]

The rule analyses the collected metrics according to its configuration. It is important to ensure that the selected collection contains loss metrics appropriate to the model.

An XGBoost debugging example

We combined metric collection and a rule in an XGBoost estimator:

from sagemaker.estimator import Estimator

sess = sagemaker.Session()
save_interval = 5
xgboost_estimator = Estimator(
    role=role,
    base_job_name=base_job_name,
    instance_count=1,
    instance_type='ml.m5.4xlarge',
    image_uri=container,
    max_run=1800,
    sagemaker_session=sess,
    debugger_hook_config=DebuggerHookConfig(
        s3_output_path=bucket_path,
        collection_configs=[
            CollectionConfig(
                name='metrics',
                parameters={
                    'save_interval': str(save_interval)
                }
            ),
            CollectionConfig(
                name='predictions',
                parameters={
                    'save_interval': str(save_interval)
                }
            ),
            CollectionConfig(
                name='feature_importance',
                parameters={
                    'save_interval': str(save_interval)
                }
            ),
            CollectionConfig(
                name='average_shap',
                parameters={
                    'save_interval': str(save_interval)
                }
            )
        ]
    ),
    rules=rules
)
xgboost_estimator.set_hyperparameters(
    max_depth=5,
    eta=0.2,
    gamma=4,
    min_child_weight=6,
    subsample=0.8,
    silent=0,
    objective='binary:logistic',
    num_round=100
)
xgboost_estimator.fit(
    {
        'train': s3_input_train,
        'validation': s3_input_validation
    },
    wait=False
)

Setting wait=False submits the job and returns control to the notebook while training continues.

For a supported configuration, SageMaker also starts the processing needed to evaluate the configured rules.

Checking training progress

We could inspect the training job through the SageMaker client:

import time
from time import gmtime, strftime

job_name = (
    xgboost_estimator.latest_training_job.name
)
client = (
    xgboost_estimator
    .sagemaker_session
    .sagemaker_client
)
print('Training job name: ' + job_name)
while True:
    description = client.describe_training_job(
        TrainingJobName=job_name
    )
    primary_status = description[
        'TrainingJobStatus'
    ]
    secondary_status = description[
        'SecondaryStatus'
    ]
    print(
        '{}: {}, {}'.format(
            strftime('%X', gmtime()),
            primary_status,
            secondary_status
        )
    )
    if primary_status in (
        'Completed',
        'Failed',
        'Stopped'
    ):
        break
    time.sleep(15)

Debugger’s built-in rules are not limited to resource utilisation. They also address common training issues. Custom rules are useful when the checks required by an application are not covered by the built-in options.

Custom rule documentation:

https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-custom-rules.html

Monitoring deployed models

Training diagnostics tell us what happened during training. Production monitoring helps us understand what happens after deployment.

SageMaker Model Monitor can analyse captured endpoint data and compare it against a baseline.

Capturing requests and responses

We enabled data capture when deploying an endpoint:

from sagemaker.model_monitor import (
    DataCaptureConfig
)

endpoint_name = (
    'DEMO-xgb-churn-pred-model-monitor-'
    + strftime('%Y-%m-%d-%H-%M-%S', gmtime())
)
print('EndpointName={}'.format(endpoint_name))
data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri=s3_capture_upload_path
)
predictor = model.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge',
    endpoint_name=endpoint_name,
    data_capture_config=data_capture_config
)

The model in this example comes from the monitoring lab.

Data capture can store request payloads, response payloads or both, along with metadata. The captured records are written to the configured S3 location.

A sampling percentage of 100 requests capture of all traffic, subject to the service’s capture behaviour and limits. For a production system, we would also consider storage costs, sensitive data and access controls.

Sending test traffic

We then sent sample requests to the endpoint:

from sagemaker.predictor import Predictor
import sagemaker
import time

predictor = Predictor(
    endpoint_name=endpoint_name,
    serializer=sagemaker.serializers.CSVSerializer()
)
!head -120 test_data/test-dataset-input-cols.csv > test_data/test_sample.csv
print(
    'Sending test traffic to the endpoint {}.'
    .format(endpoint_name)
)
with open('test_data/test_sample.csv', 'r') as f:
    for row in f:
        payload = row.rstrip('\n')
        response = predictor.predict(data=payload)
        time.sleep(0.5)
print('Done!')

Captured data is organised in S3 by endpoint, variant and invocation time.

Capture continues while it remains enabled and the endpoint receives traffic.

Creating a baseline

To detect changes, we first need a reference dataset.

We uploaded the baseline data to S3 and asked Model Monitor to generate baseline statistics and constraints:

baseline_prefix = prefix + '/baselining'
baseline_data_prefix = baseline_prefix + '/data'
baseline_results_prefix = baseline_prefix + '/results'

baseline_data_uri = 's3://{}/{}'.format(
    bucket,
    baseline_data_prefix
)
baseline_results_uri = 's3://{}/{}'.format(
    bucket,
    baseline_results_prefix
)
print('Baseline data URI: {}'.format(
    baseline_data_uri
))
print('Baseline results URI: {}'.format(
    baseline_results_uri
))
s3_key = os.path.join(
    baseline_data_prefix,
    'training-dataset-with-header.csv'
)
with open(
    'test_data/training-dataset-with-header.csv',
    'rb'
) as training_data_file:
    boto3.Session().resource('s3').Bucket(
        bucket
    ).Object(s3_key).upload_fileobj(
        training_data_file
    )
from sagemaker.model_monitor import (
    DefaultModelMonitor
)
from sagemaker.model_monitor.dataset_format import (
    DatasetFormat
)
my_default_monitor = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type='ml.m5.xlarge',
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600
)
my_default_monitor_baseline = (
    my_default_monitor.suggest_baseline(
        baseline_dataset=(
            baseline_data_uri
            + '/training-dataset-with-header.csv'
        ),
        dataset_format=DatasetFormat.csv(
            header=True
        ),
        output_s3_uri=baseline_results_uri,
        wait=True
    )
)

The baseline dataset must match the feature schema used by the monitoring configuration.

Creating a baseline does not, by itself, enable recurring monitoring. A monitoring schedule must also be configured to analyse captured traffic against the baseline.

Understanding drift

Consider a model developed using Singapore postal codes. If it later receives UK postcodes, the input format and distribution may differ significantly from the training data.

Data-quality monitoring can help identify this type of unexpected change.

However, detecting a change in input data is not the same as proving that predictive performance has deteriorated.

• Data drift concerns changes in the input distribution.
• Model-quality monitoring evaluates prediction performance, usually requiring ground-truth labels.
• Concept drift concerns changes in the relationship between inputs and the target.

These distinctions matter when deciding what to monitor and how to respond to an alert.

Conclusion

The workshop provided a practical introduction to the machine learning lifecycle in SageMaker.

My main takeaway was that training a model is only one part of the process. Reliable machine learning systems also need careful data preparation, appropriate evaluation, reproducible training, controlled deployment and ongoing monitoring.

SageMaker provides tools for each stage, while allowing different levels of control:

• Built-in algorithms for a quick start.
• Custom scripts for framework-level flexibility.
• Custom containers for control over the runtime.
• Autopilot for automated model exploration.
• Debugger for training diagnostics.
• Model Monitor for production checks.

The temporary workshop account was available for three days and did not include GPU access. To retain the work, we could export the notebooks and code or copy them into another environment.

My workshop repository is available here:

https://github.com/shanmo/amazon-sagemaker-immersion-day

The instructors also discussed broader cloud use cases, including Apple hardware through EC2 Mac instances and cloud-based media workflows.

That’s the end of the workshop, folks!