As machine learning models become more widely deployed, ML practitioners have shown increasing interest in MLOps. In our introductory blog, we give a brief background on how we think about MLOps at Edge Analytics.

So far in this series we have stored raw data and curated methods for accessing it, processed the data in preparation for modeling, trained and tuned a variety of predictive models, and tracked those models to find the ones that perform best. The last step we cover is how to get those models into the hands of users. This post is the last part of a five part series.

You can find the other blogs in the series by following the links below:

• Part 1: Data Storage with AWS S3 and Boto3
• Part 2: Data Processing Pipeline
• Part 3: Model Development
• Part 4: Model Tracking
• Part 5: Model Deployment

Running in a Jupyter notebook is typically not the end state of a trained model. To be useful in the wild, it must be deployed somewhere for inference. Where and how the model is deployed depends on how it is used. Some general questions to consider when figuring out how to serve a model to users include:

• Who needs access to the model outputs?
• How much does inference time matter?
• How often does the model need to be run?
• Where will the model be hosted?

There are many ways to package and deliver a TensorFlow, PyTorch, or SKLearn model, including solutions from a variety of cloud providers. When a user needs occasional access and has an active internet connection, we often set up an API endpoint on AWS Lambda. If we need a GPU for faster inference, we may also use AWS SageMaker. When the user needs continuous input and real-time output on an edge device, we might use TensorFlow Lite to package the model and deploy it to a microcontroller. We will briefly touch on each of these three deployment methods in this blog, with more focus on deployment using AWS SageMaker.

Model deployment in AWS Lambda

AWS Lambda is an event-driven, serverless compute resource. When creating an AWS Lambda function, the developer identifies a particular event that will trigger a provisioning of resources on AWS to run a model inference function. Because you are only charged for the time it takes to run your hosted function, this serverless method keeps running costs low for models that are only occasionally accessed. Lambda also handles scaling, memory allocation, and security, making it a safe and easy-to-use model deployment resource. One drawback is that Lambda does not currently offer GPU resources, which can be a bottleneck for model inference. See this blog we wrote for an in-depth look at getting started with AWS Lambda!

Model deployment in AWS SageMaker

Unlike AWS Lambda, with AWS SageMaker we can provision persistent instances to run models. These instances are always running, so you can avoid the cold start you may observe with code changes or updates to an AWS Lambda function. SageMaker offers many of the perks of AWS Lambda but with the added bonus of being a model-first service and providing large, GPU-based instances as options.

Recall that for our example MLOps pipeline, we used the Blood Cell Images dataset from Kaggle to generate a model that predicts cell type from a microscope image. We can think of deploying this model to SageMaker in three steps:

1. Formatting the model for deployment
2. Deploying the model
3. Testing the deployed model

Here we take a closer look at each of these steps.

Step 1: Formatting the model for deployment

Formatting your model for deployment will vary depending on which deep learning framework you choose. Although there are several file formats for saving a TensorFlow model, SageMaker requires the SavedModel format. The four components of the SavedModel for our example look like this:

variables 
    |--variables.data-00000-of-00001
    |--variables.index
keras_metadata.pb
saved_model.pb

If the ModelCheckpoint callback was used in your TensorFlow model training script, it will automatically save models using this layout. Conveniently, if you used Weights and Biases for model tracking and included the WandbModelCheckpoint, the checkpoint models will also be saved with this format. Otherwise, if the model is saved in a different format, such as an H5 file, you will need to load and save it again:

Once you have the model files in the proper format, they need to be placed in a larger folder structure for export to the SageMaker endpoint. The overarching file structure should look like this:

export_folder 
    |--001 
        |--variables 
            |--variables.data-00000-of-00001
            |--variables.index
        |--keras_metadata.pb
        |--saved_model.pb

You can give the “export_folder” any name you want, but the naming conventions for its contents are more strict. Importantly:

• The model files, especially the “variables” files, are sensitive to name changes and should be named as shown above.
• The folder containing the model files (in this example, the “001” folder) should be named with only the model version number. Even changing the name to “model_001” may cause the model deployment to fail.

Once the model has been properly saved in the “export_folder,” the folder needs to be converted to a tar.gz file. This is done easily with the tarfile package in Python:

Step 2: Deploying the model

If the setup in Step 1 has been done correctly, actual deployment to SageMaker is simple and can be done in four lines of code using the Python SageMaker SDK.

A few notes about the last two lines of code:

• For the TensorFlowModel instance, make sure you know what version of the TensorFlow package was used to create the model. This will be your framework_version argument.
• The variable my_sagemaker_role can be found by calling get_execution_role() from the SageMaker SDK.
• Once the model is deployed, it will run constantly on the number and type of instances that you specify. Make sure to select an appropriate initial_instance_count and instance_type so you don’t incur unexpected costs. Here is a list of SageMaker instance types and their prices.

Step 3: Accessing the deployed model

With the model deployed to a SageMaker endpoint, we can now access it to make live predictions on new data. First, make sure to preprocess the same way you did for the training data. To do this, we recommend using the same data processing pipeline that was set up prior to model development.

There are three main methods for sending images to the model endpoint and retrieving results. We can use the SageMaker SDK, the Boto3 SDK, or the CLI. Each method follows the same general process:

1. Serialize the input images.
2. Send the images to the model endpoint.
3. Receive and parse the output.

With the SageMaker SDK, the first and third steps are mostly handled by functions in the package. For example, we can import a json_serializer and json_deserializer from the SDK. When included as arguments for the Predictor class, these will serialize/deserialize data to/from the model endpoint. Remember that if our input images are numpy arrays, they need to be converted to Python lists to be JSON serializable.

Using the Boto3 SDK requires us to serialize the input and deserialize the output. Also, recall from our data storage blog that the Boto3 client returns metadata along with the model response, so the output needs to be parsed.

To use the CLI, the serialized input images need to be saved to disk (e.g. as input_images.json). Similarly, the model response will be written to disk before it can be parsed (e.g. as model_response.json). Then, when running the CLI from the same directory as the saved input_images.json, we call:

All three of these SageMaker access methods are simple and effective. You should use the one that is most appropriate for your application!

Model deployment to the edge

Low powered edge devices, including wearables and smartwatches, are ubiquitous. Many wearables benefit from models making predictions on the hardware itself. Deploying models to an edge device, like a microcontroller, is a very different process from cloud deployment. You’ll likely be working in a low level programming language like C, C++, or Rust, and there may be specific constraints related to the ARM architecture or firmware compiler. Memory and power constraints also range from moderate to severe; thinking about RAM in terms of kilobytes or even bytes is commonplace. Managing data inputs and outputs also requires careful attention.

In general, deploying models to microcontrollers is a bespoke process that depends heavily on the hardware and firmware setup. Someone experienced in working with low level languages for embedded devices is necessary. We’ve had success writing our own C implementations of algorithms but have also found that TensorFlow Lite for Microcontrollers can reduce development cycle times once the libraries are integrated into the code base.

What now?

The raw data has been stored, and data I/O methods have been developed. The data has been processed and used to train and tune a large number of models. Those models have been logged and tracked to find the most effective one. Now, that selected model has been deployed and is ready to be integrated into your application.

However, the work of MLOps doesn’t stop here. The deployed model still needs to be rolled out to your users. It must be monitored to ensure there are no performance issues or unexpected outputs. And as new data comes in, it should be stored, processed, and used to tune your models. The need to maintain and improve ML models is constant, but as this work continues, we hope you return to the five pillars of MLOps we’ve explored in this series.

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | Model Deployment was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

As machine learning models become more widely deployed, ML practitioners have shown increasing interest in MLOps. In our introductory blog, we give a brief background on how we think about MLOps at Edge Analytics.

Here we look at the fourth pillar of our MLOps pipeline: model tracking. Tracking generally means logging relevant information to a database or other storage for future use. As we train and test ML models, integrating model tracking software allows us to easily compare and contrast our model runs to make high value insights quickly and with confidence. This post is part four of a five part series.

You can find the other blogs in the series by following the links below:

• Part 1: Data Storage with AWS S3 and Boto3
• Part 2: Data Processing Pipeline
• Part 3: Model Development
• Part 4: Model Tracking
• Part 5: Model Deployment

ML modeling experiments are most useful if we keep a record of what we tried and how it turned out. Tracking model configurations and results per experiment trial helps us:

1. Efficiently identify parameters of the best performing models
2. Reproduce these models at a later date

The second point is particularly important for us to be confident that any models we develop will behave as expected when deployed “in the wild.”

Two questions help guide our model tracking:

1. What components do we want to track?
2. What data formats and services should we use to track them?

There are many considerations to take into account when answering these questions. For example, certain components (e.g. metrics) help identify best performing models. Other components (e.g. hyperparameters) are necessary for reproducibility. We should understand the storage requirements for components we plan to track, and we should have a sense of what dashboarding tools would make our lives easier. We will examine these questions and others more closely in the context of our example MLOps pipeline.

Components we track

There are three types of components that we might care to track. They are parameters, metrics, and artifacts. Parameters are any inputs that impact how a modeling experiment is set up and executed. Metrics are the results by which we gauge performance of the modeling run. An artifact is a catch-all term for any file we want to log with the experiment, which may include model weights, figures, or text summaries.

The set of components tracked for an experiment should fully specify every aspect of the experiment you may want to assess in the future; you want to ensure there are no information gaps in the data you log. Still, some attention should be taken to avoid exploding data storage costs. For example, tracking a dataset ID tag as a parameter may be a better solution than logging an entire training dataset as an artifact.

The types of tracked components differ across different areas of an MLOps pipeline. The table below provides a brief discussion of how we might consider what to track at each major stage. Note we discuss potential tracked data in terms of images, as our example dataset is the Blood Images Dataset from Kaggle.

Organization

When we run thousands of trials, organizing everything can get messy. Third-party MLOps software platforms that help with this problem have matured in recent years. Integration with custom project code only requires a few lines of code, and dashboards help sort through modeling trials easily. The best part is that all of the complexity of logging and database management is taken care of under the hood. Weights and Biases (W&B), MLflow, and neptune.ai are three well known services that share many model tracking features. While all three platforms are excellent, we explore W&B functionality for this example pipeline because setup is easy, the documentation is extensive, and they have a broad suite of visualization features.

Here we give a short introduction to how we implemented W&B into our example pipeline.

Weights and Biases set up

Getting started with W&B is simple. Once you sign up for an account on their website (it’s free for individuals!), you will be guided through the steps to download the W&B Python SDK.

A basic W&B implementation in Python looks like this:

Regarding the wandb.init() call above:

• The project argument is the name for the overarching project. Multiple modeling runs with different configurations can be saved to a single project.
• The config argument is a dictionary of model configuration values. The keys and values in this dictionary will be used to group and compare the various model runs within the project.
• The name argument is the name of a single model run. If left empty, a random name is generated.

Additionally, any objects and files you log within the wandb_run will be saved to the given run name, under the given project, and will be viewable on the W&B dashboard.

Logging

Within a W&B modeling run, you can log any arbitrary object or file as an artifact. Keep in mind, however, that when doing so, the default behavior will copy the object to W&B cloud storage (GCS in the United States) This convenience works well for some use cases, but here we prefer to handle cloud storage on our own to avoid tight coupling with third party tools. Instead, we log reference artifacts to URIs in our own S3 buckets. In doing so, only the URI reference is saved as an artifact to W&B. This gives us better control over our stored data and increases security for tracking sensitive files.

Viewing the dashboard

W&B provides a nice dashboard for keeping track of logged artifacts and viewing modeling results. To navigate to the dashboard, go to wandb.ai/home and select your project. The first tab will show some training results for each modeling run within the project. For the dashboard below, we used the WandbMetricsLogger TensorFlow callback to track the recall, precision, and accuracy at each epoch.

A second tab on the dashboard organizes these results into a table. This format allows you to easily sort and filter modeling runs based on hyperparameters, metrics, or other logged items.

These are just two of many nice features in the W&B dashboard. It also provides great visualizations for organizing hyperparameter searches, allows you to keep track of artifact data and metadata, generates reports with your results, and more. We recommend you spend some time getting familiar with all the dashboard has to offer!

Hyperparameter search

One powerful feature of W&B is its ability to track and visualize hyperparameter searches. You can use the W&B online dashboard to set up a hyperparameter search, or you can integrate W&B into your own search code with their SDK. To keep our example pipeline flexible, we integrate W&B logging into our existing hyperparameter search code.

During model development, we used the KerasTuner Python package to perform hyperparameter searches. The high level set up of a KerasTuner search looks like this:

When calling tuner.search(), a new model is built and trained for each trial in the num_search_trials via a run_trial() method. We can write a custom tuner class that inherits from RandomSearch and overrides run_trial(). The custom method contains the logic to set up a W&B run.

Now when we instantiate a WandBTuner and call its search() method, our custom run_trial() method will be used for the hyperparameter search. Then, W&B can access performance metrics and hyperparameters to create beautiful parallel coordinates charts like the one below.

This image shows one of the many ways W&B gives you to track and visualize your model development. These features help you sort through many modeling trials and makes W&B a great tool for gaining deep insights on your ML model.

Once we have found the best model and made sure we can recreate its results, it’s time to deploy that model for use in the real world. We’ll discuss this process in the final blog in our series.

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | Model Tracking was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Edge, we love taking notes the old-fashioned way — by hand, and with pen and paper. The classic drawbacks to this approach are that handwritten (non-digital) notes are difficult to share and index for later use. Thumbing back through your notebook to digitize handwritten notes to a computer for sharing and indexing is a painful process. Advances in AI can help with this.

In a recent blog, we shared our solution for automatically converting handwritten notes to a usable, reformatted text summary using OCR and GPT. Today we will be expanding on this idea by building a serverless, event-driven app for handwritten notes summarization using AWS Lambda, S3, Textract, and GPT-4.

The tools

Unless you have been in hibernation this winter, you have likely heard of OpenAI’s GPT-4, a multimodal large language model that has taken the internet by storm. We will be using GPT-4 to format and summarize our notes in this application.

We will also use a suite of AWS tools. This includes AWS S3 for cloud storage and AWS Textract for optical character recognition (OCR). While we could also use an open source OCR model (like Tesseract or TrOCR), we opted for the reliability and flexibility of the input format of AWS Textract.

The last tool we will be using is AWS Lambda, a serverless, event-driven compute service. Lambda is low cost and flexible, making it perfect for this event-driven application. Event-driven functions are compute events that occur automatically in response to triggers. Also known as Function-as-a-Service, Lambda allows the developer to write backends without having to reserve and maintain servers. Lambda handles scaling, memory allocation, and security.

We have opted to use AWS web tools for ease of use, but please note that you can do all of the following with the AWS CLI instead.

Now that we have defined the tools we will be using for this application, let’s get started!

Step 1 — Set up S3 buckets

The first step in building our application is setting up our S3 buckets. We will create two buckets: one for the input image — “scribble” — and one for the Markdown output — “summary”. While we could use the same S3 bucket for both the input and the output of our function, it is not recommended by AWS because it can cause recursive invocations, which could increase Lambda usage (and the associated costs). Here is a user guide for creating an S3 bucket. Name your bucket and configure it with all the default settings. Note the region you create the S3 bucket in — you will want to create your Lambda in the same region.

Once you have created your buckets, note the Amazon Resource Name (ARN) for each bucket. You will need these to set the IAM permissions for your Lambda. You can find the ARN by navigating to your bucket > Properties > Bucket overview.

Step 2 — Set up Lambda to trigger on S3

Next, you will create your Lambda function. You should choose “Author from scratch” and choose your Runtime (we’re usingPython 3.9 in this example). Click “Create function”.

To set the Lambda up to trigger on your S3 input bucket, click “Add trigger” in the Function overview tab.

To configure the trigger, choose “S3” as the source and choose the bucket you want to serve as the event source. For event types, select “All object create events”. Optionally, you can add a prefix or suffix that will allow you to limit notifications to objects with keys that start/end with the specified characters (if you only wanted to run the function on .png files, for example). For this example, you can leave these blank. Click “Add”.

You are now going to add IAM permission policies for your Lambda function. This will give your Lambda access to the S3 buckets you configured above.

Navigate to the Configuration tab > Permissions tab. Click on the link under Execution role > Role name. This will take you to the IAM role associated with the Lambda function.

In the IAM role, select “Add permissions” > Create inline policy. In the JSON tab of the editor, add the following policy (make sure to add your bucket name where you are adding the input file). Click “Review Policy” and add a name to your policy. Alternatively, you can use the Visual editor and add the ARN you copied for your bucket above.

You will repeat the above step for your bucket where you will send the output of your Lambda function, this time with the action being “s3:PutObject”. See the JSON below:

Now you should be connected to your S3 buckets. If you upload a file to your input S3 bucket, it should trigger the Lambda function. You can easily test this with your lambda_handler function (this is the function that is triggered by the S3 upload and handles the event by default. You can change which function in your code handles the event in Code > Runtime settings > Handler),

In your function’s CloudWatch logs (more about this in a later section), you should see an event when you upload a file to your S3 bucket. Now we are ready to build our app!

Step 3 — Integrate AWS Textract

Integrating Lambda and AWS Textract is incredibly straightforward. While we are using AWS Textract, you could easily replace this with a different OCR solution.

In order to utilize AWS Textract, we need to return to our IAM role (following the instructions above) and create a new inline policy. The JSON for the policy is listed below:

From the event defined in our lambda_handler() function, we will get the current event record’s bucket name and filename, which we will use in our call to the Textract API using AWS’s SKD for Python, boto3. Since we only need to detect text in the document, we will use the detect_document_text() function.

The response from the above call to the Textract API requires some reformatting to the preferred input to GPT-4. The following function will format the response from Textract into a single string for input to GPT-4:

Step 4 — Integrate OpenAI GPT

To use the OpenAI API, we will need to install the Python openai library. One nuance of AWS Lambda is the Python environment is natively configured with select packages only (like boto3). While you can zip up packages with your code and upload directly to AWS, Amazon’s recommendation is to add a Layer. Layers are additional code and content (e.g. libraries, a custom runtime, other dependencies) that can be uploaded directly to your Lambda function. Layers can be used across lambda functions and are version controlled.

The following steps will walk you through the process of creating your own Layer. In the same environment you used to develop your code, follow these steps in your command line:

It is very important to name your secondary repository ‘python’. Otherwise, AWS will not know where to find your library.

Now that you have a zipped file (named package_layer.zip), you can upload it as a Layer to AWS. In AWS Lambda, under “Additional Resources” you will find the Layers tab, where you can view all of your layers. You will create a new layer by clicking the “Create layer” button.

You will upload your .zip file. Give your layer a name and optionally add a runtime.

If you go back to your lambda function, you can now select the “Layers” tab under “Code”. Choose “Custom layers” and choose the layer you just created from the dropdown.

Another item to configure is an environmental variable for your OpenAI API key. You can do this by selecting Configuration > Environmental variables > Edit > Add environment variable.

You can call this variable in your code like so:

Now you are ready to make calls to the OpenAI API, following their documentation.

For our GPT-4 application, we used the ChatCompletion.create() function. GPT-4 inputs include the output-string from Textract and a prompt telling GPT-4 that it is a note formatter and summarizer with specific instructions on formatting.

Step 5 — Writing to S3

Again, boto3 makes it easy to write our resulting Markdown text (.md file extension is needed) to our output S3 bucket as shown.

Step 6 — General configurations

Depending on the application you build, you may have to increase the timeout and/or memory of your Lambda function. You can do this by navigating to Configuration > General configuration > Edit.

For this application, I needed to increase the timeout because the default 3 second timeout period was not long enough for the API calls to Textract and OpenAI.

Helpful Hint: If you are confident that your code runs but are still getting errors when testing, try increasing the timeout or memory of your Lambda function.

Step 7— Testing

There are two options for testing. You can configure a test event in AWS Lambda, or you can upload a file to your input S3 bucket. For debugging, you should become familiar with CloudWatch logs, which will log all requests handled by your function. You can access your logs by selecting the “Monitor” tab and clicking “View CloudWatch logs”. Here is a guide to using CloudWatch logs.

Conclusions

In this tutorial, we integrated AWS Lambda, S3, Textract, and GPT-4 to create a useful and fun application for formatting and summarizing your handwritten notes. From here, you can connect the app to inputs/outputs other than S3 — a front-end web interface, a GitHub CI/CD tool, or even a Slack bot (blog on creating a Slack bot coming soon…).

If you build something using this tutorial, let us know! We want to hear from you — what are you using these tools for?

Building tools with GPT at Edge Analytics

Edge Analytics has helped multiple companies build solutions that leverage GPT models. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

Tutorial: Building a serverless event-driven handwritten notes summarization app was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

As machine learning models become more widely deployed, ML practitioners have shown increasing interest in MLOps. In our introductory blog, we give a brief background on how we think about MLOps at Edge Analytics.

In previous posts, we examined tools for data storage and I/O and data processing; here, we look at model development. During this step, machine learning models are trained and evaluated on the processed data. This is part three of our five part series on building an MLOps pipeline.

You can find the other blogs in the series by following the links below:

• Part 1: Data Storage with AWS S3 and Boto3
• Part 2: Data Processing Pipeline
• Part 3: Model Development
• Part 4: Model Tracking
• Part 5: Model Deployment

We first need to decide on a framework for training and evaluating models. This framework will need to answer the following questions:

• What library do we want to use for modeling (TensorFlow, PyTorch, JAX, etc.)?
• What modeling architecture is appropriate for our data format and size?
• How do we define a successful model?
• What are appropriate evaluation metrics?

Recall that we are using the Blood Images Dataset from Kaggle as our example problem where we predict cell type from a microscope image. We use convolutional neural networks (CNNs) for this task since they are the standard architecture for image classification. To build the model, we can choose from a number of deep learning platforms like PyTorch, JAX, or TensorFlow. In this example, we use Tensorflow.

Model configuration

CNNs, even simple ones, are composed of many configurable hyperparameters. We can build limitless flavors of CNNs by varying the number of convolutional layers, max pooling layers, dropout layers, etc. Furthermore, each of those layers has its own hyperparameters, including convolutional kernel size and stride, pooling kernel size and stride, dropout rate, and more.

We believe that a good pipeline for model training should have a fair amount of structure while giving an ML engineer access to the knobs necessary to succeed on a project. For that reason, we typically wrap the code to create and compile the model into a main script that accepts an easily edited configuration file. We generally use a YAML file for configuration because its structure is human-readable, and its contents translate into intuitive Python base types. The configuration for our simple CNN is written in the YAML file like this:

cnn_config:
    # The first three lines specify the input data shape.
    input_x_pixels: 480 
    input_y_pixels: 480
    input_channels: 1
    # The convolutional layers.
    num_convolutional_layers: 3
    num_convolutional_filters: 5
    convolutional_kernel_size: 3
    convolutional_kernel_stride: 1
    convolutional_activation_fxn: "relu"
    # The max pooling layers.
    num_max_pool_layers: 1
    max_pool_size: 2
    max_pool_stride: 2
    # The output activation and number of labels.
    output_activation_fxn: "softmax"
    output_len: 6

We can also use this YAML configuration file to specify other model-related information such as batch size, number of epochs, loss functions, and evaluation metrics. We can produce an entirely new model by simply editing the config file and passing it from the command line, like this:

python train.py --config model_config.yaml

Having all this information accessible in a single file makes editing the model structure convenient and reproducible since we log the config file with the other outputs of the training run.

The training process

Model training has been simplified significantly in recent years by packages like TensorFlow, PyTorch, and JAX. High level interfaces in these packages abstract away many of the pain points of deep learning, so that we can more rapidly iterate on new ideas rather than reinventing the wheel each time. We start with our model configuration file and the processed data (split into training and testing sets):

When we run cross validation, we repeat the training process by splitting the data into train and validation sets and complete the process of model building, training, and evaluating. Note that most projects will typically have an additional holdout test dataset that is only used for evaluating the final model prior to deployment.

For each training run, we log relevant information to help us reason about the model in post-hoc analysis. In this basic example, these outputs include the training log, model file, and model evaluation results. Altogether, they help us keep track of the models we have built and how well they perform. We’ll dive deeper into model logging and tracking in the next blog.

Optimizing model hyperparameters

All models have hyperparameters that affect model performance. In the case of our simple CNN, we can modify the model layers until we are happy with how it performs on the validation dataset (note: not the final holdout test set!). Manual tuning of hyperparameters is often not feasible, so we turn to solutions for scaled hyperparameter searching. These solutions will train and evaluate many architectures to find those with the best performance.

The KerasTuner package is built on Keras functionality and provides simple and efficient methods for hyperparameter searching. With it, we can specify ranges of hyperparameter values over which to search. This includes discrete values and continuous distributions for any given hyperparameter. We often configure our pipelines to run searches using the same configuration file as a typical model training, where the hyperparameters of interest contain a list rather than a single value. Downstream, the code contains the logic to treat these lists as axes of a search space.

KerasTuner allows the user to specify how many trials to run, which metrics to use for model evaluation, and which search method to use for selecting hyperparameters (e.g. random search, bayesian optimization, hyperband, etc.). And conveniently, if the tuner chooses an impossible model configuration, it will skip that trial without interrupting the entire search. The KerasTuner package also includes a class for hyperparameter searches on Scikit-learn models!

As with running the simple train.py script above, all values needed for the hyperparameter sweep are saved in a single tune_config.yaml file. So if we need to change our sweep values, we can do so in the config file and run:

python tune.py --config tune_config.yaml

Remote training with Ray

Local development is excellent for writing code and unit testing models, but it is typically resource-limited for training and tuning purposes. Ray is a library that offers the ability to scale your Python applications with ease, spinning up remote instances on AWS, GCP, Azure, and more to handle workloads of any size. Ray strikes an impressive balance between being easy to use and configurable. At Edge Analytics, we’re big fans of Ray, and we’re in good company.

Getting started with Ray is easy. We recommend this tutorial to launch your own Ray cluster on AWS. Using the tutorial will give the Ray workers access to your S3 buckets auto-magically. Once the cluster is properly configured, you can send your local Python training script to be run on a remote instance.

Ray offers interfaces for Python and command line. With our goal of maintaining as much independence from any individual platform as possible, we use the CLI here. Since our data I/O for this example is cloud-based, a simple command can kick off a training job on the Ray cluster without any modifications to the local training code:

ray job submit --working-dir . \
               --runtime-env runtime.yaml \
               --entrypoint-num-gpus 1 \
               -- python train.py --config model_config.yaml

Here are a few helpful tips when working with Ray:

1. Spin up a cluster with a small head (m5.large) to keep running costs low. This instance will run constantly and wait for new requests. You can also stop the instance when not in use.
2. Virtual CPU allowance with AWS needs to be high enough to accommodate the number of workers you need, and you will likely need to request more.
3. S3 permissions for your Ray workers on AWS can sometimes create issues. This thread is extremely helpful for troubleshooting.
4. Setting up a runtime environment YAML file can help clean up your job submission command while adding environment variables and specifications for pip/conda environments.

Many other great resources for training models on remote instances exist, such as Vertex AI at Google and SageMaker at AWS. We have experience running model training scripts on AWS SageMaker Script Mode instances and have found it to be an involved and restrictive process. Comparatively, Ray is easy to set up and flexible, making it a great resource for our example pipeline.

Up next

In order to get our model just right, we might have to train countless variations of it. When we have hundreds or thousands of model results to sort through, how do we find the best ones? And once we’ve identified those models of interest, how can we ensure reproducibility? These questions are addressed by our next step in the pipeline: model tracking.

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | Model Development was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

As machine learning models become more widely deployed, ML practitioners have shown increasing interest in MLOps. In our introductory blog, we give a brief background on how we think about MLOps at Edge Analytics.

In this post, we’ll focus on approaches for transforming raw data into a form we can feed into a model. Cleaning, standardizing, and appropriately formatting raw data are all critically important in the machine learning development cycle. Building data processing pipelines that are flexible, robust, easy-to-understand, and scalable is one of the best investments an ML team can make. This post is part two of our five part series on building an MLOps pipeline.

You can find the other blogs in the series by following the links below:

• Part 1: Data Storage with AWS S3 and Boto3
• Part 2: Data Processing Pipeline
• Part 3: Model Development
• Part 4: Model Tracking
• Part 5: Model Deployment

In the first part of this series, we discussed how we can store our raw data with AWS S3, a widely used object storage service. Now, we turn our focus to preparing that data for modeling. Some key considerations for choosing data processing tools include:

• What format is our raw data currently in?
• What format does the data need to be in for modeling?
• How fast does our processing pipeline need to be?
• How will we version our processed dataset so that we can re-use it in the future?

Of all the steps in an MLOps pipeline, data processing is the least likely to be plug-and-play and often requires a bespoke pipeline. The answers to the questions above will likely depend heavily on the data type (images, time series, tabular, etc.), file format, and size. As such, there is no single best tool for all of your processing needs. Still, we’ll highlight a few of our favorite data wrangling tools in Python — native dataclasses, abstract base classes, and pyserde.

Dataclasses

One of the first considerations when building a data processing pipeline is how the raw, intermediate, and fully processed data will be represented in memory. We find that structuring the data into Python dataclasses results in highly expressive data structures with clean syntax and minimal engineering overhead. Recall from the prior blog, we’re demonstrating components of our MLOps pipeline on the Blood Images Dataset from Kaggle. Representing white blood cell images and associated metadata can look like this:

For each image, we create an Image instance as an information rich format for storing data. The full dataset can then be represented as a list of Images. This approach has slightly more overhead than dealing with raw arrays, but is much less likely to introduce bugs. Dataclasses also have lower engineering overhead than other solutions, such as protocol buffers. Flatter data structures and protocol buffers are effective tools that we use when appropriate, but for quick and effective pipelines we prefer native dataclasses.

Abstract base classes

Although data manipulation methods will vary from project to project, certain overarching steps are standard across data pipelines. For example, we expect to use methods for loading raw data, standardizing it, and dividing the processed data into train and test sets. Python abstract base classes help in such cases where code is project specific, but we want a broadly applicable pipeline architecture.

An abstract base class acts as a roadmap for all children classes that inherit from it. An example abstract base class for a data processing pipeline might look as follows:

Note that the BasePipeline class inherits the ABC class from the abc package, and all major methods we will define for a specific data processing pipeline are decorated with @abstractmethod.

Now say we want to write an example data processing pipeline for our white blood cell images and call it BloodImagesPipeline. We can write this class as a child of BasePipeline. Doing so will enforce that the three abstract methods above are always implemented. The BloodImagesPipeline class might look like this:

We can now instantiate this BloodImagesPipeline and use it to download, process, and split our dataset for modeling. If we had failed to define any of the three abstract methods in the parent, or attempted to give any of them a different name, the BloodImagesPipeline would fail to instantiate. In this way, abstract base classes can be used to guide development of data processing pipelines for various projects, no matter how different they are.

At the end of the data processing pipeline, we may find it necessary to store the resulting dataset prior to model training. It is a common practice for us to store processed datasets to ensure future modeling experiments are trained on exactly the same data. We may also need to capture additional information including the one-hot label encoder (if one is used), the random state used to generate the train/test splits, and other metadata associated with processing steps. Once again, we can create a dataclass for structuring this information.

Serialization with pyserde

We invest heavily in dataclasses that give our projects high structure, so we also need methods to serialize those dataclasses to disk. The pyserde library provides easy-to-use functions for serializing/deserializing Python objects into common formats including JSON, YAML, and MessagePack. To use these functions, we add the @serde decorator to our dataclass. Then, we can easily write the serialized object to the appropriate AWS S3 bucket or local storage. Pulling objects from S3 and deserializing is similarly straightforward.

There are a couple of caveats to using pyserde:

• Certain package-specific objects, such as the OneHotEncoder from the scikit-learn package, cannot be serialized. When using pyserde, it is best to stick to primitives, containers, standard library objects, and user defined dataclasses. You can find supported data types in their documentation.
• Serializing with pyserde is a good option assuming the entire dataset can be loaded into memory all at once. With large datasets, this may not always be the case. Under those circumstances, other storage solutions like Tensorflow sharded datasets will be more appropriate.

With pyserde and dataclasses, we can turn entire train and test sets into serialized objects that can be uploaded to and downloaded from cloud storage. In this way, we avoid processing raw data with each new model training. This helps save compute time and increases confidence in results by allowing for versioned datasets to ensure the exact same data is being fed into repeated model experiments.

In this post, we looked at our solutions for setting up data pipelines using dataclasses, enforcing structure with abstract base classes, and serializing/deserializing with pyserde. In the next blog, we’ll dive into model training and development.

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | A Data Processing Pipeline was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

As machine learning models become more widely deployed, ML practitioners have shown increasing interest in MLOps. In our introductory blog, we give a brief background on how we think about MLOps at Edge Analytics.

Here we look at the first pillar of an MLOps pipeline: data storage. This post is part one of a five part series. For the example dataset below, we store raw data for model training and evaluation in an AWS S3 bucket and access it using the Boto3 SDK in Python.

You can find the other blogs in the series by following the links below:

• Part 1: Data Storage with AWS S3 and Boto3
• Part 2: Data Processing Pipeline
• Part 3: Model Development
• Part 4: Model Tracking
• Part 5: Model Deployment

All ML projects start with data. This data, whatever file type it is, however clean it is, must be stored somewhere. When deciding where to house the data, you might consider the following questions:

1. What is the scale (MB / GB / TB) of our dataset?
2. What format is our data in (tabular, images, time series, text, etc.)?
3. What security features do we need for our dataset?
4. Do we want the dataset to be publicly available or restricted in any way?
5. What functionality exists to access the dataset as quickly as possible (I/O)?

The answers to these questions can begin to provide guidance on how data should be stored. At Edge Analytics, we have experience using both data warehouses (e.g. Google BigQuery) and data lakes (e.g. AWS S3). For this example pipeline, we’ll lean towards data lakes for their ease of use and flexibility.

AWS S3

Object cloud storage services like AWS S3, Microsoft Azure Blob, and Google Cloud Storage are widely used and very flexible as data lakes. Although all three provide similar features at similar prices, AWS has led the cloud market in recent years. For this example pipeline, we’ll store all data in AWS S3 buckets. Regarding the questions above:

1. What is the scale (MB / GB / TB) of our dataset? AWS S3 storage is scalable. Although there are limits to how many buckets can be created per account (up to 100, unless a service limit increase is requested), there is no upper limit to the size of a bucket. Additionally, it is relatively inexpensive to request more storage space.
2. What format is our data in (tabular, images, time series, text, etc.)? S3 can be used to store data for countless use cases and arbitrary data types. Setting up a data lake of raw or processed data files for your project is very straightforward.
3. What security features do we need for our dataset? S3 has many security features that ensure the protection of data like automatic encryption and Macie, which discovers and reports on sensitive data in your buckets.
4. Do we want the dataset to be publicly available or restricted in any way? Access to S3 buckets is easily managed through services that dictate who can see what data and when. Credentials with varying levels of access can be granted by your AWS manager. Contrarily, if you want to make the data public, S3 will allow that too!
5. What functionality exists to access the dataset as quickly as possible (I/O)? There are several ways of accessing files stored in S3 buckets including a command line interface (CLI), the AWS S3 GUI console, or the Boto3 software development kit (SDK) in python.

The product offerings for S3 are expansive. We touch very briefly on a few features we use frequently. AWS offers more comprehensive documentation on the S3 landing page.

An example storage solution

Prior to interacting with data in S3, you’ll first need to set up AWS credentials and install the AWS CLI. Once you have credentials for accessing S3, run `aws configure` in your local terminal. A `config` and `credential` file will be created, and you’ll be free to use the S3 CLI. If working in Python, you can install the Python Boto3 SDK instead.

The data for our blood images classifier example was borrowed from this Blood Cell Images dataset on Kaggle. It consists of 410 JPEG microscope images of white blood cells, 410 XML files with image annotations (one per image), and a single CSV file that gives the cell type label for each image. These files were downloaded to a local machine, then uploaded as-is to an AWS S3 bucket via the CLI. The file structure is shown here:

Helpful tip!

S3 buckets are data lakes; they have a flat file structure. So although it is easier to think in terms of a hierarchical file structure, like the one above, the reality is a bit different. Rather than treating files as located within folders, S3 treats cascading directories as prefixes to file names. So the file structure in S3 actually looks like this:

Because of this flat structure, there is a tradeoff between finding files easily in the AWS GUI console and accessing files quickly via code. Using a hierarchical folder naming convention can help a user manually find files but slows programmatic access if many files must be read. Alternatively, giving files unique names without folder prefixes speeds up recursive file reading but makes it harder for a user to identify a file based on name.

Although the AWS S3 CLI and GUI console are great for mobilizing data files, the Boto3 SDK in Python works best for loading them into working memory of a Python application, which is our use case here. The functionality of Boto3 is extensive, so much so that we decided to distill and simplify only the functions we needed into a separate class called S3Operations. This class includes ergonomic methods for:

As mentioned, the methods in the S3Operations class are wrappers over core Boto3 functionality. We find these convenience methods speed up development time by abstracting hard-to-remember Boto3 syntax into method names we can easily reference. It turns out, for most data I/O we do, we typically use a fairly small subset of Boto3’s total functionality.

Here is a short snippet of how we use the S3Operations class.

Helpful tip!

The Boto3 SDK requires either a “client” or “resource” to be instantiated in your Python program (e.g. as client = boto3.client(“s3”)) for making S3 service requests. We recommend always using “client”. The Boto 3 “client” is a low-level interface that has broad utility and returns metadata for your S3 request along with the data. The “resource” is a high-level interface built on top of the “client” that makes accessing files simple at the expense of some flexibility. As of January 2023, AWS does not intend to add new features to “resource.”

Up next

Now that we have a place to store the raw data and a way to access it, we will move to the next step in our MLOps pipeline: data processing. Stay tuned for the next blog which we expect to publish the week of April 24.

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | Data Storage with AWS S3 and Boto3 was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Edge Analytics, we strive to develop machine learning applications that are transparent and reproducible. Machine learning projects are often composed of many parts, and the rapidly available software solutions for managing these parts are collectively called Machine Learning Operations (MLOps).

We’ve found investing in MLOps infrastructure accelerates our ability to extract meaningful insights for the problems we solve. A robust MLOps infrastructure optimizes our iteration time and increases our trust in the models we develop for clients. In this blog series, we’ll discuss an example MLOps pipeline using some of our favorite tools.

How we use MLOps at Edge Analytics

Our team develops and deploys ML models to both edge devices and the cloud. Tracking model development and deployment is critical to ensure algorithms behave as expected in the wild. We’ll demonstrate one example of a full pipeline using tools that have worked well for our team. The example is a modular architecture with functionality broken into five major steps. These steps include:

1. Data Storage and I/O
2. Data Processing
3. Model Development
4. Model Tracking
5. Model Deployment

The specific structures of MLOps pipelines will vary from project to project. At Edge Analytics, we use the best tools for the job at hand; however, we consistently use the above five points as guiding pillars for MLOps pipeline development.

Considerations for pipeline development

Building an MLOps pipeline comes with countless trade-offs for balancing structure and flexibility. In selecting the best tools for our clients, we are guided by two major principles:

1. Code abstractions for third-party tools should be simple, consistent, and well documented.
2. No single platform has the best of all solutions, and new features are frequently available. We should maintain a flexible pipeline capable of interchanging third-party platforms.

There are several services that provide end-to-end MLOps support, such as AWS SageMaker and Ray. However, we aim to avoid tight coupling with any given service by enabling integration with a variety of Python packages and services.

Lastly, it should be acknowledged that there is no one way to build an MLOps pipeline. There are many incredible tools that can assist in developing and tracking ML models. We hope this example pipeline gives you a good place to start and encourages you to seek out the methods that best serve your project!

A blog series to showcase our process

Over this series of blog posts, we will examine each of the five central MLOps pipeline pillars more closely in the context of an example project classifying white blood cell images by cell type. And with that, let’s take a look at data storage…

Machine learning at Edge Analytics

Edge Analytics helps companies build MLOps solutions for their specific use cases. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

MLOps at Edge Analytics | Introduction was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Edge Analytics, we have years of experience developing machine learning models built for human movements and deploying those models in low power, compute constrained environments like wearable devices. Specifically, we’ve worked extensively with inertial measurement unit (IMU) data (i.e. three axis accelerometer and gyroscope channels) collected on the body with applications in consumer electronics and digital health and medicine. Applications we’ve worked on in the past include sleep tracking, gesture recognition, walking speed, and prediction of physiological phenomena that can’t be directly captured from movement.

As we’ve gone through iteration cycles of building and deploying models, we realized the importance of collecting our own pilot data for rapid experimentation and algorithm development in early phase product cycles. Demonstrating proof of concept early can ensure viability of an idea before investing heavily in expensive and time consuming data collection efforts. We made an internal investment to build out firmware to enable easy collection of IMU data using the ST SensorTile.box prototype board and ST LSM6DSOX sensor module. Building these tools has paid dividends multiple times over, as we can now iterate from idea to data collection to proof of concept in an afternoon for algorithms built on IMU data.

We had a silly idea recently to see if we could build a machine learning model to help track one of the author’s favorite pastimes: riding wheelies on a mountain bike. Going through a rapid iteration cycle of building a convolutional network with 97% accuracy at predicting between wheelie, pedal, coast, or other/unknown classes reveals a lot about Edge’s process for building early prototypes. From no data to final model iteration, we built out this proof of concept in spare time over three days.

Having a platform for data collection is a prerequisite for building a proof of concept wheelie predictor. Our solution to this was a prior investment in building firmware tools and utilities that enable the SensorTile.box, a prototype IoT board with ARM Cortex-M4 microcontroller, to work with the LSM6DSOX, an integrated circuit with accelerometer and gyroscope. Look out for a future blog where we outline how we approached building the firmware to support data collection with this hardware.

Collecting data

As is often the case when working with data from sensors on the human body, data collection was perhaps the most challenging aspect of this project. We placed the SensorTile box on the lateral side of the right thigh to capture the pedaling motion. We assumed a priori that approximately three minutes of training data would be sufficient for an initial model. Collecting data for pedaling, coasting, and other/unknown was straightforward. We simply started collecting data from a stop, then began bicycling. Obtaining labeled data for wheelies was more challenging since the author cannot hold a wheelie indefinitely. Instead, he’d repeat sets of wheelies as long as possible (typically 5–20 seconds). Then, later during data processing, we manually labeled periods of each recording as one of the four labels. Extracted windows of each class are shown below.

Test data was collected using a similar protocol, though each test recording is distinct from the training recordings to show some generalization.

We note the proof of concept demonstrated here is on a sample of n=1 persons. If Edge decided to invest more heavily in this work, we would recruit from a much larger pool of participants for data collection while carefully considering our inclusion criteria to ensure the training data accurately reflects the end user population to avoid model bias.

Creating a model to predict wheelies

Next, we trained a 1D convolutional neural network (CNN) on windows of IMU data labeled by activity class. We chose a three layer 1D CNN since this architecture has certain favorable characteristics (e.g. a 0.5 second receptive field with temporal invariance of IMU features within the window) and is fairly amenable to deployment to a microcontroller. In fact, TensflowLite for Microcontollers provides a “Magic Wand” tutorial that uses a similar 1D CNN configuration. Deploying models like this to low power microcontrollers is gaining traction but still remains fairly cutting edge. We may do a follow up blog showing how we’d deploy such a model (if you would like to see this blog, let us know in the comments!).

As anticipated by training data visuals in the previous section, our CNN does quite well at learning to detect wheelies. The video below shows a single recording including all four model activity classes. Model inference is run on a desktop with a stride of two samples, but any stride can be chosen. Note the model is quite confident during steady state activity like wheelies but experiences slight delays at transition periods. This is expected since the model sees a window extending half a second in the past.

The video above shows how the model performs when predicting activity by rolling over a recording of data. Below, we show quantitative results of model performance on 1,966 samples from five test recordings. Overall, the model achieves 97% accuracy. Note the model has slightly lower recall for coast and pedal, which the model occasionally predicts as other/unknown.

Meaningful applications in digital health

Despite having a lot of fun on this project, this exact application’s impact could only provide value to a minuscule subset of the total population. One of our driving goals at Edge is to ultimately build products that help a broad range of people, and we are very interested in deploying algorithms for digital health to wearables. IMU sensors are commonly used in consumer devices as well as digital health research including sleep tracking, activity monitoring, disease incidence, and diagnostics. Often used for remote monitoring and management of various conditions, IMU sensors can improve healthcare outcomes, which is very exciting to our team. We are particularly interested in using IMU sensors to improve activity detection in certain populations, such as in elderly care, or in edge case activities like pushing a stroller (stay tuned for an upcoming blog on this!).

Where we would take this next

Developing a 1D CNN wheelie detector for a single user was a fun proof of concept, but where would we take this next? At Edge, we like to explore the full scope of problems to see how much performance we can extract by using more complex algorithms and, equally importantly, see how simple a model can be while retaining acceptable performance. We’re interested in applying sequence model architectures (e.g. LSTM), or even transformers, to this task as an exploration of more complex models. On the other hand, we wonder how far we could get using a simple tree or heuristics based approach. Furthermore, we are always interested in pushing models to microcontrollers. Stay tuned for follow up blogs where we will explore these concepts and more.

Machine learning at Edge Analytics

Edge Analytics has helped multiple companies build solutions using IMU data on the body. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

Building a CNN to predict bicycle wheelies: insights into rapid prototyping was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a digital health data scientist at Edge Analytics, I am always on the lookout for the best machine learning methods to solve healthcare problems. In this article, I delve into the applications of generative ML models in digital medicine. With the potential to generate synthetic data for algorithm training, support analysis of existing health data, personalize medicine through simulation, and enhance patient education, these models hold great promise. I am particularly interested in exploring the use cases of generative models in mental health and will provide an in-depth analysis of this application.

Overall, this article highlights potential future use cases for generative models in digital medicine.

Improving models through data generation

Training machine learning models for medical applications is challenging. One of the primary reasons for this is a lack of available training data. Machine learning models require large amounts of training data to make accurate predictions. Medical data is particularly challenging to obtain and use for a number of reasons. First, data should be procured with patient permission to ensure patient privacy, a process that is often very time-consuming and resource-intensive. Second, medical data is often highly imbalanced, and poor data coverage makes modeling rare conditions difficult. For example, gathering data on a specific type of cancer might yield just a few cases amidst hundreds or thousands of records. Lastly, available medical data is highly biased. Machine learning models exhibit the same biases as the data used to train them, so critical to minimize bias in training datasets.

Generative models can be used to synthetically create balanced realistic data with minimal bias and no privacy restrictions. A few examples for how generative synthetic data could be used in digital medicine:

• Generative models can be used to create synthetic medical images similar to their real counterparts, but with a known ground truth. This can help in the development and evaluation of medical image analysis algorithms. For example, generative adversarial networks (GANs), have already been used to produce synthetic X-ray, computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography as well as retinal, dermoscopic, and ultrasound images [Source, Source].

Caption: Synthetic chest X-rays using NVIDIA’s StyleGAN2-ADA model [Source]

• Generative models can be used to synthesize patient data for use in research and development while maintaining patient privacy. For example, privacy regulations often limit access to electronic health record (EHR) data. Using synthetic EHR data generated using GAN or GAN-style models can eliminate this risk [Source, Source].
• To help researchers and clinicians better understand disease progression and effective treatments, generative models can be used to predict the progression of diseases by generating synthetic data that simulates the disease process. A recent study used a generative Markov-Bayesian-based model to generate 5,000 synthetic illness trajectories to infer the progression of complications in type 2 diabetes patients [Source].

In addition to synthetic data generation, generative models can be used to augment existing data. Augmenting existing datasets with additional data points may be helpful in cases where the available data is limited, or when the model needs more training examples. A recent study uses GANs to augment the Cleveland dataset, a small and imbalanced dataset for heart disease prediction [Source]. Another recent study proposes a longitudinal-diagnostic generative adversarial network (LDGAN) to predict multiple clinical scores at future time points using incomplete longitudinal MRI data [Source].

Data imputation is often done when data points are missing due to measurement error or patient noncompliance. While traditional data imputation methods are simple (i.e. mean or median imputation), they are often not representative of the actual data and can have significant impacts on the resulting model. In some instances, generative models can be used for data imputation. Imputation using generative models has been shown to have better performance than traditional methods in clinical research applications [Source].

Supporting analyses of existing health data

The incredible power of generative large language models could be harnessed for analysis of existing datasets in digital medicine. For example, a model can scan through hundreds of pages of medical records to extract key information for personalized treatment or health plans. The development of a large language model for EHRs has been used for clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference, and medical question answering [Source]. Models fine-tuned on biomedical data, like BioGPT (fine-tuned on GPT-2) or AWS Comprehend Medical, will play a central role in language modeling applied to medicine.

Generative models could also be used to aid healthcare providers through clinical decision support, like identifying potential drug interactions or recommending medication dosages. Early stage research shows great promise in predicting drug-drug interactions using generative models [Source]. Research using generative models to design drug combinations showed that the generated drug combinations collectively cover the disease module similar to FDA-approved drug combinations and could potentially suggest novel pharmacology strategies [Source].

Research in digital medicine could also greatly benefit from generative models: by analyzing large amounts of data and identifying patterns and trends, generative models could aid in the development of new insights into the causes of diseases, methods for early detection of disease, and novel treatments. Early research shows promising results using GANs to assess progression of Alzheimer’s disease [Source].

Personalized medicine with a digital twin

A digital twin is a virtual model designed to accurately reflect a physical object. Used in applications from modeling wind turbines to space capsules, digital twins are extremely useful for providing insights into a variety of sensor systems and their environment.

The human body is incredibly complex and thus challenging to simulate. With generative models, we may get one step closer to having our own health digital twin. Health digital twins are virtual representations of patients that are generated from multimodal patient data, population data, and real-time updates on patient and environmental variables [Source]. For example, generative models could be used to predict the outcomes of different treatment options and predict an individual’s response to different treatment options, personalizing medicine. The startup world is already investing in this idea: Unlearn AI has raised $50 million to build their platform that simulates potential health outcomes for individual patients using generative models.

Patient education

Generative models could also be used for patient education and to help patients manage their health and wellness. Imagine an empathetic WebMD, interactive educational modules to explain medical concepts, or a disease management system to help patients manage chronic conditions, such as diabetes or hypertension. While not reliable enough to provide healthcare management education yet, generative models hold significant promise to enable personalization of education and health management.

Applying generative models to mental health

The mental health subfield of digital medicine is one area with numerous applications for generative models. Therapies could be enhanced greatly using generative modeling.

• Art is a common mode of therapy. Image generative models can be used to generate art in collaboration with patients. Patients providing inputs to generate unique artwork could be used as a form of creative expression and therapy. At the time of writing this article, I could only find one app claiming to use generative AI art for art therapy, NightCafe.

• Generative models could be used to generate images representing a patient’s emotions to serve as a starting point for discussion during traditional therapy sessions.
• Image generative models can be used to create realistic virtual environments for patients to experience. These virtual environments can be used in exposure therapy to help patients confront their fears or anxieties in a controlled setting.
• “Bot therapists” built on large language generative models could be used for 24/7 therapy sessions. People who are worried about judgment or biased care may be more willing to share their concerns with a bot. The Chat GPT subreddit is filled with individuals using Chat GPT as therapy [Source]. It is important to note, however, that, according to OpenAI’s policies, Chat GPT is not fine-tuned to provide medical information and should not be used to provide diagnostic or treatment services for health conditions. In the future, one could imagine a world where a Chat GPT-like model is fine-tuned or trained with medical data to provide therapy.

Potential challenges

There are a number of challenges with using generative models in digital medicine that will need to be addressed as these technologies transition from R&D to clinical settings. In addition to technical challenges [Source, Source], there are ethical considerations that I would be remiss to not note:

Bias

Bias in the training dataset of generative models will result in biased model outputs. Models should be trained using representative data from the population they intend to be used on to mitigate downstream bias and potential harm.

Misuse

Misuse of these models in healthcare is another potential challenge. These models have incredible potential to augment healthcare, but these models should never be intended to replace a healthcare provider or be used for nefarious purposes.

Conclusions

In conclusion, generative models hold immense potential in digital medicine. They can augment models with synthetic data, aid in the analysis of existing data, and personalize medicine through the creation of a “digital twin”. Additionally, generative AI can improve patient education and therapy, particularly in the domain of mental health. While there are challenges to overcome, I believe we are only just beginning to see the benefits to healthcare generative models will provide.

What are you planning to use generative models for?

We want to hear from you! How do you plan to use generative models? Leave us a comment or send us an email at info@edgeanalytics.io. While you’re here, check out our blog series on Getting the Most Out of GPT-3-based Text Classifiers: Part 1, Part 2, Part 3 or check out our recent blog on GPT-Edit.

Generative AI at Edge Analytics

Edge Analytics has helped multiple companies build solutions that leverage generative models. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

Generating Better Health Outcomes: Harnessing Generative ML Models in Digital Medicine was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.

*** EDIT: As of 3/23/23, OpenAI has discontinued GPT Edit. You can use GPT-4 to format and summarize notes in one go with the following prompt: “You are a note formatter and summarizer. You format notes as Markdown and fix grammar and spelling. Then you append a 3 bullet summary to the end of the note.” ***

In the age of digitization, many still prefer taking notes by hand rather than typing. This is great until a co-worker asks you to send your notes from last week’s meeting, but the notes only exist on paper. Thumbing back through your notebook to transcribe (and reformat) those notes to a computer is a painful process. Advances in AI can help with this.

In this blog, we’ll show our solution for automatically converting handwritten notes to a usable, reformatted text summary using a common OCR-based handwriting-to-text application and GPT-Edit.

What is GPT-Edit?

GPT-Edit is one of the newest models from OpenAI. Currently in beta release, GPT-Edit is a powerful tool for editing text, fixing spelling errors and grammatical mistakes, and reformatting text. The model takes to-be-edited text and an instruction prompt as inputs and outputs the edited text body.

Example instructions for GPT-Edit include:

• “Fix the spelling mistakes”
• “Format like a letter”
• “Change text into first person”

You can learn more about the GPT-Edit API here, or you can test out GPT-Edit in the OpenAI Playground here.

The challenge with optical character recognition models

Optical character recognition (OCR) models are a model class that predicts characters, i.e. letters and numbers, from images. If you are a data scientist or an ML engineer, chances are one of the first deep learning models you built was an OCR model using the MNIST handwritten digits dataset. It’s a favorite prompt for introductory courses and tutorials. They are widely used in applications ranging from data entry automation, verifying and processing documents, and recognition of labels/number plates.

We’ll show an example of OCR used to convert images of handwritten notes to machine-readable text and highlight areas for improvement. While the MNIST dataset is a fairly simple and highly curated setting for developing toy models, real world applications tend to be more difficult. For example, the position of characters provides important context for how the machine text should be organized. Many commercial applications exist for handwritten text transformation, including Microsoft OneNote, PDFElement, and Evernote. Applications for the iPhone and iPad that convert handwriting to text include GoodNotes 5, Notability, Notes Plus, Pen to Print, Text Scanner and WritePad for iPad in addition to the native iOS Notes application. For Android users handwriting conversion apps include Adobe Scan, CamScanner, Google Keep, Readiris and Smart Lens. While these technologies are widely available and commonly used, we have found that the output text is often poorly formatted and riddled with grammatical errors, adding friction to note-taking and review.

In the figure below, we show an example handwritten note on the left. On the right, we show the transcription from the native Notes app on iOS. Observe the grammar and formatting problems in the transcription.

Using GPT-Edit to improve the quality of transcribed text

We hypothesized that results from OCR models could be improved using GPT-Edit. To test this hypothesis, we fed the transcribed text into GPT-Edit (text-davinci-edit-001).

We added line breaks to the prompt to tell GPT-Edit where the line breaks occurred in the output of the OCR model (note: these are not the line breaks seen in our original handwritten text).

Here is the edited response from GPT-Edit:

The result from GPT-Edit looks much more similar to the original handwritten note than the output from the OCR model alone. The grammar and spelling is improved; line breaks are true to the original text; and the Markdown formatting makes the note more readable. This note could be more easily shared and understood than the output from the OCR model alone.

Challenges with using GPT-Edit

The results from GPT-Edit are highly stochastic. We have found that the model requires prompt engineering and experimentation to produce more repeatable results. Experimenting with the temperature and top_p GPT variables are also ways to control randomness and diversity, respectively. You can add these to your prompt to the API like this:

Fine tuning the model is potentially another way to generate more repeatable results for particular use cases. For example, the model could be tuned for a specific type of formatting like Markdown or formatting notes to your personalized note-taking style. While fine tuning is not currently supported by the GPT-Edit API like it is for the GPT base models, we are eagerly anticipating the release of fine-tuning for GPT-Edit.

One step further — note summarization using GPT-3

Can we use the output of OCR + GPT-Edit as the input in GPT-3 (text-davinci-003) in order to summarize our note? Let’s try it!

Summary from GPT-3 using GPT-Edit output as the input along with the prompt “summarize the following note”:

At the meeting on 1/30/23, Bob proposed a 3 point plan to build cool stuff, work with awesome people, and eat lots of pizza. Bob was assigned tasks to write a blog, edit a blog, and order a pizza. His favorite pizza toppings are pineapple, ham, and mushrooms.

Conclusions

We hypothesized that we could use the brand new GPT-Edit model from Open AI to improve output from OCR models on the task of transcribing handwritten notes to text. In this blog, we demonstrated this, using a commonly used OCR-based handwriting-to-text application and the GPT-Edit model to automatically convert a handwritten note to a useable, reformatted note that can be shared easily with others. Additionally, we were able to summarize the note with GPT-3, taking “scribbles” to summaries!

What use case do you have for GPT-Edit?

We want to hear from you! How do you plan to use GPT-Edit or other large language models? Leave us a comment or send us an email at info@edgeanalytics.io. While you’re here, check out our blog series on Getting the Most Out of GPT-3-based Text Classifiers: Part 1, Part 2, Part 3.

GPT-3 at Edge Analytics

Edge Analytics has helped multiple companies build solutions that leverage GPT-3. More broadly, we specialize in data science, machine learning, and algorithm development both on the edge and in the cloud. We provide end-to-end support throughout a product’s lifecycle, from quick exploratory prototypes to production-level AI/ML algorithms. We partner with our clients, who range from Fortune 500 companies to innovative startups, to turn their ideas into reality. Have a hard problem in mind? Get in touch at info@edgeanalytics.io.

From Scribbles to Summaries: Enhancing OCR Models with GPT-Edit for Handwritten Notes was originally published in Edge Analytics on Medium, where people are continuing the conversation by highlighting and responding to this story.