Even the best metabolomics pipelines have a degree of variance, which can cause poor peak integration between samples. This reduces your ability to accurately quantify a metabolite, and usually means that you have to manually check the quality of every peak (of interest).

Kelsey Chetnik, Lauren Petrick, and Gaurav Pandey have developed a new framework and R package called MetaClean. This combines eleven peak quality metrics and eight machine learning algorithms to automatically detect poorly integrated peaks.

We interviewed Petrick and Pandey about the recent paper in Metabolomics and learned how MetaClean works and how you can apply it to your data.

Current automatic peak integration is unreliable

Liquid chromatography coupled mass spectrometry (LC-MS) is a standard tool for identifying and quantifying compounds in metabolomics workflows. In LC-MS, retention time and peak shapes of identical compounds (metabolites) will vary slightly from one measurement to another. Slight fluctuations are normal, but automated integration can still mischaracterize peaks and complicate sample to sample comparison.

In many cases, up to 30% of peaks integrated by standard post-processing tools may fail manual inspection. Low-quality peaks can be products of coeluting analytes, faulty alignment, or background noise. As a result, you can’t expect any two samples to have the exact same peaks, even when their makeup and concentrations match. But in order to compare concentrations, you still need to accurately integrate peaks from different samples.

One way to evaluate peak integration is to review manually and judge the peak shape and included areas. This isn’t just extremely time-consuming; it’s practically impossible when you’re analyzing thousands of peaks per batch. As a result, manual QC is often done on a small subset of data and often only on peaks of interest. While this may remove false positives from your analysis, it can’t address false negatives and poorly integrated peaks can slip through.

Machine learning can automate peak evaluation

Individual quality metrics can quantify individual peak quality. However, by themselves, these metrics are often insufficient to fully differentiate between low and high quality integrations.

Machine learning models can find complex and valuable combinations of metrics that can aid this differentiation. All you need are the metrics (your features) and labels that describe whether a peak integration is high or low quality (your target variable).

With these features and the target variables, you can train models and evaluate how well they predict the peak integration of unseen examples. If you find the models are highly accurate, you can easily apply those models to every single peak in a dataset (containing thousands of peaks). You can then filter out the low-quality peaks before you analyze your data.

The MetaClean features: Eleven peak quality metrics

Chetnik, Petrick and Pandey explored whether a combination of quality metrics could work better as an effective feature set for a peak quality classifier. The team created MetaClean to generate these classifiers and evaluate their performance against several datasets.

The team used a set of four metrics from Zhang et al. (M4 Metrics) and seven metrics from Eshghi et al. (M7 Metrics), as well as the combined M11 set consisting of all the metrics. These metrics were used to quantify overall peak shape and retention-time consistency between samples.

The MetaClean training data: 500 manually reviewed peaks

The team classified 500 pre-processed peaks by hand from 89 blood plasma samples: almost 40% failed manual visual inspection. They ensured a variety of different peak shapes were in the training data so the model would be trained on a representative set of peaks.

The MetaClean model: The best of eight ML classification algorithms

The next step was to find the machine learning algorithm that best classifies peak quality. MetaClean evaluates eight of the most commonly used algorithms, each paired with the M4, M7, and M11 metrics to develop 24 different peak quality classifiers. The performance of each classifier is assessed with five-fold cross-validation, repeated ten times, and results are averaged. For this, MetaClean evaluates:

• Decision Tree;
• Logistic Regression;
• Naive Bayes;
• Neural Network;
• SVM with linear kernel;
• AdaBoost;
• Model Averaged Neural Network;
• Random Forest.

After comparing the performance of each model, the team observed the AdaBoost algorithm using the M11 metric set performed the best. They called this model the “Global Peak Quality Classifier,” and it achieved almost 85% accuracy in classification on the development dataset.

However, just testing the model on the same batch of samples doesn’t provide an accurate picture of how well the model will perform in a real situation. An effective model also needs to perform well cross-batch and cross-platform:

• Cross-batch: Other samples, from a different batch, measured on the same machine with the same settings.
• Cross-platform: On samples measured on different machines, with different settings.

So the team evaluated 500 peaks from four more datasets to test. They evaluated the Global Peak Quality Classifier on:

• a different dataset from the same instrument to test same-platform performance;
• three datasets from different platforms to test cross-platform performance.

MetaClean performs well in analyses of data from the same platform

The Global Peak Quality Classifier evaluated a different dataset from the same instrument (Test 1). The classifier categorized peaks accurately in almost 81% of cases, even though this dataset was from a different sample type.

MetaClean performs reasonably well in cross-platform analyses

The team applied the Global Peak Quality Classifier to four publicly accessible datasets, each using different MS instruments, LC columns, and/or ionization modes:

Even with these platform and sample matrix variabilities, the classifier correctly categorized between 65–80% of peaks accurately. However, the reduced performance for these datasets showed that cross-platform analyses are still quite challenging.

Train MetaClean on a per-platform basis

The team used two of these datasets (Test 3 and Test 4) from a lab that is different from the one that generated the development set as an independent case to re-train and evaluate models on their data. They trained two new models with the positive and negative mode data from the same platform, evaluating each against the other. They showed that using each model against the other dataset produces great cross-mode results. Each model can be optimized in a platform-specific fashion, and can even be applied to different ionization modes.

Comparing MetaClean with RSD filtration

Random analytical errors can be difficult to address, especially in complex samples. Often, they are the product of background noise or sample carryover from a previous run and are not relevant to the question at hand. These peaks are often highly variable, and their integrations are not consistent.

One way to find these errors is through Relative Standard Deviation (RSD) filtration, in which pooled samples are analyzed multiple times. Random peaks are discarded and common peaks are evaluated for consistency. When the standard deviation of peak area is outside of a cutoff, they are removed.

Since MetaClean operates independently from RSD filtration, the team thought that first subjecting the data to RSD filtration could serve dual purposes:

• Comparing the performance of MetaClean against an established method;
• Increasing the performance of MetaClean through combination with RSD filtration.

The team found that MetaClean outperformed RSD filtration in all test cases, accurately classifying the quality of the peak better than this widely used method.

While RSD does a great job of ensuring that random analytical error is minimized, MetaClean focuses on removing integration issues that may pass RSD filtration. Since RSD and MetaClean are complementary approaches to data quality filtering, they work very effectively in combination.

Use Metaclean with your data

Training a MetaClean model on your data could be valuable for your workflow. With little up-front time investment, you can analyze all peaks from all the runs in your lab rapidly and greatly reduce the rate of false integrations without dedicating hours to manual peak reviews.

Kelsey, Lauren, and Gaurav provide an open R package that evaluates your data against the eight ML and three quality metric sets to develop a model. You can download the package today and train your model to evaluate your LC-MS integrations.

Be sure to read the MetaClean article published in Metabolomics to learn more about the platform.

MetaClean Automates Peak Quality Assessments was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Spectral similarity scores are commonly used as a proxy for structural similarity. But in many cases the two differ significantly. This can call the results of your library comparisons into question. Florian and his team have built a deep learning model that successfully predicts structural similarity directly from tandem mass spectra.

Spectral comparisons are part of every metabolomics workflow

Comparing your mass-spectral measurements against library measurements is a crucial step in any metabolomics analysis pipeline. You need reliable spectral comparisons to,

• Identify your compound: search a reference library for a compound matching your fragmentation pattern.
• Search for analogs: find structurally related compounds through similar fragmentation patterns.
• Aid in structure elucidation: link your molecule to a molecular class, to narrow potential assignments.

With tandem mass spectrometry, you typically compare mass spectra by calculating the cosine score between them. A high cosine score tells you that the fragmentation profiles of two compounds overlap well.

You might assume a high cosine score correlates to high structural similarity. Unfortunately, cosine scores only tell you about the similarity of spectra. But when you want to annotate your compounds, you’re interested in structural similarity. And that’s a crucial difference.

Spectral and structural similarity: Apples and oranges

Let’s look at the differences:

1. Spectral similarity, for example cosine similarity, measures the overlap between two mass spectral profiles.
2. Structural similarity, like the Tanimoto score, quantifies the overlap of two molecular fingerprints using a 2D-vector representation of the molecular structures.

Spectral similarity can differ significantly from structural similarity. This can lead you to the wrong conclusions from your library searches. Let’s see how that can affect your work.

Take for example, the antibiotic daptomycin (top) and a hypothetical analog (bottom):

Daptomycin (top) and it’s analog (bottom) have shifted fragmentation patterns, despite being very similar to each other overall. A cosine score comparison would give a low score and fail to detect that they are highly similar structures.

• High structural similarity: The structure of the analog (bottom) is still highly similar to it’s un-modified form: daptomycin (top).
• Low spectral similarity: The modifications in the analog carry over to many of the fragments and shift many of the mass-spec peaks. As a result, few overlapping peaks remain, so the spectral similarity score is low.

This means that if your sample included a daptomycin analog, you would NOT get a hit from either a library or an analog search, even if you had a daptomycin reference spectra in the library!

This means if you rely solely on cosine similarity,

• You can’t be sure that matches with the highest cosine score are really the best structural matches.
• Your library might contain a close structural match, but you won’t see it because the match received a very low spectral similarity score.

However, Florian Huber and his team have found a way to predict structural similarity directly from tandem mass spectra.

How MS2DeepScore predicts structural similarity from tandem mass spectra

MS2DeepScore can predict when compounds are closely related structural analogs, even when other comparison methods, like cosine similarity, may fail.

Florian Huber and his team trained MS2DeepScore using over 100,000 spectra from over 15,000 unique compounds. Once trained, the MS2DeepScore model predicts the Tanimoto similarity scores between two compounds directly from their respective MS/MS profiles.

Using what it learned, MS2DeepScore consistently outperforms both cosine similarity and Spec2Vec in relating structurally similar compounds, even when the compound’s fragmentation profiles are shifted. Importantly, the compounds on which the scores’ accuracy were compared were not included in the MS2DeepScore training dataset.

In our example, the cosine score between daptomycin and our analog was only 0.24. From this, we would assume that these compounds are not structurally close. But the MS2DeepScore model predicts a Tanimoto score of 0.95, very close to the actual Tanimoto score of 0.98.

Working with MS2DeepScore

Getting started with MS2DeepScore is simple, and you can begin comparing spectra with it in minutes.

You can use MS2DeepScore no matter what instrument you collect with. MS2DeepScore works with the matchms module, so you can use a choice of several open source data types like .mgf, .mzxml and .mzml.

Florian and his team designed MS2DeepScore as a python package that includes Jupyter notebook tutorials to get you started. You can begin comparing your spectra using the model pre-trained on 109,000 spectra of 15,000 compounds. To learn more about MS2DeepScore and its development, be sure to check out the preprint publication.

What’s next for MS2DeepScore?

We’re building new ways to leverage MS2DeepScore for library matching, including an API, an easy-to-use interface, and continuously updated models.

This new version will provide,

• A new model trained on over 230,000 spectra;
• Library matching against GNPS public databases;
• A new negative mode comparison.

If you want to try it out, don’t hesitate to contact us to receive beta access.

Going Beyond Spectral Similarity With MS2DeepScore was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

All machine learning teams face the same tasks: Manage a preprocessing pipeline; train and test models; and deploy models as APIs.

And nearly every team builds their own hodgepodge collection of internal tools and scripts. The alternatives are to either buy into an expensive and limiting proprietary platform or spend months learning and configuring open source products. The former “just work” but are limiting; the latter are tough to set up, but are flexible.

We wanted “flexible and easy” so we built Open MLOps

Open MLOps is a set of terraform scripts and user guides for you to set up a complete MLOps platform in a Kubernetes cluster. And all of it is open-source.

Who is Open MLOps for?

Open MLOps is for any engineering team. We built it so it’s easy to set up and can accommodate pretty much any machine learning use case. You’ll find Open MLOps useful if you’re,

• A machine learning research team: You know you should be using best practices from MLOps. But you’re not sure how to start and you lack the engineering capacity to build your own framework.
• A machine learning startup: You need to release your PoC quickly. But it also has to be scalable. You know Kubernetes could solve your problems but you don’t have much expertise.
• An individual machine learning practitioner: Even on your own, you battle to track experiments you’ve run and you waste time and effort on repetitive work. You need an easier, faster way to wrap your models in APIs, track experiments, and manage data pipelines.
• A machine learning team at a larger enterprise: Pre-built solutions aren’t flexible enough for you. You know you should build your own but you don’t want to start from scratch.
• A machine learning student: If you want to make the jump from academia to industry, you can test the waters by setting up Open MLOps and following our guides.

Open MLOps lets you deploy a configurable set of tools into a Kubernetes cluster. If you need to hook up your own authentication or configure the services in a specific way, you can simply fork the repository and build off the existing Terraform scripts.

What is Open MLOps?

Open MLOps is an open source repository of Terraform scripts and walk-through tutorials to set up a variety of machine learning tools in a Kubernetes cluster.

By default this sets up:

• Prefect: For scheduling tasks and building DAGs of dependent tasks. You can use it for cleaning data, training and deploying models, or other tasks that you need to automate. Tasks are defined using Python and you can easily build them into pipelines using intuitive decorators.
• JupyterHub: A shared Jupyter notebook environment for your team to run experiments, share code, and perform data wrangling.
• MLFlow: To track experiments and models, keeping a close record of results and runs, so you know what’s been tried and what’s working.
• Dask: To easily parallelize your heavy computation tasks across multiple cores or machines.
• Seldon: To convert your models into production REST APIs and serve them at scale.

What do I need to get started with MLOps?

The easiest way to get started with MLOps is with AWS. All you need is access to an AWS account and permission to create new services.

You also need some experience using command-line tools. Specifically, you’ll use Kubectl and Terraform during the set-up process, but our guides walk you through everything so you needn’t be an expert.

You can get everything running and configured in a few hours using our set up guide. Within another hour, you’ll already be training your first model.

Why should I use Open MLOps?

MLOps is an emerging but fast-growing field to help machine learning teams iterate quickly, deliver results reliably, and collaborate efficiently. Your team will benefit in several ways from the established open-source tools that make up Open MLOps:

• Hire developers more easily. Developers don’t want roles that rely on proprietary software. They know these could be dead-end jobs where they can’t learn transferable skills.
• Don’t reinvent the wheel. It’s often tempting to develop an internal platform that does everything you need. But most teams find it’s more complicated than they expected. You can read our post on why machine learning engineering is more complicated than software engineering. Even if you have intricate, custom requirements, you can start with Open MLOps as a base layer and build your custom solution on top.
• Build reproducible models. The biggest complaint machine learning teams have is the difficulty of reproducing their work, across different systems or over time. By using widely adopted frameworks and tools with MLOps practices you can ensure you’re never left with that large model binary that no one can build again.
• Collaborate efficiently. If everyone on your team uses the same system, it’s easier to work together, share, or hand over work to others.

Feedback, issues, or feature requests?

If you’ve given Open MLOps a try and need support or want to share your experience, please reach out. Alternatively, feel free to open an issue or a pull request directly on the Open MLOps repository.

Open MLOps: Open Source Production Machine Learning was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you build machine learning models, it’s common to run dozens or hundreds of experiments to find the correct input data, parameters, and algorithm. The more experiments you run, the harder it gets to remember what works and what doesn’t.

Sometimes you see a great result but because it took hours or days to produce, you’ve already changed the code. And now you can’t remember what parameter you used to get that result!

Of course, this is an eternal problem for all scientific research. It’s one that’s traditionally solved by the Lab Notebook: a journal where scientists carefully record every experiment they run.

Many data scientists follow a similar approach, keeping notes of their experiments digitally or by hand. But this doesn’t always work for machine learning. You need to keep track of three things: the data, the code, and the model. And even if you’re super precise in your note-taking, it’s not feasible to record everything in full.

Luckily, you can automate experiment tracking with MLFlow. Not only is it simple to set up, and it adapts easily to your existing workflow.

What is MLFlow?

MLFlow is an extensive machine learning platform with several components. We’ll just focus on its experiment tracking features for now. We’ve talked before about model registries and why they’re valuable. There’s definitely some overlap between experiment tracking and model registries, but they aren’t the same.

Experiment tracking vs model registries

Model registries are for system managers and focus on models that make it to production. On the other hand, experiment tracking tools are for scientists to track all experiments, including unsuccessful ones.

Registries track and visualise these models’ performance and associated metadata. But for every model in production, there are usually hundreds of failed attempts. Experiment tracking is designed for scientists and researchers to help make sense of unsuccessful and not-yet-successful experiments. It makes the research process more efficient by ensuring that work isn’t lost or duplicated.

MLFlow’s Python library and dashboard

The two components of MLFlow you’ll use for experiment tracking are its Web UI and Python library. You can get them both by running a single install command:

pip install mlflow

You can immediately view the dashboard where all your experiments will live by running mlflow ui and visiting http://localhost:5000 in your browser.

This dashboard already has a bunch of similar features to a Lab Notebook: you can take notes, search through previous data, and set up new experiments.

Things get more interesting when you use this in conjunction with the same mlflow Python library you’ve just installed. By adding a few lines to your existing machine learning code files, you can easily track every single run. This automatically saves the parameters you used, the results, and even the full model binary.

The screenshot below shows how you can start to automatically track your experiments in around ten lines of code.

The highlighted lines are MLFlow-specific, while the rest is a standard scikit-learn example to predict wine quality.

Each time you run an experiment with this code, it logs an entry you can view in the dashboard. Note the following:

• In the Python code, we referred to an experiment_id`. You can see this in the dashboard after you create a new experiment in the UI. One experiment can have many runs, so you won’t need a new experiment ID each time you try new hyperparameters or other variations.
• Your existing training and evaluation code needs to be within the mlflow.start_run block.
• You can log parameters, metrics, and the entire model with the methods log_param, log_metric and sklearn.log_model.

From here, you can change parameters as you like, without worrying about forgetting which ones you’ve used. For example, in the screenshot you can see that MLFlow tracked the result of two experiments where we changed the alpha and l1_ratio parameters, making it easy to compare how these metrics affected the r2 and rmse performance.

Keeping free-form notes

The reason many scientists keep their own notebook rather than using a structured platform is they feel that pre-created templates aren’t flexible enough.

Something we like about MLFlow is that you can keep notes at different levels: notes relating to an entire experiment, or to a specific run.

How MLFlow works internally

MLFlow can be set up in different ways. You can do it locally, as we did in the example above, or on a remote server, which lets you share a single dashboard among a whole team of scientists.

If you install it locally, you’ll notice it creates a directory called `mlruns` in the same directory where you executed your Python code. Inside this directory you can find the full model binaries (as .pkl files) for every experiment run.

If you want to track lots of models and you need more space, you can configure MLFlow to store these files on S3 or another cloud storage provider instead.

Scaling up your machine learning infrastructure

We use MLFlow as part of our Open MLOps architecture. You can use MLFlow on its own but it also works well in conjunction with other MLOps components, such as Prefect for scheduling and managing tasks, and a cloud-based Jupyter Notebook environment for easy collaboration.

We love discussing the challenges of building and keeping track of machine learning models. Contact us if you’d like to discuss your team’s approach to machine learning.

How We Track Machine Learning Experiments with MLFlow was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

‍What is MLOps?

You’ll find conflicting definitions on MLOps: Is it a movement, a philosophy, a platform, or a job title? Most are either far too vague — a “philosophy” — or far too specific, just referring to one specific toolset.

Here’s our definition of MLOPs:

MLOps is a collection of industry-accepted best practices to manage code, data, and models in your machine learning team.

This means MLOps should help your team with the following:

• Managing code: MLOps encourages standard software development best practices and supports continuous development and deployment.
• Best practice: Guidelines ensure you move seamlessly from ideas, to experiments, to deploying reliable models in production.
• Managing data: A framework and workflow help to process, save, and track versions of your datasets.
• Efficient collaboration: Teams can share code, data, models, and experiments; run and understand each other’s work; and iterate on previous work.
• Managing models: You can easily train models, track experiment results, and deploy robust APIs.

MLOps is a broad field, so we’ll take a high-level view of the landscape then dive into topics you’ll encounter when adopting it.

The MLOps landscape

Deciding on the best tools to use and how to get started can be hard. Our Machine Learning Tools page offers an overview and simple explanations of the most useful tools.

These MLOps tools cover everything from data wrangling, visualisation, and task automation, to training and deployment. We’ve focused on open source options but with so many proprietary platforms available too, is it worth shelling out?

MLOps: Build vs buy

Numerous commercial platforms aim to make MLOps simpler and “build vs buy” is a question many teams ask. People often “buy” because they lack confidence in building their own tools or using open source options. Expensive proprietary platforms promise to simplify all the complexity.

But proprietary tools often fail on their promises and end up costing and limiting your team instead. The shortcuts they guarantee are often impossible; and your team will still need internal MLOps expertise to use them effectively.

This is why we’re strong proponents of Open Source tooling for MLOps. Free, open source tools are often the correct choice: They result in lower costs, more flexibility, more learning, and easier onboarding — in spite of what the proprietary platforms would have you believe.

If you trust your engineering team, we recommend building your own solution from existing open source components over trying to fit your unique needs into someone else’s proprietary platform.

But if you want to build your own, how do you get started without wasting months researching all the options? We looked for an open source MLOps blueprint but couldn’t find one, so we built it.

Now you can skip the months of research and engineering we did and set up an open source, production-focused MLOps framework in a few hours.

An open-source, ready-to-go MLOps architecture

If you want the power and flexibility of your own solution but the simplicity and speed of a managed proprietary solution, take a look at our Open MLOps architecture. It’s a set of terraform scripts to set up the same system we use internally on Kubernetes and you can be set up in under a day.

Here are the advantages of Open MLOps:

• Free, flexible, and open source: We built Open MLOps entirely using open source tooling. This means that if you have other needs, it’s easy to adapt: Simply swap out the components for other tools or your own custom solutions.
• Easy to start: Many MLOps tools have steep learning curves. We’ve written step-by-step guides so you can walk through an example project in a few hours, then start running your own.
• Scalable: Open MLOps runs on Kubernetes, so it’s easy to scale up or down, depending on your needs. If you’re running a huge workload, you can just add more compute power. If you’re on a budget, you can run it on a small cluster.

Open MLOps includes the following components:

• JupyterHub, a shared notebook environment for your team to collaborate.
• MLFlow to track your experiments and models.
• Prefect to manage your workflows and scheduled tasks.
• Seldon to productionize your models and turn them into APIs.

You can set up everything in your existing Kubernetes cluster or run our simple set up script to create a new one using AWS EKS. Even if you use pre-built solutions, you’ll probably still need to build internal expertise in MLOps so we’ve listed some good resources to get you started.

MLOps resources

We’ve written many articles on MLOps. Here are our favourites, to help you at different stages of your MLOps journey.

Getting started with MLOps and Machine Learning

If you’re looking to adopt MLOps and want to learn more before you start building, you should start with the following:

• The Four Ways of Doing Machine Learning: Understand the different ways to adopt machine learning at a high level.
• MLOps for Research Teams: Understand why MLOps isn’t just for industry but is important for research teams too.
• Machine Learning Architecture Components: Understand the important pieces for most machine learning solutions.
• Open Source Software for MLOps: Understand why open source is usually better than proprietary solutions.
• Software Development vs Machine Learning Engineering: Understand why machine learning engineering has additional challenges that trip up even experienced software engineers.
• MLOps for Model Decay: Understand why you can’t “fire and forget” your models, even if they perform well.
• Why is a Model Registry Valuable?: Understand how and why it’s important to track your models.
• Why You Need a Model Serving Tool: Understand how to serve your models in a production environment.

Choosing tools to set up your own MLOps platform

Once you’ve chosen to adopt MLOps, you’ll need some specific tools and platforms. Given the breadth of options, this is probably the hardest part so we’ve compared our favourites and cut through the marketing speak to make it easier for you.

• Choosing a Feature Store: Why you need a feature store and how to choose one.
• Why We Love Prefect as a Data and Workflow Platform: Why you should consider Prefect as your task scheduler and workflow tool.
• Kubeflow: Not Ready for Production?: The problems we found with Kubeflow before switching to Prefect.
• Comparing MLOps Platforms: Dataiku vs Alteryx vs SageMaker vs Databricks
• Comparing Dashboarding Solutions: Streamlit vs Dash vs Shiny vs Voila
• Comparing Workflow Platforms: Airflow vs Luigi vs Prefect vs MLflow vs Kubeflow
• Comparing Data Wrangling Tools: Pandas vs Dask vs Vaex vs Modin vs Ray

Setting up Open MLOps

Once you’ve chosen your tools, you’ll need to set up and configure them. If you want to emulate the setup we use internally at Data Revenue using Open MLOps, we’ve created some step-by-step guides to get JupyterHub, Prefect, Seldon, Dask, and MLFlow up and running quickly.

• Setting up Open MLOps: An Open Source Production Architecture for Machine Learning
• Using JupyterHub, Prefect, and MLFlow on Open MLOps
• Deploy Your Model as an API with MLFlow, Prefect, and Seldon

Getting help

If you need help setting up your team with MLOps, feel free to reach out.

MLOps: The Ultimate Guide was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s common for machine learning teams to get great results on a test set, deploy the model to a real-world setting, spend two weeks watching to ensure it still gets good results, then move onto another project.

However, because machine learning models often interact with real-world events, not just static datasets, their accuracy can degrade over time. And without automated monitoring, it’s very possible that no one will notice for quite some time.

At some point, probably far later than ideal, you might pick this up. Your team will go back, retrain the model, push out an update to production, monitor manually for a couple of weeks, then move on again.

While this optimistic mindset of “assume things are working as expected” can work in cases where the results aren’t that important, it can be disastrous in higher-impact scenarios.

Here’s how you can automatically detect model decay, update a model, and deploy it to production using MLOps.

How to think about automated monitoring

The first step to automated monitoring is to detect the issues. Proactively identifying model decay, not simply reacting when problems arise, is key. You can prevent a lot of issues, even if you need to invest some time to manually retrain and redeploy a model.

In theory, this is straightforward. You set up a monitoring system and it pushes you an alert when your system’s accuracy falls below a certain threshold.

In reality, it’s more difficult. You might have a manually annotated “gold” dataset for training, but in production, you don’t know if your model is getting things wrong: if you did, you wouldn’t need the model in the first place.

Three strategies for monitoring models

To calculate your predictions’ accuracy, you can do one of the following:

• Monitor a metric that correlates with accuracy. If a spam classifier generally predicts 30% of messages as spam but one day predicts 75%, for example, there’s probably not been a huge rise in actual spam; it’s more likely the model is broken.
• Human-in-the-loop sample. For example, you might manually check 5% of all predictions, and monitor the accuracy of those..
• Monitor retrospectively. If you make predictions about the future, like whether a patient will recover in a month, then you’ll be able to verify your accuracy later on.

Once you have a metric that acts as a reasonable quality check, you’ll need a tool to monitor it. This can either be a generic monitoring tool like DataDog or a specialized machine-learning monitoring tool. A specialized tool makes it easier to get more useful metrics for machine learning scenarios, like an error distribution or Mean Absolute Error. But if your use case is simpler and you’re already set up with another monitoring tool, it’s probably faster to start with that.

Automatically retraining and deploying your models

Detecting decay is the most important step, but it’s not that valuable unless you can do something about it. Manually retraining a model and deploying it is one option, but then your team needs to be constantly “on call”, waiting for the alarms to trigger.

Manual retraining involves a lot of effort and carries more risk of human error, especially if you need to do this frequently.

If you have a full MLOps architecture set up, you should have everything in place to automatically retrain your models and deploy them — either on a fixed schedule or when you detect model decay.

In software engineering, a CI/CD or “continuous integration/continuous deployment” framework is common. This means you’ve set things up so that new code can be automatically integrated into existing code, tested, and deployed.

In MLOps, you also want to retrain your models, so you need CT/CI/CD, or “continuous training/continuous integration/continuous deployment”.

You’ll need to invest time upfront to set up a CT/CI/CD pipeline, but in our experience, most teams do things manually for too long.

Tools you can use against model decay

There are plenty of MLOps tooling options, but here’s what we use in automatically retraining our models to avoid model decay:

• Prefect as a workflow and dataflow tool. This lets us define a graph of tasks to be run and then automatically execute them, monitoring and re-running if necessary. You can read more about what we love about Prefect. Prefect not only handles the automatic retraining of models, but also the scheduled monitoring to help us analyse and detect decay.
• Feast as a feature store. With Feast, you can track changes to data and share features between models. Learn more about feature stores in our article, Choosing a Feature Store.
• Seldon for serving models. With Seldon, you can easily turn any model into an API and serve it at scale, without writing serving code from scratch.

Do you need help setting up MLOps for your research team?

What often makes teams hesitate to invest in a proper architecture is the number of unknowns: they don’t know how long it will take or how to start. We’ve done it before and would enjoy helping you skip the grunt work, so contact us if you want to chat.

MLOps Can Help Fix Model Decay was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Machine learning teams can use proprietary platforms like Knime or SageMaker, or build their own using open source tools. Companies often sell proprietary platforms as more powerful, more efficient, and easier to use. But in reality, they’re often more complicated and less powerful than open source alternatives.

We build our own MLOps architecture from purely open source foundations. Here’s why ownership of your platform is important.

Open source is higher quality

People are sometimes suspicious of free tools but open source software is often higher quality than paid alternatives. Megacorps like Google and Microsoft rely heavily on open source software and frequently have their best engineers working on these projects.

Open source projects also benefit from having many more eyes on them than closed-source alternatives. If a developer finds a bug, they can contribute a fix that benefits everyone.

Finally, open source software is usually not marketed, meaning incentives are better because quality is the key factor determining success. By contrast, the success of commercial projects can depend far more on marketing, partnerships, and branding. This means companies are often incentivized to spend more time and effort on these areas than engineering.

“You get what you pay for” is therefore a real misnomer when it comes to open source software.

Open source leads to transferable skills, which makes hiring easier

Engineers want to learn open source tools because they can use this knowledge at other companies, too. No one wants to be trapped in their current job because they spent ten years learning the ins-and-outs of some internal platform.

Training your team to use tools valuable to other companies, like TensorFlow and Kubernetes, might make some people nervous, but really it’s the only sustainable way to attract top engineering talent.

Your team can benefit from these transferable skills in other ways, too: it’s far easier to find expert consultants or even simple community-sourced help if you use the same tools as everyone else.

Open source solutions are more modular

Being “all-encompassing” is a key way enterprise platforms try to differentiate themselves. Instead of building only a training platform, or only a deployment tool, they sell themselves as “all-in-one” solutions or “the last tool your team will need”.

But being the best at everything is unrealistic. And the monolithic nature of these solutions makes it harder to swap out individual components. If your team wants to use the best model registry, often they just can’t plug it into their existing platform.

By contrast, open source software is usually more granular and has a strong focus on integrating with other platforms. This means an open source solution is more like playing with lego: if part of it’s giving you trouble, you can simply detach it and switch in an alternative.

There are hidden costs in proprietary platforms

Proprietary platforms are incentivized to constantly sell upgrades and new features to existing customers. In many cases, this means they’re not upfront about their limitations.

During sales, they might convince you that a low or mid-level plan suits your needs perfectly. Only after buying in do you realise they’ve strategically removed key features, or added specific limits designed to cripple your workflow until you pay for the next upgrade.

These platforms can also raise their pricing with next-to-no notice, so your team can either scramble to rewrite everything or pay more than you budgeted.

When does it make sense to use proprietary platforms?

If your team doesn’t have much engineering expertise, especially when it comes to DevOps and creating and managing infrastructure, then setting up and maintaining open source solutions can be challenging.

Open source tools tend to be built “by developers for developers”, so non-technical teams like marketing might also prefer proprietary platforms in some cases.

Finally, if you’re working on small internal projects or proof-of-concept applications, there’s often little need for infrastructure. If people aren’t using your solution as a product, an off-the-shelf tool for something like a quick visualisation is often good enough.

Does your team need help adopting open source MLOps?

We’re a machine learning agency that loves helping research teams set up open source MLOps infrastructure. Contact us any time if you’d like to discuss MLOps tools.

Why Open Source Beats Proprietary Software for MLOps was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Calculating the similarity between two MS/MS mass spectra is an essential step in most untargeted metabolomics workflows.

However, it’s long been known that cosine similarity — the most popular similarity measure — has significant weaknesses.

Justin van der Hooft and his team have developed a new approach to calculating spectral similarity: Spec2Vec. This is really helpful for analyzing untargeted metabolomics results in biomarker discovery or natural product research.

Let’s see how Spec2Vec works and why it’s a good alternative to cosine similarity.

Why mass spectral similarity metrics matter

A datapoint in an untargeted metabolomics dataset is, at first, just a measured entity. We don’t know yet what molecule it might be. Hence, we can’t interpret its significance: Is it something new? What is it’s metabolomic function?

But if you can match this entity against a database, like the Human Metabolome Database, then you can add (annotate) important information to that entity. And in order to match your measurement and the entries in the database, you need a similarity metric.

This is how mass spectral similarity metrics are mainly used:

• Spectral library matching: Matching a compound against a spectral library (such as HMDB).
• Analogue search: Finding the top hits in a large spectral library, to look for structurally related molecules.
• Mass spectral networks: Clustering related compounds in a mass spectral network.

Any mistake can be costly: If you can’t find a match, then your compound might seem to be an exciting new drug candidate, when in reality, it’s a well-known molecule; If you only discover this at a much later stage of your research, then you’ve wasted a lot of effort.

The better your similarity metric is, such mistakes will happen less frequently.

The drawbacks of cosine similarity

Cosine similarity measures how much one mass spectrum overlaps with another mass spectrum. If two mass spectra have near perfect overlap, then the cosine similarity is high — and their molecular structures are likely very similar.

Sometimes though, you have structurally similar molecules whose mass spectra do not overlap — and in those cases, cosine similarity will likely fail. Here’s how that happens:

1. Large molecules modified in several places: Large molecules — as often found in plants and microorganisms — can easily get several small modifications in their molecular structures.
2. Several fragments are shifted: If you fragment a molecule that has several modifications, then not one, but several of its fragments also retain a modification. As a result, several fragment peaks shift on the mass spectrum.
3. Few peaks overlap perfectly: If lots of fragments are shifted, the overall spectrum no longer overlaps neatly with the unmodified molecule.
4. Low cosine similarity: In terms of cosine similarity (overlap), the two molecules therefore seem unrelated and you’ll see a low cosine score.
5. Wrong conclusions: As a result, you might think a library has no molecule similar to your compound, when in fact it does. This could provoke the conclusion that you’ve found a novel molecule, even though what you’ve found is just a slight modification of a well-described molecule.

This is a well-known problem but no fundamentally alternative mass spectral similarity metrics have been proposed as solutions, until Spec2Vec.

Spec2Vec: Unsupervised learning of spectral similarities

At the basis of Spec2Vec is a powerful new assumption: If two fragmentation peaks often show up together, across thousands of mass spectra, then that’s probably because they come from the same molecular substructure.

So those two peaks are related in some way. And now, if you only see one of these peaks in one molecule and the other in another molecule, you can make a guess that the same substructure, maybe with small modifications, is present in both molecules.

Spec2Vec implements precisely this idea. But on a large scale, Spec2Vec takes a dataset of many thousands of mass spectra and learns from this dataset what peaks are related. Then you can use what Spec2Vec learned to calculate spectral similarity between any two mass spectra.

If this works, then a molecule with several small modifications — like the previous example — shouldn’t fool Spec2Vec. Now, does this work?

Comparing Spec2Vec with cosine similarity

Let’s look at the results of two tests. For the tests, we use a large molecular library from GNPS that contains molecule structures and their spectra.

Test 1: How well do Spec2Vec scores correlate with structural similarity?

First you need a ground truth against which you can compare Spec2Vec and cosine similarity. Justin and his team took 12,797 unique compounds and used their known molecular structures to directly calculate the structural similarity score (Tanimoto score) for each pair of molecules.

For each of the same pairs (81,875,206 pairs in total), they also calculated cosine, modified cosine, and Spec2Vec scores, but this time, only using the library spectra, not the known structures. This was the result:

• Spec2Vec correlates considerably better with structural similarity;
• Only when spectra overlap almost perfectly do all metrics perform similarly well.

Spec2Vec’s high correlation with structural similarity (Tanimoto scores) is very promising, but does this translate to better library matches?

Test 2: Can Spec2Vec find the right matches in a library search?

In a second test, Justin’s team selected 1,000 unique molecules from the spectral library that also had at least four planar structural equivalents (these look the same in 2D) in the library. These near-equivalent molecules remained in the library.

The task then was to figure out if the molecules that had the same 2D structure would get the highest similarity scores.

The results were in line with the first test: Again, Spec2Vec found considerably more reliable matches than cosine similarity, across all score ranges.

But Spec2Vec has even more advantages.

Spec2Vec is much faster

Cosine similarity calculations are very expensive on mass spectra because of the preprocessing they need. They’re so expensive that it’s usually impractical to make comparisons against an entire large database.

A Spec2Vec model, once it’s trained, is extremely fast: roughly 100-times faster than cosine similarity scoring. Even if you include the time to train the Spec2Vec model, it’s still around 10-times faster overall (without taking any possible optimizations in cosine similarity into account).

This means that with Spec2Vec, you can easily run all-vs-all searches against large libraries, without any limitations.

And Spec2Vec isn’t difficult to implement, even though it’s a much more advanced approach to similarity scoring.

How to integrate Spec2Vec into your workflow

You can implement Spec2Vec with an easy-to-use, Python open-source package that is freely accessible on Github: https://github.com/iomega/spec2vec. You can install it with Anaconda (recommended) or pip.

You can even download the pre-trained Spec2Vec model Justin used for the tests above, from Zenodo. And you can calculate mass spectral similarities without training your own model.

>> Important to note: Spec2Vec is an unsupervised algorithm. This means you can train it on any set of MS/MS spectra you have, without any additional information.

This is just the beginning

Spec2Vec is a great approach and can probably be easily extended:

• GC-MS: So far, Spec2Vec has only been tested on LC-MS data. It might be even more valuable for GC-MS studies because in GC-MS measurements, you can’t limit library searches by pre-filtering by the similarity of precursor ions.
• Supervised: It’s great that Spec2Vec is unsupervised. But you could also build a supervised model: For example, you could train a model that predicts the Tanimoto score directly from the spectral embeddings. This might give you even more accurate models.
• Improved Mass2Motifs: Spec2Vec vector embeddings could potentially be used to train more powerful Mass2Motifs.
• Improved Network Annotation Propagation (NAP): Similarity scores are essential in building mass spectral networks, so if Spec2Vec produces more useful networks, it could also improve your annotation and speed up structural elucidation.

In general, any tool that currently uses cosine structural similarity can probably benefit from integrating Spec2Vec as an alternative similarity metric.

Further Reading

If you want to learn more about Spec2Vec:

• Try the Spec2Vec python package;
• Try Spec2Vec within GNPS;
• Read the Spec2Vec paper; and
• Check out Florian Huber’s Spec2Vec Tutorial.

Do you need help applying machine learning to omics research?

We can help your research team set up the right infrastructure for reproducible and easily deployable machine learning models. Just get in touch.

Spec2Vec: The Next Step in Mass Spectral Similarity Metrics was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Zebra Medical Vision has 7 FDA-approved solutions — here’s how they did it

The peculiar lack of clinical AI diagnostic solutions

Many research studies have shown that machine learning models can perform as well as trained doctors on a range of tasks — especially in radiology — and can even find patterns a human doctor would miss.

But for many years, AI-supported diagnostic solutions were not actually being used in hospitals. In other fields, we’re used to a quick translation from research to practice: after a breakthrough in facial detection technology, you might see Facebook using the same technology to tag your friends in less than six months. But in medical diagnostics, innovation simply didn’t translate into practice. Why not?

This is exactly the situation Eyal Toledano faced in 2014, when Zebra Medical Vision was born: lots of breakthrough research, an obvious need for AI support in diagnostics, plenty of talent, but almost no solutions in day-to-day clinical use.

So they went looking for the reasons behind this reality. They found that research alone couldn’t address some of the critical challenges which needed to be solved before AI diagnostics could be applied in clinical contexts. Doctors and clinicians needed:

1. Models trained on datasets that represent the population;
2. Seamless integration into clinical workflows;
3. A technology platform to connect research and hospitals.

Challenge 1: Building a representative dataset

The first issue Eyal and his team discovered was that most academic research relied on small datasets. Not only were they small, but they also didn’t represent the variety of patient cases clinicians see in hospitals on a day-to-day basis. For real-world diagnostic solutions, representative data is absolutely crucial.

So the first challenge is getting a large, diverse dataset that looks exactly like what you’d expect to see in a hospital.

That’s why, for the first two years, Zebra Medical Vision hardly did any machine learning at all. Instead they focused on developing data partnerships with over 30 hospitals in Israel, the US, and India.

Collecting a full view of the patients

While many studies might simply compare doctors’ and machines’ diagnostic performance using one type of data, Zebra Medical Vision had to go further. They collected data on:

• Radiology images in many different modalities (X-Ray, CT, MRI, Mammography, PET, and Nuclear Medicine);
• Lab analyses;
• Patient admission and discharge;
• Doctors’ diagnoses;
• Clinical outcomes (meaning how patients fared after discharge, possibly years later).

The key factor here was that the samples Zebra-Med collected had to represent the exact same diversity and complexity doctors face in the hospital every day.

Proof-of-concept vs. a clinical diagnostic solution

Let’s say you build a system that can take a radiology image and then correctly judge whether the image shows signs of lung cancer or indicates a healthy patient. At first sight this might seem very useful. But it wouldn’t be useful in a hospital.

So far, this is just a proof-of-concept — you’ve demonstrated that an algorithm can process radiology images and differentiate between two clearly described groups of patients. But does this scenario (lung cancer vs. healthy patient) really represent the problem a doctor would face?

In fact, this isn’t a question that comes up in a hospital. It’s more likely that a patient comes to you because they have some symptoms already, and you need to diagnose the cause: ground-glass opacity, pulmonary embolism, COPD, emphysema, and lung cancer might all present similar symptoms.

If the machine learning model is going to be of any use to the doctor, it needs to be able to differentiate among radiology images of patients with symptoms — not between healthy and unhealthy people.

To put it another way: the data you train the diagnostic model on needs to represent the population you expect to see in the clinic.

But this isn’t the only hurdle to making a useful model for everyday hospital diagnosis.

Challenge 2: Integrating seamlessly into clinical workflows

It’s safe to assume that doctors, like anybody else, want new solutions to make their work easier — not more complicated.

But academic research leaves this issue unresolved. For example, you might develop a very accurate machine learning model that only performs well when the image is taken with specific machine settings — settings a doctor wouldn’t normally use in a routine exam.

Solving the problem in this way is certainly a valuable research result, but it’s still far from a workable diagnostic model for clinical practice, because it requires a non-standard workflow.

In practice, the model has to fit seamlessly into the existing workflow. This means:

Using the data produced in the standard workflow

A clinically useful model should not require the clinician to perform any additional steps. It should only require the exact data — in the exact format — that’s already produced in the hospital’s normal diagnostic workflow.

Working with the software doctors already use

Zebra Medical Vision didn’t build a new software tool for doctors to add to their workflow. Instead they partnered with diagnostic workstations and integrated their predictions directly into the software tools the doctors were already familiar with.

This seamless integration is essential — not only because asking doctors to change their workflow is unrealistic, but because the whole point of using AI is to save time, not to create additional work.

But this wasn’t easy. There are a wide variety of workstation providers and software, as well as different kinds of environments (e.g., web-based or Windows based). Plus a lot of the software is quite old. Nevertheless, Zebra-Med has to integrate with each of them.

Separating integration from computation

Zebra-Med built a smart infrastructure that allows for flexibility while minimizing double work. The hospital only has to install an agent in their local system once, and this agent deals with privacy, security, and anonymity, as well as providing a local user interface.

On the other hand, all the heavy lifting and prediction happens in the cloud. This also makes it very simple for Eyal and his team to roll out new versions of the model. They simply update the model on their side, and the agents installed in the hospital communicate with the centrally hosted model. This means hospitals don’t need to do anything to receive updates.

Plus Zebra-Med calculates the predictions in advance, so when a physician clicks on an image, they can see the result immediately.

So let’s say you’ve managed to build a model that accurately represents the population, seamlessly integrates into diagnostic workstations, and fits the doctor’s workflow.

Even then, you still can’t be sure that the doctor would benefit from using your diagnostic model. Here’s why.

Saving the doctor time

Let’s say there are 6 cancer diagnoses for every 1,000 mammographies, and you’ve trained a machine learning model that can assess the images in real-time. The doctor now sees the AI model’s assessments next to all of their scans and can prioritize the cases that have been flagged.

Now we face two crucial considerations:

The false-negative rate. How often does the model think a patient is healthy when in fact they are not? This kind of error is obviously very dangerous. If the doctor doesn’t look at a scan because the system says the patient is healthy when they aren’t, then the patient might miss their chance to get treated. This is a serious error that must be avoided at all costs.

The false-positive rate. How often does the system flag a scan when the patient is in fact healthy? This error is much less problematic — the doctor will cross-check the scan because the system flagged it, and will discover it was a false alarm.

But overall, the false positive rate still determines whether the system saves the doctor time. Imagine you see that a scan is flagged. You’ll inspect it diligently, try to understand why the AI system might have deemed it problematic, and spend a lot of time deliberating before you conclude: “No, the AI made a mistake. This patient is actually healthy.” Let’s say this is the case for 49 out of every 50 scans the AI flags.

In this case, it doesn’t matter how much more quickly the AI can make a single assessment if the doctor has to spend extra time checking and refuting most of the cases the AI flagged.

To assess whether an AI solution can truly speed up a particular diagnostic workflow, you need to consider:

• Integration. How seamlessly can your model integrate into the current tools? Does an AI assessment fit into the workflow?
• Presentation. Can you present the predictions in a way that makes sense to the doctor?
• Clinical prevalence. How many cases do you expect in a given number of scans? And can you reach a false-positive rate that the doctor can tolerate?

So far, we’ve highlighted how important it is to get real clinical data as the basis for building a model that can perform on the exact population and data present in the clinic. And we’ve shown that simply having an automated diagnostic model doesn’t necessarily mean you’re saving a doctor time — seamless integration and a manageable false-positive rate are essential.

But if we want to connect research with the clinic — in a way that satisfies regulatory requirements — we’re still missing something.

Challenge 3: Building the technological backbone of an AI diagnostic solution

A proficient data science and research team is essential, but it’s not sufficient.

Once Eyal and his team had both the data and a means of getting solutions into clinicians’ hands, they had to build a highway on top of this bridge: the technological backbone that would make everything else possible.

Annotating and curating data at scale

Machine Learning models learn from the dataset. So if there are any mistakes in your data — like a wrong diagnosis — then the model will also learn those mistakes. Before you run any experiments, you need to double check and correctly annotate all the data points in your study.

For their studies, Zebra Medical Vision had to coordinate support from up to 60 different expert annotators worldwide — all working on the same clinical mission. At this scale, Eyal had to build internal tools to collect, compare, and consolidate all those annotations.

Running thousands of experiments at a time

In the past, a researcher might take the time to design the perfect experiment, implement it, and then assess whether their approach solves the problem. But this doesn’t work for machine learning: there are so many ways to slice the data and build a model, you would never reach the end.

So Eyal [and the team] needed to build a platform to allow researchers to test not one, but thousands of experiments at the same time. They also built tools to track all of these experiments and then compare the results.

Separating research and clinical systems

Many companies talk about feedback loops — where the model improves via user feedback. For clinical solutions, that’s not usually realistic or responsible.

A diagnostic model is considered a medical device, and any change has to go through the process of regulatory approval.

Zebra Medical Vision’s two systems for research (model training and testing) and clinical use are entirely separate. There is a firewall between them, and they’re even hosted in two separate physical locations.

Of course, the team still appreciates getting feedback from doctors so they can understand how the model is performing and put that learning to use in future iterations.

But this is just one of the hurdles of working in a regulated environment.

Fulfilling regulatory requirements

Everything connected to the development and use of a diagnostic model needs to be traceable, including:

• Product definition and system requirements;
• Design;
• Data curation;
• Research, development, and engineering;
• Testing;
• Release processes;
• Customer complaints.

Zebra Medical Vision built a global quality management system which provides this information to the FDA on an ongoing basis, gaining trust through transparency. They also adhere to and are tested on three different ISO standards, as well as SOC 2 Type 2 internal controls reports issued by third-party auditors.

This required a huge investment in security, as well as making sure everything is logged and documented.

Now we covered partnerships, data, integration, and the technical backbone. They are all necessary. But these elements are still not sufficient to solve a machine learning diagnostics problem — or what Eyal calls a clinical mission.

The Band: An interdisciplinary group with a clinical mission

Zebra Medical Vision very quickly learned that data scientists can’t build a useful medical device all on their own. What’s more, if you pair clinicians with data scientists you’re still stuck, because even when they think they’re talking about the same thing, they’re usually not. Someone needs to fill the gap.

Translating between two worlds: The role of the clinical information manager

Zebra Medical Vision needed someone with clinical trials management experience — someone who could speak the languages of both the clinicians and the data scientists. They dubbed this role the “Clinical Information Manager” — and this person usally has a PhD in biomedical engineering or clinical research.

Each clinical mission also needs a project manager, a research engineer, and an operations engineer.

Eyal calls this unique interdisciplinary team the band: different talents, same clinical mission.

Each band member has a specific role:

• The project manager keeps the project on track.
• The clinician brings knowledge about the complexity of everyday reality in the hospital.
• Data scientists and researcher engineers formulate and test assumptions on the data and also know how to train and validate machine learning models.
• The clinical information manager understands clinical research and bridges the gap between clinicians and researchers.
• Operations engineers implement the clinical solution in a robust and scalable way.

Having a band that works closely together means everything can be much more dynamic. As Eyal discovered, this flexibility is absolutely essential.

Dynamic problem solving: The mission always changes along the way

In university studies, the problem you’re working on is often fixed. But Eyal and the band found that they very often make discoveries along the way. And because those discoveries improve their understanding of the problem, they often change the clinical mission.

The band is very well suited to adjust to these changes. One reason is Zebra Medical Vision’s technological backbone, which allows them to work fast and to easily modify and rerun experiments — without starting from scratch.

With all this infrastructure and this team in place, Zebra Medical Vision can move at a dazzling pace. They’re already on their way to their seventh FDA-approved diagnostic solution, with more coming soon.

Now let’s look at two examples of diagnostic solutions Eyal and his team developed, and find out what they learned along the way.

The difference between real-world and research approaches to medical image solutions

Example 1: Early warning for coronary artery disease

Half of all cardiovascular-related deaths are due to coronary artery disease. Cellular waste products, proteins, and calcium stick to blood vessel walls and combine with fat to form plaque. If this happens in the arteries that supply blood to the heart muscle, then it can limit or stop the supply of oxygen to the heart and cause a heart attack.

Unfortunately, the buildup of calcifications in coronary arteries is often only diagnosed after a heart attack or similar cardiac event.

The academic research approach: When Zebra Medical Vision looked into this problem, they found an academic paper saying that if you manually segment the areas of interest, and have a gated CT scan (a scan focused only on the heart), and measure with and without contrast in a specific protocol, then you could build a model that can provide the equivalent of the Agatston score (a risk measurement from 0 (low risk) to 400 (very high risk)).

Essentially this is a segmentation problem: you segment the white clouds of calcification in the coronary arteries.

The problem with this approach: This particular CT scan protocol is something you would only run on a patient who is already known to have a risk of a heart attack. And therefore, this would not help all the other patients who are at risk but have no symptoms yet.

Considering how few patients have symptoms before a heart attack, it would be more helpful to find a way to diagnose a much larger group of patients.

The more practical approach: Zebra Medical Vision realized that patients get CT scans for many other diseases. These so-called untargeted or ungated scans can include many organs, and also the heart. Eyal and his team found an approach where they could take these much more frequent scans and still achieve a similar accuracy — in predicting risk for heart disease — to the model that the researchers build for the targeted scans.

This means doctors now have an early-warning system for heart disease running in the background. This system automatically alerts them if a patient is high risk — even though they took the scan for another reason and may never have looked at the heart. Hence many patients are diagnosed earlier and receive preventive treatment.

Today this is one of Zebra Medical Vision’s leading solutions, and it’s proven to be effective on a large part of the population. It has even outperformed other solutions that only work on gated CT scans.

Example 2: Vertebral compression fractures

Early diagnosis and treatment for osteoporosis is essential. But vertebral compression fractures — a reliable symptom of osteoporosis — are often missed during routine exams.

Vertebral compression fractures (VCFs) — a condition in which part of a vertebra bone in the spine collapses — are often simply ignored by radiologists. It’s not part of their standard workflow, the diagnosis is often not acute, and they’re tedious to diagnose: The radiologist has to check another section of the scan and then compare each vertebra’s height to its baseline height. In the end, 75% of all VCFs go undiagnosed or unreported.

But most people in high-risk groups already have a scan somewhere on file. That means the pixels are there and the compression fractures are already captured in the data — they’re just not reported.

Even though the problem isn’t sexy and doesn’t receive much attention, the benefits of diagnosing VCFs — and consequently osteoporosis — are immense for the patients. For example, 50% of patients who fracture a hip die of complications in the next 10 years. Then there’s also the immense burden of rehabilitation.

So Zebra Medical Vision built an algorithm that locates and identifies these compression factors. This helps the clinicians who really care, who deal with osteoporosis prevention and treatment programs, to identify patients with VCFs.

Now hospitals that run screening programs on patients who are at risk of osteoporosis can run this model in the background and are automatically alerted to patients who likely have a fracture. Then doctors can confirm these cases with a manual check.

As Eyal says, “Sometimes we look at the problems that are meaningful and not sexy.” This is exactly the kind of opportunity you find when you look at large datasets with an exploratory mindset — a data scientist’s mindset.

Accomplishing the impossible with naive optimism

In May 2014, at the very beginning, Eyal and his colleagues went to a radiology conference in Dallas. Everyone told them: “There will never be anything like machine vision in radiology. That’s a fantasy. You’re nice Israeli guys. Enjoy your vacation, go back to Israel, and find something else to do.”

But their naïve optimism saved them. Eyal and his co-founder thought they could solve this problem, and fast. In the end, even if it took a bit longer, it would be an extremely meaningful challenge, and the impact and value of the project would be undeniable.

This mindset pushed them on through myriad challenges. But as Eyal said, “You need to be extremely stupid — in a positive way.”

How Zebra Medical Vision Developed Clinical AI Solutions was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Machine learning teams often face the same challenges. MLOps is a set of processes most machine learning teams should follow to address these problems.

To make MLOps more concrete, we’ll look at what problems it solves for research teams.

Why does your team need MLOps?

Many machine learning teams focus on building novel algorithms or on training high-performing models. While these are fundamental components of any machine learning solution, they’re relatively small compared to all the surrounding processes needed in a real solution, such as data engineering and monitoring.

Research teams tend to focus more on the machine learning code than the surrounding infrastructure. With an increasing need to collaborate, share results, and re-use work done on large datasets, research teams quickly reach the limit of this approach. MLOps solves these problems and allows research teams to achieve their goals despite the complexity that comes from dealing with large datasets, code, and machine learning models.

What challenges might you face without an MLOps architecture?

Challenge #1: You can’t rebuild existing models

Your team has a trained model that works well but can’t be rebuilt. Data and software versions have changed, or the team member who set up the training pipeline on their local machine has left. Your team can share the trained model file directly but you can’t improve or update it, because some of the steps to build it are now lost.

Challenge #2: You can’t effectively audit or monitor your models

The results of your team’s model were excellent at an evaluation stage. But there is no ongoing auditing or monitoring to ensure the model’s predictions still make sense. Your team needs to exercise a blind faith that the model is operating as expected.

Challenge #3: You can’t easily share models

Each member of your team has built their own pipeline to manage data and train models. It’s difficult to collaborate or share results because you can’t easily work on each other’s models, or re-use each other’s intermediate data.

Challenge #4: Your proof-of-concept looked promising but training at scale crashes your hardware

You were able to train your model on a small dataset on your laptop, but it’s difficult to scale, either by pushing it out to a more powerful machine or by taking advantage of compute clusters.

This means translating the proof-of-concept to a production solution takes significant effort.

Challenge #5 You find errors too late

Because your team builds infrastructure and glue code from scratch for each project, bugs and other issues occur frequently and are often only caught after results have been released. The same problems reoccur for each project because you can’t easily re-use code.

How can MLOps architecture address these challenges?

Implementing MLOps practices and using a standardized architecture can help solve all of these challenges. Most machine learning teams should have an architecture that includes the following:

An experimentation hub

An experimentation hub lets your researchers develop notebooks, experiment with new models and architectures, and validate hypotheses. It helps team members share code and ensures they can reproduce each other’s models.

You only need to set things up once because you’re all working in the same environment. The shared hub prevents headaches over hardware-specific issues, like team members using different operating systems or having underpowered development machines.

We use JupyterHub.

A model registry and experiment tracker

A model registry stores each model your team produces with a name and one or more versions. An experiment tracker is similar but works at a higher level: defining names, versions, and metadata for your team’s experiments.

By storing and versioning every single model and experiment your team produces, you can always reproduce results performed in a specific experiment. You won’t be blocked because you don’t know which model produced specific results or what parameters you used for a historical experiment.

This will also help you comply with any necessary legal requirements to justify your results.

We use MLFlow.

A model serving tool

A model serving tool automatically deploys your model to staging or production environments and makes a unified API accessible to your team and your end users.

A unified API and infrastructure across all of your models make it easier to use and monitor them. Automatically deploying your models to staging and production environments means the latest versions can be put to use immediately.

We use Seldon.

A dataflow tool

A dataflow tool keeps track of every step in your pipelines, and can monitor and rerun steps as required. This helps prevent cases in which a model can’t be rebuilt, because you can easily rerun the exact same steps you used each time.

It also saves your teams’ time and prevents human errors in any repetitive work.

We use Prefect.

A feature store

Sometimes your team needs the exact features used to train a specific model but the underlying data has since changed. Or maybe you’ve spent time building specific features and now want to re-use these for a new model. A feature store keeps versions of every feature for your team to re-use and therefore collaborate more easily.

We use FEAST.

Is MLOps ‘nice-to-have’ or is it essential?

While some research teams still operate without MLOps tools or best practices, we believe MLOps has become an essential ingredient for nearly all teams. Unless your team is very small or working only on trivial problems, your machine learning research also needs to be reproducible, accountable, collaborative, and continuous. Without MLOps, meeting these goals is very challenging.

Do you need help setting up MLOps in your research team?

We love finding the right MLOps architecture for machine learning research teams. Contact us to find out more.

MLOps for Research Teams was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.