MLOps Models: Developing, Deployment and Monitoring
A big part of MLOPs is model development, training, deployment and afterwards monitoring of the model performance. In a graphical way this could look like this.
Experimental Environment and Tracking
When you are developing models, you need access to raw or some curated will be experimenting with a lot of different models and parameters and sometimes it is hard to remember what model with what parameters had which performance. This is where experiment tracking comes in play: It enables your team to track your experiments, promoting them, using it for production ready training pipelines. An example how this could look like can be seen here
Training and Retraining (Pipeline)
Once you have decided which model you want to use can register that model in a model registry, and pull that model to run a training pipeline. You run that training pipeline either against curated production data, that has been run through a processing pipeline, or directly access it from a feature store as mentioned in the first diagram. For promoting the pipeline to higher environments there ideally should be some automated process e.g when specific pull requests are created, the CI/CD triggers the process. This way you still have the control of when to promote the pipeline, but still have an automated process.
After the training pipeline is done, the model artifact is created and stored in the model registry, which then can be used in the deployment pipelines.
Retraining can then either be triggered periodically, e.g once a day or by detecting performance losses or drifts.
Model Registry- why we need it ?
The model registry is essential to glueing the experimentation phase, the training and deployment pipelines together. It is where the model artifact handover happens. Moreover, a good model registry has a good model versioning, so in case a model is performing badly in production, you can easily revert back to a previous version.
Deployment
Once your model artifact is created , it is time to package the model (normally in a container) inside an application. How this is done depends on the deployment type, so to better understand this, here is a graphic:
Batch: In the batch you apply your trained models that are packaged in a container as part of an ETL process. In my experience this is normally done on a schedule after some ETL process has run, and the predictions are saved in a storage which is then used by another application to load the predictions. I would say the majority of use cases I encountered used this deployment type.
Stream: Here you apply your model in a stream processing pipeline:
While Data is continuously piped through your 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀, an application with a loaded model continuously applies inference on the data and the model can write to results into a stream which is then consumed by other applications. To speed up this process, this deployment type can include a real time lookup of precomputed features to retrieve statistical features.
Request Response: Here you expose your model as backend service via for example REST Api. For speedup this can also include a real time lookup of precomputed features to retrieve statistical features. Inference can then requested by any application as long as it is able to form a correct request with respect to the API contract
Monitoring
Once the model is deployed, it is time to monitor certain things to ensure that the model is still working, because once your model is deployed, it faces many challenges that can affect its performance and require continuous monitoring.
Monitoring is divided into two levels: Functional and operational:
Functional Monitoring
At the functional level one can monitor three categories:
Input Data: Models depend on the data received as input. If a model receives an input it does not expect, the model may break. Monitoring the input data is the first step to detecting functional performance problems and extinguishing them before they impact the performance of the machine learning system. Items to monitor from an input data perspective include:
Get Sammer Puran’s stories in your inbox
Join Medium for free to get updates from this writer.
Data Quality: to ensure data integrity, one should validate production data before it sees the model. This validation steps can include: data type check, data schema validation, change in column names etc… Changes in these can break the model and should be counteracted in a change of data pipeline.
Data Drift: Changes in distribution between the training data and production data can be monitored to check for drift: this is done by detecting changes in the statistical properties of feature values over time. Data comes from a never-ending, ever-changing source called the real world. As people’s behavior changes, the landscape and context around the business case you’re solving may change. At that point, it is time to update your machine learning model.
Model
Performance: For your business to derive value, the model must maintain a performance above a certain threshold. As this is not directly measurable at prediction time, one can use following metrics:
Model drift: Model drift is the decay of a model’s predictive power due to alterations in the real-world environment. Statistical tests should be used to detect drift, and predictive performance should be monitored to evaluate the model’s performance over time.
Output
To understand how the model performs, you must also understand the predictions the model outputs in the production environment. A machine learning model is put into production to solve a problem. Thus, monitoring the model’s output is a valuable way to ensure it performs according to the metrics used as KPIs
Ground truth: For some models you can acquire (probably delayed ground truths). If you are predicting arrival times of planes for example, you will get the delayed ground truth after the plane has arrived, and you can use this as ground truth to evaluate the model. However, acquiring ground truth is not always possible, so an alternative method could be to measure
Prediction drift:
When it is not possible to acquire ground truth labels, predictions must be monitored. If there is a drastic change in the distribution of predictions, something has potentially gone wrong. For example, if you are using a model to predict fraudulent credit card transactions and suddenly the proportion of transactions identified as fraud shoots up, then something has changed. Perhaps input data structure has been altered, some other microservice in the system is misbehaving, or maybe there is just more fraud in the world.
Operational monitoring
At the operational level you are concerned that the resources concerned with your ML service are healthy to ensure smooth running. In a MLOps setup, engineers are responsible if there is an unhealthy state, so monitoring the ressources ensures, that a problem can be detected.
System performance
Because issues in the ML stack can affect the whole application, it is important to monitor System performance metrics such as:
Memory use
Latency
CPU/GPU use
Data and Model Pipeline
Two crucial pipelines should be monitored: the data pipeline and the model pipeline. Failure to monitor the data pipeline may raise data quality issues that cause the system to break. Regarding the model, you want to track and monitor the factors that may lead to the model failing in production, such as the model dependencies.
Costs
From data storage to model training and more, there are financial costs involved in machine learning. While machine learning systems can generate lots of value for a business, it is also possible for leveraging machine learning to become excruciatingly expensive. Constantly monitoring how much your machine learning application costs your organization is a responsible step to ensuring costs are maintained.
For example, you can set budgets using a cloud vendor such as AWS or GCP since their services track your bills and spending. The cloud provider will send alerts to inform the team when budgets are maxed.
If you are hosting the machine learning application on-premises, monitoring the system usage and cost could provide greater insight into what component of the application is the most costly and whether or not you can make certain compromises to cut costs.
Conclusion
In this article we talked about how we develop, track machine learning experiments, where to store our artifacts, how to package and deploy them depending on the use case and some considerations for monitoring.
In a later blogpost, I will delve deeper in the tools and details of monitoring and how to track the metrics mentioned before.
