Monitoring discriminative ML models using Amazon SageMaker AI with MLflow

The effectiveness and accuracy of machine learning (ML) models decreases almost as soon as the training job finishes. Changes in consumer behavior, releases of new products, upgrades in sensor technology, and a shifting economic and political landscape are all examples of uncontrollable factors that change the patterns and probabilities the model learned during training. By actively monitoring models deployed in production for changes in accuracy and baseline statistics, you can intervene before the drop in accuracy becomes problematic. Model monitoring can be combined with AI observability tools that track latency, application availability, and other metrics used to identify problems in the overall system.

This post focuses on discriminative machine learning models used for classification and regression use cases. For generative AI models, see Production-Ready Real-Time Monitoring Solution for LLMs on Amazon SageMaker AI Endpoint inference. The factors that cause a reduction in quality for discriminative ML models can be broadly split into two categories:

Figure 1: How data drift and model drift fit in an ML workflow

Amazon SageMaker AI is a fully managed machine learning service that helps organizations build, train, deploy, and manage both discriminative and generative ML models. Although SageMaker AI is fully managed, you might need a more customizable approach. For example, you might want to manage the entire modeling life cycle cost-effectively, monitor unique use cases that managed services do not support, or integrate your model monitoring into other UI or observability pipelines. Therefore, this post introduces a model monitoring architecture based on the open source Evidently Python library and Amazon SageMaker AI with MLflow for calculating data and model drift. The results of this model monitoring solution can be integrated into your preferred dashboard, can be used to send alerts to relevant stakeholders, or can trigger automatic model retraining pipelines.

This solution demonstrates how to implement model monitoring in the machine learning workflow, from model training to model deployment. Figure 2 shows the workflow for a batch inference use case, up to and including alerting in Slack and visualizing the model results in MLflow.

Figure 2: Model monitoring workflow for batch inference

The workflow consists of the following steps:

You can take a similar approach for use cases where real-time endpoints are used for model deployment. Figure 3 shows the workflow for real-time endpoints. The key difference is that the endpoint must have data capture enabled, which logs the inputs and outputs of the endpoint to an S3 bucket. Instead of a processing job, this architecture uses AWS Lambda functions to deploy the monitoring code. Both architectures can use either processing jobs or Lambda, based on your preference. The AWS Lambda functions that calculate data and model drift can run on a regular schedule. You can also trigger them when data from the endpoint lands in the S3 bucket.

Figure 3: Model monitoring workflow for real-time endpoints

This pattern for real-time endpoints can also be used with Amazon SageMaker Hyperpod, where you can provision clusters to use for model training as well as model inference. You can enable data capture for Hyperpod at the endpoint level, at the load balancer level, or at the model pod level. Inference requests and responses are automatically written to an S3 bucket, where you can access them to perform model monitoring.

Both previous diagrams focus only on the model monitoring process. However, model monitoring is normally implemented as part of a larger MLOps workflow. Figure 4 shows how this model monitoring solution fits in an end-to-end MLOps scenario, where the processing job used to calculate drift is deployed alongside the endpoint through a CI/CD workflow. For more information on implementing MLOps with SageMaker AI, see MLOps foundation roadmap for enterprises with Amazon SageMaker AI.

Figure 4: How model monitoring integrates with an MLOps architecture

This section provides a step-by-step explanation for setting up the model monitoring solution for batch transform. See the full repository for other examples, including a sample for real-time endpoints.

Before following the steps in this walkthrough, you will need:

To continue with the walkthrough, open the notebook predictive_ml_experimentation_data_model_monitoring_evidently.ipynb. Note that the solution uses the SageMaker Python SDK v3. If you use the DefaultMLFlowApp, the notebook code identifies the correct app automatically. Otherwise, edit the code with the name of your MLflow App.

The example notebook uses the Bank Marketing dataset from the UCI Machine Learning Repository. This dataset contains information about marketing phone calls made on behalf of a Portuguese bank, with the goal of predicting if the customer will subscribe to a term deposit. It is a binary classification use case where the target variable has a value of 1 if the customer subscribed and 0 if a customer did not subscribe.

In the first half of the notebook, the data is cleaned, processed, and split into training, validation, and test sets. An XGBoost model is trained on the training and validation datasets, with logs and metrics sent to MLflow and the final model object registered in the MLflow model registry.

Two key actions in this data processing and model training section set up the model for monitoring later in production:

Figure 5: Model metrics (precision, recall, AUC) displayed in MLflow

Next, the notebook creates the model object from the training job output and sets up a batch transform job which runs the test features through the model for inference.

Evidently has various presets for calculating different types of data drift, all of which can be customized to suit the particular ML use case. The sample notebook uses the DataDriftPreset and DataSummaryPreset and creates a helper function to save the Evidently reports (both HTML and JSON) in MLflow and extracts specific drift values to save separately as metrics in MLflow. This enables you to compare different runs or generate alerts based on certain values.

Figure 6: An Evidently data drift report as an artifact in MLflow

Figure 7: Data drift metrics and parameters in MLflow

Another advantage of saving each data drift calculation as a run in MLflow is that it allows us to add parameters such as the model name, the name of the training job, the size of the dataset, and other important information.

Evidently also provides presets for calculating model quality. This notebook uses the ClassificationPreset, which calculates accuracy, precision, recall, F1 score, and more based on model predictions and the ground truth dataset. Instead of the ClassificationPreset with its broad range of metrics, Evidently also supports custom reports that you can target to calculate the metrics most relevant to a particular use case. For example, because of imbalanced labels in the marketing dataset used in the sample notebook, accuracy is a less relevant metric than precision, recall, and AUC.

Similar to data monitoring, the Evidently reports for model quality are added as artifacts to MLflow, and the individual metrics can be extracted into MLflow metrics. Note that Evidently does not calculate model drift, meaning the difference between metrics from the original training job and metrics calculated from the predictions. However, you can add drift calculations in the monitoring code, and we provide an example of this in the real-time endpoint sample, where model drift, when detected, triggers an alert via Amazon SNS.

Figure 8: Model quality report artifact in MLflow

Views in MLflow can be customized and individual runs can be compared based on certain attributes. Figure 9 shows a view of multiple data drift, model quality, and comprehensive quality runs together with the original training job. You can see which training run the monitoring runs relate to, how many drifted columns were detected in the consecutive data drift runs, and how much the accuracy dropped between training and running the model quality job.

Figure 9: Multiple monitoring runs in MLflow

Furthermore, you can plot the metrics captured in MLflow over time. As shown in Figure 10, this lets you view the entire model life cycle in one place, covering model experimentation, model training, and model deployment and monitoring.

Figure 10: Model life cycle tracking metrics view in MLflow

Although the first notebook demonstrates in depth how to train a model, deploy it, and run monitoring with Evidently, the second notebook batch_monitoring_pipeline.ipynb shows how to scale inference and monitoring through jobs and pipelines. It creates a pipeline with two steps:

Figure 11: A pipeline with one batch transform step and one monitoring step

Finally, the notebook demonstrates how pipeline executions can be run on a regular schedule with Amazon EventBridge. For an example of how to trigger pipeline executions based on inference data becoming available in S3, see the real-time endpoint sample.

Each notebook has a section at the end to help you clean up the resources you deployed, including any endpoints and model objects. If you wish to delete the data artifacts in S3 or the runs logged to MLflow, you need to remove them manually.

Implementing a data and model monitoring solution is necessary to maintain prediction accuracy and help achieve the best outcome for your machine learning use case. This post shows how you can use open source Evidently together with Amazon SageMaker AI to generate monitoring reports, organize and compare the results in MLflow, scale through pipelines, and trigger drift notifications. The AWS sample GitHub repository provides implementations for both batch predictions and real-time endpoints, with scaling done either through SageMaker AI pipelines or AWS Lambda. To learn more about MLOps on Amazon SageMaker AI, see the Amazon SageMaker AI MLOps workshop.