Machine learning, particularly its subsets, deep learning, and generative ML, is currently in the spotlight.

This surge in interest from both businesses and the general public has been fueled by the excitement around latest AI technologies, such as ChatGPT.  While many businesses believe that ML-driven automation is key to their success, a survey by McKinsey shows that only 15% of businesses’ ML projects ever succeed.

One of the main reasons for this failure is the lack of ML model testing to ensure that the products remain secure from potential threats while maintaining reliability, robustness, and scalability at the deployment stage.

In fact, the top 10 strategic technology trends identified by Gartner show that businesses should prioritize AI trust, risk, and security management over everything else. This can only be done by rigorous and continuous testing of your ML models.

However, testing practices designed for software systems, are often inadequate for ML models, and it is not yet as mature and well-understood as traditional testing.

We are all still trying to figure out how to test machine learning models. Source

In this article, we will explore the concept of machine learning model testing along with some of the tools specifically designed for testing ML models, which will significantly enhance your ML pipeline.

What is Machine Learning Model Testing?

Machine learning model testing is a critical process designed to ensure the model's reliability, robustness, and performance. The main objectives of machine learning model testing include identifying errors, evaluating performance metrics, ensuring the model’s ability to generalize across diverse datasets and avoid potential security threats, and enhancing interpretability to better understand the model’s decisions. Essentially, the idea is to develop a model that can efficiently detect and handle uncertainties and anomalies over time which might degrade the prediction capability of the model.

These objectives are crucial for building trust in the model’s effectiveness and applicability in real-world scenarios.

Evaluation Vs. Testing: Are They Different?

In machine learning, the model evaluation focuses on performance metrics and plots to summarize the correctness of a model on an unseen holdout test data set.  This helps monitor changes in model outcomes across versions but may not reveal the reasons behind failures or specific behaviors.

Model testing, on the other hand, includes explicit checks to verify if the model’s learned behavior aligns with the behavior we expect the model to follow. It is not as rigorously defined as model evaluation, but covers a wider scope, ensuring all ML system components work together for expected result quality even after deployment.

In practice, a combination of the two is commonly employed, where evaluation metrics are calculated automatically, and some model “testing” is done manually through error analysis. However, it is better to use tools that can automate and test the models in a systematic manner.

Benefits of Testing ML Models

ML model testing is crucial to creating a robust production-ready model for diverse real-world data. Some of the benefits include:

1. Finding anomalies in the dataset: Testing can identify outliers or unusual data points in the dataset that could negatively impact the model’s performance.
2. Detecting model and data drift: Patterns and relations in data may evolve with time. Therefore, frequently monitoring and testing the models on evolving data helps to prevent the models from becoming obsolete over time.
3. Detect bugs and errors that degrade the prediction capabilities of the model: Testing can pinpoint and eliminate bugs and errors that degrade the model’s predictive capabilities, and encourage retraining to maintain performance.
4. Finding new insights: Testing can uncover unexpected patterns or relationships in the data, offering valuable insights.

Challenges in Testing ML Models

Though Testing ML Models is beneficially, it's not always easy to test a machine learning model. In fact, there are several challenges in testing ML models. Some of the key challenges include:

1. Lack of Transparency: Many machine learning models function as black boxes, making it difficult to understand their internal mechanisms.

2. Indeterminate Modeling Outcomes: Models that rely on stochastic algorithms may yield inconsistent results upon retraining.

3. Generalizability: It’s crucial to ensure that models perform consistently outside their training environment.

4. Unclear Testing Coverage: Unlike traditional software development, defining testing coverage for machine learning models lacks standard metrics and often relates to factors like input data and model output distribution.

5. Resource-Intensive Testing: Continuous testing of machine learning models requires substantial time and resources.

Different Types Machine Learning Model Tests

When you're testing an ML Model, it's important to understand that there are two main types of model tests:

1. Pre-train tests: These tests help identify bugs early, potentially avoiding unnecessary training jobs. They can be run without trained parameters and include checks for model output shape, output ranges, loss decrease after a single gradient step, dataset assertions, and label leakage between training and validation datasets

2. Post-train tests: These tests use the trained model artifact to inspect behaviors for various important scenarios. They aim to interrogate the logic learned during training and provide a behavioral report of model performance.

The authors of the paper "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" present three different types of model tests to understand behavioral attributes:

• Invariance Tests: These tests describe a set of perturbations that should not affect the model’s output.
• Directional Expectation Tests: These tests define a set of perturbations to the input which should predictably affect the model output.
• Minimum Functionality Tests (aka data unit tests): Similar to software unit tests, these tests quantify model performance for specific cases found in your data.

The authors recommend structuring your tests around the “skills” expected of the model during its learning task. For instance, a sentiment analysis model might be expected to understand vocabulary, parts of speech, robustness to noise, named entities, temporal relationships, and negation of words. An image recognition model might be expected to learn concepts such as object rotation, partial occlusion, perspective shift, lighting conditions, weather artifacts, and camera artifacts.

The below image shows a machine learning model testing pipeline proposed by Jeremy Jordan that incorporates these best practices:

Tools for Testing Machine Learning Models

1. Deepchecks (3.3k GitHub stars)

Deepchecks is an open-source Python tool for validating machine learning models and data. It operates in three phases:

• Research Phase: Offers tests for data integrity, data splits, data drifts, model integrity, and model performance evaluation.
• Deployment Phase: Provides test suites for model performance across metrics and data segments, and insights into model behavior.
• Production Phase: Enables real-time model monitoring post-deployment, alerts for model failures, and root cause analysis for detected problems.

Deepchecks also offers customizable test suites for automated model testing, allowing for the addition, removal, or editing of checks within a suite and conditions within a check.

Installation

The latest version of deepchecks can be installed by:

pip install deepchecks

Key features

1. Built-in automated checks:

Deepchecks provides checks to validate data and model integrity, detect data drifts and leakages, and perform root cause analysis. It can identify inconsistencies in your data, evaluate performance metrics, and provide a detailed view of the model’s behavior.

For instance, you can use the DataDuplicates check to find duplicates in a dataset and the WeakSegmentsPerformance check to identify the model’s weakest segments. The FeatureDrift check can be used to inject synthetic drifts into the dataset and observe the impact on the model’s performance.

The below code snippet shows the DataDuplicates and WeakSegmentsPerformance checks on a tabular dataset used for a binary classification task:

import pandas as pd
from deepchecks.tabular.checks import DataDuplicates
from deepchecks.tabular import Dataset
from deepchecks.tabular.datasets.classification import breast_cancer

path_to_data = "data.csv"

dataset = Dataset(pd.read_csv(path_to_data), label="target", cat_features=[], label_type="binary")

check = DataDuplicates().add_condition_ratio_less_or_equal(0.1)
result_data = check.run(dataset)
result_data.show()

model = breast_cancer.load_fitted_model()
check = WeakSegmentsPerformance(feature_1="mean radius", feature_2="mean concave points", max_segments=3)
result_model = check.run(dataset, model=model)
result_model.show()

The above check will generate a report on the data duplicates found along with the number of times they appear and a visualization of underperforming segments based on selected features of the dataset:

The FeatureDrift check can be used to insert a corruption (or a change) to the test dataset and calculate the drift between the train dataset and the test dataset per feature, using statistical measures.

import pandas as pd
from deepchecks.tabular.checks import FeatureDrift
from deepchecks.tabular import Dataset

path_to_train_data = "train.csv"
path_to_test_data = "test.csv"

train_dataset = Dataset(pd.read_csv(path_to_train_data), label="target", cat_features=[], label_type="binary")

test_dataset = Dataset(pd.read_csv(path_to_test_data), label="target", cat_features=[], label_type="binary")

check = FeatureDrift(columns=["mean radius"], show_categories_by="largest_difference").add_condition_drift_score_less_than(max_allowed_categorical_score = 0.2, max_allowed_numeric_score = 0.2)
result = check.run(train_dataset, test_dataset)
result.show()

2. ML validation continuity:

Deepchecks ensures continuity from research to production. The same checks used during research can be used for CI/CD and production monitoring. This ensures that the knowledge gained by your data science team is utilized by the ML Engineers in later stages of the model/data lifecycle.

Source

3. Open-source and user-friendly:

Deepchecks monitoring tool is open-source, free, and easy to use. It also has a growing community of users.

4. Versatile Support:

Deepchecks supports both classification and regression models and can handle computer vision and tabular datasets.

Pricing plans

Deepchecks offers four pricing tiers for its monitoring tools:

• Open-source: Offers almost all the features accessible to paid customers such as testing and monitoring support for tabular data, NLP data, but limits custom metrics and checks. It supports one model per deployment with basic security.
• Startup plan: Tailored for commercial use, this plan allows 1-10 models per deployment and includes custom metrics and checks, priced at $89 per model.
• Dedicated and partnership: These plans cater to large-scale commercial use, offering advanced security features and support for an unlimited number of models.

For detailed pricing, you can request a quote from the deepcheck website.

Drawbacks

1. Limited Scope: Deepchecks is primarily designed for tabular data and may not be as effective or comprehensive for other types of data or more complex ML models.

2. Cost: Since the open-source version of the monitoring tool allows only 1 model per deployment, you have to go for a pricing plan if you are to scale up, which might be a barrier for small businesses or individual users.

2. CheckList (2k GitHub stars)

Inspired by principles of behavioral testing in software engineering, CheckList was introduced in the paper "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" by Marco Tulio et.al.

It is a task-agnostic, open-source Python library for testing NLP models. CheckList guides users on what to test, offering a list of linguistic capabilities applicable to most tasks. It introduces different test types to break down potential capability failures into specific behaviors.

CheckList includes multiple abstractions, such as templates, lexicons, general-purpose perturbations, visualizations, and context-aware suggestions, to facilitate the easy generation of a large number of test cases.

Installation

pip install checklist
jupyter nbextension install --py --sys-prefix checklist.viewer
jupyter nbextension enable --py --sys-prefix checklist.viewer

Note: Use --sys-prefix to install into python’s sys.prefix, useful in virtual environments, such as with conda or virtualenv. If you are not in such environments, switch to --user to install into the user’s home jupyter directories.

Key features

1. Built-in functions:

You can test a range of NLP model capabilities including but not limited to vocabulary+POS (important words or word types for the task), taxonomy (synonyms, antonyms, etc), robustness (to typos, irrelevant changes, etc), NER (appropriately understanding named entities), fairness, negation, coreference, semantic role labeling (understanding roles such as agent, object, etc), and logic (ability to handle symmetry, consistency, and conjunctions).

2. Generating test cases at scale:

With CheckList, you can create test cases from scratch or by perturbing an existing dataset. It provides templates for generalizing test cases and perturbations, and a masked language model for expanding templates as shown in the example below:

Source

3. Offers a range of test types:

The capabilities can be evaluated with three different test types: Minimum Functionality tests, Invariance, and Directional Expectation tests.

4. Visualization and summary report on test cases:

A breakdown analysis of all test cases and types with failure rates and context-aware suggestions can be visualized with built-in functions.

Source

5. Free and open source:

The tool is open source and free of charge, allowing for customization without any pricing schemes.

Drawbacks

1. No real-time monitoring: Does not facilitate continuous, real-time monitoring to analyze data and model drifts.

2. Rigidity and Inflexibility: In dynamic environments, tasks may evolve, and a rigid checklist may struggle to adapt

3. Limited Scope: CheckList is primarily designed for NLP models and may not be as effective or comprehensive for other types of data or more complex ML models

4. Less user-friendly: The checklist could be overwhelming for some users, especially those who prefer focusing on thoughts and visualizations over lists.

3. TruEra (1.5k GitHub stars)

TruEra is a sophisticated platform that enhances model quality and performance through automated testing, explainability, and root cause analysis. It provides a suite of features for model optimization and debugging, through automated testing, explainability, and root cause analysis.

TruEra has multiple products designed to test ML models during two different stages of model development:

• TruEra Diagnostics: This tool is designed to speed up the model development. It offers automated testing for performance, stability, and fairness, along with metrics for model comparison. It aids in identifying and resolving model bias and can be deployed and scaled across various cloud environments.
• TruEra Monitoring: Provides extensive monitoring, reporting, and alerting of model performance, outputs, and inputs post-deployment. It enables quick and accurate debugging of model drift, overfitting, bias, and high error segments with unique AI root cause analysis.

Installation

To utilize TruEra’s AI quality and observability services, register for an account at app.truera.net.

Upon registration, you’ll gain immediate access to a TruEra deployment that can be shared within your organization.

Tutorials and documentation are available for guidance.

Note that the free plan excludes access to the TruEra Monitoring platform.

Key features

Source

1. Scalable monitoring: TruEra provides comprehensive monitoring, reporting, and alerting of model performance, outputs, and inputs, and can be seamlessly integrated into your existing infrastructure and workflow.

2. Fast, accurate debugging: TruEra enables debugging of model drift, overfitting, bias, and high error segments through unique AI root cause analysis. It also offers automated and systematic testing to ensure performance, stability, and fairness during model development and production.

3. Responsible AI and AI explainability: TruEra helps operationalize your Responsible AI (RAI) goals without compromising performance or efficiency. It tracks the evolution of model versions, enabling faster and more effective model development by providing insights. It also helps identify and pinpoint specific features contributing to model bias.

4. Enterprise-level scaling: TruEra is optimized for speed, reliability, and scalability, making it suitable for cloud multi-tenant workloads.

5. Security and Governance: TruEra provides foundational security and governance capabilities, including Role-Based Access Control (RBAC) and end-to-end data encryption in transit and at rest.

Pricing plan

TruEra offers two pricing plans for users who sign up for their platform.

1. Basic Plan: This free plan allows users to manage up to 5 projects with a maximum of 3 models per project. It supports NLP models, allows for collaboration and role assignment, and provides community support. However, it only grants access to TruEra Diagnostics, not the Monitoring tool.

2. Premium Plan: This plan allows users to manage over 50 projects with more than 50 models per project. It includes access to the Monitoring tool, enhanced community support, and improved security. For pricing details, users can contact the TruEra sales team.

Drawbacks

1. Limited scope: TruEra provides a wide range of tools for ML model management, but its primary focus on specific LLM and NLP models may not cater to all use cases.

2. Platform complexity: New users may find the platform complex as account creation is required to access the free version and tutorials. The process of test creation is not straightforward as well.

3. Cost and Accessibility: While the free version of the tool is accessible upon account creation, the monitoring tool is not freely available.

4. Kolena (38 GitHub stars)

Kolena is an end-to-end machine learning testing and debugging platform built to deal with multiple machine learning model categories including computer vision, NLP, tabular, audio/speech, and multimodal models such as image-text retrieval and video-text retrieval.

Kolena is to manage and curate datasets, test cases, and metrics, and compare the performance of different machine learning models on different smaller subclasses of a dataset to enable the ML models to extract features and representation at a much granular level. Kolena believes that this will increase the performance of the model as well by balancing both the bias and variance such that the model generalizes well in the real-world scenario.

Installation

You can interface with Kolena through the web at app.kolena.com by creating an account or programmatically via the Kolena Python client.

You can install Kolena directly from PyPI using any Python package manager such as pip or Poetry. Here is how you can do it:

# Python
pip install kolena

# Poetry
poetry add kolena

Remember to set the KOLENA_TOKEN variable in your environment after generating an API token from the Developer page.

export KOLENA_TOKEN="your_token_here"

A more detailed guide can be found in the documentation.

Key features

1. Explore high-resolution test results:

Kolena can conduct detailed evaluations of machine learning models based on their behavior, allowing users to delve into model performance based on various subsets of the dataset.

For instance, the screenshot below shows how two models perform for different subclasses of the CIFAR-10 dataset in the Debugger section of Kolena.

The root cause of model failures can be explored based on data points and model results using Kolena Studio as shown below:

2. Focused test set creation and curation

With the Test Case Studio, users can efficiently slice through data and assemble test cases. It also enables the cultivation of high-quality tests by eliminating noise and enhancing annotations. Quality standards can be defined with test cases and metrics to view improvements and regressions across scenarios.

The example below shows how different classification model performances can be observed according to tests created on dataset slices.

3. Automatically detect surface failure modes and regressions:

Kolena identifies regressions and specific issues to be addressed, and extracts commonalities among failures to understand model weaknesses. It tracks model behavior changes over time and visualizes model evaluations for a clear understanding of model performance.

4. Seamless integration into the workflow:

Kolena can be integrated into existing data pipelines and CI systems using the kolena-client Python client, ensuring that data and models remain under user control at all times.

Pricing plan

While Kolena provides free access to its web application with an account creation, as of now, the pricing details for Kolena are not publicly available. For accurate pricing information, it’s recommended to contact the Kolena team directly or request a demo on their official website.

Drawbacks

1. Subscription-based model: Kolena operates on a subscription-based model, but the specific pricing details are not publicly disclosed1.

2. Less community support: As of now, Kolena has limited community support, which might make problem-solving and learning more challenging.

3. Managing fine-grained tests: With Kolena, managing detailed tests can be a complex data engineering task, particularly as your dataset expands and your understanding of your domain evolves.

5. Robust Intelligence

Robust Intelligence is an end-to-end platform designed to maintain the integrity of AI systems by identifying and eliminating risks inherent to production AI. It integrates into your AI pipeline, providing a comprehensive solution for testing data and model performance at both training and post-deployment stages.

The Robust Intelligence platform consists of two complementary components, which can be used independently but are best when paired together:

1. AI Validation: Automate evaluation of AI models, data, and files for security and safety vulnerabilities and determine required guardrails for secure AI deployment in production.
2. AI Protection (AI Firewall): Guardrails for AI applications in production against integrity, privacy, abuse, and availability violations with automated threat intelligence platform updates. AI Protection is possible with the AI Firewall, which wraps a protective layer around your applications to protect against model vulnerabilities discovered during AI Validation.

Installation

The model testing agent is installed through a Helm chart into a Kubernetes namespace. Robust Intelligence provides a self-guided installation process through the web client, which will automatically generate a pre-filled Helm values file for your agent based on your cloud configuration.

Key features

1. Security risk identification during development:

Robust Intelligence uses specialized tests and algorithmically generated red teaming attacks that can run in the background in a CI/CD manner, to detect model vulnerabilities and unintended behaviors automatically.

2. Detect new risks in production models:

The platform periodically tests models and analyzes outputs over time, to identify and provide alerts on a range of issues including novel security threats, data drift, biased predictions, and anomalous data.

Source

3. Model validation:

The platform automates the time-consuming process of manual testing, providing comprehensive model validation and translating statistical test results into clear outputs that align with major AI risk frameworks and regulatory requirements. The AI Firewall can scan model outputs to ensure they do not contain sensitive information, hallucinations, or other harmful content.

4. Custom AI risk standards enforcement:

The platform allows for customization, enabling users to define their own specifications of model failures and business criteria and to create monitors for their custom tests.

5. Real-time blocking of malicious inputs:

The AI Firewall inspects every input and automatically blocks malicious payloads before they can harm your model. Prompt injection, prompt extraction, and personally identifiable information (PII) detection are some of the risks that can be blocked by the AI firewall.

Source

6. Broad domain support:

The platform supports tabular, NLP, and computer vision models, ensuring robustness across various domains.

Pricing Plan

As of now, the pricing details for Robust Intelligence are not publicly available. For accurate pricing information, it’s recommended to contact the Robust Intelligence team directly or request a demo on their official website.

Drawbacks

1. Only for enterprises: Robust Intelligence is primarily designed for enterprise-level applications, which may limit its accessibility for individual users or small businesses.

2. Limited details available online: Information about Robust Intelligence is somewhat limited on the internet, making it challenging for potential users to fully understand its capabilities and usage.

3. The setting up process is not straightforward: The installation and setup process for Robust Intelligence can be complex and may require a significant amount of time and technical expertise.

Conclusion

Machine Learning Model Testing is a crucial aspect of the AI development process.

It involves rigorous evaluation of ML models to ensure their robustness, reliability, and performance under various conditions. Despite the challenges in testing ML models, such as lack of transparency, indeterminate modeling outcomes, and resource-intensive testing, different model tests and best practices have been developed to address these issues along with tools to automate the tests.

In this article, we went through five tools that are gaining popularity in the field of ML model testing. Each tool has its unique strengths and is suited to different situations:

• Deepchecks excels in providing a comprehensive suite of automated checks for data and models, making it ideal for users seeking an all-in-one solution.
• CheckList is a powerful tool for NLP models, offering a range of tests to check for linguistic capabilities and potential biases, making it a great choice for NLP applications.
• TruEra stands out for its ability to provide interpretability and fairness metrics, making it a good fit for applications where transparency and fairness are prioritized.
• Kolena has the ability to perform high-resolution model evaluation and track behavioral improvements and regressions over different types of models from computer vision to NLP, making it suitable for users who need detailed insights into model behavior.
• Robust Intelligence is known for its stress testing capabilities and its focus on identifying and eliminating risks in production AI, making it a strong choice for applications requiring robust security and risk mitigation.

Choosing the right tool depends on the specific needs and constraints of your project. By understanding the key features, strengths, and limitations of each tool, you can make an informed decision that best suits your ML model testing needs.