Skip to content

Giskard

Giskard is an open-source AI quality management system dedicated to ML models.

Giskard provides a suite of tools for scanning, testing, debugging, and monitoring all AI models, from tabular to LLMs.

With DagsHub and Giskard, you can easily debug your models and view Giskard's vulnerability and testing reports. This functionality is based on the MLflow server that comes with each DagsHub project

Check out the example repo to see a Tabular and LLM testing example, or open the examples directly in Colab:

Tabular Example:
Open in Colab

LLM Example:
Open in Colab

How does Giskard work with DagsHub?

By setting DagsHub as the logger of the experiment, it authenticates your DagsHub user and uses MLflow and DagsHub Client to log the information of the experiment to your DagsHub repository. We use built-in Giskard callbacks to log the metrics and parameters of every run using MLflow, and the artifacts, as in data and trained model, using either MLflow or DVC. You can find the source code of the logger in the Giskard repository.

How to log Giskard Vulnerability & Testing Reports to DagsHub?

Setup

We will start by installing Giskard, DagsHub, and MLflow by running the following command from the CLI

pip install giskard mlflow dagshub

Next, log the Giskard report to your repository's MLflow server, simply use dagshub.init() as following: 

# Only DagsHub related lines you need:
import dagshub 

# This will work if you have write access to the repo below, if you cloned it, please change the repo_owner to your user name
dagshub.init(repo_name="your_repo_name", repo_owner="your_username")

Perform the evaluation

To make sure everything is installed correctly, and Giskard is now part of MLflow's evaluators, run: 

import mlflow
mlflow.models.list_evaluators() # ['default', 'giskard']

The configuration of the Giskard evaluator can be done entirely through the evaluator_config argument that can yield 3 keys:

  • model_config
  • dataset_config
  • scan_config

Read Giskard's guides for more information on how to set these variables.

Here’s the integration in a nutshell: 

evaluator_config = {"model_config":   {"classification_labels": ["no", "yes"]},
                    "dataset_config": {"name": "Articles"},
                    "scan_config":    {"params": {"text_perturbation": {"num_samples": 1000}}}}

with mlflow.start_run(run_name="my_run") as run:
  model_uri = mlflow.sklearn.log_model(..., pyfunc_predict_fn="predict_proba").model_uri
  mlflow.evaluate(model=model_uri,
                  model_type="classifier",
                  data=df_sample,
                  evaluators="giskard",
                  evaluator_config=evaluator_config)

View the report

If you're working with Python files, you can commit them to your DagsHub repository, or if you're in a Colab notebook, simply use: 

dagshub.notebook.save_notebook(repo="repo_owner/repo_name")

To save it to DagsHub.

To view it, click on the "Go to MLflow UI" button in the Experiments Tab: Go To MLflow UI

Then click on the experiment with the model you want to view the report for, and you'll see is at "giskard-scan-results.html" In the artifacts view: Giskard Report

Congratulations, you’re all set to track & view your Giskard reports using DagsHub!