AI has been serving as the forefront for modern software engineering. Therefore, it's important to efficiently develop, deploy, and maintain a machine learning model. To do so, teams implement a Machine Learning Operations (MLOps) pipeline to automate their model management. This is particularly true in Computer Vision.

Such models need to be updated frequently with new data and need to be at peak performance to deliver desirable results. But, constructing a comprehensive solution tailored for computer vision using Machine Learning Operations (MLOps) is often demanding, and requires a strategic integration of open-source tools.

So, it's important that you understand how to correctly leverage open-source tools to build a functional MLOps pipeline to seamlessly manage your computer vision project, and reduce your operations costs significantly.

Therefore, this guide will walk you through the key stages of building a robust MLOps pipeline, from data collection to model deployment, using popular open-source software (OSS) platforms.

MLOps Lifecycle

The MLOps lifecycle consists of seven iterative stages, as shown in the graphic below, which is crucial for the success of a machine learning system.

Each stage requires careful execution, and setbacks may necessitate revisiting previous steps to address issues.

Here's a breakdown of each stage:

Data Collection & Management

Leveraging Min.IO

Efficient data collection sets the foundation for any successful computer vision model. Leveraging Min.IO an open-source platform that ensures scalable and reliable storage solutions. With data being the cornerstone of any machine learning project, organizing and accessing it seamlessly is critical for effective model development.

Key features of MinIO

• Compatibility with S3 applications, high throughput, and low latency.
• MinIO can be easily deployed on various platforms, including on-premises hardware or in the cloud.
• It is suitable for a wide range of use cases, such as data lake storage, backup and recovery, and content delivery.
• It provides robust security features, including encryption and access control, making it a versatile solution for organizations with diverse storage needs.

To understand how MinIO can be used for computer vision projects, let's break down key concepts such as Object Storage, Buckets, and S3 in the context of MinIO with a traffic sign detection example.

Object Storage: In MinIO, data is organized as objects, where each object typically represents a file or a piece of data. Objects can be of any type, such as images, videos, documents, etc.

Buckets: Buckets help in organizing and managing data, acting as a top-level namespace for objects. You can create multiple buckets within a MinIO instance, and each bucket has its own unique namespace.

S3 Compatibility: MinIO is compatible with S3 API, which is the standard interface for interacting with object storage in the AWS ecosystem. S3-compatible services, like MinIO, allow you to use existing S3 tools and libraries, making it easy to integrate MinIO into applications that were originally built for AWS S3.

DagsHub also supports connecting AWS S3, GCS, and MinIO to Dagshub Repos

Organizing Data with Buckets:

Buckets help you organize your object detection data. For example, you might create different buckets for different projects, teams, or types of datasets. Within each bucket, you can store related object detection data, making it easier to manage and retrieve.

I have created a MinIO storage bucket to store our traffic sign detection dataset, which includes images, annotations, configuration files, and other metadata. Each annotated image or dataset file is treated as a separate object within a bucket.

MinIO documentationMinIO console gives you a quick overview of the overall state of your MinIO deployment.

Data Labeling

Harnessing the Power of Label Studio

Accurate labelling of data is critical for training reliable computer vision models. Label Studio, an open-source data labelling tool, provides a versatile and efficient solution for annotating images, ensuring the high-quality labelled datasets necessary for training precise models.

Key features of Label Studio include:

Versatility: Supporting multiple annotation tasks such as image classification, object detection, named entity recognition, and more. .

Collaboration: It facilitates collaboration among teams by providing a user-friendly interface for annotators and supporting multiple annotator roles.

Integration: Label Studio can be integrated with different machine learning frameworks and libraries, allowing users to seamlessly incorporate labelled data into their training pipelines. It supports popular formats for exporting labelled datasets.

Label Studio supports integration with Amazon S3 for storing and managing labelled data.

Ease of Use: The platform is designed to be user-friendly, making it accessible to both technical and non-technical users.

Customization: Users can create custom annotation interfaces tailored to their specific use cases.

Here, I have used Label Studio to make annotations to our traffic sign dataset.

Automate your labeling and annotation process with Label Studio.

Data Visualization and Curation

Voxel51 for Enhanced Insights

Voxel51 offers a powerful MLOps framework for data visualization and curation. With its comprehensive suite of tools, you can explore and curate your datasets, gaining insights into the characteristics of your data that are crucial for developing effective computer vision models.

Features of Voxel51:

• Dataset curation
• Model evaluation
• Error identification
• Embedding visualization
• Expediting the transition to production, ultimately aiding teams in achieving better model performance.

Voxel51 provides relatively many functions and plugins for data deduplication, data sampling, deleting samples with respective tags, filtering data, dimensionality reductions, correcting annotations with errors, building data-centric pipelines, and integrations with your favorite ML tools like Colab, Label Studio, Jupyter, TensorFlow, Google Cloud, AWS, etc.

In the Voxel51 dashboard, we have the capability to visualize our traffic sign dataset along with annotations. Leveraging the platform's plugins and functions, we can actively engage in refining our data to create an unbiased and balanced dataset.

This process is instrumental in preparing the dataset for optimal performance during model training, testing, and validation stages. The interactive features provided by Voxel51 empower us to make informed adjustments, ensuring the dataset is well-curated and aligned with the specific requirements of our computer vision model.

Model & Experiment Tracking

Using MLFlow for Development

MLflow simplifies the management of machine learning experiments. With MLflow, you can track and compare experiments, log parameters, and store model artifacts. This open-source platform ensures reproducibility and accountability throughout the model development process.

The MLflow documentation on models provides the functionality related to managing and deploying machine learning models within the MLflow platform.

Key Features:

1. Model Packaging: MLflow supports packaging machine learning models in a standardized format, allowing for easy distribution and sharing.
2. Model Formats: MLflow is versatile and supports various machine learning frameworks and libraries, enabling users to work with models developed using different technologies.
3. Model Registration: MLflow provides a model registry to keep track of different versions of models, making it easier to organize, document, and reproduce experiments.
4. Model Deployment: MLflow facilitates the deployment of machine learning models to various deployment environments, including cloud platforms and containerized environments, ensuring a smooth transition from development to production.
5. Tracking Models: MLflow allows users to log and track experiments, parameters, and metrics associated with model development, aiding in reproducibility and collaboration.

You can create an integration with DagsHub and MLflow as DagsHub provides a free hosted MLflow server with team-based access control for every repository.

You can authorize GitHub to integrate your model repository into DagsHub, and as shown in the above image, you can host your repository for experimentation and tracking on MLflow.

Model Deployment

BentoML for Seamless Integration

In the ML pipeline, model deployment is a crucial stage where open-source tools like BentoML simplify the deployment of models, allowing you to turn your trained models into scalable and production-ready APIs. Its modular and flexible design ensures seamless integration with various deployment environments, making it an ideal choice for computer vision applications.

Our trained and developed Object detection model can now be deployed using BentoML. There are a few steps that you need to follow to get the model deployed.

1. Save our trained model into local storage and load it back to BentoML.

model.save("C:/Users/your_path/Downloads/archive/assets/CNN.keras")

Install dependencies:

pip install bentoml
pip install tensorflow
pip install requests

After we save our trained model locally, load and save our model to BentoML.

from pathlib import Path
from tensorflow import keras
import bentoml

def load_model_and_save_to_bentoML(model_file: Path) -> None:
    model = keras.models.load_model(model_file)
    bento_model = bentoml.keras.save_model("keras_model", model)
    print(f"Bento model tag = {bento_model.tag}")
    
load_model_and_save_to_bentoML(Path("C:/Users/minur/Downloads/archive/assets/CNN.keras"))

Run this code on your command line to check the list of BentoML models saved.

bentoml models list

Create an endpoint to access Bento model and use the script to make predictions using BentoML.

import numpy as np
import bentoml
from bentoml.io import NumpyNdarray

BENTO_MODEL_TAG = "keras_model:ze7wq5oh3g6nm5b2"


recognition_runner = bentoml.keras.get(BENTO_MODEL_TAG).to_runner()

traffic_recognition_service = bentoml.Service("traffic_sign_recognition", runners=[recognition_runner])


@traffic_recognition_service.api(input=NumpyNdarray(), output=NumpyNdarray()) # use input outputs appropriate to your data
def recognize(input_data: np.ndarray()) -> np.ndarray():
    return recognition_runner.predict.run(input_data)

Run the following service script to get predictions by parsing the test dataset to our model using a service request to the bentoml server.

Note: Use bentoml serve service.py on the command line to run the script.

from typing import Tuple
import json

import numpy as np
import requests
# import test dataset from model training folder from the local drive
from achieve.train import test_traffic_sign_images  


SERVICE_URL = "http://localhost:3000/traffic-signs"


def sample_random_recognition_data_point() -> Tuple[np.ndarray, np.ndarray]:
    _, _, test_images, test_labels = test_traffic_sign_images() # Prepare your data to return images and labels
    random_index = np.random.randint(0, len(test_images))
    random_test_image = test_images[random_index]
    random_test_image = np.expand_dims(random_test_image, 0)
    return random_test_image, test_labels[random_index]


def make_request_to_bento_service(
    service_url: str, input_array: np.ndarray
) -> str:
    serialized_input_data = json.dumps(input_array.tolist())
    response = requests.post(
        service_url,
        data=serialized_input_data,
        headers={"content-type": "application/json"}
    )
    return response.text


def main():
    input_data, expected_output = sample_random_recognition_data_point()
    prediction = make_request_to_bento_service(SERVICE_URL, input_data)
    print(f"Prediction: {prediction}")
    print(f"Expected output: {expected_output}")


if __name__ == "__main__":
    main()

2. Create a BentoML Service

You can build your BentoML Service by going through their documentation. To keep things simple in this article, I won't be diving deep into BentoML, but feel free to get the sample code from their documentation.

3. Build Model + Service Into the BentoML

After you've built your service, you'd have to build your model and service into a Bento. To do so, check out their documentation for a clearer implementation. Afterward, you can directly server your model through BentoML

4. Dockerise Bento & Run BentoML Service via Docker

Finally, you can distribute your Model as a packaged application using container based platform such as Docker.

Model Monitoring and Error Analysis

Finally, after you've deployed your model, you must start monitoring it. This ensures that your model is consistently looked at and issues are detected early on.

To do so, you can leverage EvidentlyAI, an open-source library, that provides tools for model monitoring and error analysis.

Note, EvidentlyAI is not specifically designed for computer vision models, however, it can be adopted to monitor computer vision models.

With its capabilities in statistical analysis and visualization, you can proactively identify deviations and ensure your computer vision model stays at its peak performance.

You can integrate your hosted models on MLflow with EvidentlyAI to automate model quality evaluations.

Conclusion

This guide outlines the construction of a comprehensive MLOps pipeline for computer vision using open-source tools, covering key stages from data collection to model deployment. Leveraging tools such as Min.IO, Label Studio, Voxel51, MLflow, BentoML, and EvidentlyAI, the guide emphasizes the importance of strategic integration for an effective workflow.

Daghub acts as a unifying platform, seamlessly integrating with these open-source tools to provide a cohesive solution for managing the end-to-end MLOps lifecycle in collaborative environments.