Since the launch of the ImageNet dataset, computer vision has become mainstream in a rapidly evolving technical landscape. It helped build applications around image classification, object detection, face recognition and so much more!

As you can see, the ImageNet database revolutionized computer vision and has become a catalyst for computer vision tasks! Therefore, in 2024, you will very much run into apps driven by computer vision. For instance, in healthcare industries to diagnose cancers, in vehicles that self-drive (such as Tesla), in security to identify intruders, and more!

However, developing and deploying computer vision projects can be challenging due to the variety of digital content utilized. In fact, the project's success also depends on various factors, such as:

1. The algorithms and frameworks used
2. The data used for training
3. The performance metrics
4. The deployment platform

So, it's important that you manage your computer vision projects well to ensure success. If not, you might delay your projects or even burn more money only to deliver sub-optimal project outcomes. let's discuss five best practices for building scalable and future-proof computer vision projects and how to avoid some common mistakes.

1. Planning and Defining the Project Scope

The ultimate success of any project depends on two things:

1. How well the requirements have been gathered by the business team
2. How well the requirements have been understood by the technical team.

If either of the two have any mistakes, it would mean that the entire implementation would go wrong. Therefore, it's important to remember that planning and defining a scope is a fundamental step in any product development lifecycle, and is equally important for computer vision.

If you don't gather your requirements properly, you'd end up with a scenario as shown above.

One common mistake in computer vision projects is diving into coding straightaway while the task needs to be more defined. Sure, you might get some results initially, but interpreting and improving those results would get tricky without clearly knowing the purpose of the model.

Some of the key points that can guide you in translating the business problem into technical terms and defining project objectives, specifications, and milestones are discussed below.

Purpose of the Model

Knowing the business context and expected functionalities of the computer vision product is important.

For instance, in a healthcare application where computer vision is used to identify signs of early disease using medical images, understanding the specific diseases to focus on, the variety of medical images (e.g., X-rays, MRIs) to be analyzed, and the criticality of false negatives versus false positives is crucial.

In such cases, developers must set boundaries to ensure the model excels at identifying early signs of the targeted diseases, recognize edge cases such as rare conditions or unusual image artifacts, and determine acceptable accuracy levels to prioritize patient safety.

Inputs and Outputs

Another important aspect to consider is identifying the inputs fed to the computer vision system and the expected output from the system.

Some of the key considerations related to inputs and outputs are:

1. Quality and quantity of data:

The quality and quantity of input data directly impact the performance of a computer vision system. For instance, a self-driving car project would require high-resolution images or videos to accurately detect and recognize objects in various lighting and weather conditions.

Some of the tests you can conduct to measure the quality of images are:

• Noise levels present: Signal-to-Noise ratio
• Sharpness: Measuring the derivatives
• Lens distortion: Address Pincushion and Barrel effects

2. Variety of data:

Regardless of the algorithm’s quality, it won’t perform well if the inputs don’t mirror the variations in the real-world data encountered at inferencing.

For example, in computer vision tasks for classifying skin diseases, it’s crucial to consult with experts to identify the specific pathologies the system should classify, because some diseases are rare and you may not have enough training data to learn from. Another example would be pupil detection systems. Ron Soferman, the CEO of RSIP Vision states that while at first glance the task appears to be simple, the accuracy of the system may highly vary due to factors such as the width of the eye-opening, skin color, occlusion, and reflections.

It is best to consult the clients and domain experts at the beginning and throughout the project regularly to include diverse training data that reflects these conditions.

3. Desired output type:

The desired output type (classification, detection, localization, segmentation) will dictate the choice of the model. For example, if the client expects the system to classify images into predefined categories, a convolutional neural network (CNN) might be suitable. However, if the task is to segment an image into several parts, a model like U-Net or Mask R-CNN would be more appropriate.

4. Level of accuracy of the output

The level of accuracy expected by the client is another important consideration. If the client requires high precision, you might need to invest more time in model training and hyperparameter tuning. However, accuracy is not the only metric to evaluate a computer vision system. Depending on the use case, you might also need to consider other metrics, such as recall, specificity, or F1 score. For example, if the system is used for medical diagnosis, you might want to prioritize recall over precision, as missing a positive case could have serious consequences.

Algorithmic Techniques

Deciding whether to go for a pre-trained model, customize an existing one, or build from scratch depends on the project requirements, timelines, budget, and the resources that are available. Consider a retail scenario where computer vision is applied for inventory management through object detection. A pre-trained model like YOLO (You Only Look Once) might be efficient for general object recognition. However, for specific inventory items with unique shapes or sizes, fine-tuning an existing model with additional training on a curated dataset of store items could significantly improve accuracy.

This decision impacts not just the technical development but also the data collection strategy and required computational resources.

For example, using Vision Transformers (ViT) for a self-driving car application would require large amounts of data representing various driving scenarios, weather conditions, and potential obstacles, and consume significant computational resources. Tesla, for instance, relies on a cluster of NVIDIA A100 GPUs to train their vision-based autonomous driving algorithms.

How Do You Measure Success?

Different success metrics are tied to different edge cases and failure modes of a computer vision model. It's crucial to establish suitable metrics so that achieving good performance genuinely indicates an improved product, avoiding reliance on vanity measures. For instance, accuracy or error rate may not be reliable measures when dealing with imbalanced data, multiple classes, or localization tasks. Precision and recall are valuable for assessing a model's ability to identify relevant objects and avoid false positives, while the F1 score helps balance the trade-off between precision and recall, facilitating comparisons between different models. To measure the model's ability to localize objects, the intersection over union (IoU) metric is often employed. The dice coefficient can be used to measure image segmentation performance.

Legal and Ethical Considerations

Data privacy and user privacy are important aspects to consider when defining the scope of a computer vision project. They can affect the ethical, legal, and social implications of the project, as well as the trust and satisfaction of the users. Therefore, it is essential to identify the potential risks and benefits of collecting, processing, and sharing data from images or videos. For example, if the domain is healthcare, the project managers must pay extra attention to medical images, where the patient's consent and confidentiality are of the utmost priority.

Managers can proactively tackle ethical issues by setting up an ethical review framework. This includes privacy assessments, consent processes, and fairness audits during model creation. Engaging stakeholders like legal advisors and impacted communities can help identify potential ethical hurdles.

Clearly understanding the needs of the business, its business purpose, how the business operates, and the goal of the requested computer vision product is essential to building a pipeline optimized to solve the task at the intended level of success.

2. Managing the Data

Data is the lifeblood of a computer vision project. The robustness and generalizability of the project highly depend on the diversity, amount, and quality of data gathered.

Some of the considerations in managing data are:

Data Collection

• Set clear objectives and requirements for the data collection process to ensure the data reflects the target problem, covering diverse scenarios and variations expected during inference.
• Use the same devices for data collection as those used for inference to obtain better performance.
• Ensure that the distribution of training data reflects the distribution of real-world data. This means the variety, frequency, and characteristics of examples in the training data should closely match what the model will encounter when deployed.
• Gather enough data for training, validation, and testing of the models. According to the Deep Learning book by Goodfellow, Bengio, and Courville, as of 2016, a rough rule of thumb is that a supervised deep learning algorithm typically needs about 5,000 labeled examples per category for acceptable performance. To surpass human performance, it may require a dataset of at least 10 million labeled examples.

Labeling and Annotating

• Create well-defined annotation guidelines and provide thorough training for annotators.
• Implement a feedback loop to review annotation inconsistencies and maintain quality control.
• Utilize tools like DagsHub Annotations for managing labeling flow, ensuring reproducibility, scalability, and efficient version control of annotations and data, while also supporting auto-labeling for enhanced active learning of the model.

Preprocessing and Augmentation

While building a dataset may seem tedious, having an adequately preprocessed and cleaned dataset will help train your models in an optimized manner.

recommended to pre-process your dataset with techniques

Therefore, it's important to pre-process your data before you begin training. As a generic rule of thumb for pre-processing a computer vision data set, here are some techniques that you could explore:

1. Normalization
2. Denoising
3. Color equalization
4. Separation Techniques (To remove noise, correct errors and standardize formats).

Apply preprocessing techniques such as normalization, denoising, color equalization, and separation techniques to remove noise, correct errors, and standardize formats.

Normalization

Use Z-score normalization to standardize intensity values across diverse images. This minimizes the model's sensitivity to variations in lighting conditions.

Denoising

Use the Non-Local Means Denoising algorithm to reduce image noise while preserving essential details like edges.

Color Equalization

Leverage histogram equalization to enhance the contrast in images, especially when dealing with significant variations in lighting conditions.

Separation Techniques

Use techniques like color space conversion to isolate features relevant to the model's task such as converting RGB images to HSV (Hue, Saturation, Value) to enable better detection and classification of objects based on color.

For datasets with high bias, use augmentation techniques like geometric transformations (flipping, random cropping, rotation, and translation), color distortion, and kernel filters tailored to your project's need.

Geometric transformations such as random cropping and rotation introduce variability, mimicking different object orientations to make the model invariant to position and angle variations.

Color distortion techniques, including adjustments to brightness, contrast, and saturation, prepare the model for diverse lighting conditions it may encounter during inference.

Kernel filters like Gaussian blur simulate motion blur or focus variations, enhancing the model's capability to recognize objects under less-than-ideal photographic conditions.

Consider integrating a library like Augmentor or Albumentations to streamline the application of these techniques via customizable pipelines, ensuring efficient and effective augmentation of your dataset.

Data Version Control

When data undergoes several stages, keeping track of the different versions is essential. With proper data version control for proper coordination, you will eventually retain project control.

Implement a version control system to track changes in datasets, annotations, and preprocessing steps, enabling reproducibility and understanding of data evolution during training.

This is where tools like Dagshub Data Engine come into play. Dagshub Data Engine is a central platform for effectively managing data, metadata, labels, and predictions. Data Engine allows users to efficiently manage extensive and dynamic datasets by uncovering biases through smart metadata querying, visualizing and tracking subsets of data, reviewing and modifying datasets with a simple click, and collaborating on subsets in annotation workspaces. Additionally, the platform aids in minimizing costs and optimizing memory usage for data storage.

3. Selecting the Appropriate Frameworks and Deployment Platforms

Although it is tempting to delve immediately into deep development right away, it is more effective, despite being counterintuitive, to start with a minimal but functional end-to-end pipeline. This means using even a small or partial dataset, loading it for minimal preprocessing, just enough to run and train initial models and identify any pitfalls regarding data, preprocessing techniques, and performance metrics selected.

In general, the deployment can either be in the cloud or at the edge, depending on the use case.

Cloud Deployment

Using cloud deployment involves running a model on a remote server accessed via API. The key benefits of using cloud deployment models include:

1. Capability to scale on an infinite basis and balance the load across multiple instances with no hassle.
2. Simplified management of model redeployment since models are online and ready for modifications.

However, some drawbacks of using cloud deployment models include:

1. The latency in waiting for results due to remote API access
2. Handling resource groups and the cost of maintaining an always-on compute instance can be complex, especially if the model isn't consistently in use.

But, if you're looking to deploy your computer vision projects in the cloud, some of the cloud services tailored for computer vision projects are Google Cloud Vision AI and AWS Rekognition.

So, if you're going ahead, only go ahead if you're part of the following use cases:

1. You don't need real-time processing.
2. You're dealing with a substantial storage of data such as video recordings from Zoom. and real-time inference is unnecessary.
3. You're in need of consistent high-speed bandwidth.

Edge Deployment

An edge device offers a notable advantage by minimizing latency in using the model's processed results. Another benefit is the ability to keep data entirely private on edge deployments.

However, edge devices often have limited computing resources, necessitating a smaller model that can reduce accuracy and throughput. Additionally, managing edge devices can be challenging, making it tougher to monitor model health and update performance.

Some of the frameworks that can help in the edge deployment of computer vision projects are PyTorch mobile, Intel OpenVINO, and ONNX runtime.

Choose edge deployment for the following use cases:

• If you need real-time processing that triggers an immediate action
• If you don't need internet access to make actionable decisions.
• If you need to periodically perform image inference in remote locations without internet access.

4. Training, Validating, and Testing the Model

The model used for image or video analysis is at the heart of any computer vision project. Depending on your needs, you can opt for a pre-trained model, customize an existing one, or create a new one from scratch.

Adhering to best practices in machine learning is crucial—this involves tasks such as data splitting, selecting appropriate metrics and loss functions, hyperparameter tuning, applying regularization and optimization techniques, and closely monitoring the training process.

For instance, consider a medical image segmentation project where transfer learning with a pre-trained Convolutional Neural Network (CNN) like Mask R-CNN can significantly accelerate development while maintaining accuracy instead of building and training a segmentation model architecture from scratch without pre-trained weights.

Some of the other important points to pay special attention to at the implementation phase are:

• Establishing benchmarks: Simply reaching a 90% accuracy or F1 score isn't informative unless you have a baseline to measure it against. Without knowing what a random guess, heuristic, or current state-of-the-art model would score, it is challenging to determine if your results are good or bad. Establishing these benchmarks is essential before you can accurately assess the performance of your system.
• Fine-tuning the model by parallelizing work: Once the end-to-end pipeline is finalized, fine-tuning the model can be done iteratively by splitting the problem into pieces where people can work in parallel to optimize different aspects. For instance, one person could focus on enhancing the dataset by adding more relevant data, while another improves the validation procedures or the model itself. The parallelism increases the rate at which you can deliver better models.
• Post-training evaluation: After training, assess the performance and accuracy by testing it on fresh, unseen data. Ensure that the test data is either from the same source as the production data or is similar in nature. Check its robustness by testing in various scenarios, considering factors like noise, occlusion, distortion, lighting, angle, or scale changes. Evaluate speed, memory usage, and power consumption to ensure it meets your needs.
• Deployment: Once tested, deploy the model to your chosen platform—whether a web server, mobile app, or edge device like NVIDIA Jetson, ensuring your computer vision project is accessible to the end users.
• Concise documentation: Concise, relevant, and updated documentation is essential in computer vision project management. Proper documentation should include reference to the model architecture, hyperparameters, datasets, evaluation metrics, model adjustments, testing protocols, issues identified, and solutions introduced to mitigate the issues. Documentation ensures transparency, reproducibility, and effective collaboration throughout the project.

5. Continuously Updating and Maintaining the Model

Active learning is the ability of a model to learn over time from a stream of data. This means supporting the model's continuous learning and adaptation in production as new data comes in.

The reason for continual learning to be a vital feature in machine learning models is simple. Data is ever-changing for reasons such as:

1. Changes in trends
2. Preferences of users
3. Business logic.

Therefore, this can lead to issues such as data and model drift and will indefinitely require you to update your model again.

As your system processes more images and learns from its predictions, you can adjust to enhance accuracy and accommodate new product variations. By consistently checking for errors, gathering user feedback and data, and addressing performance issues, your computer vision project stays adaptable and relevant to changing industry standards and requirements.

In a well-established MLOps workflow, monitoring involves three main aspects:

• Technical/system monitoring ensures the correct functioning of the model infrastructure.
• Model monitoring continually assesses the accuracy of predictions.
• Business performance monitoring evaluates whether the model contributes positively to the business. It's crucial to monitor the impact of changes in a release.

You can employ tools like MLflow, TensorBoard, or Prometheus for effective updates and maintenance.

Conclusion

For a scalable and up-to-date computer vision project, it’s crucial to establish user requirements clearly and gather and preprocess relevant data enough to obtain good results in the project's training, validation, and testing phases.

Selecting robust hardware and infrastructure, incorporating cloud services for scalable resources, and keeping algorithms and models updated with advancements in deep learning and AI to enhance accuracy is also essential. Regularly retrain models with fresh data for improved performance and update the pipeline to address long-term ethical and legal considerations, ensuring compliance with evolving regulations to maintain public trust.

Implementing the best practices outlined in this article will enable you to construct a seamless and future-proof pipeline for your computer vision project, ensuring customer satisfaction and encouraging repeat engagements.