New Organization
CY4GATE: Improving Model Deployment Efficiency
Learn how CY4GATE, a leading cybersecurity solutions provider, used DagsHub to reduce their time-to-production by 20% and improve their collaboration on AI projects.
By managing their large-scale data projects with DagsHub, CY4GATE ensured their projects' reproducibility, were able to experiment faster, and spent 40% less time coordinating collaborative machine learning work. DagsHub enabled CY4GATE to manage their entire project lifecycle in one platform, streamlining the ML workflow, and delivering better outcomes for CY4GATE.
In this case study, we explore CY4GATE's use case, main MLOps challenges, and how DagsHub provided a comprehensive solution for them.
Company overview
CY4GATE is a leading provider of cybersecurity solutions. The company's mission is to design, develop, and produce technologies, products, and services that meet the rigorous needs of Cyber Intelligence & Cyber Security. Their client base includes armed forces, police, intelligence agencies, and civilian companies in Italy and abroad. With its one-of-a-kind business model, CY4GATE offers proprietary products that cover the entire cyber market, focusing on data collection, analysis, and security. Their comprehensive offerings ensure clients receive tailored solutions to enhance their cybersecurity posture.
How CY4GATE applies ML in their work
CY4GATE leverages machine learning (ML) in two key areas.
Firstly, in the cybersecurity domain, they apply ML models to analyze cybersecurity data and logs, enabling advanced threat detection and response capabilities. This helps safeguard their clients' systems and networks from potential cyber threats.
Secondly, in the realm of decision intelligence, CY4GATE uses ML algorithms to extract valuable insights from unstructured data sources like PDFs, images, and videos. This enables organizations and military entities to make informed decisions based on the extracted knowledge. Additionally, CY4GATE actively participates in European research projects, contributing to and publishing their ML models, which further advances the field of cybersecurity.
The challenges
The ML team at CY4GATE faced several challenges that hindered their productivity. These challenges included coordinating experimentation & collaboration, difficulties in ensuring accountability and reproducibility, and inefficient data management and storage for large-scale projects.
Coordinating experimentation & collaboration
The ML team at CY4GATE faced challenges in coordinating experimentation and promoting effective collaboration. Working on multiple projects required involving multiple team members, but distributing the workload while ensuring comprehensive knowledge sharing and progress documentation proved challenging. The situation was compounded by the team members' varying familiarity with different tools and their individual logging styles, making result comparison and interpretation difficult. This resulted in more siloed workflow, hindering effective collaboration.
“This led to some confusion inside the team because not all the members had the complete information.”
Accountability and reproducibility
Ensuring accountability and ensuring reproducibility were ongoing challenges for the ML team. The absence of standardized logging structures hindered the ability to extract essential information from the logs, such as the specific architecture or tools used during modeling. This lack of standardized documentation impeded the ML team's ability to review, validate, and compare experiments accurately, potentially leading to inconsistencies in results and analysis.
Inefficient data management and storage for large-scale projects
The ML team at CY4GATE faced a critical issue related to the management and sharing of their large datasets. Without a centralized repository like DagsHub, each researcher had to individually send links or files to team members requiring their data. This approach proved inefficient, especially when dealing with sensitive information that sometimes required physically transferring data using thumb drives, introducing potential security concerns.
“The process has always been solid, but now there's way less stress involved.”
The problem worsened as data was added or modified, as there was no proper tracking system in place. Without a data tracking system in place, work was unreproducible and different versions of datasets were scattered across multiple locations and team members, making it difficult to determine who was working with which version. This lack of organization and version control hindered collaboration and productivity within the team.
The solution
CY4GATE discovered that DagsHub offered a comprehensive end-to-end platform for managing their ML projects and addressing their wider range of needs in the ML lifecycle. While initially exploring Label Studio integration for an NLP project, CY4GATE recognized that DagsHub could provide much more than just labeling capabilities. The user-friendly interface and cost-effectiveness of DagsHub compared to other solutions impressed the team, leading them to choose DagsHub as their primary tool for ML project management.
Here are the main ways they used DagsHub as their ML solution:
Accelerated experimentation:
DagsHub allowed CY4GATE to conduct experiments at a faster pace. They could now test different models, architectures, and hyperparameters more efficiently, enabling them to iterate and optimize their solutions more rapidly.
“We are now able to move through the MLOps processes more easily, and we can apply updates, say, to add a last-minute feature, to a version ready to be put in production much more quickly than before.”
Standardization of experiment tracking and reproducibility:
By utilizing DagsHub, CY4GATE implemented a standardized logging system across their projects. Every team member followed the same logging structure, ensuring consistency and making it easier to interpret and compare results. This enhanced the accountability and reproducibility of their experiments.
Enhanced collaboration and coordination:
DagsHub facilitated better collaboration and coordination among team members. Multiple people could now work on the same project simultaneously, utilizing different machines and logging their experiments in real-time while being able to share intermediate results and data versions. This streamlined the workflow, improved communication, and allowed for better knowledge sharing.
“We were able to store all of our projects in one location and this led to an increment in collaboration, and consequently productivity, on top of a more smooth transfer of code between different departments of the company.“
Efficient data storage and management:
With DagsHub, CY4GATE resolved its data storage challenges. They no longer relied solely on local machines to store their data. DagsHub provided a central source of truth where they could store and manage their datasets effectively. This ensured a more scalable and organized approach to data storage.
Optimized code sharing and secure internal data transfer
By implementing DagsHub, the ML team resolved their challenges in sharing project information and achieved secure internal data transfer within the company network. They replaced time-consuming and more risky methods like email or USB drives with a streamlined sharing process accessible to all teams.
“The biggest change is how quickly we can share code, pushing and cloning even big projects in a few minutes at worst.”
DagsHub eliminated old transfer methods with file size limitations and storage constraints, enabling seamless sharing of large files and datasets. With VPN access and internal network connectivity, team members could retrieve information from anywhere without physical transfers or external devices.
“We never had any leaks before, the process has always worked fine, but now there's much less stress.”
The results / impact
With the adoption of DagsHub as their MLOps platform, CY4GATE witnessed significant results and impacts, including:
Improved efficiency in model deployment:
By eliminating the back-and-forth between development teams and production environments, CY4GATE achieved a 20% reduction in time-to-production. They could identify and address compatibility issues early on, ensuring smoother transitions from development to production, and enhancing their ability to deliver cybersecurity solutions to clients.
“The model test is an important stage in model deployment. To test the model in a test environment, we share and try with the DevOps department and by using Dagshub this stage of the deployment has been straightforward.”
Improved collaborative work and time savings:
By using DagsHub, CY4GATE reduced the time needed for coordinating collaborative work by 40%, meaning they could do more in the same amount of time. The platform's collaborative features facilitated teamwork and coordination among team members, leading to increased productivity.
“We have more or less 20% Time gain in model deployment for production and over 40% time gain for collaborative work”
By integrating various tools, including Git, DVC, MLflow, and Label Studio, DagsHub provided CY4GATE with a comprehensive solution that streamlined its workflow from project inception to deployment. With DagsHub, they successfully designed data pipelines, automated training with DVC, managed hyperparameters with MLflow, and versioned their data efficiently.
Overall, DagsHub played a pivotal role in centralizing CY4GATE's machine-learning workflow, improving collaboration, accessibility, and efficiency. The platform's features significantly reduced time-consuming tasks and accelerated the development and deployment processes, leading to increased productivity and better outcomes for the organization.
