Real-Time ML Monitoring: New Relic Integration with DagsHub

Training deep learning models is time-consuming. Even after starting your run, in some cases, you must wait for the run to end to analyze outcomes. In other cases, you can visualize real-time metrics, but you need to manually check the status of your run, leading to checking your experiment management tool every 3 minutes as if you’re waiting for some test result. Don’t even get me started about creating custom or ad-hoc metrics.  

Benefits of Real-Time Experiment Monitoring

For these reasons, using a dedicated monitoring solution to observe the performance of experiments in real-time offers significantly more efficiency for your workflow. This is the motivation behind our new integration with New Relic and DagsHub making it easy to monitor, visualize, analyze as well as synchronize all data from your runs in real-time.  

The new integration offers several benefits for data science workflows:

• Monitor: Monitor your experiment performance in real-time with New Relic.
• Alert: Easily set alerts for the behavior you care about. Whether it’s a metric going below or above a certain value or an error popping up. Get notifications and react without needing to sit in front of your computer throughout the training.
• Customize: Visualize performance data in DagsHub and create custom visualizations in New Relic. Define ad-hoc custom metrics that suit your needs.
• Integrate: Use DagsHub to seamlessly integrate between experiment tracking with MLflow, experiment monitoring and alerting in New Relic, and data, and model management using DVC and automation using Jenkins. All without DevOps work.

Overview of New Relic and DagsHub

New Relic is an observability platform that helps you build better software. You can bring in data from any digital source so that you can fully monitor, analyze, and understand your metrics and how to improve them.

DagsHub is a platform for data scientists and machine learning engineers to version their data, models, experiments, and code. It allows you and your team to easily share, review and reuse your work, providing a GitHub experience for machine learning. DagsHub is built on popular open-source tools and formats, making it easy to integrate with the tools you already use, for example, New Relic.

Getting Started

To get started, sign up for a DagsHub account, and create a new project. Select the New Relic integration in DagsHub, you can find more details here. To connect your repository and start monitoring in New Relic documentation is available here

Key Features

Here are few things to try with the new DagsHub – New Relic Integration::

• Add an alert: Alerts can be created from conditions from a chart. You can name and customize the condition. Once you've created an alerts condition, you can choose how you want to be notified in varied channels.
• Analyze your metrics: You can write and execute custom queries to fetch information from all your sources and analyze results. Here is an example:

• Build a dashboard: Visualize all your training and evaluation metrics in real-time using the New Relic dashboard. There are dozens of charting options to both understand and share results. All experimental data is also automatically synchronized to DagsHub.

Join us For a Live Demo and MLOps Discussion

Join Dean Plean, Co-Founder / CEO, DagsHub and Dr. Ori Cohen, Lead Researcher, New Relic for a discussion into the state of MLOps, demo of the new integration and practical insights from research into MLOps. Sign up here

_{Join: Meetup – The State of MLOps}

Full Transparency in Your Experiments in Real-Time

Monitoring and alerting offer new ways to optimize your training workflow, adding transparency, customizability and efficiency. You can get started quickly here.