dagshub-docs/docs/reference/new_relic_monitoring.md

New Relic real-time monitoring and alerting

This document explains how to connect DagsHub and New Relic to monitor and analyze machine learning training metrics in real-time. We will cover configuring DagsHub and New Relic, visualizing your metrics in a dashboard, creating and analyzing custom metrics, and setting alerts for them.

Prerequisites

Log training or evaluation metrics

To log experiments to New Relic in real-time, you must first log metrics in real-time to DagsHub. This is done using MLflow. For a guide on how to log experiments with MLflow please refer to this doc.

Configure DagsHub to send metrics to New Relic in real-time

Retrieve your Insights insert key using New Relic's integration wizard. From your New Relic home screen hit "Add more data" in the top right corner of the screen.

Then select the DagsHub Integration.

Copy the necessary API key.

Take note of your account id, and your API key, and go to your DagsHub Repository. From there, go to the integrations page of the repository settings.

(You can find the integrations page in the following URL: https://dagshub.com/{username}/{reponame}/settings/integrations)

Follow the wizard, where you will be asked to enter your Account Id and Insight insert key.

DagsHub is now sending metrics to New Relic in real-time!

Visualize metrics in a dashboard

You can visualize all your training and evaluation metrics in real-time all in one place using dashboards. When finishing the integration wizard on New Relic, a default dashboard like the one below will appear.

Create a custom dashboard

From the New Relic home page, click on Dashboards in the main menu. On the top right corner of the page, click "Create a dashboard". Insert a name for your dashboard and hit "Create".

To add a metric, click on one of the boxes in your dashboard, where the label "Add a new chart" appears.

Then select "Add a chart".

Now it's time to write NRQL, this is the query language used by New Relic to fetch information from all its sources. Use this example query as a reference to display a metric, replace metric_name with the name you gave to your metric in your training/evaluation pipeline.

SELECT latest(<metric_name>) FROM Metric TIMESERIES SINCE 1800 seconds ago

Click "Run" to preview the result, or Save to add the chart to your dashboard.

Monitor in real-time and analyze custom metrics

Some metrics are lineary dependent, but they give a better grasp of the data than looking at a set of raw orthogonal metrics. You might want to look at squared values or absolute values for instance. Logging those metrics seems like data duplication, and often you will manipulate your metrics postmortem. The query language used to fetch your metrics is called NRQL, and it can be leveraged to calculate custom metrics out of your raw metrics. You can log orthogonal metrics, and monitor any metric created out of them in real-time, as long as it's computable using NRQL.

The process to monitor a custom metric is the same as monitoring a metric, besides the query itself. Let's monitor the squared value of our metric named "Loss Objectness". First, duplicate the "Loss Objectness" chart by clicking on the ellipsis, and duplicate.


Then, on your duplicated chart, click on the ellipsis again, then edit.

The query editor will pop. Instead of the original query, let's multiply the value by itself;

SELECT latest(loss_objectness)*latest(loss_objectness) as loss_objectness_squared FROM Metric TIMESERIES SINCE 1800 seconds ago

Give an informative name to your chart and hit "Save". This is it, from now on you can see the squared value of "Loss Objectness" while the metric is being logged during your training sessions. To see the full extent of NRQL capabilities, please refer to NRQL syntax, clauses, and functions.

Set an alert

You can use alerts to receive notifications that a metric reached a specific value or fills a certain condition. The simplest way to set an alert is from an existing metric on your dashboard. Click on the ellipsis on the top right corner of the chart, and select "Create alert condition".


Then set the condition and the policy of the alert following Step 3 and Step 4 of this guide.