Alerting

Welcome to the alerting documentation, here you’ll learn how to set up effective alerts and notifications for you team.

Creating alerts

Click a project’s “Alerts” menu item to view the alerts defined, then click “Add alert” to get started.

No alerts defined

To configure an alert you’ll need to give it a descriptive name, a query used to filter events, an operator and event count threshold. The following example configuration alerts when one or more events matches error and program = "api" within a 5 minute window.

Configuring a new alert

After setting up the alerting query and threshold you’ll need to notify your team. Depending on the size of your organization you may want to notify teams separately, such as a frontend or backend team. The previous example notifies the operations team when any errors originate from the “api” program.

If you want to notify your entire development team you could simply alert on the query string error which will report anything with the level above warning.

Note that program is an example custom field, your logs may vary as you can decide on your own event field conventions.

Muting alerts

Clicking the mute button in the alert listing will prevent the alert from running until it is unmuted.

Muting an alert

Creating notifications

The alert creation page allows you to create a notification, however, they are separate entities and are managed by clicking the “Notifications” menu item. Currently the following integrations are supported:

Email

The email alert notification integration allows you notify team members via email addresses.

Email alerting

PagerDuty

The PagerDuty alert notification integration lets you create complex on-call schedules with various forms of alerting, including SMS, phone calls and others.

PagerDuty alerting

Slack

The Slack alert notification integration allows you to keep your team informed by receiving alert messages in your Slack channel(s) by providing a Slack Webhook URL.

Slack alerting

The Slack is a great way to notify your team of non-error events as well, such as user acquisition or product subscriptions.

SMS

The SMS notification integration allows you to receive SMS text messages on your mobile phone.

SMS alerting

Due to message size restrictions only a summary of the alert is provided. Using the email integration on your mobile phone is recommended over SMS.

Webhooks

The webhook integration allows you to create custom alert integrations by providing a URL for alerts. The URL you provide will receive a POST request for each alert triggered.

Custom webhook alert

The following is an example webhook request payload for a single error performing monitor check log event:

{
  "alert": {
    "id": "1eM71gypfa9NLh7fl4WZ1LNAYcJ",
    "project_id": "web_production",
    "name": "Errors",
    "description": "All production errors",
    "severity": "error",
    "query": "level > warning",
    "operator": ">=",
    "threshold": 1
  },
  "events": [
    {
      "id": "1eM7jxcd1ZuEIGp6NmlQ9PmYkiw",
      "level": "error",
      "message": "error performing monitor check",
      "fields": {
        "aws": {
          "log": {
            "group": "/aws/lambda/vitals_check_processor",
            "stream": "2020/07/07/[110]2ba51fea382243e2bef29b32b3a0d24c"
          },
          "region": "eu-west-1"
        },
        "check_id": 16604,
        "user_id": 92038,
        "error": "timeout exceeded",
        "function": "check_processor"
      },
      "timestamp": "2020-07-07T11:02:16.995Z"
    }
  ],
  "triggered": true
}

Currently triggered is always true, however, your integration should check this value as future releases may also send requests when resolving an alert.