Alerting on Elasticsearch

“Alerts make your life easier when you create them and hell when you receive them” ~ Shreshth Pal.

An Alert comprises of three part in Elasticsearch ( v7.10.2-oss lest you ask):

• Monitor
• Trigger
• Customised Notification

First part the Monitor, for our use case we are using the extraction query to monitor our index.

{
    "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "error_occurred": true
          }
        },
        {
          "match": {
            "svc.name": "timescale"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
}

}

what this query hopes to achieve is two conditions are to be met in a time interval of 1 hr and it return 10 logs/ hit for the same.

Paste this query or form your own according to your use case.

Trigger specifies the threshold value when our alert goes off.

Trigger condition of receiving >100 errors in an hour

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Get Shreshth Pal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Customised notification is what makes an alert message relevant. Set up your destination before moving forward

Fields of a message can be accessed using ctx data structure

ctx.results.0.hits.hits.<index_number 0-size>._source
example:
ctx.results.0.hits.hits.0._source.message

the following query above can access the message of our hit.

Built upon it, is a template.

{
    "@type": "MessageCard",
    "@context": "http://schema.org/extensions",
    "themeColor": "d76100",
    "summary": "xyz",
    "sections": [{
        "activityTitle": "Alert",
        "activitySubtitle": "Trigger: {{ctx.trigger.name}}",
        "activityImage": "https://img.icons8.com/?size=512&id=110240&format=png",
        "facts": [{
            "name": "Period start",
            "value": "{{ctx.periodStart}}"
        }, {
            "name": "Period end",
            "value": "{{ctx.periodEnd}}"
        }, {
            "name": "Error",
            "value": "{{ctx.monitor.name}}"
        }, {
            "name": "Error_message-1",
            "value": "{{ctx.results.0.hits.hits.0._source.message}}" 
        },{
            "name": "Error_message-2",
            "value": "{{ctx.results.0.hits.hits.1._source.message}}" 
        },{
            "name": "Error_message-3",
            "value": "{{ctx.results.0.hits.hits.2._source.message}}" 
        }],
        "markdown": true
    }]

}

Expected response of the template.

Before you go on creating your own alerts :(

https://xkcd.com/2342/