Grafana Mail Alert Configuration

In this article, I will demonstrate how to configure an alert in the Grafana monitoring tool. I already published about Grafana installation and configuration on my other post. These can be accessed from the below links.

Grafana installation (TIG)

Grafana installation with MySQL database.

Here we will do step by step configuration of Grafana alert configuration using mail server. Grafana supports various types of alerting systems like mail, Slack, PagerDuty, etc.  In this post, I will use Gmail as my mail server to send mail. So first we need to add mail server details in the Grafana configuration file. Open the config file and add the following details.

[smtp]

enabled = true

host = smtp.gmail.com:587

user = <emailid>@gmail.com

password = XXXXXXXX

;cert_file =

;key_file =

;skip_verify = false

from_address = admin@grafana.localhost

from_name = Grafana

 

Here I am not using certificate file, key file and verify so I commented out it. Once all the above details are added restart the Grafana server to load new changes in the configuration file.

$ sudo systemctl restart grafana-server

:~$ sudo systemctl status grafana-server

● grafana-server.service - Grafana instance

   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)

   Active: active (running) since Sat 2019-10-12 15:12:04 UTC; 1min 4s ago

     Docs: http://docs.grafana.org

 Main PID: 6831 (grafana-server)

 

Next, Login to the Grafana Home and click on the Bell icon on the left panel and choose the “Notification channel” to configure a new channel.

 

Add a name for your Alert channel and if you want image along with the notification then enable it. Under the Email, address section adds all the mail address that you want to receive the notifications.

I have added admin@techiescorner.in as my recipient. Click and “Send Test” button to make sure that the mail server is working.

When I clicked the first time I have got below error message on my Grafana screen.

To know more about the error I have tailed my Grafana log file(

/var/log/grafana/grafana.log) and found the following error message.

t=2019-10-13T11:00:31+0000 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=[admin@techiescorner.in] t=2019-10-13T11:00:31+0000 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="Failed to send notification to email address: admin@techiescorner.in: 535 5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials 26sm12959121iog.10 - gsmtp"

 

I am pretty sure that my username and password are correct, but the message shows that it is wrong. 🙁 To overcome this we need to make sure we haven’t enabled 2 step verification. If you have enabled it you need to generate a new password. Also, we have to enable “Less secure app access”

 

 

All the above details can be found under your Google account security section.

Now, go back to Grafana and click the “Send Test” button to see it’s working or not. But again I got the same error notification, but this time the Grafana error message was different. Adding error message below.

t=2019-10-13T12:20:17+0000 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=1 uname=admin error="Failed to send notification to email address: admin@techiescorner.in: 534 5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbty\n5.7.14 Xt90DgmP28mdkuXBGZM-eZ1V_Cf\n5.7.14 ZPcc7w1UZXH8Y07NWw7lLVYGpzaygCsqlXfZJ66edBH4-DFCBAkwXZPmTUmVDD> Please\n5.7.14 log in via your web browser and then try again.\n5.7.14  Learn more at\n5.7.14  https://support.google.com/mail/answer/78754 i26sm13465460ion.40 - gsmtp" t=2019-10-13T12:20:17+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=106.51.111.165 time_ms=1304 size=48 referer=http://xx.xxx.xxx.xxx:3000/alerting/notification/new

This error is because I haven’t enabled google to allow access from remote servers. To enable click the below link.

https://accounts.google.com/DisplayUnlockCaptcha

 

Yes, try again to see still we are getting any error message. Go to the Grafana dashboard and click the “Send Test” button.

Yes, this time I got a green signal

Check the recipient’s email address to make sure that we have received the test email. You will get a test mail like below.

 

Next, we will create a Graph and set an alert for it.

Go to Grafana home and create a new dashboard and click on “Add Query”.

 

Then configure the parameter as shown in the below image.

 

After you add these value your graph start pull the values from the host and display. next, click on General and give a name and description for your graph.

 

 

Next, click on the bell icon and create an alert rule.  Here the Grafana will send an alert if the CPU usage reaches 50% or more.

Save the dashboard. Yes, we have configured an alert.

Next, we will do a test to make sure that our alert configuration is working as expected, for this I will create load on the host.

Execute below command and it will generate load on the server.

dd if=/dev/zero of=/dev/null

Monitor the graph and make sure that the graph line is showing an increased load on the server. Here you can see a sudden spike has happened, if it continues for 1 minute ( we configured in the graph) then Grafana will trigger a mail notification.

Check your mail inbox and you can see an mail like below.

Grafana will send another mail notification once the system load falls back to below 50%. To check that you can cancel the command that we running and wait for few minutes to settle down the load.

That’s all guys we have configured mail alert for Grafana using Google mail server.