Last time, in Part 1, we discussed setting up a firewall and an email relay so notifications from the firewall could get to us.
Now, in Part 2, we’re going to talk about more signal. Server monitoring and alerting. Our primary software for monitoring is Monit.
Monit has a single configuration file, but many distributions, including mine, set up a /etc/monit.d folder so you can divide your monit configuration into different files.
Once it is running, you can monitor its status by running
monit status
It will show the status of whatever is monitoring. There is also an optional web component, if you want to check status in a web browser.
What can you monitor?
Monit can monitor any program and restart it if it crashes.
check process nginx with pidfile /var/run/nginx.pid
start program = "/bin/systemctl start nginx.service"
stop program = "/bin/systemctl stop nginx.service"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout
As you can see, the simple scripting language allows you to not only restart, execute programs, but alert the user.
Not only can it make sure something is running, but it can monitor its resource usage, as well as system resource usage. It can monitor processes, network connections, programs and scripts, files, directories, etc.
An Alternative to Email Alerts
The default for an alert is to send an alert email, but for bigger emergencies, a phone push notification is also useful.
Monit provides a simple instruction on how to set it up for Pushover. There is also the alternative of PushBullet.
Pushover costs $5 per platform(Android, iOS, Desktop) to use on as many devices as you want. There is a per application limit of 7,500 messages per month. Pushbullet is, by comparison, free. The basic difference as I see it is that Pushbullet is more geared toward the consumer, and Pushover is more geared toward developers in how it was initially set up. They do have similar feature sets though.
Here is Monit’s suggested Pushover script, which can be run instead of an email alert.
/usr/bin/curl -s
-F "token=your_mmonit_or_monit_app_token"
-F "user=your_pushover_net_user_token"
-F "message=[$MONIT_HOST] $MONIT_SERVICE - $MONIT_DESCRIPTION"
https://api.pushover.net/1/messages.json
Here is an alternative version for Pushbullet
curl -u <your_access_token_here>: -X POST https://api.pushbullet.com/v2/pushes --header 'Content-Type: application/json' --data-binary '{"type": "note", "title": "$MONIT_HOST", "body": "$MONIT_SERVICE - $MONIT_DESCRIPTION"}'
Conclusion
In all cases, monit allows you to monitor your system and take action based on a change in performance. The complexity of your rules is entirely up to you. But, if you give thought to their setup, you can not only be told when there is a server emergency, but the system can take action to fix it.