Good ol' Nagios. The monitoring system that just won't die and for good reason it does what we want and does it well.
I found there were not very many good resources on setting up nagios alerts to publish to SNS topics for SMS subscriptions so I did the leg work and am presenting it here.
Note: Its expected that you already have Nagios set up on a server and understand at least the basics of configuration definitions.
Setting up the SNS topic in AWS
The first step we need to do is actually set up the SNS topic via the AWS dashboard.
Make sure you get the Topic ARN after creating it, we will need to create a service account and IAM policy to publish the nagios alerts to the topic. It will look like
arn:aws:sns:us-east-1:101010101010:nagios-publish where 101010101010 is your account number.
While you're in the Topics dashboard create subscriptions for the engineers that may want to be alerted to nagios alerts by clicking the "create subscription" button
go to
https://console.aws.amazon.com/iam/home?region=us-east-1#/users and create the user that will have access to publish. I always use explicit naming conventions to easily manage them via IAM. In this case it will be
service.nagios. Save the access and secret key values that are created for the user.
Now create a policy
policy.nagios-sns with the following permissions.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": [
"arn:aws:sns:us-east-1:101010101010:nagios"
]
}
]
}
Nagios Integration
first thing you are going to want to do is add the
aws_access_key_id and
aws_secret_access_key to the server nagios is on under the nagios user (or whatever user the nagios service was set up to run under)
By default the user that runs the nagios service has its shell set to
/bin/nologin but you can get around that by issuing as root
su - nagios -s /bin/bashIf you're already on an AWS instance you can just issue command
aws configure if nagios is not running in amazon install the awscli client by following steps here
http://docs.aws.amazon.com/cli/latest/userguide/installing.html once you add the keys they are stored in
~nagios/.aws/credentials and the policy you created for the service account tied to these keys will be permitted.
We are going to create a separate contact template and command inside our nagios configuration just for being alerted via SNS. Navigate to the objects directory inside your nagios installation. my path is
/usr/local/nagios/etc/objects and edit the file
templates.cfg in search for
generic-contact we are pretty much copying everything from that generic contact default entry and just changing names around and passing a different command. The bottom is our copied and edited entry that we need named sns-contact.
# Generic contact definition template - This is NOT a real contact, just a template!
define contact{
name generic-contact ; The name of this contact template
service_notification_period 24x7 ; service notifications can be sent anytime
host_notification_period 24x7 ; host notifications can be sent anytime
service_notification_options w,u,c,r,f,s ; send notifications for all service states, flapping events, and scheduled downtime events
host_notification_options d,u,r,f,s ; send notifications for all host states, flapping events, and scheduled downtime events
service_notification_commands notify-service-by-email ; send service notifications via email
host_notification_commands notify-host-by-email ; send host notifications via email
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
}
define contact{
name sns-contact ; The name of this contact template
service_notification_period 24x7 ; service notifications can be sent anytime
host_notification_period 24x7 ; host notifications can be sent anytime
service_notification_options w,u,c,r,f,s ; send notifications for all service states, flapping events, and scheduled downtime events
host_notification_options d,u,r,f,s ; send notifications for all host states, flapping events, and scheduled downtime events
service_notification_commands notify-service-by-sns ; send service notifications via email
host_notification_commands notify-host-by-sns ; send host notifications via email
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
}
The real magic behind the nagios aws sns integration is within the command itself. Its how we publish alerts to the topic.
edit the configuartion file
/usr/local/nagios/etc/objects/commands.cfg and add this under sample notification commands.
# 'notify-host-by-sns' command definition
define command{
command_name notify-host-by-sns
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | aws sns publish --topic-arn arn:aws:sns:us-east-1:101010101010:nagios --message "$NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$"
}
# 'notify-service-by-sns' command definition
define command{
command_name notify-service-by-sns
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | aws sns publish --topic-arn arn:aws:sns:us-east-1:101010101010:nagios --message "$NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$"
}
We're almost done now. The last thing we have to do is define a contact group and contact to inherit the sns-contact object definitions.
Wherever you have your contacts defined add the entry
#################################################################################
# SMS Admin contact group #
#################################################################################
define contactgroup{
contactgroup_name smsadmins
alias SMS Nagios Administrators
members sms-distro
}
define contact{
contact_name sns ; Short name of user
use sns-contact ; Inherit default values from generic-contact template (defined above)
alias SNS Alert ; Full name of user
service_notification_options c,r
}
From here on out when you define a new server check that you want to get SMS notifications for you would just add the contact group we just created to the check.
# Define a service to "ping" the local machine
define service{
use remote-service,srv-pnp ; Name of service template to use
host_name serverName
service_description PING
contact_groups snsgroup,emailadmins
check_command check_ping!250.0,20%!500.0,60%
}
Finally after all the work you can now publish nagios alerts to the SNS topic and get those alerts through SMS subscription.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
