James Permenter

Good ol' Nagios. The monitoring system that just won't die and for good reason it does what we want and does it well.

I found there were not very many good resources on setting up nagios alerts to publish to SNS topics for SMS subscriptions so I did the leg work and am presenting it here.

Note: Its expected that you already have Nagios set up on a server and understand at least the basics of configuration definitions. 

Setting up the SNS topic in AWS

The first step we need to do is actually set up the SNS topic via the AWS dashboard.




Make sure you get the Topic ARN after creating it, we will need to create a service account and IAM policy to publish the nagios alerts to the topic. It will look like arn:aws:sns:us-east-1:101010101010:nagios-publish where 101010101010 is your account number.

While you're in the Topics dashboard create subscriptions for the engineers that may want to be alerted to nagios alerts by clicking the "create subscription" button



go to https://console.aws.amazon.com/iam/home?region=us-east-1#/users and create the user that will have access to publish. I always use explicit naming conventions to easily manage them via IAM. In this case it will be service.nagios. Save the access and secret key values that are created for the user.

Now create a policy policy.nagios-sns with the following permissions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            
            "Effect": "Allow",
            "Action": [
                "sns:Publish"
            ],
            "Resource": [
                "arn:aws:sns:us-east-1:101010101010:nagios"
            ]
        }
    ]
}

Nagios Integration



first thing you are going to want to do is add the aws_access_key_id and aws_secret_access_key to the server nagios is on under the nagios user (or whatever user the nagios service was set up to run under)

By default the user that runs the nagios service has its shell set to /bin/nologin but you can get around that by issuing as root su - nagios -s /bin/bash

If you're already on an AWS instance you can just issue command aws configure if nagios is not running in amazon install the awscli client by following steps here http://docs.aws.amazon.com/cli/latest/userguide/installing.html once you add the keys they are stored in ~nagios/.aws/credentials and the policy you created for the service account tied to these keys will be permitted.

We are going to create a separate contact template and command inside our nagios configuration just for being alerted via SNS. Navigate to the objects directory inside your nagios installation. my path is /usr/local/nagios/etc/objects and edit the file templates.cfg in search for generic-contact we are pretty much copying everything from that generic contact default entry and just changing names around and passing a different command. The bottom is our copied and edited entry that we need named sns-contact.

# Generic contact definition template - This is NOT a real contact, just a template!

define contact{
        name                            generic-contact         ; The name of this contact template
        service_notification_period     24x7                    ; service notifications can be sent anytime
        host_notification_period        24x7                    ; host notifications can be sent anytime
        service_notification_options    w,u,c,r,f,s             ; send notifications for all service states, flapping events, and scheduled downtime events
        host_notification_options       d,u,r,f,s               ; send notifications for all host states, flapping events, and scheduled downtime events
        service_notification_commands   notify-service-by-email ; send service notifications via email
        host_notification_commands      notify-host-by-email    ; send host notifications via email
        register                        0                       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
        }


define contact{
        name                            sns-contact         ; The name of this contact template
        service_notification_period     24x7                    ; service notifications can be sent anytime
        host_notification_period        24x7                    ; host notifications can be sent anytime
        service_notification_options    w,u,c,r,f,s             ; send notifications for all service states, flapping events, and scheduled downtime events
        host_notification_options       d,u,r,f,s               ; send notifications for all host states, flapping events, and scheduled downtime events
        service_notification_commands   notify-service-by-sns ; send service notifications via email
        host_notification_commands      notify-host-by-sns    ; send host notifications via email
        register                        0                       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
        }

The real magic behind the nagios aws sns integration is within the command itself. Its how we publish alerts to the topic.

edit the configuartion file /usr/local/nagios/etc/objects/commands.cfg and add this under sample notification commands.

# 'notify-host-by-sns' command definition
define command{
        command_name    notify-host-by-sns
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | aws sns publish --topic-arn arn:aws:sns:us-east-1:101010101010:nagios --message "$NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$"
        }

# 'notify-service-by-sns' command definition
define command{
        command_name    notify-service-by-sns
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" |  aws sns publish --topic-arn arn:aws:sns:us-east-1:101010101010:nagios --message "$NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$"
        }

We're almost done now. The last thing we have to do is define a contact group and contact to inherit the sns-contact object definitions.

Wherever you have your contacts defined add the entry

#################################################################################
# SMS  Admin contact group                                                      #
#################################################################################

define contactgroup{
        contactgroup_name       smsadmins
        alias                   SMS Nagios Administrators
        members                 sms-distro
        }



define contact{
        contact_name                    sns            ; Short name of user
        use                             sns-contact         ; Inherit default values from generic-contact template (defined above)
        alias                           SNS Alert          ; Full name of user
        service_notification_options    c,r
        }

From here on out when you define a new server check that you want to get SMS notifications for you would just add the contact group we just created to the check.

# Define a service to "ping" the local machine

define service{
        use                             remote-service,srv-pnp         ; Name of service template to use
        host_name                       serverName
        service_description             PING
        contact_groups                  snsgroup,emailadmins
	check_command			check_ping!250.0,20%!500.0,60%
        }

Finally after all the work you can now publish nagios alerts to the SNS topic and get those alerts through SMS subscription.