syslog: Real-Time Email-Notification For Critical syslog Events

One of the big advantages of syslog is the separation between the log request and the logging action.

The following steps will show how to :

   1. Write all critical events to a local logfile
   2. Log this to the console
   3. Send an email notification of the event. 

In this example, we'll take all critical messages written from all facilities and (in addition to logging) send them to the mail recipient, named@example.com. 

First, create a log file for critical messages, for example:

 # touch /var/adm/critmessages
 # chmod 600 /var/adm/critmessages

Next, configure syslog to log all critical messages written from all facilitites to this log file. add the following statement to your syslog.conf file.

One thing to note when editing /etc/syslog.conf, you can't use spaces, you must use tabs.

 *.crit                             /var/adm/critmessages

The final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i've included an example below, let's call it /usr/bin/syslogMailer:

 #!/bin/bash
 # syslogMailer: a script to read stdin and turn each line into an alert
 # email typically this is used to read a named-pipe written to by syslog
 #
 #   example usage: syslogMailer.sh < /var/adm/critmessages
 #

 hostname=`hostname`
 alertRecipient="named@example.com"      # the mail recipient for alerts
 TMOUT=1                                 # don't wait > 1 second for input

 # process each line of input and produce an alert email
 while read line
 do
    # remove any repeated messages
    echo ${line} | grep "message repeated" > /dev/null 2>&1
    if test $? -eq 1
    then
       # send the alert
       echo "${line}" | mailx -s "Critical Error on syslog | $hostname" ${alertRecipient}
       cat /dev/null > /var/adm/critmessages
    fi
 done

This allows you to schedule it in cron to run, say, every 1 min of every hour with a statement like:

 * * * * * /usr/bin/syslogMailer.sh < /var/adm/critmessages > /dev/null

After changing the configuration, make sure that you restart syslogd, on Solaris 10 Update 1 or newer you'd use svcadm restart svc:/system/system-log:default.

Test it by simply running the following logger command.

 logger -p auth.crit "test"