The importance of Logging (Is your machine trying to tell you something?)

The AIX operating system produces a lot of useful diagnostic information which can prove invaluable to the system administrators, however many of this messages are missed simply because the logging is not correctly configured.

You should regularly check the following:

• Startup and console logs
• AIX system error logs
• Syslog messages

AIX maintains special circular files known as alog's. These logs are available both from smit (smit alog) or from the command line using the alog command. You can choose from the following: 

• boot - This file records the LED codes displayed on the front of the machine
• bosinst - These messages were generated when the system was installed
• nim - Network Install Manager messages
• console - Messages sent to a virtual console device
• cfg - Output of the plug-and-play cfgmgr utility
• mdmplog - optional
• lvmt - Output of the logical volume manager daemon
• lvmcfg - The status of the lvmo utility. This manages logical-volume performance
• dumpsymp - Information about system dumps created

      To display the contents of the console alog file from the command line:

      alog -o -f '/var/adm/ras/conslog'

The errpt command can be used to view the AIX system error messages, errpt -a gives more details.

The syslog daemon needs to be configured to record system messages and the log files MUST exist BEFORE you start the process, otherwise the messages will be lost. Log files grow indefinitely by default and so needed to be manually copied and truncated, however the latest version of syslog includes the "compress" and "rotate" options which can be used to better manage your logfiles. An example is:

user.debug     /var/log/syslog.out         rotate size 100k files 4
user.notice    /var/log/syslog.out         rotate size 200k files 16

Messages are sent to syslog text files according to predefined alert types and status levels and the default behaviour is to  record everything and above, so if you set DEBUG, you will get EVERYTHING unless you use the "!" switch.

Example:

kern.info;kern.!err          /var/adm/kernel-info

save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info. Everything from err and higher is excluded.

Facilities, priorities, and destinations

The configuration file "/etc/syslog.conf" determines how messages from a particular source and of a set priority will be dealt with. Every system program that uses syslogd will transmit messages at a pre-defined "facility". Those facilities are:

kern	kernel messages
user	various user-level programs
mail	sendmail
daemon	system daemons, including ftpd
auth	authorization messages
authpriv	authorization messages whose viewing should be restricted to root
syslog	messages generated internally by the syslog daemon
lpr	lpd (printer subsystem)
news	nntp (news) server messages
uucp	uucp subsystem messages
cron	crond messages
local0 - local7	facilities available for administrator-defined use

Each "message" sent to syslogd also has a severity or priority attached to it. Those priorities, in order of severity from highest to lowest, are:

emerg	daemon or subsystem failure has occurred or is pending
alert	immediate action is required to prevent failure
crit	a critical condition has occurred
err	an error has occurred
warning	a warning has occurred
notice	a normal, but significant event has occurred
info	informational messages
debug	debug-level messages

Rotation

Keyword	Description
rotate	This keyword must be specified after the action field.
size	This keyword specifies that rotation is based on size. It is followed by a number and either a k (kilobytes) or m(megabytes).
time	This keyword specifies that rotation is based on time. It is followed by a number and either a h(hour) or d(day) or w(week) or m(month) or y(year).
files	This keyword specifies the total number of rotated files. It is followed by a number. If not specified, then there are unlimited number of rotated files.
compress	This keyword specifies that the saved rotated files will be compressed.
archive	This keyword specifies that the saved rotated files will be copied to a directory. It is followed by the directory name.

An example syslog config file:

#Log all warnings

*.warn  /var/adm/messages       rotate size 500k files 7 compress archive /var/adm/syslog_archive

#Log security messages

auth.debug      /var/adm/security.messages      rotate size 500k files 7 compress archive /var/adm/syslog_archive

#Send interesting security events to syslog on bigserver

auth.notice            @bigserver

#Notify everyone of real probs

*.alert     *

*.alert     @bigserver

#Log all messages but mail messages to /var/log/debug

*.debug;mail.none              /var/log/debug

# record messages from all facilities at severity "alert" or higher in

# the AIX errlog ...

*.alert     errlog

# record messages from all facilities at severity "err" or higher in

# the log file /var/adm/errorlog

*.err   /var/adm/errorlog       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "mail" facility at severity "info" or

# higher in the log file /var/adm/mail.log

mail.info                /var/adm/mail.log       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "auth" and "authpriv" facilities at

# severity "info" or higher in the log file /var/adm/auth.log

auth,authpriv.info      /var/adm/auth.log       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "daemon" facility at severity "info" or

# higher in the log file /var/adm/daemon.log

daemon.info     /var/adm/daemon.log     rotate size 500k files 7 compress archive /var/adm/syslog_archive