More logging

AIX has some optional logging that can prove invaluable when debugging a problem.

1. A good example of this is SNMP. You can verify operation either locally or remotely as follows:

# snmpinfo -m get sysdescr.0
1.3.6.1.2.1.1.1.0 = "IBM PowerPC CHRP Computer
Machine Type: 0x0800004c Processor id: 00C9B8FA4C00
Base Operating System Runtime AIX version: 06.01.0008.0000
TCP/IP Client Support version: 06.01.0008.0001"

If this command fails, or gives unexpected results, you can configure logging by editing the "/etc/snmpd.conf" as follows and then restarting the deamon:

logging         file=/usr/tmp/snmpd.log         enabled
logging         size=100000                     level=4
The default logging level is "0" which means no logging. The log is circular and restricted to the maximum size in bytes.

Once you restart the snmpd daemon you should start to see all kinds of useful messages.

2. The inetd super daemon can produce all kinds of debugging information. Simply restart the daemon with "-d" option. Messages are logged by syslog.


3. AIX 6.1 can now manage the logs produced by the cron daemon. This is controlled by the "/etc/cronlog.conf" file. You must restart cron in order for the changes to take effect.

Checking your BIOS and microcode levels

It is vital that you regularly visit IBM Fix Central and check for firmware updates and fixes as failing to keep them up to date can invalidate your support agreement. Whenever you install an AIX update such as a TL or SP it may include some firmware updates and there is a danger that the firmware on your cards is newer than that in the frame, and this can cause unpredictable results. IBM provides the invscout utility which generates XML reports that can be uploaded to the IBM, webserver which then automatically diagnoses your firmware and produces a report. This is the IBM invscout manual page: http://pic.dhe.ibm.com/infocenter/aix/v7r1/index.jsp?topic=%2Fcom.ibm.aix.cmds%2Fdoc%2Faixcmds3%2Finvscout.htm You can also view a summary of your microcode from the command-line as follows: /usr/sbin/lsmcode -Ac

sys0!system:SF240_417 (t) SF240_403 (p) SF240_417 (t) ent0!14108902.DV0210 ent1!14108902.DV0210 ent2!1410ff01.SCU015 sisscsia0!44415255.070A0011 ent3!1410ff01.SCU001 ent5!14106902.GOL021 cd0!IBM-DROM002054.P633 rmt0!C105 hdisk2!ST37320.4A553042.43373041 hdisk3!ST37320.4A553042.43373043 ent4!14106902.GOL002 hdisk0!ST37320.4A553042.43373041 hdisk4!ST37320.4A553042.43373041   Your system has two copies of firmware in case an update fails or you want to test two versions. The version that the system boots from can be selected from the firmware.

The importance of Logging (Is your machine trying to tell you something?)

The AIX operating system produces a lot of useful diagnostic information which can prove invaluable to the system administrators, however many of this messages are missed simply because the logging is not correctly configured.

You should regularly check the following:

• Startup and console logs
• AIX system error logs
• Syslog messages

AIX maintains special circular files known as alog's. These logs are available both from smit (smit alog) or from the command line using the alog command. You can choose from the following: 

• boot - This file records the LED codes displayed on the front of the machine
• bosinst - These messages were generated when the system was installed
• nim - Network Install Manager messages
• console - Messages sent to a virtual console device
• cfg - Output of the plug-and-play cfgmgr utility
• mdmplog - optional
• lvmt - Output of the logical volume manager daemon
• lvmcfg - The status of the lvmo utility. This manages logical-volume performance
• dumpsymp - Information about system dumps created

      To display the contents of the console alog file from the command line:

      alog -o -f '/var/adm/ras/conslog'

The errpt command can be used to view the AIX system error messages, errpt -a gives more details.

The syslog daemon needs to be configured to record system messages and the log files MUST exist BEFORE you start the process, otherwise the messages will be lost. Log files grow indefinitely by default and so needed to be manually copied and truncated, however the latest version of syslog includes the "compress" and "rotate" options which can be used to better manage your logfiles. An example is:

user.debug     /var/log/syslog.out         rotate size 100k files 4
user.notice    /var/log/syslog.out         rotate size 200k files 16

Messages are sent to syslog text files according to predefined alert types and status levels and the default behaviour is to  record everything and above, so if you set DEBUG, you will get EVERYTHING unless you use the "!" switch.

Example:

kern.info;kern.!err          /var/adm/kernel-info

save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info. Everything from err and higher is excluded.

Facilities, priorities, and destinations

The configuration file "/etc/syslog.conf" determines how messages from a particular source and of a set priority will be dealt with. Every system program that uses syslogd will transmit messages at a pre-defined "facility". Those facilities are:

kern	kernel messages
user	various user-level programs
mail	sendmail
daemon	system daemons, including ftpd
auth	authorization messages
authpriv	authorization messages whose viewing should be restricted to root
syslog	messages generated internally by the syslog daemon
lpr	lpd (printer subsystem)
news	nntp (news) server messages
uucp	uucp subsystem messages
cron	crond messages
local0 - local7	facilities available for administrator-defined use

Each "message" sent to syslogd also has a severity or priority attached to it. Those priorities, in order of severity from highest to lowest, are:

emerg	daemon or subsystem failure has occurred or is pending
alert	immediate action is required to prevent failure
crit	a critical condition has occurred
err	an error has occurred
warning	a warning has occurred
notice	a normal, but significant event has occurred
info	informational messages
debug	debug-level messages

Rotation

Keyword	Description
rotate	This keyword must be specified after the action field.
size	This keyword specifies that rotation is based on size. It is followed by a number and either a k (kilobytes) or m(megabytes).
time	This keyword specifies that rotation is based on time. It is followed by a number and either a h(hour) or d(day) or w(week) or m(month) or y(year).
files	This keyword specifies the total number of rotated files. It is followed by a number. If not specified, then there are unlimited number of rotated files.
compress	This keyword specifies that the saved rotated files will be compressed.
archive	This keyword specifies that the saved rotated files will be copied to a directory. It is followed by the directory name.

An example syslog config file:

#Log all warnings

*.warn  /var/adm/messages       rotate size 500k files 7 compress archive /var/adm/syslog_archive

#Log security messages

auth.debug      /var/adm/security.messages      rotate size 500k files 7 compress archive /var/adm/syslog_archive

#Send interesting security events to syslog on bigserver

auth.notice            @bigserver

#Notify everyone of real probs

*.alert     *

*.alert     @bigserver

#Log all messages but mail messages to /var/log/debug

*.debug;mail.none              /var/log/debug

# record messages from all facilities at severity "alert" or higher in

# the AIX errlog ...

*.alert     errlog

# record messages from all facilities at severity "err" or higher in

# the log file /var/adm/errorlog

*.err   /var/adm/errorlog       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "mail" facility at severity "info" or

# higher in the log file /var/adm/mail.log

mail.info                /var/adm/mail.log       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "auth" and "authpriv" facilities at

# severity "info" or higher in the log file /var/adm/auth.log

auth,authpriv.info      /var/adm/auth.log       rotate size 500k files 7 compress archive /var/adm/syslog_archive

# record messages from the "daemon" facility at severity "info" or

# higher in the log file /var/adm/daemon.log

daemon.info     /var/adm/daemon.log     rotate size 500k files 7 compress archive /var/adm/syslog_archive