Using syslog in a chroot jail

If your system uses chroot jails you should enable separate logging within them in order to catch important messages such as break-in attempts.

1. Setup the chroot jail.
The jail requires a dedicated filesystem (in this example “/var/local/chroot”), and a small selection of executables and libraries. These should be copied directly from the installed OS, and updated following each major OS patch or update (There are many scripts available on the web that automate the building of a jail so I won't list all the steps).

mknod /var/local/chroot/dev/null c 2 2
chmod 666 /var/local/chroot/dev/null

cp -p /usr/bin/ksh /var/local/chroot/usr/bin/
cp -p /usr/lib/libcrypt.a /var/local/chroot/usr/lib/
cp -p /usr/lib/libc.a /var/local/chroot/usr/lib/
cp -p /usr/lib/libodm.a /var/local/chroot/usr/lib/
cp -p /usr/lib/libsrc.a /var/local/chroot/usr/lib/

cp -p /usr/sbin/sftp-server /var/local/chroot/usr/sbin
cp -p /usr/sbin/syslogd /var/local/chroot/usr/sbin

2. A chroot’ed syslog daemon is required for each jail in order to enable jailed events to be logged to the global syslog. These processes should be started AFTER the main syslog daemon and thus placed at the end of the “/etc/inittab” as follows:

syslogJAIL:23456789:once:chroot /var/local/chroot /usr/sbin/syslogd -f /etc/chroot.conf -p /dev/log


Note:    The “chroot*.conf” exists within the jailed environment and can be used to send alerts to a remote/central log file e.g.

/var/local/chroot/etc:
*.debug;local3.none                 /var/adm/sftp.log               rotate size 1m files 21 compress


3. The sshd daemon runs from within each jail and thus requires its own “sshd_config” file:

/var/local/chroot/etc/sshd_config:
Protocol 2
PermitRootLogin no
IgnoreUserKnownHosts yes
PermitEmptyPasswords no
HostbasedAuthentication yes
LogLevel VERBOSE
SyslogFacility AUTH
UsePrivilegeSeparation yes
PasswordAuthentication no
ChallengeResponseAuthentication no
AllowGroups netadm netsadm  sftpgrp
Subsystem      sftp    /usr/sbin/sftp-server -f AUTH -l VERBOSE
#AuthorizedKeysFile     %h/.ssh/authorized_keys

Match User jailed_user
        PasswordAuthentication yes

A login file prevents the jailed users from spawning a shell e.g.

“/var/local/chroot/./home/jaileduser/.profile”:

trap "" 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
echo "This account may not be used for shell login!";exit 0
trap 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16


AIX normally knows that user is jailed because they have a "." within there home-directory path in "/etc/passwd". This can create a problem when trying to SSH or SFTP to the parent system as the jailed user. In this case you need to find the AIX chroot patch for your version of SSH, and then recompile it.

Network subnetting

IP subnetting can be a real black art at the best of times. Here is a really useful network cheat sheet that can save a lot of time whilst increasing your understanding of CIDR (Classless Internet Domain Routing).

Open Source Projects

Here is a collection of the more interesting AIX/Opensource projects currently available in the market. (There are of course others but they are not always actively maintained etc).

Note: Many of these projects also provide a commercial/paid and supported version.

Scripts for AIX
Snare
Wireshark network monitoring
Berlin job scheduler
Ganglia monitoring system
GraphLVM
AIX Partitions and Systems Performance
General Graphics Interface
OpenSmart system monitoring and reporting tool
IBM MQ explorer tool
System configuration collector
OpenSmart system monitoring tool
AIX Administration scripts
AIX LPP package generator


general libraries and tools
Boxbackup

UK government names Linux the most secure platform

http://www.techrepublic.com/blog/linux-and-open-source/linux-the-clear-choice-for-security/?tag=nl.e099&s_cid=e099&ttag=e099&ftag=TREd8c0fa8

This may not seem directly relavent to AIX, however what it does show is that a correctly configured Unix/Linux system is clearly far more secure than Windows and this is highly significant when choosing your corporate infrastructure.

AIX currently lags behind Linux in one major security area and that is the lack of support for iptables. Iptables is a rule-based firewall built directly into the Linux Kernel. AIX does have ipfilt, however this is currently nowhere near as widely supported nor understood.

One other AIX/Linux feature that is definitely not given enough attention is IPSec. IPSec can be used to create a secure private network/tunnel between servers, and because everything is encapsulated at pack level (layer-2), you do not need to worry about complex firewall rulesets. AIX IPSec performance can also be greatly enhanced by offloading the encryption overhead to the actual network hardware/card.

The importance of BIOS Updates

I regularly answer questions on it.toolbox and also find it an invaluable source of information. Yesterday somebody was complaining that they could not see the NPIV virtual adapters in AIX, and this reminded me of a similar problem I had recently with a new Ethernet card in a p710+

When the machine was delivered from IBM it came with almost the latest system BIOS but the required level of HMC/VIOS was not yet available so I had to go through an unbelievable route of downloading endless versions of VIOS and HMC, and then had to patch them in a particular order before everything would play nicely together. That eventually meant that I could do anything I wanted with the system and cards at the hardware level, but once I tried to configure my highly available SEA in VIOS, things quickly went pear-shaped.

After a lot more research I download the latest BIOS/firmware for the actual card and manually added it to my VIOS, et-voila the card was correctly recognised and worked as expected. I then tried to build a NIM server to push out my OS images, and the same problem re-occurred. Again the only way to see the card was to patch AIX to the very latest tech-level, and then to apply the firmware update again.

Finally everything worked as expected!

The thing I wanted to stress is that you must keep your BIOS an Tech-Levels up to date, otherwise when you try to install new hardware it may not work at all.

Pstree

This is not the newest or most powerful command in AIX, and I know it won't be news to many people, however I do believe its power and simplicity are overlooked and it can be a great way to visualise how services such as the SRC (System Resource Controller) damon actually work.

Consider this example:

# ps -T 1835256
     PID    TTY  TIME CMD
 1835256      -  0:00 srcmstr
 1441898      -  0:00    |\--ksh
 3473544      -  0:00    |    \--tlmagent.bin
 2752612      -  0:00    |\--cimssys
 3276934      -  0:00    |\--cimssys
 3670134      -  0:00    |\--snmpdv3ne
 3866750      -  0:00    |\--sendmail
 4456486      -  0:00    |\--pmserviced
 4653200      -  0:00    |\--tftpd
 5374142      -  0:00    |    \--tftpd
 4718738      -  0:00    |\--inetd
 5898340      -  0:00    |    \--bootpd
 4849820      -  0:00    |\--xntpd
 4980896      -  0:00    |\--snmpmibd
 5177356      -  0:04    |\--rmcd
 5308630      -  0:00    |\--portmap
 5963950      -  0:00    |\--pnsd
 6029524      -  0:00    |\--nimesis
 6095064      -  0:00    |\--sshd
 6225990      -  0:00    |   |\--sshd
 3801168  pts/1  0:00    |   |    \--ksh
 9699392  pts/1  0:00    |   |        \--ps
 9044072      -  0:02    |    \--sshd
 8454202  pts/0  0:01    |        \--ksh
 5439694  pts/0  0:00    |            \--man
 3932274  pts/0  0:00    |                \--sh
 9437218  pts/0  0:00    |                    \--more
 6291580      -  0:00    |\--ksh
 7340090      -  0:00    |    \--ksh
 4915214      -  0:23    |        \--java
 6815790      -  0:00    |\--qdaemon
 1704024      -  0:00    |    \--ksh
 4325572      -  0:00    |        \--piohpnpf
 7405594      -  0:00    |\--IBM.HWCTRLRMd
 7667732      -  0:00    |\--ksh
 9175138      -  0:00    |    \--nonstop_aix
 7864320      -  0:39    |        \--java
 7733296      -  0:19    |\--java
 7930002      -  0:00    |\--writesrv
 8192066      -  0:00    |\--IBM.DRMd
 8257564      -  0:00    |\--IBM.CSMAgentRMd
 8388690      -  0:00    |\--hwsdagent
 8519766      -  0:00    |\--aso
 8650880      -  0:00    |\--IBM.AuditRMd
 8781936      -  0:00    |\--lpd
 8847384      -  0:00    |\--IBM.ServiceRMd
 8912930      -  0:00    |\--IBM.DMSRMd
 8978462      -  0:00    |\--pmloadcheck
 9633962      -  0:00     \--syslogd

You can clearly see each of the active subsystems ans child processes such as shell or Java, and this can be very useful when trying to diagnose system performance issues. You can also extend this functionality by installing the pstree RPM:

# wget http://www.oss4aix.org/download/RPMS/pstree/pstree-2.36-1.aix5.1.ppc.rpm
# rpm -Uvh ./pstree-2.36-1.aix5.1.ppc.rpm
pstree                      ##################################################

pstree enables you to see everything that is running on your system, and how processes relate to each other, and this helps you to identify things that should not be running e.g.

# pstree
-+- 00001 root /etc/init
 |--= 524354 root aioPpool
 |--- 1376382 root [4]ldmp_process
 |--- 1507478 root /usr/ccs/bin/shlap64
 |--= 1638530 root [2]kbiod
 |--= 1769576 root aioLpool
 |--= 1900694 root /usr/lib/errdemon
 |--- 2031730 root [17]/usr/sbin/syncd 60
 |--= 2097274 root efs_tkr_gc
 |--- 2424924 root ethchanproc
 |--- 3014860 root random
 |--= 3080408 root /opt/ibm/icc/cimom/bin/dirsnmpd
 |--- 3145902 root /opt/ibm/director/cimom/bin/tier1slp
 |-+= 3276968 root /usr/sbin/srcmstr
 | |--= 2293910 root /usr/sbin/snmpmibd
 | |--= 2555958 root /usr/sbin/portmap
 | |--= 2818118 root /usr/sbin/xntpd
 | |--= 3408008 root /usr/sbin/inetd
 | |--= 3473518 root /usr/sbin/syslogd -r
 | |--= 3604590 root sendmail: accepting connections
 | |-+= 3932308 root [2]/usr/sbin/tftpd -n
 | | \--= 3866836 nobody /usr/sbin/tftpd -n
 | |--= 4063364 root /usr/sbin/aso
....
...

This machine is a NIM Master so I need tftpd to provide network boot images, otherwise this daemon should not be active!

IBM Profits slump

Is anyone actually surprised that IBM's earnings are down? The senior management can blame the global ressesion as much as they like, and make so called "big", but "empty" gestures like forgoing the odd million $ bonus here and there, but I believe the reasons for their problems are entirely self inflicted.

http://www.bbc.co.uk/news/business-25836875

In the past few years Linux and all-things cloud have become the must-have solutions for most businesses as they look to reduce costs and rid themselves of costly and inflexible IT outsourcing deals.

Linux is NOT free!

I have lost count of the times that people have told me that AIX is expensive and Linux is free whilst nothing could actually be further from the truth. The reality is that the IBM model is like owning a car, whilst the Linux/Redhat and cloud-types model are more akin to leasing it. This may seem a strange comparission but the point is that if you have a support-contract/lease model, once you stop paying paying you have nothing, coupled with the problem of how you retreive and retain your data, and what you do with it next?
I have always been a huge Linux fan and can readily see that it has some great features and clear advantages over AIX, but it also has some issues. I find the easiest comparison is Apple/IOS and Android. Apple (like IBM) controls both the hardware and software, and to a great extent what can run on it, whereas with Android, anything is possible, but when you have a problem, you are on your own.

The solution?

I believe that IBM is certainly not helping itself and the way to claw-back market share and to re-invigorate AIX is:

1. Create a developer licence which enables individual users, and SMEs to buy budget, and even second-hand hardware, and use it to test, develop, and port solutions to AIX.
2. Create a Freshmeat/Sourceforge type hub where developers are encouraged and helped to exchange code and ideas, and market new ideas.
3. Add support for YUM (YellowDog Update Manager) to AIX in order to make installing and managing RPMs easier on AIX.
4. Create an open-source version of Fix-Central where developers and prospective customers can download AIX, patches, trial products, etc. This would closely follow the way in which companies such as Redhat operate.

More syslogging

The AIX version of syslog seems to be quite tricky to configure and has a lot of features that are not particulalry well explained, and dare I say a little buggy.

Syslog is normally started from the "/etc/rc.tcpip" file and dynamically creates the "/dev/log" device.  You can add startup options as follows:

# Start up syslog daemon (for error and event logging)
# Added -r to prevent remote logging
start /usr/sbin/syslogd "$src_running" "-r"

In this example the "-r" option is passed to the syslogd process.

If you want to dynamically pass options to syslogd whilst testing use:
# startsrc -s syslogd -a "-m 5 -r -e -d"

Here is an example of an "/etc/syslog.conf" file that logs pretty-much everything:

# Log messages from all facilities at severity "alert" or higher in
# the AIX errlog (alert,emerg)
*.alert errlog

# Log all warning messages (err,crit,alert)
*.warn  /var/adm/log/error.log  rotate size 8m compress files 7

# Log security messages
auth,authpriv.info      /var/adm/log/auth.log   rotate size 8m compress files 7

# Log cron messages
cron.info       /var/adm/log/cron.log   rotate  size 8m compress files 7

# Log printer messages
lpr.info        /var/adm/log/lpr.log    rotate  size 8m compress files 7

# Log email activity
mail.info       /var/adm/log/mail.log   rotate  size 8m compress files 7

# Log news messages
news.info       /var/adm/log/news.log   rotate  size 8m compress files 7

# Log syslog messages
syslog.info     /var/adm/log/syslog.log rotate  size 8m compress files 7

# Log UUCP messages
uucp.info       /var/adm/log/uucp.log   rotate  size 8m compress files 7

# Log deamon activity
daemon.info     /var/adm/log/daemon.log rotate size 500k files 7 compress archive /var/adm/syslog_archive

# Application specific logging
local0.info     /var/adm/log/wrappers.log       rotate size 8m compress files 7
local7.info     /var/adm/log/sftp-server.log    rotate size 8m compress files 7

# Log non-critical messages only
*.debug /var/adm/log/messages.log       rotate  size 8m compress files 7

# ASO log configuration
aso.notice      /var/log/aso/aso.log    rotate size 128k time 7d
aso.info        /var/log/aso/aso_process.log    rotate size 1024k files 7
aso.debug       /var/log/aso/aso_debug.log      rotate size 8m compress files 7

You can group priorities together using the ";", ",", and "none" keywords to filter output and restrict message duplication, however this seems a little buggy and inconsistent e.g:

# output to console

*.err;mail,kern.notice;daemon,auth.debug;user.info   /dev/console

According to the syslog manual it is possible to start the syslogd using the "-e" option, however this does not seem to work correctly at AIX 6.1?

-e
            Specifies enhanced rotation. All compressed and uncompressed files
            that are available in the log directory and that are created by
            the syslogd daemon are considered for rotation.

The other issue that I have seen is that some logs such as the "aso.debug" and "aso.info" contain a lot of duplicate messages and it does not seem possible to filter them using the ".none" syntax. I would be really interested to hear from anyone that has made this work correclty.

Advanced message logging

AIX logs many messages to several places, however this often vital information is lost unless the system is correctly configured.

1. The principle subsystem for collecting messages is syslog

2. syslogd must be started (Check /etc/rc.tcpip)

# grep syslog /etc/rc.tcpip
# Start up syslog daemon (for error and event logging)
start /usr/sbin/syslogd "$src_running" -r

3. Ensure that "/etc/syslog.conf" exists and that any files referenced exist BEFORE syslogd was started.

4. Ensure there is sufficient space in the filesystem where the logging is taking place

5. Check there are no stray control or special characters in the configuration file as this can confuse the system and cause message loss.

6. Fine tune and test your configuration (eg.)

# cat /etc/syslog.conf
auth.info    /var/adm/messages rotate size 8m compress files 7
local0.info             /var/adm/wrappers.log rotate size 8m compress files 7
local7.info             /var/adm/sftp-server.log rotate size 8m compress files 7
mail.debug                      /var/adm/mail.log               rotate  size 8m compress files 7
*.debug;local0.none;local7.none /var/adm/messages               rotate  size 8m compress files 7

In this example the log files are limited to 8MB and when a file exceeds this it is backed-up and upto seven rotated copies are maintained e.g.:


# ls -l /var/adm/messages*
-rw-r--r--    1 root     system      2434639 Jan 20 11:50 /var/adm/messages
-rw-r--r--    1 root     system       485237 Dec 19 11:31 /var/adm/messages.0.Z
-rw-r--r--    1 root     system       540774 Dec 19 11:26 /var/adm/messages.1.Z
-rw-r--r--    1 root     system       328879 Dec 19 10:06 /var/adm/messages.2.Z
-rw-r--r--    1 root     system       588315 Dec 19 09:22 /var/adm/messages.3.Z
-rw-r--r--    1 root     system       421498 Dec 18 12:20 /var/adm/messages.4.Z
-rw-r--r--    1 root     system       439781 Dec 18 09:43 /var/adm/messages.5.Z
-rw-r--r--    1 root     system       445143 Dec 16 16:05 /var/adm/messages.6.Z

Further copies can be created by using the archive keyword.

7. By default each entry logs all messages at and ABOVE the level described e.g. "*.debug" will log every single system message to this file. To restrict logging only to that level use the ".none" keyword. e.g. "*.debug;local0.none;local7.none".

Note: If you want to specify multiple logging levels use the ";" to separate them.

8. By default the syslog daemon accepts messages from other systems. If you wish to override this behaviour start syslogd with the "-r" option.