Unix system monitoring with SAMS

You can access a live demo here

I ) General overview

SAMS is a set of scripts allowing a SysAdmin to monitor performance of Unix servers. It consists of 2 main tasks :
- Collects performance data from servers. As of version 1.0, linux, SCO and Solaris clients are available.
- Records performance data in rrdtool db and Produce graphs. This is the main engine. It has been coded on Linux RedHat 9 but should run on all flavour of linux.

Here is an simplified view of SAMS architecture :

Features list :

CPU utilisation : %usr, %sys, %wio and %idle

Number of connected users

%buzy for disks

disk space used per filesystem in % (alerts can be sent)

number of memory page swapped and free

swap in and swap out information

number of network packets in and out

I.1 ) Collecting data

Every single server that you would like to monitor will have to run a SAMS client. This client uses system commands and sar information to produce a text file locally. This text file will be collected by the SAMS server at a later time. Here is the list of the clients :

for Linux : getdata_linux.bash

for SCO : getdata_sco.ksh

for Solaris : getdata_sol.ksh

Every 5mns, SAMS server will collect the data file from its clients and starts processing the data.
All these information are obtained through standard Unix commands : vmstat, sar, df ... To be able to monitor closely the servers, it is suggested to configure sar to run every 5mns (for more information, please report to section II of this page). IMPORTANT : SAMS server uses the command "rcp" to copy the data files from its clients to its local directory, so you need to configure your servers to allow rcp command between them.
All the performance information are stored in RRD databases. You have to installed this software prior to the installation of SAMS. You will find everything about RRD by clicking here.

I.2 ) Graph generation

The script generating the graphs is producegraph.ksh. It reads SAMS configuration file to know wich graph to produce. For each graph category (memoire, disques, CPU, users...) it produces 4 graphs : last 24 hours, last 7 days, last month and last year.

II ) SAMS configuration

To be able to work, SAMS needs a certain configuration that is done manually. First, you need to configure the servers which will be monitored, then you need to configure SAMS server.

II.1 ) Configuring the servers to monitor

You need to configure sar on the server you want to monitor. Insert the following 2 lines in user sys's cron :
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/lib/sa/sa1
5 18 * * 0-6 /usr/lib/sa/sa2 -s 08:00 -e 18:01 -i 1200 -A
These lines will execute sar every 5 mns.

Then you need to create a user (we will use the user samsuser in this exemple).
The last thing to do is to activate the client. For example, if you would like to monitor a SUN Solaris server, you will have to copy the script getdata_sol.ksh (from sams/clients directory) to the home directory of the samsuser and activate it in the cron of samsuser will the following line :
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /users/nagios/getdata_sol.ksh > /dev/null
NOTE:/users/nagios needs to be replace to whatever the home directory of samsuser is.

II.2 ) SAMS configuration

We can now configure SAMS on the monitoring server. We begin with the creation of the user samsuser. Then decompress SAMS archive in /usr/local directory.
You should see the sams/conf directory. Change directory to it and edit sams.conf.
This is the main configuration file. It is comprised of section. Each section begin with the keyword servername= and each following line correspond to what you would like to monitor. Let's look at the exemple below :
servername=server01:SUN
server01:REMOTEDATA:/users/nagios/data.collected
server01:NET:hme0
server01:FS:/:90:md2 server01:FS:/disk1:85:md8 server01:FS:/db:90:md11 server01:FS:/c4:80:md13 server01:FS:/backup:80:md17 server01:FS:/tmp:95:st11
This section tells SAMS that we want to monitor server01. Its OS is SUN Solaris. Please, note that :SUN is for future use. The following line tells SAMS where to collect the data file from. This is usefull when you have a lot of different servers with home directory located at different places.
The next line tells SAMS which network interface you would like to monitor. As of SAMS version 1.0, only one network interface can be monitored.
The following lines describe several file systems to be monitored. Let's look at the first line. It's comprised of 5 fields. Each field on this line is separated with ":". The first field indicates the server name. The second one indicates that it is a file system (FS). SAMS can support up to 9 file systems per server. The third field is the name of the filesystem.
The fourth field is the threshold for the file system utilization. If you want to be warned if the file system is over 80% of its capacity, indicates 80 in this field. The last field is the device name as reported by sar -d.

Creates as many servers as you want. In theory there is no limit. I use it with 5 servers.

III ) database initialisation

We need to initialize the databases before using them. We go to /usr/local/sams/bin and execute the script : create_rrd_db.ksh. IMPORTANT : edit this file and change the first few environment variables to match your configuration :
DBPATH=/usr/local/sams/db
RRDTOOL=/usr/local/rrdtool-1.0.46
If rrdtool is not installed in the above directory (RRDTOOL=) you have to edit every single script and change them.
If everything is OK, create_rrd_db.ksh will create all necessary databases with the information you provided in sams.conf.

IV ) Web page creation

Although this is not required, producehtml.ksh can generate automatically all html pages necessary to easily view all the graphs. This script creates all necessary web pages in /usr/local/sams/html. Each server will have a daily_.html. For example daily_server01.html.
To view these web pages, you will need to configure your web server. As a example, the following lines should be added to the httpd.conf file of an Apache server :
Alias /perf/ "/usr/local/sams/html/"

Options None
#AllowOverride AuthConfig
Order allow,deny
Allow from all

NOTE : Replace "/perf/" with whatever you need. To access your monitoring pages, open your favorite browser and go to : http://my_web_server/perf/daily_server01.html
If you have more than one server, I recommand the creation of an index.html file that would be comprised of links to your different server pages.

V ) SAMS Automation

The last thing to do is to activate samsuser's cron. When you uncompressed the archive, you should have a file named cronlist in sams/conf. It is already in cron format. So to activate, tape in "crontab cronlist".

As you can see, the installation is rather long and extremely manual, but the result is worth the effort ! You will be able to monitor your servers closely.
If you have comments or need more information, please send me an email : jerome.boismartel@primus.ca.

VI ) Debugging SAMS

If you're having trouble with the installation, or if your graphs are empty, here are several things to check :

Data extraction on the client : The first thing to do is to check if the client is collecting its data. Check if the file data.collected is generated in samsuser's home directory on the client. For example, if you monitor a sun server, log into it, and go to /home/samsuser to check if data.collected is there.

Data collection from sams server : To see if you get all the data from the servers you added in sams.conf, go to sams/bin and manually run : ./getdata.ksh -d This calls getdata in debug mode. You will see a lot of output.

If everything is fine, you can run ./producegraph.ksh -d

VII ) Downloading SAMS

The latest version of SAMS is 1.0. This package is provided to you freely. You can use it or modify it the way you want.
If you decide to use SAMS, you do that on your own risks and I cannot be responsible for your use of SAMS package or any damages it may cause.
Click here to download SAMS.