System Monitoring Tutorial for Newbies.
System Monitoring Tutorial for Newbies.
Dear AO. This tutorial is a brief introduction on how to monitor your linux systems and to let various logfiles make sense to you. I want to introduce 3 kinds of monitoring.
1 – How to protocol different system messages and how to configure the Syslog Daemon.
2 & 3– Knowing some of the different Protocol files on your system.
4 – Programs used to monitor disk space.
Please forgive any bad English or weird typos that i might make :D
1 – The Syslog Daemon.
The syslog daemon is used by many other system services, to log many different events. The advantage of this daemon is that there is only one file which holds the protocols/logs, making it easier to maintain an overview.
The Syslog Daemon takes system messages from the different services and logs them depending on its configuration file which can be found in /etc/syslog.conf
# /etc/syslog.conf – Configuration file for syslogd.(8)
# For more info about the format of this file, see “man syslog.conf”.
# print most on tty10 and on the xconsole pipe
# enable this, if you want that root is informed
# immediately, e.g. Of logins
The file /etc/syslog.conf has a rule for each line. Every rule is split up into 2 fields which are seperated either with a space or by pressing the TAB button. The first field contains the category, which is seperated by decimal points, asigning it that way into a certain priority. The second field is what will happen with the system message once it appears. The category lists the subsystem, which provides the system message.
Every application which you want syslog to log for you will be categorized into one or more of the following categories:
authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, local0 to local7.
The definitions of the categories are as follows:
authpriv – is used by all services that have to do with system security or system authorization. All messages coming from PAM uses this categorie. The ssh daemon still uses the category “auth”, which actually should in combination with the category “security”, be triggered by “authpriv”.
cron – takes the messages from the “cron” and the “at” daemons.
daemon – is used by different other daemons, which dont have its own category (eg. Th ppp-daemon).
kern – is its own category for kernel messages.
lpr – for printer messages.
mail – is the category used only by the mail systems. This is important since many system messages can show up.
news – is the category used by the news systems. Same like in the mail systems, many messages can occur within short time.
syslog – Internal messages generated for the syslog daemon.
user – General category for system messages from the users. eg. The program “login” uses this category to record all failed logins (and successful logins if “login” is configured to do so).
uucp – are the messages from the uucp-systems.
local0 to local7 – There are 8 categories available, which can each be used for a single application. This way you can maintain different applications individually within the /etc/syslog.conf file.
The priority defines the importance of each system message. They are as follows listed from the smallest priority to the highest priority:
debug – should really only be used for error analysis, since all other priorities are much higher.
info – is the priority used to get information only.
notice – is for system messages which you should pay attention to.
warning – is for messages which occur if anything outside the normal happens.
err – This priority shows any system errors that might occur.
crit – Shows any critical system messages.
alert – If this shows up in your logfiles, then a system administrator's intervention is required, in order to keep the system running.
emerg – Well, if you get this message, then its normally to late. This means that the system is not usable.
Like i mentioned earlier, the second field is used to how to let the system know what to do once an error message occurs.
A - Specification of a file:
With the “-” character infront of the filename, you can specify that the file is NOT synchronized after every system message entry.
B – Specification of a device file for tty:
All the wanted system messages are to be displayed to a tty of your choice. The default for suse linux is tty10.
C – Specification of a userlist:
All logged in users get an entry inside the tty specified above (tty10 in this case). (This does not work on all terminal types.)
D – Specification of a hostname labelled with the @ character:
System messages will be forwarded to the specified hostname's syslog daemon, and logged there depending on its configuration.
E - *
All logged in users get a message using “wall”:
The rules which you setup are always valid for the listed prioritym and all other higher priorities:
A – By adding the “=” character in front of the priority the rule is set only to this priority.
B – If you use the “!” character in front of the priority, then this and all higher priorities are excluded from the logging.
C – If you use the “*” character, then its for ALL categories and priorities. If you wish to assign them all, but exclude one or the other, then use the priority “none”.
In addition you can specify start parameters for the syslog daemon by activating the “SYSLOGD_PARAMS” inside the /etc/rc.config file. eg. If you want your syslog daemon to accept logs from remote hosts, you would have to activate the syslogd_params with the parameter -r.
Below is the example for the /etc/rc.config:
# if not empty: parameters for syslogd
# for example SYSLOGD_PARAMS=”-r -s my.dom.ain”
2 – Important Logfiles.
The log files with the most entries is the /var/log/messages. Here you will find almost any messages regarding your system. (Its wise to not let the firewall log's entries to be posted in here, so its easier to have a clean overview of your system messages.) Inside you /var/log/messages you will see the word “--MARK--”. These entries are from your syslog daemon. If they are missing inside the messages logfile, then it means that the syslog daemon is not running or is not working correctly. The easiest way to view the logfiles is by using the command “tail”.
instronics:~# tail /var/log/messages
With tail -10 it will show you the last 10 lines within your /var/log/messages. Any number you specify will show you the amount of entries inside the logfile. To see it in real time, use the -f option.
instronics:~# tail -f /var/log/messages
You can also combine it to show you for example the last 20 lines AND also real time monitoring.eg:
instronics:~# tail -20 -f /var/log/messages
The -f means that it will show you the file in interactive mode. To leave the interactive mode press CTRL + C.
(Isn't linux shell a wonderfull thing ;) .)
The messages from the mail systems are listed in /var/log/mail, for the news services, its in /var/log/news. The boot messages are inside /var/log/boot.msg Since the boot messages are displayed in a rather weird format, its advisable to view them using “dmesg”. Another important logfile is /var/log/wtmp. This file is in binary, and can be viewed using the “last” command. The syslog daemon posts all the entries from these various logfiles into one tty in real time.
3 – Size of the Logfiles.
The size of the logfiles are guarded automatically. The script /etc/cron.daily/aaa_base_rotate_logs is executed daily by the cron daemon. It gets its information from /etc/logfiles, thats were the logfiles and logfile sizes are set. (You can define how big you wish your logfiles to become.) Once the logfile has reached its maximum size, its automatically compressed with gzip and a new empty logfile is created. The way the new empty file is created can also be set inside /etc/logfiles.
Below is a small copy paste of my /etc/logfiles (modified for my security reasons).
# This file tells cron.daily, which log files have to be watched
# File max size mode ownership service reload if changed.
/var/log/mgetty.* +1024k 644 root.root
/var/log/messages +2048k 644 root.root
/var/log/vgetty.* +1024k 644 root.root
/var/log/isdn.log +1024k 644 root.root
/var/log/myfirewall +2048k 644 instronics.instronics
Not only the size is being watched, but also the age of the logfile. For this the script /etc/cron.daily/aaa_base_rotate_logs reads the variable MAX_DAYS_FOR_LOG_FILES from the /etc/rc.config file. If the entry there is greater than 0, then all logfiles (also the older compressed ones) will be checked against their age. All files which are older than the ones set in MAX_DAYS_FOR_LOG_FILES will be deleted automatically. (Be sure to keep a backup of them before they get deleted.) Some files must be restarted/rehashed when re-created. The files which require this are also set inside the /etc/logfiles on the right hand side named “Service reload if changed”. The administrator can also set inside the MAX_DAYS_FOR_LOG_FILES how long the logfiles are to be kept before automatic delete takes place.
4 – Monitoring Disk Space.
To monitor the amount of disk space used you can use the command “df” (disk free). This command is valid for any drive that is mounted. It shows how much space is used and how much space is still available. With the -h option, the df shows its entries in human readable form.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda7 6433796 1903224 4530572 30% /
/dev/hda5 23302 4680 17419 22% /boot
/dev/hda1 2100360 1737904 362456 83% /windows/C
shmfs 63272 0 63272 0% /dev/shm
Pretty confusing if your not aware of your 101 on bits, bytes, kilobytes, megabytes etc.... So now try the human readable version.
instronics:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda7 6.2G 1.9G 4.4G 30% /
/dev/hda5 23M 4.6M 18M 22% /boot
/dev/hda1 2.1G 1.7G 354M 83% /windows/C
shmfs 62M 0 62M 0% /dev/shm
If you would like to know the size of single files or folders, then nuse the command “du'(disk usage). That will show you the exact size of every file or folder within the path where you enter that command. Again here you can use the -h option to make it readable for humans. There are some more very interesting options that can be used in combination with the du command. Do a man du for more info on them. An example of du is:
instronics:~#> cd Desktop
So in human form it would look like this:
Another helpful program in general is called xlogmaster. Im not sure on which linux distributions it can be found, its on my SuSE Linux. I cannot describe how to use it, since i uses shell scripting todo all that for me. But in any case its worth a look at. Always remember, that logfiles are VITAL to a healthy system and sometimes they are your only help in finding out information if you have been compromised or if your system has problems. Also do not forget to read your logfiles regularly. Read them!
I hope this has been of help for some of your newbies out there. Good luck.