Qmail log analysis tool

Ive been slaving over this for two days now and Ive finally got it to a level I think could be useful to me. Our servers process tens of thousands of emails per day - not all of them spam, and grep-ing the maillog always causes me headaches, so a nice little visual PHP app should keep me in the manner to which Im accustomed.

Download the maillog analyser tool

All I have to do now is remember to collect my log files every few days from each server and the app will make sure I have a continuous summary of all mail activity through each server. The script runs locally for security, can read gzipped logs and breaks up multi-day files into more manageable 24 hour blocks.

To get this working under windows you need to unzip the files into a directory of your choosing. Inside this directory, you should have a folder called "resources", with a subfolder "mail".

Inside this mail folder are two more folders called "cache" and "mess". (Mess is for the message viewer, where you can drag any queued messages from the server into here and scrutinise the mail headers).

Note: In Linux you will need to CHMOD the cache folder to 777. Its already writeable in windows.

Drag your maillog, maillog.processed and any other logfiles into the local mail folder. In Plesk, you'll find these in /usr/local/psa/var/log/.

Messages that are in the queue can be found in /var/qmail/queue/mess/* - these go into your local mess folder - complete with directory tree (eg: messages are stored within numbered folders in qmail, you can just drag the folders across into "mess").

Now open your browser in the correct place. You should see a couple of drop-down boxes at the top of the page.

The first of these is a "cache generator". You should see a list of files in that correspond to files in your local mail folder that are waiting to be processed, so select the first one listed - the page reloads... nothing... nothing... this could take a while depending on the size of your log files.

If you receive an "Out of memory" error, you must have a monster of a maillog, so you need to tweak a few settings in the first few lines of the application - this is explained in the troubleshooting bit at the bottom of this post.

If all goes well, you should see a couple of notices informing you that cache files have been created. You may need to reload the page to see these in the second drop-down which contains the list of cache files, so click the "refresh" link (Do not click your browsers refresh button or this will reprocess the script!).

If your maillog updates daily, only 1 date will have been entered here followed by a notice that either says "Complete" or "Part". This relates to whether there is a full complement of 24 hours records in this file or not. Select the newly created cache (date) from the list and depending on how many records you are displaying, you should see a page that resembles the pictures above, along with four tabs.

From here, you can figure out what information is recorded in the four tabs. The analysis captures all POP and IMAP logins, SMTP logins and keeps track of all messages as they are either delivered, rejected or minced through spamassassin. It also keeps track of "relaylocks" to help you build firewall rules.

On the POP stats tab, the column abbreviations E, S, K, Q and F relate to Emails received by the system (E), Tagged as Spam (S), OK/Delivered (K), Queued (Q) and Failed (as in delivery failure - F). These stats are against every email address the system knows about.

The table for POP/IMAP logins records the interval between each login. This is fairly smart, its not just an average over the 24 hours as this figure would be skewed by somebody say, turning off their PC for 8 hours, or logging out of webmail. Instead it keeps a running average from first to last login making it a far less blunt instrument. OK still not razor-like but good enough.

The next drop down box is a list of messages in your mess folder, unless its empty of course. Selecting a message will reload the page and display the message headers in a text box. This is most useful for analysing individual messages that have been stuck in the queue for a while.

Download the maillog analyser tool

Notes

This was written with no intention of it becoming a polished tool, it serves a purpose for me but Im sure it has semantic bugs in it - if you notice something thats not calculating correctly, Id like to know about it so I can keep improving it. It comes with no warranty or license, but if you find it useful or have an idea to improve it in some way, let me know. Feel free to contribute any code too.

Troubleshooting

Timeouts

With large files, you may need to adjust some settings. The default values PHP defines are 32Mb and a 60 second timeout, this will just about cope with 150,000 maillog records. This app uses 3 minutes and 128M so should comfortably handle 450,000 records. If you have more you can either reduce SCANLIMIT (this forces the script to break out of the loop before the end) or up your memory_limit to 256M, then try again. To do this, open index.php and find:

ini_set("max_execution_time", 180);
ini_set("memory_limit", "128M");
define("SCANLIMIT", 350000);


and adjust accordingly. eg: 256M doubles the memory limit. 360 doubles the execution time to 6 minutes, or lowering SCANLIMIT forces the script to end prematurely.