Although every care has been taken to provide accurate information, I do not warrant the accuracy or completeness of the information included in this website. Any use, or reliance on, such information is entirely at user's own risk.

Combining and Sorting Chronologically Apache Log Files

Thanks to: Royans K Tharakan (rkt@pobox.com)

There are 2 reasons why I need these scripts...

I have 2 webservers behind my firewall and the primary Apache webserver (which receives all requests for port 80) proxies for the IIS webserver that only has an internal IP address (*See Below). I therefore am able to log requests for the IIS server using the Apache logs.
I like having separate logs for every domain that I host. This lets me run analysis for just one domain (using AWStats, Webalizer, Relax, etc.). This script will allow me to combine all traffic and analyze the logs combined.

Script: /usr/local/bin/combinelogs.sh		Additional Information: This script merges all the logs into one file but doesn't sort the content.
S C R I P T	cd /var/log/apache/ /usr/local/bin/mergelogs.pl edward.alfert.com-access_log sysadmin.info-access_log cubalibre.org-access_log truelove.net-access_log prvacation.info-access_log > allmywebsites-access_log

Script: /root/mergelogs.pl		Additional Information: Many of the log analysis programs that I use (AWStats, Webalizer, etc.) require that the log entries be in chronological order. This is required in order to track individual visitor sessions. This script sorts the logs entries chronologically.
S C R I P T	#!/usr/local/bin/perl # # By Royans K Tharakan (rkt@pobox.com) # Date: July 16 2001 # Updated: Aug 21 2001 # Version: $Id: mergelogs.pl,v 1.2 2001/10/29 20:28:10 control Exp $ # # Usage: mergelogs.pl log1 log2 log3 log4....... #################################################################3 use FileHandle; $\|=1; $ignore[$#ignore+1]="10.1.10.13"; $ignore[$#ignore+1]="10.1.10.14"; $ignore[$#ignore+1]="10.1.10.15"; $ignore[$#ignore+1]="10.1.10.16"; $ignore[$#ignore+1]="10.32."; $ignore[$#ignore+1]="10.1.10"; @files=@ARGV; #print "@ARGV $#ARGV $#files -------"; #print $#files; for $i (0..$#files) { # print "checking file $files[$i] $i\n"; $DATAIN[$i]=IO::File->new("<$files[$i]")\|\|die "Can't open $files[$i]"; } $dataoutfile="$files[$#files]"; #open(STDOUT,">$dataoutfile")\|\|die "Cant send data"; $stop=0; &get_time_stamps(); while ($stop == 0) { #for $tt (keys %time) {print "COMPARE: $tt $time{$tt} \n ";} # print "\n"; @time1=sort {$time{$a} cmp $time{$b}} keys %time; #print "@time1\n"; # for $tt (@time1) {print "$tt $time{$tt}=";} # print "\n"; if ($time{$time1[0]} != 93104863665) { addlog($time1[0],$time{$time1[1]}); } else { $stop=1; } } for $i (0..$#files-1) { $DATAIN[$i]->close(); } #close(STDOUT); sub addlog() { my ($FD,$timestamp)=@_; $check_time=0; # print "=== $FD $timestamp , $time{$FD} === \n"; while (($timestamp>=$time{$FD})&&($time{$FD}!=93104863665)) { # print "=== $FD $time{$FD} $timestamp === \n"; print STDOUT "$datain[$FD]"; my $td="--"; while($td eq "--") { $td=readline($DATAIN[$FD]); #print "$#ignore @ignore \n"; for $ignore_i (0..$#ignore) { if ($td=~/$ignore[$ignore_i]/) {$td="--";} } #print "+1 $td $FD \n"; } $datain[$FD]=$td; #print "+2 $td $FD \n"; if ($datain[$FD]==0) { $time{$FD}=93104863665; } else { $time{$FD}=&gettime($datain[$FD]); #print "==== 5 \n"; } } #print "==== 6 \n"; } sub get_time_stamps() { for $i (0..$#files) { my $td="--"; while($td eq "--") { $td=readline($DATAIN[$FD]); for $ignore_i (0..$#ignore) { if ($td=~/$ignore[$ignore_i]/) {$td="--";} } #print "+3 $td $FD \n"; } $input=$td; $time{$i}=&gettime($input); $datain[$i]=$input; #print ":::===== $i $#files ====== ",$time{$i}," $datain[$i]\n"; } } sub gettime() { my $i="@_"; # print "=========== $i ===========\n"; $i=~m/(\S+) - - \[(.?):(.?)\] \"\S+ (\S+) .?\" (\d+) (\S+)(?: \"(.?)\" \"(.?)\")?/ ; #print "=============== $1 $2 $3==============\n"; my $date = $2; my $time = $3; if($date=~m/(\d+)\/(\w+)\/(\d+)/i) { $day = $1; $month = &month2num(lc($2)); # print "$month = $2\n"; $year = $3; } if($time=~m/(\d+):(\d+):(\d+)./) { $hour = $1; $min = $2; $sec = $3; } $total=$year31536000+$month2592000+$day86400+$hour3600+$min*60+$sec; #print "4 ===\n"; return $total; } sub month2num{ my $mon = shift; $mon =~ s%Jan%0%io; $mon =~ s%Feb%1%io; $mon =~ s%Mar%2%io; $mon =~ s%Apr%3%io; $mon =~ s%May%4%io; $mon =~ s%Jun%5%io; $mon =~ s%Jul%6%io; $mon =~ s%Aug%7%io; $mon =~ s%Sep%8%io; $mon =~ s%Oct%9%io; $mon =~ s%Nov%10%io; $mon =~ s%Dec%11%io; return $mon; }

Config File: /etc/apache/httpd.conf

Additional Information:

Here is a snip from the Apache config file to show you how I use Apache to proxy requests to the internal IIS webserver. This is necessary because I only have one static IP address on the Internet and therefore only have one port 80 that I can redirect to one webserver.

Since I have a website (cubalibre.org) that I wrote long ago using ASP, I have to keep an IIS server on my network. I know, I can buy ChiliSoft ASP, but I don't have thousands of dollars to spend. I'm glad I'm now using PHP for all present and future websites I design, but I don't feel like redoing my ASP code.

By having the Apache webserver as the frontline webserver, this has the additional benefit that the IIS server is not exposed to the Internet and is protected from CodeRed and any future malicious attacks that target IIS.

C
O
N
F
I
G