Logfile rotation!

 
Post new topic   Reply to topic    Aprelium Forum Index -> Tutorials
View previous topic :: View next topic  
Author Message
MarkusL
-


Joined: 12 Nov 2005
Posts: 12

PostPosted: Tue Nov 22, 2005 12:08 pm    Post subject: Logfile rotation! Reply with quote

If Abyss Web Server is running as a service:

Create a .bat file:

Code:

net stop AbyssWebServer

rem (move the log file in another folder, example: access.22.11.2005.log)
move "X:\web\server\log\access.log" "X:\web\server\logbackup\access.%date%.log"

rem (Abyss start and create a new log file)
net start AbyssWebServer


Use the Scheduled Tasks Wizard to run the batch file at the desired time.

Markus
Back to top View user's profile Send private message
Anonymoose
-


Joined: 09 Sep 2003
Posts: 2192

PostPosted: Wed Nov 23, 2005 10:19 am    Post subject: Reply with quote

Nice...

Does DOS/Windows cmd shell automatically replace the /'s in the date with .'s ? %DATE% normally returns in the form dd/mm/yyyy, which would be illegal in a filename.
_________________

"Invent an idiot proof webserver and they'll invent a better idiot..."
Back to top View user's profile Send private message
aprelium
-


Joined: 22 Mar 2002
Posts: 6800

PostPosted: Wed Nov 23, 2005 12:20 pm    Post subject: Reply with quote

Anonymoose wrote:
Nice...

Does DOS/Windows cmd shell automatically replace the /'s in the date with .'s ? %DATE% normally returns in the form dd/mm/yyyy, which would be illegal in a filename.

The output of %DATE% depends on the locale of the current user. So if you're using some European ones, the date separator is . and not / and that's why MarkusL's batch worked fine for him.
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Send e-mail
Anonymoose
-


Joined: 09 Sep 2003
Posts: 2192

PostPosted: Wed Nov 23, 2005 1:36 pm    Post subject: Reply with quote

Whoops...

Here's a sample of how to workaround for UK users or anyone else whos date is spat out with illegal characters. The extra stuff tagged on at the start strips out the /'s - or more correctly it actually strips out the numbers in the date and rebuilds minus /'s

Code:

for /f "tokens=1-3 delims=/ " %%a in ("%DATE%") do (
set dd=%%a
set mm=%%b
set yyyy=%%c)

net stop AbyssWebServer

rem (move the log file in another folder, example: access.22.11.2005.log)
move "X:\web\server\log\access.log" "X:\web\server\logbackup\access.%dd%%mm%%yyyy%.log"

rem (Abyss start and create a new log file)
net start AbyssWebServer


I believe this will only work in NT4/2K/XP, but that's OK, because Abyss wouldn't be running as a service in 9x :)
_________________

"Invent an idiot proof webserver and they'll invent a better idiot..."
Back to top View user's profile Send private message
wspollack
-


Joined: 28 Dec 2003
Posts: 108

PostPosted: Thu Dec 01, 2005 4:17 pm    Post subject: I wrote a log-switching program Reply with quote

Not to dismiss the elegant use of bat files, but for anyone interested note that I wrote a Windows-based program that does this sort of thing almost two years ago. See "abyss-restarter" at:

http://www.billanddot.com/downloads.htm

As with the bat file, Abyss is down for only a few seconds, i.e., long enough to stop the server, rename the log file, and restart the server. All subsequent work is done with the server running. And as with the bat file, this program is intended to be run by whatever scheduler is built into the O/S or a scheduler of your choice.

The advantages of the program are:

1) It works with Abyss running as a service or not, e.g., on older O/Ss.
2) You specify monthly or daily files, and data is appended, if appropriate, to existing files, or new ones created. For instance, if you want to run this program several times a month (perhaps because you want your log files archived to a different drive, reducing the size on your log-file drive) and you want monthly-based log files, subsequent runs for a given month have data added to an existing file for that month.
3) The daily or monthly archived files are based on data inside the log file, as opposed to the bat-file method (which relies on the date/time the bat file is run).
4) You can optionally remove log lines based on one or more IPs.
5) It backs up the config file, in case anything nasty happens.

Anyway, those are the reasons I wrote it, and several folks are using this (without incident, as far as I know).

Of course, all of this will soon be moot, as I've read here that Aprelium is planning on having log-rotation built in, in a future release.

Regards,

Bill
Back to top View user's profile Send private message Visit poster's website
AJWesley
-


Joined: 13 Jan 2008
Posts: 39
Location: Kentville, Nova Scotia, Canada

PostPosted: Sat Jan 26, 2008 10:44 pm    Post subject: Rotate Abyss web server logs by month in 32-bit Windows? Reply with quote

Rotate Abyss web server logs by month in 32-bit Windows/-???

Assumptions:

The Abyss web server log file is in C:\Program Files\Abyss Web Server\log\access.log.
Windows has a yyyy/mm/dd date format from its Date and Time Control Panel applet.
Windows has a fully functioning Task Scheduler (all related services are running).
Steps:

Create a batch file with the following contents.

for /f "tokens=1-3 delims=/ " %%a in ("%DATE%") do (
set yyyy=%%a
set mm=%%b
set dd=%%c)

rem the order of %%a, %%b and %%c are system dependent! (control-panel, date/time settings)

echo %dd%
echo %mm%
echo %yyyy%

if not exist "ac%yyyy%%mm%.log" (
net stop AbyssWebServer
rem (move the log file in another folder, example: access.22.11.2005.log)
ren "access.log" "ac%yyyy%%mm%.log"
rem (Abyss start and create a new log file)
net start AbyssWebServer
)
pause

Create a multiple time scheduled task (or make it run daily) to run the above batch file/-???
Back to top View user's profile Send private message MSN Messenger
loloyd
-


Joined: 03 Mar 2006
Posts: 435
Location: Philippines

PostPosted: Thu Feb 07, 2008 6:22 pm    Post subject: Reply with quote

AJWesley, that snippet of text looks awfully familiar.

Oh yeah, I wrote it over at http://www.loloyd.com/ (doesn't run on Abyss though, but http://home.loloyd.com/ does!). Of course, I used and modified Anonymoose's batch script. Thanks for bringing it up here.
_________________

http://home.loloyd.com/ is online if the logo graphic at left is showing.
Back to top View user's profile Send private message Visit poster's website
AJWesley
-


Joined: 13 Jan 2008
Posts: 39
Location: Kentville, Nova Scotia, Canada

PostPosted: Mon Feb 11, 2008 8:21 am    Post subject: Reply with quote

loloyd wrote:
AJWesley, that snippet of text looks awfully familiar.

Oh yeah, I wrote it over at http://www.loloyd.com/ (doesn't run on Abyss though, but http://home.loloyd.com/ does!). Of course, I used and modified Anonymoose's batch script. Thanks for bringing it up here.


Exactly! Hence the "?" ... I searched high and low for a solution to a 32mb log file! (I was away from my server for a week... my access.log was so HUGE that I couldn't open it.) I need a solution. =)
Back to top View user's profile Send private message MSN Messenger
puertoblack2003
-


Joined: 08 Oct 2006
Posts: 87

PostPosted: Mon Feb 11, 2008 8:47 pm    Post subject: Reply with quote

i use this baretail program for viewing files on real time and it opens large files. something that wordpad can't do.


http://www.baremetalsoft.com/baretail/index.php

good luck
Back to top View user's profile Send private message
AJWesley
-


Joined: 13 Jan 2008
Posts: 39
Location: Kentville, Nova Scotia, Canada

PostPosted: Tue Feb 12, 2008 1:14 am    Post subject: Abyss web server logs - the 32MB solution? Reply with quote

puertoblack2003 wrote:
i use this baretail program for viewing files on real time and it opens large files. something that wordpad can't do.
http://www.baremetalsoft.com/baretail/index.php
good luck


baremetalsoft.com - what a gem! Is there a method to compress, save, export, and delete original log content, without closing the server?
It's wild what you can find online...

Thanks :)
:: AJ ::
_________________
"A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila..."
Back to top View user's profile Send private message MSN Messenger
loloyd
-


Joined: 03 Mar 2006
Posts: 435
Location: Philippines

PostPosted: Tue Feb 12, 2008 2:35 am    Post subject: Reply with quote

This is a bit off-topic (not about log file rotation), but...

Aprelium showed us the way to Baretail back in 2007 via the thread
Quote:
http://www.aprelium.com/forum/viewtopic.php?t=14252&highlight=baretail


AJWesley wrote:
Is there a method to compress, save, export, and delete original log content, without closing the server?
It would be best to close the server first before doing anything with its logs because a conflict might occur when you're doing a long operation on your huge log file and Abyss would want to add to it simultaneously. If you want to operate on the log file without stopping Abyss, try copying the file first - if you have an advanced enough OS and filesystem combo, it would/should be intelligent enough to queue machine instructions on a file upon copying and appending onto it (I don't have paper references to this though). You can then operate on the copy.
_________________

http://home.loloyd.com/ is online if the logo graphic at left is showing.
Back to top View user's profile Send private message Visit poster's website
AJWesley
-


Joined: 13 Jan 2008
Posts: 39
Location: Kentville, Nova Scotia, Canada

PostPosted: Tue Feb 12, 2008 4:19 am    Post subject: Reply with quote

loloyd wrote:
This is a bit off-topic (not about log file rotation), but...
Aprelium showed us the way to Baretail back in 2007 via the thread http://www.aprelium.com/forum/viewtopic.php?t=14252&highlight=baretail

It would be best to close the server first before doing anything with its logs because a conflict might occur when you're doing a long operation on your huge log file and Abyss would want to add to it simultaneously. If you want to operate on the log file without stopping Abyss, try copying the file first...


Thanks much :) How is log rotation working for you... and could I get an update to the script, or relevant links? (This is a LARGE board. I am trying to read as much as I can before I post... still reading. Combined logs exceed 2gig now...)
_________________
"A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila..."
Back to top View user's profile Send private message MSN Messenger
loloyd
-


Joined: 03 Mar 2006
Posts: 435
Location: Philippines

PostPosted: Tue Feb 12, 2008 5:05 am    Post subject: Reply with quote

It's working quite perfectly well. Let me show you the log directory of my office PC:

Code:
 Volume in drive C is Office_Issue_C
 Volume Serial Number is DCEC-0304

 Directory of C:\Program Files\Abyss Web Server\log

2008/02/04  11:00 AM    <DIR>          .
2008/02/04  11:00 AM    <DIR>          ..
2006/06/08  11:15 AM             9,030 ac200509.log
2006/06/08  11:17 AM            80,345 ac200510.log
2006/06/08  11:17 AM             2,184 ac200602.log
2006/06/08  11:18 AM             6,567 ac200603.log
2006/06/08  11:23 AM               373 ac200604.log
2006/06/08  11:24 AM           517,151 ac200605.log
2006/07/03  11:56 AM         1,161,497 ac200607.log
2006/08/03  09:24 AM           462,931 ac200608.log
2006/08/30  03:43 PM           163,974 ac200609.log
2006/10/04  10:11 AM           174,464 ac200610.log
2006/11/02  10:33 AM           485,027 ac200611.log
2006/11/29  09:50 PM           182,453 ac200612.log
2006/12/28  04:26 PM            24,029 ac200701.log
2007/02/01  09:52 AM         1,737,378 ac200702.log
2007/03/01  10:10 AM         5,377,397 ac200703.log
2007/04/02  10:54 AM         1,126,963 ac200704.log
2007/05/02  09:38 AM           781,976 ac200705.log
2007/06/05  10:34 AM         1,942,858 ac200706.log
2007/07/02  10:22 AM           753,581 ac200707.log
2007/07/27  02:40 PM           372,552 ac200708.log
2007/08/31  02:17 PM           166,253 ac200709.log
2007/09/25  03:35 PM           766,176 ac200710.log
2007/11/05  10:48 AM           744,921 ac200711.log
2007/12/03  07:54 AM           952,153 ac200712.log
2008/01/02  10:26 AM           661,607 ac200801.log
2008/02/04  08:50 AM         1,042,394 ac200802.log
2008/02/12  10:03 AM           157,676 access.log
2007/10/11  10:54 AM            15,965 cgi.log
2005/09/23  10:12 AM                 0 isapi.log
2006/07/03  12:16 PM         1,090,656 report.txt
2006/07/03  12:17 PM         1,082,060 report4.txt
2006/07/03  11:07 AM               478 rotate.bat
              32 File(s)     22,043,069 bytes

I am just working on a glitch that happened just this month. Otherwise, it's all good.

What update on the script do you need? Or maybe you wanted to divide your 2 Gig log file into smaller pieces?
_________________

http://home.loloyd.com/ is online if the logo graphic at left is showing.
Back to top View user's profile Send private message Visit poster's website
AJWesley
-


Joined: 13 Jan 2008
Posts: 39
Location: Kentville, Nova Scotia, Canada

PostPosted: Tue Feb 12, 2008 3:01 pm    Post subject: Reply with quote

loloyd wrote:
It's working quite perfectly well... I am just working on a glitch that happened just this month. Otherwise, it's all good.

What update on the script do you need? Or maybe you wanted to divide your 2 Gig log file into smaller pieces?


Nice! Yes... something similar to logrotate but for windows. If I could archive to a specific drive or folder... an option to delete/wipe the original... split larger files (my head is tuned to server log maintenance for linux servers - but not so swift when it comes to windows...)
Thanks
_________________
"A computer lets you make more mistakes faster than any invention in human history - with the possible exceptions of handguns and tequila..."
Back to top View user's profile Send private message MSN Messenger
loloyd
-


Joined: 03 Mar 2006
Posts: 435
Location: Philippines

PostPosted: Wed Feb 13, 2008 2:43 am    Post subject: Reply with quote

You can use file splitters, such as File Split Stream or HJSplit to split that huge log file of yours. You can Google them or go here for the 40kb File Split Stream:

http://shareware.pcmag.com/product.php%5Bid%5D68209%5Bcid%5D112%5BSiteID%5Dpcmag

It would not be as good as having it rotated by the date-format of your choice, and it would split the file in less than desirable areas (in the middle of lines); but, ultimately, it will make your log file operations more manageable nonetheless.

Another idea is to dump the grep of your huge log file onto smaller files. Windows XP has the DOS command "find" somewhat emulating the grep directive in Linux. Or you can use BareMetal's BareGrep to export the results to various files. This requires manual operations though, and not much like the automations through scripting provided by logrotate.

Then there's also the Windows emulation for grep over at http://unxutils.sourceforge.net/.

Whatever route you choose, good luck.
_________________

http://home.loloyd.com/ is online if the logo graphic at left is showing.
Back to top View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Aprelium Forum Index -> Tutorials All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB phpBB Group