Cleaning up log files on Unix/Linux

We run many different tomcat instances on a Unix box and need to keep the log files from getting to big and filling up our filesystem. Typically we have logging turned up high since we need the information for debugging.

Instead of using a cron job, we decide to create a Hudson job that periodically loops through the tomcat log directories and removes any old log files. Here's the shell command we periodically run:

/usr/bin/find /opt/*/*/logs/* -type f -mtime +2 | grep -v cruise | grep -v /opt/tools/confluence | xargs --no-run-if-empty rm -v

Note: /opt/tools/confluence is a sym link so we exclude it since it should already be cleaned up via it's real directory name. "cruise" is a directory that I don't want touched by this so I use grep -v to exclude it.

This is a development server so we don't need log files over 2 days old (not modified in over 2 days).  We also compress files that are over a day old to save space.  You can then use zgrep to grep through compressed files.  The compression is done with this shell command:

/usr/bin/find . -type f -mtime +1 | grep -v cruise | grep -v /opt/tools/confluence |xargs --no-run-if-empty /bin/gzip -v

Note 2:  I use "rm -v" and "gzip -v" so that the hudson console output includes verbose output about what was removed or compressed.

Hudson v1.347 Kills Background Processes :(

I recently upgraded out Hudson instance after not touching it for quite a while. This quickly broke the set up we have in which Hudson stops and starts our Tomcat servers after a successful build to deploy the new code. The problem seems a change in the Hudson code that kills any background processes that a Hudson job leaves around. There's documentation about this issue at:

http://issues.hudson-ci.org/browse/HUDSON-2729
http://hudson.gotdns.com/wiki/display/HUDSON/Spawning+processes+from+build

We use the following shell script to stop and start Tomcat servers. The key part of the script is the setting of the environmental variable BUILD_ID=dontKillMe which overrides the default environmental variable setting Hudson gave the variable and thus prevents it from killing the background processes.

#!/bin/bash
#.Name
# startTomcat.sh
#.What
# Starts a tomcat container. This script handles some nuances of Hudson
# in which it kills background processes that you want to have stay alive
# after a build job (e.g. Restart Tomcat) finishes.
#
#.See
# http://issues.hudson-ci.org/browse/HUDSON-2729
# http://hudson.gotdns.com/wiki/display/HUDSON/Spawning+processes+from+build

exec 2>&1;

BUILD_ID=dontKillMe; export BUILD_ID;

if [ $# -ne 1 ]
then
echo "Usage: $0 full_path_to_tomcat_directory"
exit 1
fi

if [ ! -d ${1} ]
then
echo "Not a directory: ${1}"
exit 2
fi

echo $0 starting $(date)
set -x

pwd

cd $1

if [ $? -ne 0 ]
then
echo "Directory not found or has wrong permissions: $1"
exit 3
fi

pwd

./tomcat.sh stop

sleep 5

./tomcat.sh start >> ${1}/logs/start.log 2>&1

echo $0 completed $(date)

exit 0