Cleaning up log files on Unix/Linux

We run many different tomcat instances on a Unix box and need to keep the log files from getting to big and filling up our filesystem. Typically we have logging turned up high since we need the information for debugging.

Instead of using a cron job, we decide to create a Hudson job that periodically loops through the tomcat log directories and removes any old log files. Here's the shell command we periodically run:

/usr/bin/find /opt/*/*/logs/* -type f -mtime +2 | grep -v cruise | grep -v /opt/tools/confluence | xargs --no-run-if-empty rm -v

Note: /opt/tools/confluence is a sym link so we exclude it since it should already be cleaned up via it's real directory name. "cruise" is a directory that I don't want touched by this so I use grep -v to exclude it.

This is a development server so we don't need log files over 2 days old (not modified in over 2 days).  We also compress files that are over a day old to save space.  You can then use zgrep to grep through compressed files.  The compression is done with this shell command:

/usr/bin/find . -type f -mtime +1 | grep -v cruise | grep -v /opt/tools/confluence |xargs --no-run-if-empty /bin/gzip -v

Note 2:  I use "rm -v" and "gzip -v" so that the hudson console output includes verbose output about what was removed or compressed.

Confluence children and pagetree macros breaking

One of my side tasks is to provide some technical support for our Confluence wiki instance. We recently had a space stop working. The symptom was that the children and pagetree macros would never load. Because of this it made it impossible to view the space pages in tree view mode and move pages out of there to find the problem page.

The problem seems to have boiled down to a page somehow getting created without a page name. From Confluence's knowledge base: http://confluence.atlassian.com/display/CONFKB/Accessing+a+Space+via+Webdav+Returns+a+Blank+Page

Resolution

You need to identify the page title(s) that is null.

Run the following database query:

SELECT CONTENTID
FROM SPACES s, CONTENT c
WHERE s.SPACEID = c.SPACEID AND s.SPACEKEY = 'YourSpaceKey' AND c.CONTENTTYPE = 'PAGE' AND c.TITLE IS NULL;

Once identified, you can delete the page from the database or update it via these queries:

update content set title = 'title' where contentid = ;

Make sure that the titles are unique to the space.

Spring WAR configuration for Development, QA, and Production

Configuring applications with Spring

This is an excellent article on how to configure a Spring web application. The main point that I would like to echo is that it's important to have your build procedure create the same exact artifact for each of your environments and just externalize the configuration where necessary. This is a point often missed by developers and operations staff.

Tips on Evaluating UI Wireframes

I received this while working with a design firm, Rolling Orange. It's an excellent concise description of how to react to an initial set of wireframes. People often skimp on this step and that's a big problem that will come back to haunt you. This description makes that clear and more importantly gives people a guide to interpret and analyze the wireframes.

Elaboration Phase: Concept & Design: Wireframes

ABOUT THIS MILESTONE
This is the first draft of the wireframes for the new site. This represents our initial recommendations for navigation, functionality, and content, based on the findings of our Research and Strategy phase. The purpose of these wireframes is to create a site structure, functionality, and messaging platform that will serve as the foundation of your site. We will address visual design ("look and feel") only after we have approved wireframes.

WHAT SHOULD I EVALUATE?
- Look at this phase as an opportunity‚ now is the time to make it right.
- This draft is a starting point, so it's flexible and we expect it will evolve. Now is the time to suggest changes and explore new solutions.
- Does the architecture reflect the recommendations made in the Research & Strategy phase?
- Does the navigation and architecture address the functional needs of the audience per the Persona document?
- Is the proposed design "buildable" based on your understanding of supporting technologies and systems?
- Can your organization sufficiently manage and sustain the proposed design?
- Did we miss anything?

WHAT'S NEXT?
We will refine these wireframes based on your feedback.

PROJECT TIP
Think like your customer! Think about user goals and their frame of mind (“I’m researching, I’m applying, I’m managing”). Pretend you don’t know any of this—would you understand it as a first-time reader? Think simple—include just what users need and no more. If it’s “nice to have,” cut it.

TCP Window Scaling with Solaris and Linux

We've been having an annoying problem in which we can not insert large blobs to our Oracle 11g database. The solution was found by one of our security engineers. We needed to turn the tcp window scaling off. To temporarily test this out you can run this command as root:

bash-3.00# ndd -set /dev/tcp tcp_wscale_always 0

The original value for this tunable option was '1'. Interesting, running this command on the Linux (Oracle 11g) side had no effect. Running it on the Solaris (Tomcat Application Server) side solved the problem.

Fyi...to view the current setting of this property, you can run:

$ /usr/sbin/ndd -get /dev/tcp tcp_wscale_always

A value of 0 is off. A value of 1 is on. I'm able to run this command as myself. I don't need to be root.

Cygwin 1.7 Upgrade

I'm a big fan of Cygwin and have used it for years. Without it, it's possible for me to be productive on Windows. The latest version of Cygwin is 1.7. This version changed a few things for.

The first was one they warned about, how Cygwin stores mount points. These are no longer stored in the Windows registry, but are instead read from /etc/fstab. Cygwin comes with a script who's purpose appears to be to create this file for you, but when I ran it nothing happened. So I was left creating my own /etc/fstab. Here's what mine looks like:

C:/cygwin/bin /bin ntfs binary,user 0 0
C:/cygwin/etc /etc ntfs binary,user 0 0
C:/cygwin/home /home ntfs binary,user 0 0
C:/cygwin/var /var ntfs binary,user 0 0
C: /c ntfs binary,user 0 0
C:/Documents\040and\040Settings/don/Application\040Data /n ntfs binary,user 0 0
C:/Documents\040and\040Settings/don/My\040Documents /m ntfs binary,user 0 0
W: /w ntfs binary,user 0 0
Y: /y webdrive binary,user 0

The second issue I ran into was that I didn't have /bin or /usr/bin in my PATH. The previous versions of bash/cygwin did this for me. I actually had a line in my .profile commented out that would have done this. I simply put it back in and was good to go.

So far these are the only issues I've had with the upgrade and they were relatively easy to fix.

Calculating size of Oracle Database

select decode(least(sum(bytes), 1000000000),
1000000000, ROUND(sum(bytes)/1000000000) || 'GB',
ROUND(sum(bytes)/1000000) || 'MB') as database_size
from dba_segments;

The above query has the nice effect of displaying the size of the database in a human readable format.

DATABASE_SIZE
------------------------------------------
151GB

1 rows selected

Hudson v1.347 Kills Background Processes :(

I recently upgraded out Hudson instance after not touching it for quite a while. This quickly broke the set up we have in which Hudson stops and starts our Tomcat servers after a successful build to deploy the new code. The problem seems a change in the Hudson code that kills any background processes that a Hudson job leaves around. There's documentation about this issue at:

http://issues.hudson-ci.org/browse/HUDSON-2729
http://hudson.gotdns.com/wiki/display/HUDSON/Spawning+processes+from+build

We use the following shell script to stop and start Tomcat servers. The key part of the script is the setting of the environmental variable BUILD_ID=dontKillMe which overrides the default environmental variable setting Hudson gave the variable and thus prevents it from killing the background processes.

#!/bin/bash
#.Name
# startTomcat.sh
#.What
# Starts a tomcat container. This script handles some nuances of Hudson
# in which it kills background processes that you want to have stay alive
# after a build job (e.g. Restart Tomcat) finishes.
#
#.See
# http://issues.hudson-ci.org/browse/HUDSON-2729
# http://hudson.gotdns.com/wiki/display/HUDSON/Spawning+processes+from+build

exec 2>&1;

BUILD_ID=dontKillMe; export BUILD_ID;

if [ $# -ne 1 ]
then
echo "Usage: $0 full_path_to_tomcat_directory"
exit 1
fi

if [ ! -d ${1} ]
then
echo "Not a directory: ${1}"
exit 2
fi

echo $0 starting $(date)
set -x

pwd

cd $1

if [ $? -ne 0 ]
then
echo "Directory not found or has wrong permissions: $1"
exit 3
fi

pwd

./tomcat.sh stop

sleep 5

./tomcat.sh start >> ${1}/logs/start.log 2>&1

echo $0 completed $(date)

exit 0

Oracle SQL Schema and Text Index Size

I recently had to determine the size of my schema and the size of the text indexes used in that schema. These SQL statements got me the results:

SQL> select sum(bytes) from dba_segments where owner='SCHEMA_NAME' and SEGMENT_NAME like '%$%';

SUM(BYTES)
----------
6663307264

SQL> select sum(bytes) from dba_segments where owner='SCHEMA_NAME';

SUM(BYTES)
----------
9464446976

Oracle SQL Developer and Oracle Text

In the SQL Editor in Oracle's excellent SQL Developer tool, if you are running queries using an Oracle Text Index, you will run into issues when using the CONTAINS operator and searching for terms such as ${MySearchTerm}. The dollar-bracket syntax is used to do stemmng, but the SQL editor interprets them as bind variables and prompts you for their value. To turn off the bind variable interpretation you can precede the SQL statement with the following command:

set define off;

You only need to do this once and it will remain in effect for the rest of your usage of that SQL worksheet.

Unix Math Calculations

I just ran across http://x-bc.sourceforge.net/index.html. If you've ever used the Unix bc calculator utility, this project provides two excellent extensions to the built in capabilities of the calculator. In particular:

http://x-bc.sourceforge.net/extensions_bc.html

http://x-bc.sourceforge.net/scientific_constants_bc.html

You'll get a wealth of math functions available to you by simply including the extension files avaiable at those URLs.

In order to have these extensions always loaded as well turn on the built in math extensions, you can alias your bc command as follows:

alias bc="bc -l ~/bin/*.bc"

Apache Tomcat DBCP Connection Pooling

I've been looking at some performance issues with our Tomcat servers (we have many) and have seen some real inconsistency in the DBCP options that are specified. In general the basic ones you'd expect to see are there: a validation query, minIdle, maxActive. However, other ones are missing. This page http://commons.apache.org/dbcp/configuration.html lists the options and gives a good explanation of each. Everyone using Tomcat and DBCP should review this list and understand them.

A couple of interesting options to point out are those dealing with the validity of idle connections in the pool:

testWhileIdle (default false)

The indication of whether objects will be validated by
the idle object evictor (if any). If an object fails to
validate, it will be dropped from the pool. NOTE - for
a true value to have any effect, the validationQuery
parameter must be set to a non-null string.

timeBetweenEvictionRunsMillis (default -1)

The number of milliseconds to sleep between runs of the
idle object evictor thread. When non-positive, no idle
object evictor thread will be run.

numTestsPerEvictionRun (default 3)

The number of objects to examine during each run of the
idle object evictor thread (if any).

minEvictableIdleTimeMillis (default 1000 * 60 * 30)

The minimum amount of time an object may sit idle in
the pool before it is eliga

A good blog post on Apache DBCP can be found on Roy's Musings under Gotchas with DBCP.

For a comparison of Apache DBCP with another connection pool see Vigil Bose's Blog.

Also check out BoneCP for a connection pool specifically written to be fast.

Change a text cell into a Hyperlink in Excel

I recently imported a CSV file I was given into Excel. The file had two columns (with many rows of data) that contained URLs in the format http://foo.com/bar?phuid=1023, etc. I wanted to make these links clickable. Excel provides a straight forward, but manual way to do this. I asked around for an automated way to do this and was quickly given this macro:

Sub URL_List()
   For Each cell In Selection
       If cell.Value <> "" Then
           If Left(cell.Value, 7) = "http://" Then
               URL = cell.Value
           Else
               URL = "http://" + cell.Value
           End If
           ActiveSheet.Hyperlinks.Add Anchor:=cell, _
             Address:=URL, TextToDisplay:=cell.Value
       End If
   Next cell
End Sub

I don't know the source of this macro it proved useful today. I hope you can make some use of it.

The directions on creating a macro in Excel can be found in Excel's help which I'll repeat here:

Create a macro using Microsoft Visual Basic

1. On the Tools menu in Microsoft Excel, point to Macro, and then click Visual Basic Editor.


2. On the Insert menu, click Module.


3. Type or copy your code into the code window of the module.


4. If you want to run the macro (macro: An action or a set of actions that you can use to automate tasks. Macros are recorded in the Visual Basic for Applications programming language.) from the module window, press F5.


5. When you're finished writing your macro, click Close and Return to Microsoft Excel on the File menu.