Apache Benchmarking with Compression

We belatedly noticing that we didn't turn gzip compression on for our Apache web server. After turning it on, we ran the Apache benchmarking tool and didn't notice any improvement. To get the ab tool to accept compression you need to have it pass the correct header as follows:

ab -n 100 -H "Accept-Encoding: gzip, deflate" http://......

It was quite simple. On our local network we didn't see any improvement since everything is fast. From home and on smart phones, you do see a difference.

Sorting Alphabetically in Java

I've recently had to fix alphabetical sorting in two different Java-based web applications and know we have the problem in other places. It's a pain that this comes up all of the time and there's really nothing to prevent it in the future except programmer diligence. The basic problem is that Strings in Java are sorted lexicographically by default (usually using String's compareTo() method). This is almost never what you want shown in your UI, but when you are writing code it's easy to just call Collections.sort() and expect it to do what you want.
Once you know you have this problem, there's a few ways to do the sorting:

• Do a case-insensitive sort on the Strings. This is very common and is basically a left-over pattern for handling this issue that shouldn't be done. You see this in Java code and JDBC/SQL code where a toUpperCase() or toLowerCase() function is called. A nuance of this is that it's technically random whether "de Lucca" or "De Lucca" comes first since when sorting they both look like "de lucca".
• Use one of Java's language-aware sorting methods such as Arrays.sort(array, Collator.getInstance()). This puts lower case letters before upper case, but on a per-letter basis so that capital Z is not before lower case 'a'. See the sorting examples below.
• Use a SQL Function to properly sort if your database supports it. For example, Oracle's NLSSORT() method. Note that you can't rely on SQL's order by clause by itself because it will also sort text values by their ascii code. And to make matters worse...some of our code did use the NLSSORT() function, but it put the results into a Java TreeSet which resorted them by their ascii code :(
• Consistently use the natural ordering of Strings and get your users to start thinking in terms of ascii and unicode.

Here's some simple code to show the differences between the above techniques. The output is in the table below.

import java.text.*;
import java.util.*;

public class coll {
  public static void main(String args[]) {

    String [] a1 = { "def", "daf", "de la Sol", "De la Sol", "de Battista", "DeBatista", "de Lucca", "deBattista", "Deloitte", "Das", "Deep" };

    String [] a2 = { "def", "daf", "de la Sol", "De la Sol", "de Battista", "DeBatista", "de Lucca", "deBattista", "Deloitte", "Das", "Deep" };

    String [] a3 = { "def", "daf", "de la Sol", "De la Sol", "de Battista", "DeBatista", "de Lucca", "deBattista", "Deloitte", "Das", "Deep" };

    Arrays.sort(a1);
    Arrays.sort(a2, Collator.getInstance());
    Arrays.sort(a3, new Comparator<String>() {
          public int compare(String s1, String s2) {
            if (s1 == null) {
              if (s2 == null) return 0;
              return 1;
            } else if (s2 == null) {
              return -1;
            }
            return s1.toUpperCase().compareTo(s2.toUpperCase());
          } 
        });

    for (int i=0; i < a1.length; i++) {
      System.out.printf("%-15s %-15s %-15s\n", a1[i], a2[i], a3[i]);
    }
  }
}

Here's the example output:

Natural Order (by ASCII code)	using a Collator	Case-insensitive Ordering
Das	daf	daf
De la Sol	Das	Das
DeBatista	DeBatista	de Battista
Deep	deBattista	de la Sol
Deloitte	de Battista	De la Sol
daf	Deep	de Lucca
de Battista	def	DeBatista
de Lucca	de la Sol	deBattista
de la Sol	De la Sol	Deep
deBattista	Deloitte	def
def	de Lucca	Deloitte

Fix for Yahoo! Mail Crashing Firefox

This post may not seem like it's related to Software Development, but if you rely on a working browser to get your job done, problems with them or small features in them, can really impact your productivity. Here's a solution the problem I was having.

A few weeks ago, Firefox was frequently crashing on me. The culprit seemed to be either the Yahoo! toolbar or Yahoo! Mail. After searching around, I removed a .dll associated with the Yahoo! toolbar, but that didn't work. I then completely removed the Yahoo! toolbar, but that still didn't work. Finally the folks at Firefox put out a new update and I thought that may address the issue, but no still luck. It was now easy to find friends experiencing the same problem and tons of questions on Yahoo! Anwsers about this.

The problem was clearly related to Yahoo! Mail's web UI now and Firefox would crash whenever I logged out or closed the tab that Yahoo! Mail was in. After living with this for a while, things got worse, my Firefox download dialog stopped showing entries. After searching for a solution to that issue, I found a link on Mozilla's site that recommended removing mimeTypes.rdf file used by Firefox. That didn't solve the problem. I then found another link that recommended deleting Firefox's downloads.rdf file. Finally after exiting Firefox, deleting both of these .rdf files and restarting firefox, I no longer am having issues with Firefox and Yahoo! Mail.

Here's the location of the .rdf files on my machine:

c:/Documents and Settings/username/Application Data/Mozilla/Firefox/Profiles/eak5tu9x.default/downloads.rdf

c:/Documents and Settings/username/Application Data/Mozilla/Firefox/Profiles/eak5tu9x.default/mimeTypes.rdf

I hope this helps someone else out there.

Recursively Adding Files With Subversion

This script is for people who use the command line version of Subversion (SVN). I often use SVN's command line tools in Cygwin rather than in my IDE, eclipse. This is just Bash script that runs the svn stat command, pulls out all of the files whose status line starts with a question mark, and runs the svn add command on them.

You need to be cautious, if you have set up your svn:ignore property or your SVN config file to ingore temporary files this will blindly add them. After that, you can use the svn revert filename command to un-add them.

###
#.Name
#   svnadd
#.What
#   recursively finds all of the new files via the 'svn stat' command
#   and runs the 'svn add' command on each one.
###

case "$1" in
    -t) SVN_CMD="echo svn"
        ;;
    -h)
        echo Usage: $(basename 0) [-t|-h]
        exit 1;
        ;;
esac

: ${SVN_CMD:=svn}

for f in $(svn stat | awk '/^?/ {print $2}')
do
    $SVN_CMD add "$f"
done

Find and Grep with Subversion and Cygwin

Subversion (SVN) creates .svn directories to store information. These get in the way of the Unix/Cywin find command. Here's a Bash shell script that will ignore all of the .svn directories for you.

#!/bin/bash
#
#.Name
#   sfind
#.Title
#   Find Command for Subversion Workspaces
#.Author
#   Don Mitchell (10/2005)
#.What
#   Strips out the .svn directories when using the Unix find command.
#

if [ $# -lt 2 ]
then
    echo Usage: $(basename $0) [path] [expression...]
fi

path="$1"
shift

find "${path}" ! -wholename '*/.svn/*' -a ! -wholename '*~' \
    -a ! -wholename "*/.*.swp" -a "$@"

I put this script in my PATH and then add a few aliases to make it easier to invoke.

alias sf='sfind'

This alias just abbreviates the actual name I gave the shell script. I should have just renamed the shell script. Example: sf . -name "*Foo*.java"

alias sff='sfind . -type f -print0 | xargs -0 grep'

This alias, sff, combines grep with find, grepping for the specified pattern in all files under the current directory. Example: sff SOME_CONSTANT_NAME

alias sfg='sfind . -name "*.java" -print0 | xargs -0 grep'

This alias, does the same thing as my sff alias, but it only greps in Java files and ignores everything else.

• Example: sfg someMethodName


• Example: sfg -l someMethodName # list just the file names containing that mehtod

alias sfn='sfind . -name '

This alias is another quick shortcut, but I rarely use it. For example, sfn "*Dao.java" will list all of the files under the current directory that end in Dao.java.

JavaOne Critique

As always, I'm enjoying the JavaOne conference this year. It's only the second day, but my usual criticism of the presentations holds true. The presenters love to show flashy demos and rarely if ever cover the gory details of how you manage the source code, build your project in a test environment, deploy to production, and manage developers using a mix of IDEs. In particular how do you handle the differences between development, test, and production environments: different databases, hostnames, mail servers, and other external resources that must be configured.

This is a minor point, but I would expect a developer's conference to talk about the development life cycle and environment. These issues get too little coverage.

Supporting multiple browsers

http://browsershots.org/

This is an interesting site I just found out about. I'm not sure how accurate the results are and if you are really paying attention to details in your page styling, then this is too slow to be usable. However, it still gives you a quick idea of how your basic web pages look and might highlight a browser that they look bad in.

You're still better off looking at what browsers your users use and then testing with them directly.

Data Visualization in everyday applications

http://musicovery.com - This is a cool internet radio station. It's flash based, not java, but it's a great example of using data visualization to display data relationships in an area you may not have expected. I like this view of what you're listening to better than what you get in Pandora or Yahoo! Music Engine.

Java Syntax Pickiness

While fixing bugs in some Java code, I've recently come across two things that I consider poor programming syntax. Both are addressed by well known refactorings. The first one is basic programming style. The second is new and addresses a common Java trick (or hack) that is not needed once you move to Java 5 and beyond. It's an example of people not doing things the new way just because they are so use to the old way and like to copy-n-paste things.

Fail-Fast If Statements

Fail-fast is the idea that you you check error or return conditions first, get it out of the way, and then continue with the main thrust of the method. If done correctly, it makes the code more readable and avoids unnecessary indentation. Here's some simplified examples of what I've been running into.

    if (x != null && SOME_CONSTANT.equals(x)) {
        // ... some code here ...
        if (y != null && y.isSomePropertyTrue()) {
            // ... some code here ...
            if (z != null) {
                // ... some code here ...
                return true;
            } else {
                // ... some code here ...
                return false;
            }
        } else {
            return false;
        }
    } else {
        return false;
    }

    return false;

When a method that contains this sort of code is big, you often have to scroll down to see what the else block is doing and thus fully understand the method. It's better if you could understand it as you read it from top to bottom. The simplest way to refactor it, is to take the fail-fast approach and handle the various if conditions quickly and individually.

    if (x == null || ! SOME_CONSTANT.equals(x)) {
        return false;
    }

    if (y == null || ! y.isSomePropertyTrue()) {
        return false;
    }

    if (z != null) {
        // ... some code here ...
        return true;
    }

    // ... some code here ...
    return false;

Static Imports

Before static imports, a lot of folks would create an interface filled with constants. When they needed to use a constant they would implement that interface and thus be able to directly refer to the constants and not scope them to a particular class. For instance:

    String username = properties.getProperty(USER_NAME_PROPERTY);

instead of

    MyConstants.USER_NAME_PROPERTY
    String username = properties.getProperty(MyConstants.USER_NAME_PROPERTY);

It's a minor difference in syntax, but it is more readable. The problem with this approach comes into play when you generate javadocs. By default you will see all of the constants you inherited from the interface in your local class' javadoc page. If you have project-wide constant file with lots of constants, you'll end up seeing these in almost every javadoc page.

Besides this javadoc issue, my real complaint with this is that it is no longer needed. You should use static imports for constants now and stopped this old interface approach.

    import static com.foo.Bar.USER_NAME_PROPERTY;

    ...

        String username = properties.getProperty(USER_NAME_PROPERTY);
    ...

If you have a lot of constants you are using can import them all with a wildcard in the static import.

    import static com.foo.Bar.*;

Free Conference Call Website

http://www.freeconferencecall.com

I just saw this for the first time and used it with a vendor we're starting to talk to. It's a great service. You have to renew your account every 120 days, but it's free and doesn't appear to have limitations. If you're having trouble doing voice calls over IM with a team of developers and your company doesn't provide a conference call service, try it out.

Mailman Mailing Lists and Spam Prevention

We recently had an issue with spam coming to a mailing list that only our application should be sending to. We use Mailman to manage our mailing list and have our own mail server. The mailing list is set up to only allow emails from two email addresses: one an administrative address we use for manual announcements we need to make, the other the name of the email list itself which we use for the automated emails.

The spammers started sending requests into our mail server with one of our allowed email addresses. I looked in the Mailman interface to see if there was some option that would block these spam attempts and didn't see anything that would apply. We were already restricting who could send to the lists and had all other addresses either blocked or moderated. Two obvious solutions were for IT to configure our spam filtering software to block these spam attempts or configure the mail server to not allow emails with our FROM address that originate outside the firewall. For various reasons, our IT group said they could not do either.

The solution we found was to add an "Approved: [password]" header as the first line in the body of the email (followed by one blank line). Mailman will treat this line as an email header and make sure the password matches the administrative password set in the Mailman UI (this is different than the moderator's email address).

Here's the article article we found at python.org describing the solution. The only piece missing was which password to set in the UI.

The password we use is essentially a shared secret, which isn't the most secure thing, so we've externalized it's value into a property file. This allows our Operations team to change the password if it gets compromised without involving development.

One last note, if you previously configured Mailman to allow (unmoderated) emails from your FROM address, you will want to to remove that (in Mailman's Privacy Options -> Sender Filters) or all emails will still be allowed, even if you have no password or an incorrect password.

Subversion (SVN) Configuration and Auto Properties

I've been very happy with Subversion, but one issue we've had with it is setting up its default behavior for an entire group of people. We have developers working on Unix, Windows, and Macs. Most developers use Eclipse for all of their editting, but other editors are used as well. The two crucial things we need set are the svn:mime-type and svn:eol-style properties.

I thought I would share the config file we currently use. We actually keep this file in Subversion and have all of our developers check it out into their home directory, thus overlaying the default config file you get with Subversion.

On Windows XP, this file lives in the C:/Documents and Settings/username/Application Data/Subversion/ directory and is simply called config. There are bunch of things you can set, but if you work with a diverse set of developers you probably have to limit what your standard defaults are. Here's the basic ones we set:

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store .*.swp jtags Thumbs.db

This is a list of file types that will be ignored by the SVN status, add, and other commands. Using this in addition to occasionally setting the svn:ignore property on a directory are very nice ways of cleaning up what you see with the status command and making sure temporary files do not get into your repository. As long as you are adding odd filenames that you would never want in your repository to this list you won't have problems doing this, but if you added a filename or pattern that is needed in one project, then you are stuck and have to remove it from this list.

use-commit-times = yes

This setting preserves the last modified time of files when you check them out of the repository. If you set this to no, then all of the files have the same timestamp of when you checked them out. Knowing which files were most recently touched is a nice convenience when looking through code so we default this to yes.

enable-auto-props = yes

If you want to use automatic properties for various file types, as I've done below, you need to enable it here.

Here's a full listing of our current config file. If you scroll to the bottom you will find our current listing of file types and the automatic properties that we set for them. One place that having automatic properties set correctly helps is with Subversion's HTTP sever. If you tell Subversion the mime-type of the file, then its HTTP server will serve up the files with the correct mime-type and your browser will correctly display the file. If you commit HTML pages or images to Subversion, this allows you to see the files displayed correctly when you navigate your repository in a browser.

### This file configures various client-side behaviors.
###
### The commented-out examples below are intended to demonstrate
### how to use this file.

### Section for authentication and authorization customizations.
[auth]
### Set store-passwords to 'no' to avoid storing passwords in the
### auth/ area of your config directory.  It defaults to 'yes'.
### Note that this option only prevents saving of *new* passwords;
### it doesn't invalidate existing passwords.  (To do that, remove
### the cache files by hand as described in the Subversion book.)
# store-passwords = no
### Set store-auth-creds to 'no' to avoid storing any subversion
### credentials in the auth/ area of your config directory.
### It defaults to 'yes'.  Note that this option only prevents
### saving of *new* credentials;  it doesn't invalidate existing
### caches.  (To do that, remove the cache files by hand.)
# store-auth-creds = no

### Section for configuring external helper applications.
[helpers]
### Set editor to the command used to invoke your text editor.
###   This will override the environment variables that Subversion
###   examines by default to find this information ($EDITOR,
###   et al).
# editor-cmd = editor (vi, emacs, notepad, etc.)
### Set diff-cmd to the absolute path of your 'diff' program.
###   This will override the compile-time default, which is to use
###   Subversion's internal diff implementation.
# diff-cmd = diff_program (diff, gdiff, etc.)
### Set diff3-cmd to the absolute path of your 'diff3' program.
###   This will override the compile-time default, which is to use
###   Subversion's internal diff3 implementation.
# diff3-cmd = diff3_program (diff3, gdiff3, etc.)
### Set diff3-has-program-arg to 'true' or 'yes' if your 'diff3'
###   program accepts the '--diff-program' option.
# diff3-has-program-arg = [true | false]

### Section for configuring tunnel agents.
[tunnels]
### Configure svn protocol tunnel schemes here.  By default, only
### the 'ssh' scheme is defined.  You can define other schemes to
### be used with 'svn+scheme://hostname/path' URLs.  A scheme
### definition is simply a command, optionally prefixed by an
### environment variable name which can override the command if it
### is defined.  The command (or environment variable) may contain
### arguments, using standard shell quoting for arguments with
### spaces.  The command will be invoked as:
###     svnserve -t
### (If the URL includes a username, then the hostname will be
### passed to the tunnel agent as @.)  If the
### built-in ssh scheme were not predefined, it could be defined
### as:
# ssh = $SVN_SSH ssh
### If you wanted to define a new 'rsh' scheme, to be used with
### 'svn+rsh:' URLs, you could do so as follows:
# rsh = rsh
### Or, if you wanted to specify a full path and arguments:
# rsh = /path/to/rsh -l myusername
### On Windows, if you are specifying a full path to a command,
### use a forward slash (/) or a paired backslash (\\) as the
### path separator.  A single backslash will be treated as an
### escape for the following character.

### Section for configuring miscelleneous Subversion options.
[miscellany]
### Set global-ignores to a set of whitespace-delimited globs
### which Subversion will ignore in its 'status' output.
global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store .*.swp jtags Thumbs.db ~$*
### Set log-encoding to the default encoding for log messages
# log-encoding = latin1
### Set use-commit-times to make checkout/update/switch/revert
### put last-committed timestamps on every file touched.
use-commit-times = yes
### Set no-unlock to prevent 'svn commit' from automatically
### releasing locks on files.
# no-unlock = yes
### Set enable-auto-props to 'yes' to enable automatic properties
### for 'svn add' and 'svn import', it defaults to 'no'.
### Automatic properties are defined in the section 'auto-props'.
enable-auto-props = yes

### Section for configuring automatic properties.
[auto-props]
### The format of the entries is:
###   file-name-pattern = propname[=value][;propname[=value]...]
### The file-name-pattern can contain wildcards (such as '*' and
### '?').  All entries which match will be applied to the file.
### Note that auto-props functionality must be enabled, which
### is typically done by setting the 'enable-auto-props' option.


# *.dsp = svn:eol-style=CRLF
# *.dsw = svn:eol-style=CRLF
*.bat        = svn:eol-style=native;svn-mime-type=text/plain
*.bmp        = svn:mime-type=image/bmp
*.bsh        = svn:eol-style=native;svn:mime-type=text/x-beanshell
*.c          = svn:eol-style=native
*.cpp        = svn:eol-style=native
*.css        = svn:eol-style=native;svn:mime-type=text/css
*.csv        = svn:eol-style=native;svn:mime-type=text/plain
*.doc        = svn:mime-type=application/msword
*.dtd        = svn:eol-style=native;svn:mime-type=application/xml-dtd
*.gif        = svn:mime-type=image/gif
*.h          = svn:eol-style=native
*.htm        = svn:eol-style=native;svn:mime-type=text/html
*.html       = svn:eol-style=native;svn:mime-type=text/html
*.ico        = svn:mime-type=image/x-icon
*.java       = svn:eol-style=native
*.jnlp       = svn:eol-style=native;svn:mime-type=application/x-java-jnlp-file
*.jpeg       = svn:mime-type=image/jpeg
*.jpg        = svn:mime-type=image/jpeg
*.js         = svn:eol-style=native;svn:mime-type=text/plain
*.js         = svn:eol-style=native;svn:mime-type=text/x-javascript
*.jsp        = svn:eol-style=native
*.pdf        = svn:mime-type=application/pdf
*.php        = svn:eol-style=native;svn:mime-type=text/x-php
*.pl         = svn:eol-style=native;svn:mime-type=text/x-perl;svn:executable
*.pm         = svn:eol-style=native;svn:mime-type=text/x-perl
*.png        = svn:mime-type=image/png
*.ppt        = svn:mime-type=application/vnd.ms-powerpoint
*.properties = svn:eol-style=native;svn:mime-type=text/plain
*.py         = svn:eol-style=native;svn:mime-type=text/x-python;svn:executable
*.sh         = svn:eol-style=native;svn:mime-type=text/x-sh;svn:executable
*.spec       = svn:eol-style=native;svn:mime-type=text/x-sh
*.sql        = svn:eol-style=native;svn:mime-type=text/plain
*.text       = svn:eol-style=native;svn:mime-type=text/plain
*.tif        = svn:mime-type=image/tiff
*.tiff       = svn:mime-type=image/tiff
*.tld        = svn:eol-style=native;svn:mime-type=text/plain
*.txt        = svn:eol-style=native;svn:mime-type=text/plain
*.xhtml      = svn:eol-style=native;svn:mime-type=text/xhtml+xml
*.xls        = svn:mime-type=application/vnd.ms-excel
*.xml        = svn:eol-style=native;svn:mime-type=text/xml
*.xsd        = svn:eol-style=native;svn:mime-type=text/xml
CHANGES      = svn:eol-style=native;svn:mime-type=text/plain
COPYING      = svn:eol-style=native;svn:mime-type=text/plain
INSTALL      = svn:eol-style=native;svn:mime-type=text/plain
Makefile     = svn:eol-style=native;svn:mime-type=text/plain
README       = svn:eol-style=native;svn:mime-type=text/plain
TODO         = svn:eol-style=native;svn:mime-type=text/plain

Subversion: Inconsistent line ending style

We use Subversion and have people working on Unix, Windows, and Macs. This occasionally causes problems with newline characters. All of us use the same Subversion config file in our home directories so that svn:mime-type and svn:eol-style get set correctly for various file types, but I don't know a way of specifying the svn:eol-style for properties that Subversion lets you set on directories. We use both svn:externals and svn:ignore on our projects. I've just started to integrate FogBugz (the bug tracking system we use) with Subversion and this requires even more Subversion properties.

For the most part everyone uses Eclipse as their IDE and editor. A few people use Vim or Emacs in addition to Eclipse. I've always been able to solve these newline problems before this week. However, this week I Subversion wouldn't let me edit or even list the properties:

$ svn propedit svn:externals .
svn: Inconsistent line ending style

$ svn propget svn:externals .
svn: Inconsistent line ending style

$ svn propget svn:externals .
svn: Inconsistent line ending style

$ svn proplist .
Properties on '.':
bugtraq:label
bugtraq:url
svn: Inconsistent line ending style

Eclipse with its Subclipse plugin was able to show all of the properties. But didn't allow me to correct this so that I could work with the properties on our common Unix test and development servers.

I did find mailing list comments about this and some people recommended changing the Subversion files under the .svn directory. This is slightly tricky because of the format and apparent byte counts in the file.

An easier solution turned out to be the propdel command. I vaguely knew which property was causing the problem and just deleted it. After that the proplist, propget, and propedit commands all worked. To get back the value I just deleted, I used the svn diff -N . command to see the old value and just copy-pasted it back in with the svn propedit command.

I've generally found that working on Windows and with Eclipse and Subclipse, I still need to have the command-line version of Subversion installed to work around issues like this or help people debug issues they run into with Subversion. TortoiseSVN is also helpful some times. But I've found it's impossible to rely solely on Subclipse.

Just fyi...on my machine I have Subclipse, RapidSVN, TortoiseSVN, and the command-line SVN all installed. I rarely use RapidSVN, but occasionally I find useful as a repository browser.