Cygnus have announced an open source Java test suite to provide a free
alternative to Sun's Java Compatibility Kit. The suite is a collaboration between the GNU Classpath project, Cygnus Solutions and Transvirtual and currently contains about 500 tests for Java conformance. Obviously this is nowhere near the 12,000 offered by Sun but it's free and open. If you need a particular test, why not write it yourself and contribute to the project.