libjava/classpath/doc/testing.framework.text

Guile Testing Framework for GNU Classpath
Written by Paul Fisher (rao@gnu.org)

GNU Classpath tests are written in Java.  Guile is responsible for
executing the tests and organizing the results.  Guile and Java
communicate through JNI.  If JNI is unavailable, see the section on
modifying the framework to allow for an alternate means of
communication. [This has not been written. -PF]

All tests must implement gnu.test.Test.  gnu.test.Test contains two
methods:

1. String getName()
2. Result test()

When getName() is called, your test should return the name of the
test.  When test() is called, your test should be performed.  Upon
completion of the test (either through success or failure), a Result
object is returned.  test() may throw runtime exceptions and errors --
if this happens, an implicit error result is returned.

There are seven predefined result types, including the POSIX 1003.3
result codes.  All result objects may optionally be constructed with a
single String argument specifying additional information about the
result.

gnu.test.Pass        : Test passed and was excepted to pass.
gnu.test.XPass       : Test passed but was expected to fail.
gnu.test.Fail        : Test failed but was expected to pass.
gnu.test.XFail       : Test failed and was expected to fail.
gnu.test.Unresolved  : Test produced indeterminate results.
gnu.test.Untested    : Test was not run -- a placeholder.
gnu.test.Unsupported : Test does not have the required support to run.

(Error is also a valid result type, but only in Guile/JNI code.)

System.out and System.err are used for directing additional
information about the running test.  System.out should be used to send
status messages when tests are expected to take large amounts of time.
System.err should be used to send messages which are logged to the
verbose log.

Example test:

import gnu.test.*;
public class StringCharAtZeroTest implements Test
{
  public getName() {
    return "java.lang.String.charAt(0)";
  }

  public Result test() {
    char ch = "foobar".charAt(0);
    if (ch == 'f')
      return new Pass();
    else
      return new Fail("zero index of \"foobar\" is '" + ch + "'");
  }
}

It's often desirable to group multiple tests together into one file.
In this case, inner classes should be used.  There's also the added
benefit that tests can easily share data through static variables in
the parent class.

For example:

import gnu.test.*;
public class TestContainer {
  public static class test1 implements Test {
     String getName() {
       return "test1";
     }
     Result test() {
       // test1 ...
     }
  }
  public static class test2 implements Test {
    String getName() {
      return "test2";
    }
    Result test() {
      // test2 ...
    }
  }
}

The testsuite contains a file known as "tests.to.run" which contains a
newline delimited listing of all tests to be executed.  Just add the
name of the new test to the file and it'll be included in future runs
of the testsuite.

Running the testsuite:
guile-jvm -s test.scm tests.to.run

(It would be more natural for the testsuite to read from standard in
if a file was not specified, but read-line in Guile 1.3a is broken.)

Classes are located via the environmental variable CLASSPATH.

Results are sent to two log files -- one summary (classpath.sum) and
one verbose (classpath.log).