UI Testing on the Sly

One of the drawbacks of testing your UI components (you are testing them, right?) is that to properly run a test, it needs to run on a display. Unfortunately, if you want to use that display for something else (like writing code and fixing bugs), you run into problems with either the tests writing your code, or the code you're writing supplying input to your UI tests.

One good solution is to have a dedicated machine for the tests, which you can continually monitor and can easily be updated with changes from your dev machine. Since that isn't always an option, I'll present a few alternatives for several different platforms which allow you to run UI tests without having a separate machine to run them on.

OS X

VNC is a great tool for controlling remote computers of any platform. If you have fast user switching enabled, you can actually use OSXvnc to display the desktop for a user other than the one currently using the display. If you configure OSXvnc to not quit when switching to another user, you will always be able to connect to the desktop where OSXvnc was first launched.

Set up a "test" user, and set up OSXvnc to launch when that user logs in. Set up the "Startup" tab like this:

Once OSXvnc is running, you can access the "test" desktop while using your normal desktop. Run your tests in the background, peek in on them to check the results.

Linux

Run vncserver, either manually or as a daemon. This will give you an additional X display which is totally separate from your physical one. Connect to the new display with vncviewer, and you have a window onto your "test machine".

Windows

Windows isn't quite as functional when it comes to sharing a display, but you can still get some mileage out of UI testing on the same machine. There's a peculiar little mode in windows, I'll call it "service mode", where you think you have a display but you don't. This prevents java.awt.Robot from working properly, but you can still get some mileage from UI tests that don't depend on screen captures or native drag and drop. Abbot (a UI testing library) will work just fine in this mode, and does so by artificially generating events directly into the AWT event queue in order to drive a UI.

The easiest way to run tests in this manner is to install cygwin sshd. It runs as a service, so by default it doesn't have access to the desktop (there is a checkbox in the services admin control panel that indicates whether a service is allowed desktop access). Once that is installed and configured (google "cygwin sshd install"), you can use ssh to get a shell prompt within the service mode "sandbox". If you've got an ant script to run your tests, just invoke it. Abbot automatically detects whether this mode is in effect, so if you're using that library, everything works out of the box. If you only run your tests interactively through an IDE, you're out of luck, since you won't be able to interact with the IDE if you launch it in service mode.

Bugs in the Library

One of the great benefits of Java is the availability of a wide variety of libraries providing all sorts of functionality. However, managing bugs, workarounds, and fixes becomes more complicated when tracking multiple third-party libraries.

With a simple source code base or a single application, it's relatively simple to take library updates to get bug fixes, or even to apply patches to a local copy of a library. There are many situations where this isn't possible and you need to determine for a given section of client code whether a bug is present. One codebase may need to support several VM releases and what's a workaround in one may introduce a bug in another.

I keep track of library-based bugs with a class Bugs which comprises static methods which check the environment for the presence of a given bug. For instance, Mac OSX between Java releases 1.4 and 1.5 incorrectly mapped mouse button events generated by java.awt.Robot. So Bugs would have a corresponding method,

// This bug has been submitted to Apple, bug #XXXXXXX
// It was officially fixed in version 1.5
public static boolean hasRobotButtonSwapBug() {
    return Platform.isMac() && Platform.JAVA_VERSION >= 0x1400
        && Platform.JAVA_VERSION < 0x1500;
}

The method contains checks which describe as precisely as possible the environment where the bug is present. In this case, several java releases were tested; the bug first appeared in the Java 1.4 release and persisted until 1.5. The method should narrowly identify only those environments shown to produce the bug (see below).

In the corresponding BugsTest, we implement a method which actually checks for the bug. This serves several purposes. First, it provides a test case to submit along with a bug report to whoever is responsible for the bug. Second, it provides a standard check when used with other automated tests to flag changes in released libraries. Checking the "bug presence" method at the start of the test avoids test failures where we already know the bug exists, and tests for its presence in any environment we haven't yet tested. A system property may be set to force running the test.

public void testRobotButtonSwapBug() throws Exception {
    // skip the test if we already know the bug exists, unless forcing a bug check
    if (!Boolean.getBoolean("check_all_bugs") || hasRobotButtonSwapBug()) return;
    // perform tests to reproduce the bug
    final List events = new ArrayList();
    Robot robot = new java.awt.Robot();
    Window w = new Window();
    w.add(new JLabel("Swap test"));
    w.addMouseListener(new MouseAdapter() {
        public void mouseClicked(MouseEvent e) {
            events.add(e);
        }
    });
    showWindow(w);
    Point where = w.getLocationOnScreen();
    where.x += w.getWidth()/2;
    where.y += h.getWidth()/2;
    robot.mouseMove(where.x, where.y);
    robot.mousePress(InputEvent.BUTTON2_MASK);
    robot.mouseRelease(InputEvent.BUTTON2_MASK);
    robot.waitForIdle();
    assertEquals("Robot generated incorrect button event for button 2",
                 InputEvent.BUTTON2_MASK, 
                 ((MouseEvent)events.get(0)).getModifiers());
    // and so on...
}

With each new library release, the test is run. If it fails with a new version or a new platform, extra checks can be added to the method in Bugs.

The Bugs method test can now be used in client code to determine whether workarounds need to be applied. Such checking allows the client code to be used consistently regardless of the environment in use on the project where it is used. If you release an application that runs on Java 1.2 and above, you may need to selectively apply workarounds depending on the actual JRE in use by a user. In the example above, a workaround to swap button definitions was applied (and still is) if the user happens to be using the affected Java release on OSX.

This pattern makes bug checks and workarounds very explicit. Compare these two implementations. First, the ad hoc workaround:

JTree tree;
...
// tree row size isn't updated automatically, so hack it
tree.setRowSize(tree.getRowSize());
tree.revalidate();
tree.repaint();

Or this, which is only slightly more structured, but entirely clear in intent. You can also tell exactly what code can go away when the bug does:

JTree tree;
...
if (Bugs.hasTreeRowSizeUpdateBug()) {
    tree.setRowSize(tree.getRowSize());
    tree.revalidate();
    tree.repaint();
}

The code explicitly identifies a bug that has been at least partially analyzed, and indicates workaround code. If a programmer needs further information on the bug, she can just look up the corresponding information in Bugs or BugsTest