Guidelines for Test Case Development

We need more testing. We appreciate all people who help by developing tests. We like their gracious contribution to be as effective as possible. There have been problems with tests in the past, and these guidelines intend to help improve this.

As ticket #5 shows, code restructuring can invalidate test cases. This is a very painful thing because the restructuring hasn't actually damaged any remainaing code but has only slightly changed the way things should be set up.

If the tests had been conservative in their assumptions, they probably would still work. Test cases which fail ungracefully in these cases make development and change a painful experience. We want to avoid that. For that reason we should hold test cases to higher standards.

• Most test cases should be exemplary for how people should use the API

• There is room for the whacky test case that tests how easy it is to screw up the system, but these do not set good examples, and they are hard to maintain on evolving systems. Either way, tests should still use the API properly even if they try something funny.

• Make no dependence to file system layout or operating system, never ever. All files should be taken from the classpath as ClassLoader?.getResourceAsStream(), not by some funny path assumptions. As a last resort, make a system property and figure out some way for ant to set these properties in a reliable fashion.

• Do not use internals. All classes ending in Impl or in impl packages are out of bounds. Instead, user the proper factories for that. Yes some of our code uses impl stuff directly, but that is a known issue to be fixed. Test cases that use internals are an antithesis for agile development.

• Make tests fail gracefully! If there is anything at all that might fail, make it fail the test case instead of bringing down the whole test run. If stuff fails hard, it blocks other tests from being run. The tests that run first aren't necessarily the most important ones. Tests that don't run are problems left undetected.

Many of the present tests violate one or more of these guidelines. They need to be fixed.