Automated testsuite?

[dtzWill]

There are many tests you run manually, but automating these would have two benefits:
a)you don't forget/do it wrong/easy to run full gauntlet
b)can throw it on a server somewhere, and track progress/regressions over time.

Doesn't need to be fancy, but would be useful to have a grip on.

Bonus points if includes performance measurements, as we're lagging behind the Classpath implementation ATM, and it'd be nice to see that fixed.