One suggestion, from Andrew Clegg:
Thought I'd put this idea in here for discussion after the lively
questions on Jiten's talk in Manchester just now...
Once functional testing (e.g. via soapUI scripting) is running, let
service owners specify how many instances of the functional test can
be run in parallel.
Then once a day, run that many copies of the test at once, with slight
random start delays, thereby simulating that number of simultaneous
On completion, count how many of the sessions returned successfully,
and display this for each day.
Service owners can indicate how 'enterprisey' their services are by
how many sessions can be safely run.
Users know whether this is a realistic claim or not, by seeing how
many sessions actually completed (on each day and over history).
Owners of small services can indicate that by setting the parallelism
level low (or at 1), but still get judged for reliability *relative to
their own capacity*.
Users also avoid getting unrealistic expectations of small services.
Let the service owner specify the time of day to run the test – so
they can tell their users that the service will be slow at that time.