Performance testing

BAT comes with a test suite which performs benchmark and performance tests. The test suite is run for each release and the results can be found here.

The test suite comprises of several types of tests which each come with a set of sub-tests. So far, only tests for the MCMC are implemented. Each test returns a status which is one of the following:

good the test is successful
acceptable the test is successful but at least one sub-test result deviates from the expectation to a level which is still acceptable
bad the test is not successful, at least one sub-test result deviates from the expectation to a level which is not acceptable
fatal the test is not successful, something is seriously wrong
unknown the outcome of the test is not known
off test is not performed

Each test first summarizes the settings used and then lists the sub-tests with the results and allowed deviations (tolerances). The tests implemented so far are:

Function1D
An analytically known 1-dimensional function is sampled from and the sampled (marginalized) distribution is compared to the function. Several quantities are calculated for the distribution and the function and compared. These are:
Function2D
An analytically known 2-dimensional function is sampled from and the sampled (marginalized) distribution is compared to the function. Several quantities are calculated for the distribution and the function and compared. These are:
Varying parameters
An analytically known 1-dimensional function is sampled from and the sampled (marginalized) distribution is compared to the function. Several quantities are calculated for the distribution and the function and compared. One parameter of the MCMC, e.g., the lag or the number of iterations, is varied.

Results of performance testing