# Testing code changes

## Automated tests

The Bio-Formats testing framework component contains the infrastructure to run automated tests against reference data. These tests check the metadata and binary data of every dataset using INI-based configuration files as the source of truth.

All representative files submitted as part of the bug report or development projects and used as supporting data for new Bio-Formats releases are stored in a curated QA repository, also known as the full data repository, managed by the Bio-Formats maintainers alongside the associated configuration files. As part of the review process, every change against Bio-Formats is tested nightly against the entire data repository and all tests must be passing in order for a contribution to get integrated.

It is also possible for contributors to generate their configuration files and run the automated tests against their own data. Assuming the data is stored under `/path/to/data`, after checking out source code and building all the JAR files (see Obtaining and building Bio-Formats), switch to the `test-suite` component and run the ant `gen-config` target to generate the configuration files under `/path/to/config`:

    $ cd components/test-suite
    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config gen-config

To run the automated tests using existing configuration files, use the ant `test-automated` target:

    $ cd components/test-suite
    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config test-automated

Multiple options can be passed to the ant `test-automated` target by setting the `testng.${option}` option via the command line. Useful options are described below.

testng.directory

  Mandatory option. Specifies the root of the data directory to be tested:

      $ ant -Dtestng.directory=/path/to/data/dataset1 -Dtestng.configDirectory=/path/to/config/dataset1 test-automated

  On Windows, the arguments to the test command must be quoted:

      > ant "-Dtestng.directory=C:\\path\to\data\dataset1" "-Dtestng.configDirectory=C:\\path\to\config\dataset1" test-automated

testng.configDirectory

  Mandatory option. Specifies the root of the directory containing the configuration files. This directory must have the same hierarchy as the one specified by `testng.directory` and contain `.bioformats` configuration files:

      $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config test-automated

testng.configSuffix

  Specifies an optional suffix for the configuration files:

      $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.configSuffix=win test-automated

testng.memory

  Specifies the amount of memory to be allocated to the JVM:

      $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.memory=4g test-automated

  Default: 512m.

testng.threadCount

  Specifies the number of threads to use for testing:

      $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.threadCount=4 test-automated

  Default: 1.

You should now see output similar to this:

    Buildfile: build.xml

    init-title:
         [echo] ----------=========== bio-formats-testing-framework ===========----------
    ...
    test-automated:
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [${logback.configurationFile}]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/opt/ome/bioformats/components/test-suite/logback.xml]
       [testng] 17:05:28,835 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
       [testng] 17:05:28,837 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
       [testng] 17:05:28,876 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.sift.SiftingAppender]
       [testng] 17:05:28,878 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SIFT]
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [loci.tests.testng] to DEBUG
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SIFT] to Logger[ROOT]
       [testng] 17:05:28,892 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[loci.tests.testng]
       [testng] 17:05:28,892 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
       [testng] 17:05:28,894 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@706a04ae - Registering current configuration as safe fallback point
       [testng] [2015-08-18 17:05:28,904] [main] testng.directory = /ome/data_repo/test_per_commit/
       [testng] 17:05:28,908 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [loci.tests.testng.TimestampedLogFileAppender]
       [testng] 17:05:28,909 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [logfile-main]
       [testng] 17:05:28,955 |-INFO in loci.tests.testng.TimestampedLogFileAppender[logfile-main] - File property is set to [target/bio-formats-test-main-2015-08-18_17-05-28.log]
       [testng] [2015-08-18 17:05:28,963] [main] testng.multiplier = 1.0
       [testng] [2015-08-18 17:05:28,964] [main] testng.in-memory = false
       [testng] [2015-08-18 17:05:28,964] [main] user.language = en
       [testng] [2015-08-18 17:05:28,964] [main] user.country = US
       [testng] [2015-08-18 17:05:28,964] [main] Maximum heap size = 455 MB
       [testng] Scanning for files...
       [testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
       [testng] [2015-08-18 17:05:32,258] [main] Total files: 480
       [testng] [2015-08-18 17:05:32,258] [main] Scan time: 3.293 s (6 ms/file)
       [testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
       [testng] Building list of tests...

and then eventually:

       [testng] ===============================================
       [testng] Bio-Formats software test suite
       [testng] Total tests run: 19110, Failures: 0, Skips: 0
       [testng] ===============================================
       [testng]

    BUILD SUCCESSFUL
    Total time: 16 minutes 42 seconds

In most cases, test failures should be logged in the main console output as:

    [testng] [2015-08-18 17:13:13,625] [pool-1-thread-1]     SizeZ: FAILED (Series 0 (expected 2, actual 1))

To identify the file, look for the initialization line preceding the test failures under the same thread:

    [testng] [2015-08-18 17:13:12,376] [pool-1-thread-1] Initializing /ome/data_repo/test_per_commit/ome-tiff/img_bk_20110701.ome.tif:

The console output is also recorded under `components/test-suite/target` as `bio-formats-software-test-main-$DATE.log` where “$DATE” is the date on which the tests started in “yyyy-MM-dd_hh-mm-ss” format. The detailed report of each thread is recorded under `bio-formats-software-pool-$POOL-thread-$THREAD-main-$DATE.log`

## MATLAB tests

Tests for the Bio-Formats MATLAB toolbox are written using the xunit framework and are located under https://github.com/ome/bioformats/tree/develop/components/formats-gpl/test/matlab.

To run these tests, you will need to download or clone matlab-xunit, a xUnit framework with JUnit-compatible XML output. Then add this package together with the Bio-Formats MATLAB to your MATLAB path:

    % Add the matlab-xunit toolbox to the MATLAB path
    addpath('/path/to/matlab-xunit');
    % Add the Bio-Formats MATLAB source to the MATLAB path
    % For developers working against the source code
    addpath('/path/to/bioformats/components/formats-gpl/matlab');
    addpath('/path/to/bioformats/artifacts');
    % For developers working against a built artifact, e.g. a release
    % addpath('/path/to/bfmatlab');

You can run all the MATLAB tests using runxunit:

    cd /path/to/bioformats/components/formats-gpl/test/matlab
    runxunit

Individual test classes can be run by passing the name of the class:

    cd /path/to/bioformats/components/formats-gpl/test/matlab
    runxunit TestBfsave

Individual test methods can be run by passing the name of the class and the name of the method:

    cd /path/to/bioformats/components/formats-gpl/test/matlab
    runxunit TestBfsave:testLZW

Finally, to output the test results under XML format, you can use the `-xmlfile` option:

    cd /path/to/bioformats/components/formats-gpl/test/matlab
    runxunit -xmlfile test-output.xml