Testing code changes
====================

Automated tests
---------------

The :source:`Bio-Formats testing framework <components/test-suite>` component
contains the infrastructure to run automated tests against reference data.
These tests check the metadata and binary data of every dataset using
INI-based configuration files as the source of truth.

All representative files submitted as part of the :doc:`bug report </about/bug-reporting>`
or development projects and used as supporting data for new Bio-Formats releases
are stored in a curated QA repository, also known as the full data repository,
managed by the Bio-Formats maintainers alongside the associated configuration
files. As part of the review process, every change against Bio-Formats is
tested nightly against the entire data repository and all tests must be
passing in order for a contribution to get integrated.

It is also possible for contributors to generate their configuration files and
run the automated tests against their own data. Assuming the data is stored
under :file:`/path/to/data`, after checking out source code and building all
the JAR files (see :doc:`building-bioformats`), switch to the :file:`test-suite`
component and run the :program:`ant` ``gen-config`` target to generate the
configuration files under :file:`/path/to/config`::

  $ cd components/test-suite
  $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config gen-config


To run the automated tests using existing configuration files, use the
:program:`ant` ``test-automated`` target::

  $ cd components/test-suite
  $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config test-automated

Multiple options can be passed to the :program:`ant` ``test-automated`` target 
by setting the ``testng.${option}`` option via the command line. Useful options
are described below.

testng.directory
  Mandatory option. Specifies the root of the data directory to be tested::

    $ ant -Dtestng.directory=/path/to/data/dataset1 -Dtestng.configDirectory=/path/to/config/dataset1 test-automated

  On Windows, the arguments to the test command must be quoted::

    > ant "-Dtestng.directory=C:\\path\to\data\dataset1" "-Dtestng.configDirectory=C:\\path\to\config\dataset1" test-automated

testng.configDirectory
  Mandatory option. Specifies the root of the directory containing the configuration files.
  This directory must have the same hierarchy as the one specified by
  ``testng.directory`` and contain :file:`.bioformats` configuration
  files::

    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config test-automated

testng.configSuffix
  Specifies an optional suffix for the configuration files::

    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.configSuffix=win test-automated

testng.memory
  Specifies the amount of memory to be allocated to the |JVM|::

    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.memory=4g test-automated

  Default: 512m.

testng.threadCount
  Specifies the number of threads to use for testing::

    $ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.threadCount=4 test-automated

  Default: 1.

You should now see output similar to this::

    Buildfile: build.xml

    init-title:
         [echo] ----------=========== bio-formats-testing-framework ===========----------
    ...
    test-automated:
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [${logback.configurationFile}]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
       [testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/opt/ome/bioformats/components/test-suite/logback.xml]
       [testng] 17:05:28,835 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
       [testng] 17:05:28,837 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
       [testng] 17:05:28,876 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.sift.SiftingAppender]
       [testng] 17:05:28,878 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SIFT]
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [loci.tests.testng] to DEBUG
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
       [testng] 17:05:28,891 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SIFT] to Logger[ROOT]
       [testng] 17:05:28,892 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[loci.tests.testng]
       [testng] 17:05:28,892 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
       [testng] 17:05:28,894 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@706a04ae - Registering current configuration as safe fallback point
       [testng] [2015-08-18 17:05:28,904] [main] testng.directory = /ome/data_repo/test_per_commit/
       [testng] 17:05:28,908 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [loci.tests.testng.TimestampedLogFileAppender]
       [testng] 17:05:28,909 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [logfile-main]
       [testng] 17:05:28,955 |-INFO in loci.tests.testng.TimestampedLogFileAppender[logfile-main] - File property is set to [target/bio-formats-test-main-2015-08-18_17-05-28.log]
       [testng] [2015-08-18 17:05:28,963] [main] testng.multiplier = 1.0
       [testng] [2015-08-18 17:05:28,964] [main] testng.in-memory = false
       [testng] [2015-08-18 17:05:28,964] [main] user.language = en
       [testng] [2015-08-18 17:05:28,964] [main] user.country = US
       [testng] [2015-08-18 17:05:28,964] [main] Maximum heap size = 455 MB
       [testng] Scanning for files...
       [testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
       [testng] [2015-08-18 17:05:32,258] [main] Total files: 480
       [testng] [2015-08-18 17:05:32,258] [main] Scan time: 3.293 s (6 ms/file)
       [testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
       [testng] Building list of tests...

and then eventually::

       [testng] ===============================================
       [testng] Bio-Formats software test suite
       [testng] Total tests run: 19110, Failures: 0, Skips: 0
       [testng] ===============================================
       [testng]

    BUILD SUCCESSFUL
    Total time: 16 minutes 42 seconds

In most cases, test failures  should be logged in the main console output as::

       [testng] [2015-08-18 17:13:13,625] [pool-1-thread-1]     SizeZ: FAILED (Series 0 (expected 2, actual 1))

To identify the file, look for the initialization line preceding the test
failures under the same thread::

   [testng] [2015-08-18 17:13:12,376] [pool-1-thread-1] Initializing /ome/data_repo/test_per_commit/ome-tiff/img_bk_20110701.ome.tif: 

The console output is also recorded under :file:`components/test-suite/target` 
as :file:`bio-formats-software-test-main-$DATE.log` where "$DATE" is the date 
on which the tests started in "yyyy-MM-dd_hh-mm-ss" format. The detailed report
of each thread is recorded under
:file:`bio-formats-software-pool-$POOL-thread-$THREAD-main-$DATE.log`


MATLAB tests
------------
.. _matlab-xunit: https://github.com/psexton/matlab-xunit

Tests for the Bio-Formats MATLAB toolbox are written using the xunit framework
and are located under :sourcedir:`components/formats-gpl/test/matlab`.

To run these tests, you will need to download or clone `matlab-xunit`_, a
xUnit framework with JUnit-compatible XML output. Then add this package
together with the Bio-Formats MATLAB to your MATLAB path:

.. code-block:: matlab

  % Add the matlab-xunit toolbox to the MATLAB path
  addpath('/path/to/matlab-xunit');
  % Add the Bio-Formats MATLAB source to the MATLAB path
  % For developers working against the source code
  addpath('/path/to/bioformats/components/formats-gpl/matlab');
  addpath('/path/to/bioformats/artifacts');
  % For developers working against a built artifact, e.g. a release
  % addpath('/path/to/bfmatlab');

You can run all the MATLAB tests using :command:`runxunit`:

.. code-block:: matlab

  cd /path/to/bioformats/components/formats-gpl/test/matlab
  runxunit

Individual test classes can be run by passing the name of the class:

.. code-block:: matlab

  cd /path/to/bioformats/components/formats-gpl/test/matlab
  runxunit TestBfsave

Individual test methods can be run by passing the name of the class and the
name of the method:

.. code-block:: matlab

  cd /path/to/bioformats/components/formats-gpl/test/matlab
  runxunit TestBfsave:testLZW

Finally, to output the test results under XML format, you can use the ``-xmlfile`` option:

.. code-block:: matlab

  cd /path/to/bioformats/components/formats-gpl/test/matlab
  runxunit -xmlfile test-output.xml
