Writing a new file format reader
================================

This document is a brief guide to writing new Bio-Formats file format readers.

All format readers should extend either :javadoc:`loci.formats.FormatReader
<loci/formats/FormatReader.html>` or :javadoc:`an existing reader <loci/formats/in/package-summary.html>`.

Methods to override
-------------------

- :javadoc:`isSingleFile(java.lang.String) <loci/formats/IFormatReader.html#isSingleFile-java.lang.String->`
  Whether or not the named file is expected to be the only file in the
  dataset.  This only needs to be overridden for formats whose datasets can
  contain more than one file.

- :javadoc:`isThisType(loci.common.RandomAccessInputStream) <loci/formats/IFormatReader.html#isThisType-loci.common.RandomAccessInputStream->`
  Check the first few bytes of a file to determine if the file can be read by
  this reader.  You can assume that index 0 in the stream corresponds to
  the index 0 in the file.  Return true if the file can be read; false if
  not (or if there is no way of checking).

- :javadoc:`fileGroupOption(java.lang.String) <loci/formats/IFormatReader.html#fileGroupOption-java.lang.String->`
  Returns an indication of whether or not the files in a multi-file dataset
  can be handled individually.  The return value should be one of the
  following:

  * :javadoc:`FormatTools.MUST_GROUP <loci/formats/FormatTools.html#MUST_GROUP>`:
    the files cannot be handled separately
  * :javadoc:`FormatTools.CAN_GROUP <loci/formats/FormatTools.html#CAN_GROUP>`:
    the files may be handled separately or as a single unit
  * :javadoc:`FormatTools.CANNOT_GROUP <loci/formats/FormatTools.html#CANNOT_GROUP>`:
    the files must be handled separately

  This method only needs to be overridden for formats whose datasets can
  contain more than one file.

- :javadoc:`getSeriesUsedFiles(boolean) <loci/formats/IFormatReader.html#getSeriesUsedFiles-boolean->`
  You only need to override this if your format uses multiple files in a
  single dataset.  This method should return a list of all files
  associated with the given file name and the current series (i.e. every file
  needed to display the current series).  If the ``noPixels`` flag is set, then
  none of the files returned should contain pixel data.
  For an example of how this works, see
  :bfreader:`loci.formats.in.PerkinElmerReader <PerkinElmerReader.java>`. It
  is recommended that the first line of this method be
  ``FormatTools.assertId(currentId, true, 1)`` - this ensures that the file
  name is non-null.

- :javadoc:`openBytes(int, byte[], int, int, int, int) <loci/formats/IFormatReader.html#openBytes-int-byte:A-int-int-int-int->`
  Returns a byte array containing the pixel data for a specified subimage
  from the given file.  The dimensions of the subimage (upper left X
  coordinate, upper left Y coordinate, width, and height) are specified in the
  final four int parameters.  This should throw a
  :javadoc:`FormatException <loci/formats/FormatException.html>` if the image
  number is invalid (less than 0 or >= the number of images).  The ordering of
  the array returned by openBytes should correspond to the values returned by
  :javadoc:`isLittleEndian <loci/formats/IFormatReader.html#isLittleEndian-->`
  and
  :javadoc:`isInterleaved <loci/formats/IFormatReader.html#isInterleaved-->`.
  Also, the length of the byte array should be [image width * image height *
  bytes per pixel].  Extra bytes will generally be truncated. It is recommended
  that the first line of this method be
  ``FormatTools.checkPlaneParameters(this, no, buf.length, x, y, w, h)`` -
  this ensures that all of the parameters are valid.

- :javadoc:`initFile(java.lang.String) <loci/formats/FormatReader.html#initFile-java.lang.String->`
  The majority of the file parsing logic should be placed in this method.  The
  idea is to call this method once (and only once!) when the file is first
  opened.  Generally, you will want to start by calling
  ``super.initFile(String)``.  You will also need to set up the stream for
  reading the file, as well as initializing any dimension information and
  metadata.
  Most of this logic is up to you; however, you should populate the
  :javadoc:`core <loci/formats/FormatReader.html#core>` variable (see
  :javadoc:`loci.formats.CoreMetadata <loci/formats/CoreMetadata.html>`).

  Note that each variable is initialized to 0 or null when
  ``super.initFile(String)`` is called.
  Also, ``super.initFile(String)`` constructs a Hashtable called
  :javadoc:`metadata <loci/formats/FormatReader.html#metadata>`
  where you should store any relevant metadata.

  The most common way to set up the OME-XML metadata for the reader is to
  initialize the MetadataStore using the
  :javadoc:`makeFilterMetadata() <loci/formats/FormatReader.html#makeFilterMetadata-->` method and populate the
  Pixels elements of the metadata store from the ``core`` variable using the
  :javadoc:`MetadataTools.populatePixels(MetadataStore, FormatReader) <loci/formats/MetadataTools.html#populatePixels-loci.formats.meta.MetadataStore-loci.formats.IFormatReader->` method::

    # Initialize the OME-XML metadata from the core variable
    MetadataStore store = makeFilterMetadata();
    MetadataTools.populatePixels(store, this);

  If the reader includes metadata at the plane level, you can initialize the
  Plane elements under the Pixels using
  :javadoc:`MetadataTools.populatePixels(MetadataStore, FormatReader, doPlane) <loci/formats/MetadataTools.html#populatePixels-loci.formats.meta.MetadataStore-loci.formats.IFormatReader-boolean->`::

    MetadataTools.populatePixels(store, this, true);

  Once the metadatastore has been initialized with the core properties,
  additional metadata can be added to it using the setter methods. Note that
  for each of the model components, the ``setObjectID()`` method should be
  called before any of the ``setObjectProperty()`` methods, e.g.::

    # Add an oil immersion objective with achromat
    String objectiveID = MetadataTools.createLSID("Objective", 0, 0);
    store.setObjectiveID(objectiveID, 0, 0);
    store.setObjectiveImmersion(getImmersion("Oil"), 0, 0);

- :javadoc:`close(boolean) <loci/formats/IFormatReader.html#close-boolean->`
  Cleans up any resources used by the reader.  Global variables should be
  reset to their initial state, and any open files or delegate readers should
  be closed.

Note that if the new format is a variant of a format currently supported by
Bio-Formats, it is more efficient to make the new reader a subclass of the
existing reader (rather than subclassing :javadoc:`loci.formats.FormatReader
<loci/formats/FormatReader.html>`).  In this case, it is
usually sufficient to override
:javadoc:`initFile(java.lang.String) <loci/formats/FormatReader.html#initFile-java.lang.String->` and :javadoc:`isThisType(byte[]) <loci/formats/FormatReader.html#isThisType-byte:A->`.

Every reader also has an instance of :javadoc:`loci.formats.CoreMetadata
<loci/formats/CoreMetadata.html>`.  All readers should populate the fields in
CoreMetadata, which are essential to reading image planes.

If you read from a file using something other than
:common_javadoc:`loci.common.RandomAccessInputStream <loci/common/RandomAccessInputStream.html>` or
:common_javadoc:`loci.common.Location <loci/common/Location.html>`,
you *must* use the file name returned by ``Location.getMappedId(String)``, not
the file name passed to the reader.
Thus, a stub for ``initFile(String)`` might look like this:

::

      protected void initFile(String id) throws FormatException, IOException {
        super.initFile(id);

        RandomAccessInputStream in = new RandomAccessInputStream(id);
        // alternatively,
        //FileInputStream in = new FileInputStream(Location.getMappedId(id));

        // read basic file structure and metadata from stream
      }


For more details, see
:common_javadoc:`loci.common.Location.mapId(java.lang.String, java.lang.String) <loci/common/Location.html#mapId(java.lang.String,java.lang.String)>`
and :common_javadoc:`loci.common.Location.getMappedId(java.lang.String) <loci/common/Location.html#getMappedId(java.lang.String)>`.

Variables to populate
---------------------

There are a number of global variables defined in
:javadoc:`loci.formats.FormatReader <loci/formats/FormatReader.html>` that
should be populated in the constructor of any implemented reader.

These variables are:

- :javadoc:`suffixNecessary <loci/formats/FormatReader.html#suffixNecessary>`
  Indicates whether or not a file name suffix is required; true by default

- :javadoc:`suffixSufficient <loci/formats/FormatReader.html#suffixSufficient>`
  Indicates whether or not a specific file name suffix guarantees that this
  reader can open a particular file; true by default

- :javadoc:`hasCompanionFiles <loci/formats/FormatReader.html#hasCompanionFiles>`
  Indicates whether or not there is at least one file in a dataset of this
  format that contains only metadata (no images); false by default

- :javadoc:`datasetDescription <loci/formats/FormatReader.html#datasetDescription>`
  A brief description of the layout of files in datasets of this format; only
  necessary for multi-file datasets

- :javadoc:`domains <loci/formats/FormatReader.html#domains>`
  An array of imaging domains for which this format is used.  Domains are
  defined in
  :javadoc:`loci.formats.FormatTools <loci/formats/FormatTools.html>`.

Other useful things
-------------------

- :common_javadoc:`loci.common.RandomAccessInputStream
  <loci/common/RandomAccessInputStream.html>` is a
  hybrid RandomAccessFile/InputStream
  class that is generally more efficient than either RandomAccessFile or
  InputStream, and implements the DataInput interface.  It is recommended that
  you use this for reading files.

- :common_javadoc:`loci.common.Location <loci/common/Location.html>` provides an API
  similar to java.io.File, and supports
  File-like operations on URLs.  It is highly recommended that you use this
  instead of File.  See the :javadoc:`Javadocs <>` for additional information.

- :common_javadoc:`loci.common.DataTools <loci/common/DataTools.html>` provides a
  number of methods for converting bytes to
  shorts, ints, longs, etc.  It also supports reading most primitive types
  directly from a RandomAccessInputStream (or other DataInput implementation).

- :javadoc:`loci.formats.ImageTools <loci/formats/ImageTools.html>` provides several methods
  for manipulating
  primitive type arrays that represent images. Consult the source or Javadocs
  for more information.

- If your reader relies on third-party code which may not be available to all
  users, it is strongly suggested that you make a corresponding service class
  that interfaces with the third-party code.  Please see :doc:`service` for a
  description of the service infrastructure, as well as the
  :javadoc:`loci.formats.services package <loci/formats/services/package-summary.html>`.

- Several common image compression types are supported through subclasses of
  :javadoc:`loci.formats.codec.BaseCodec <loci/formats/codec/BaseCodec.html>`. These include JPEG, LZW, LZO, Base64, ZIP and RLE (PackBits).

- If you wish to convert a file's metadata to OME-XML (strongly encouraged),
  please see :doc:`Bio-Formats metadata processing </about/index>` for further information.

- Once you have written your file format reader, add a line to the
  :source:`readers.txt <components/formats-api/src/loci/formats/readers.txt>`
  file with the fully qualified name of the reader, followed by a '#' and the
  file extensions associated with the file format. Note that
  :javadoc:`loci.formats.ImageReader <loci/formats/ImageReader.html>`,
  the master file format reader, tries to identify which format reader to use
  according to the order given in :source:`readers.txt <components/formats-api/src/loci/formats/readers.txt>`, so be sure to place your
  reader in an appropriate position within the list.

.. seealso::
  :doc:`external-readers`

- The easiest way to test your new reader is by calling "java
  loci.formats.tools.ImageInfo <file name>".  If all goes well, you should see
  all of the metadata and dimension information, along with a window showing
  the images in the file.  :javadoc:`loci.formats.ImageReader <loci/formats/ImageReader.html>` can take additional
  parameters; a brief listing is provided below for reference, but it is
  recommended that you take a look at the contents of
  :source:`loci.formats.tools.ImageInfo <components/bio-formats-tools/src/loci/formats/tools/ImageInfo.java>` to see
  exactly what each one does.

+--------------+----------------------------------------------------------+
| Argument     | Action                                                   |
+==============+==========================================================+
| -version     | print the library version and exit                       |
+--------------+----------------------------------------------------------+
| file         | the image file to read                                   |
+--------------+----------------------------------------------------------+
| -nopix       | read metadata only, not pixels                           |
+--------------+----------------------------------------------------------+
| -nocore      | do not output core metadata                              |
+--------------+----------------------------------------------------------+
| -nometa      | do not parse format-specific metadata table              |
+--------------+----------------------------------------------------------+
| -nofilter    | do not filter metadata fields                            |
+--------------+----------------------------------------------------------+
| -thumbs      | read thumbnails instead of normal pixels                 |
+--------------+----------------------------------------------------------+
| -minmax      | compute min/max statistics                               |
+--------------+----------------------------------------------------------+
| -merge       | combine separate channels into RGB image                 |
+--------------+----------------------------------------------------------+
| -nogroup     | force multi-file datasets to be read as individual files |
+--------------+----------------------------------------------------------+
| -stitch      | stitch files with similar names                          |
+--------------+----------------------------------------------------------+
| -separate    | split RGB image into separate channels                   |
+--------------+----------------------------------------------------------+
| -expand      | expand indexed color to RGB                              |
+--------------+----------------------------------------------------------+
| -omexml      | populate OME-XML metadata                                |
+--------------+----------------------------------------------------------+
| -normalize   | normalize floating point images*                         |
+--------------+----------------------------------------------------------+
| -fast        | paint RGB images as quickly as possible*                 |
+--------------+----------------------------------------------------------+
| -debug       | turn on debugging output                                 |
+--------------+----------------------------------------------------------+
| -range       | specify range of planes to read (inclusive)              |
+--------------+----------------------------------------------------------+
| -series      | specify which image series to read                       |
+--------------+----------------------------------------------------------+
| -swap        | override the default input dimension order               |
+--------------+----------------------------------------------------------+
| -shuffle     | override the default output dimension order              |
+--------------+----------------------------------------------------------+
| -map         | specify file on disk to which name should be mapped      |
+--------------+----------------------------------------------------------+
| -preload     | pre-read entire file into a buffer; significantly        |
|              | reduces the time required to read the images, but        |
|              | requires more memory                                     |
+--------------+----------------------------------------------------------+
| -crop        | crop images before displaying; argument is 'x,y,w,h'     |
+--------------+----------------------------------------------------------+
| -autoscale   | used in combination with '-fast' to automatically adjust |
|              | brightness and contrast                                  |
+--------------+----------------------------------------------------------+
| -novalid     | do not perform validation of OME-XML                     |
+--------------+----------------------------------------------------------+
| -omexml-only | only output the generated OME-XML                        |
+--------------+----------------------------------------------------------+
| -format      | read file with a particular reader (e.g., ZeissZVI)      |
+--------------+----------------------------------------------------------+

\* = may result in loss of precision

- If you wish to test using TestNG, ``loci.tests.testng.FormatReaderTest``
  provides several basic tests that work with all Bio-Formats readers. See
  the FormatReaderTest source code for additional information.

- For more details, please look at the source code and :javadoc:`Javadocs <>`.
  Studying existing readers is probably the best way to get a feel for the
  API; we would recommend first looking at
  :bfreader:`loci.formats.in.ImarisReader <ImarisReader.java>` (this is the
  most straightforward one). :bfreader:`loci.formats.in.LIFReader <LIFReader.java>` and :bfreader:`InCellReader <InCellReader.java>` are also
  good references that show off some of the nicer features of Bio-Formats.

If you have questions about Bio-Formats, please contact :imagesc:`the forums <tag/bio-formats>`.
