Bulk imports

The CLI import option --bulk specifies a configuration file that can be used to perform a batch of imports with the same or similar options. The file is written in a simple YAML syntax and can be named whatever you would like. It does not need to be placed in the folder from which the OMERO commands are run.

A minimal YAML file might look like:

---
path: "my-files.txt"

Assuming that my-files.txt is a list of files such as

fileA
fileB
directoryC

this is equivalent to:

$ omero import -k --transfer=ln_s fileA fileB directoryC

where the files fileA and fileB and all the files of directoryC will be imported.

Bulk-only options

Path

The path key specifies a file from which each individual line will be processed as a separate import. In the simplest case, a single file is placed per line as above. For more complex usages, path can point to a tab-separated value (TSV) or a comma-separate value (CSV) file where each field will be interpreted based on columns.

Columns

A fairly regular requirement in importing many files is that for each file a similar but slightly different configuration is needed. This can be accomplished with the columns key. It specifies how each of the separated fields of the path file should be interpreted.

For example, a bulk.yml file specifying:

---
path: "files.tsv"
columns:
- name
- path

along with a files.tsv of the form:

import-1    fileA
import-2    fileB

would match the two calls:

$ omero import --name import-1 fileA
$ omero import --name import-2 fileB

but in a single call. The same could be achieved with this CSV file:

import-1,fileA
import-2,fileB

Other options like target can also be added as a separate field:

Dataset:name:training-set   import-1        fileA
Dataset:name:training-set   import-2        fileB
Dataset:name:test-set-001   import-3        fileC

by defining columns in your bulk.yml as:

columns:
- target
- name
- path

which will create the named datasets if they do not exist. See Import targets for more information on import targets and see below for more examples of options you can use.

Include

The include key specifies another bulk YAML file that should be included in the current processing. For example, if there is a global configuration file omero-imports.yml that all users should use, such as:

---
checksum_algorithm: "File-Size-64"
exclude: "clientpath"
transfer: "ln_s"

then users can make use of this configuration by adding the following line to their bulk.yml file:

include: /etc/omero-imports.yml

Dry_run

The dry_run key can either be set to true in which case no import will occur, and only the potential actions will be shown, or additionally it can be set to a file path of the form my_import_%s.sh where %s will be replaced by an number and a file with the given name will be written out. Each of these scripts can then be used independently.

Other options

Otherwise, all the regular options from the CLI are available for configuration via --bulk:

  • checksum_algorithm for faster processing of large files

  • continue for processing all files even if one errors

  • exclude for skipping files that have already been imported

  • parallel_fileset for concurrent imports

  • parallel_upload for concurrent uploads

  • target for placing imported images into specific containers

  • transfer for alternative methods of shipping files to the server

See Import images for more information.