# Build System

Overview

The page goes into details about how the build system is configured.

Since 5.5, OMERO decouples many components and uses, for some components, an Gradle-based build. The two overarching repositories are https://github.com/ome/omero-build and https://github.com/ome/omero-gradle-plugins. See the README of each repository for more details. OMERO still uses an Ant-based build, for some components, with dependency management provided by Ivy. C++ code is built using Cmake and Python uses the traditional distutils/setuptools tools.

## Structure of the build

This is an (abbreviated) snapshot of the structure of the filesystem for OMERO:

    OMERO_SOURCE_PREFIX
    |
    |-- build.xml .......................... Top-level build file
    |
    |-- build.py ........................... Python wrapper to handle OS-specific configuration
    |
    |-- omero.class ........................ Self-contained Ant launcher
    |
    |--etc ................................. Configuration folder
    |   |-- grid ........................... Deployment files folder
    |   |-- ivysettings.xml ................ Main Ivy configuration file
    |   |-- hibernate.properties
    |   |-- build.properties
    |   |-- logback.xml
    |   |-- omero.properties
    |   \-- profiles
    |
    |-- examples ........................... User examples
    |
    \components
      |
      |--<component-name> .................. Each component has this same basic structure.
      |    |-- build.xml ................... Build file
      |    |-- ivy.xml ..................... Jar dependencies
      |    |-- test.xml .................... Test dependencies
      |    |-- src ......................... Source code
      |    |-- resources ................... Other files of interest
      |    |-- test ........................ Test source code and test resources
      |    \-- target ...................... Build output (deleted on clean)
      |
      |  NOTABLE COMPONENTS
      |
      |--tools ............................. Other server-components with special build needs.
      |    |--build.xml .................... Build scripts
      |    |
      |    \--<tool-name>
      |         |--build.xml ............... Build file
      |         \--ivy.xml ................. Jar dependencies
      |
      \--antlib ............................ Special component which is not built, but referenced by the build
          |
          \--resources ..................... Build resources
              |--global.xml ................ Global build properties
              |--hibernate.xml
              |--lifecycle.xml ............. Ivy-related targets
              \--version.xml ............... Version properties

Note

User examples are explained under Working with OMERO

Unfortunately, just the above snapshot of the code repository omits some of the most important code. Many megabytes of source code is generated both by our own DSLTask as well as by the Ice `slice2java`, `slice2cpp`, and `slice2py` code generators. These take an intermediate representation of the OME-Model and generate our OME-Remote Objects. This code is not available in git, but once built, can be found in all the directories named “generated”.

## Build tools

### Ant

`./build.py` is a complete replacement for your local ant install. In many cases, you will be fine running ant. If you have any issues (for example `OutOfMemory`) , please use `./build.py` instead. However, only use one or the other; do not mix calls between the two.

The main build targets are defined in the top-level `build.xml` file. All available targets can be listed using:

    ./build.py -p

### Ivy

The build system uses Ivy 2.3.0 as the dependency manager. The general Ivy configuration is defined in a settings file located under https://github.com/ome/openmicroscopy/blob/develop/etc/ivysettings.xml.

In order to determine the transitive closure of all dependencies, Ivy resolves each `ivy.xml` file and stores the resolved artifacts in a cache to speed up other processes. The OMERO build system defines and uses two kinds of caches:

1. the local dependencies cache under `lib/cache` is used by most resolvers

2. Maven resolvers use the Maven cache under `~/.m2/repository`

Note

When the Ivy configuration file or the version number is changed, the cache can become stale. Calling `./build.py clean` from the top-level build will delete the content of the local cache.

Resolvers are key to how Ivy functions. Multiple dependency resolvers can be defined fine-grained enough to resolve an individual jar in order to pick up the latest version of any library from a repository, a generic URL or from the local file system. Since OMERO 5.1.3, the remote repository resolvers are set up to resolve transitive dependencies.

The OMERO build system uses by default a chain resolver called `omero-resolver` which resolves the following locations in order:

1. `target/repository` which contains most artifacts published by the build system in the install step of the lifecycle

2. the local dependency repository under `lib/repository`

3. the local Maven cache under `~/.m2/repository`

4. the Maven central repository

5. the OME artifactory

Bio-Formats dependencies are resolved using a specific chain resolver called `ome-resolver` which resolves the following locations in order:

1. the local Maven cache under `~/.m2/repository`

2. the OME artifactory

To define its dependencies, each component uses a top-level Ivy file, `ivy.xml`, for the build and optionally another Ivy file, `test.xml`, for the tests.

The OMERO build system defines and uses four types of Ivy configurations:

1. build: defines dependencies to be used for building

2. server: defines dependencies to be bundled under `lib/server`

3. client: defines dependencies to be bundled under `lib/client`

4. test: defines dependencies to be used for running the tests

While building, most Java components follow the same lifecycle define in lifecycle.xml. The default dist target for each component calls each of the following steps in order:

1. retrieve: retrieve the resolved dependencies and copy them under `target/libs`

2. prepare: prepare various resources (property files, https://github.com/ome/openmicroscopy/blob/develop/lib/logback-build.xml)

3. generate: copy all resources from the previous step for compilation

4. compile: compile the source files into the destination repository

5. package-extra: package the sources and the Javadoc into Jar files for publication

6. package: package the compiled classes into a Jar file for publication

7. install: convert the component Ivy file into a pom file using makepom and publish the component artifacts

Individual components can override the content of this default lifecycle via their `build.xml`.

### OmeroTools

The Ant build alone is not enough to describe all the products which get built. Namely, the builds for the non-Java components stored under https://github.com/ome/openmicroscopy/tree/develop/components/tools are a bit more complex. Each tools component installs its artifacts to the tools/target directory which is copied on top of the `dist` top-level distribution directory.

### Jenkins

The OME project currently uses Jenkins as a continuous integration server available here, so many binary packages can be downloaded without compiling them yourself. See the Continuous Integration documentation for further details.

## Server build

The default ant target (`build-default`) will build the OMERO system and copy the necessary components for a binary distribution to the `dist` directory. Below is a comparison of what is taken from the build, where it is put, and what role it plays in the distribution.

OMERO_SOURCE_PREFIX | OMERO_SOURCE_PREFIX/dist | Comments
--- | --- | ---
components/tools/OmeroCpp/lib* | `lib/` | Native shared libraries
lib/repository/<some> | `lib/client` & `lib/server` | Libraries needed for the build
etc/ | `etc/` | Configuration
`sql/*.sql` | `sql/` | SQL scripts to prepare the database

Note

By default, OMERO C++ language bindings are not built. Use `build-all` for that.

These files are then zipped to OMERO.server-<version>.zip via `release-zip`

## Coupled development

Since OMERO 5.1.3, Bio-Formats is decoupled from the OMERO build system which consumes Bio-Formats artifacts from the OME Maven repository via Ivy.

While this decoupling matches most of the development use cases, it is sometimes necessary to work on coupled Bio-Formats and OMERO branches especially during breaking changes of the OME Data Model or the Bio-Formats API.

The general rule for coupled branches is to build each component in their dependency order and use the local Maven repository under `~/.m2/repository` to share artifacts.

### Building Bio-Formats

From the top-level folder of the Bio-Formats repository,

1. if necessary, adjust the version of Bio-Formats which will be built, installed locally and consumed by OMERO e.g. for 8.2.0-SNAPSHOT:

       $ mvn versions:set -DnewVersion=8.2.0-SNAPSHOT -DgenerateBackupPoms=false

2. run the Maven command allowing to build and install the artifacts under the local Maven cache:

       $ mvn clean install

### Building OMERO

From the top-level folder of the OMERO repository,

1. in https://github.com/ome/omero-model, adjust the version of `ome:formats-gpl` in https://github.com/ome/omero-model/blob/v5.7.0/build.gradle to the version chosen for the Bio-Formats build

2. publish locally the change using `gradle publishToMavenLocal`