Model graph operations

Overview

When the OMERO server acts on its model objects it must determine the impact on related objects. For instance, deleting an image may entail deleting users’ rendering settings for that image, also the links to any datasets that the image is in. Understanding the details of the implementation substantially assists in debugging or creating server operations that act on the directed graph of model objects.

Motivation

The OMERO model objects are interlinked. Plates may have wells whose samples come from multiple runs. Both datasets and well samples may have images, but in different ways. Datasets, wells, images, among others, may all be annotated. Images themselves are not simple: for example, they may be in a fileset, they may have ROIs drawn on them, they may share an instrument with the projection of that image. All these entities are separate objects that can be thought of as forming the nodes (vertices) of a directed graph of relationships.

Various operations supported by the OMERO server, most commonly moving objects to a different group, or deleting them, may implicitly include many related objects. For example, if one deletes a fileset, one also deletes the images from that fileset, and even the comments on those images. This section describes how the graph of model objects is traversed and how the target set of related objects is determined.

This technical detail is important to understand if one wishes to,

• adjust the set of related model objects that are included in operations

• change the types of OME-Remote Objects model objects or the permissible links among them

• fix bugs in the related request objects defined in Graphs.ice that may be submitted to OMERO sessions for execution.

Approach

Graph node states and transitions

In determining which model objects to process, and how, each corresponding graph node is in one of these states:

adjective	rule format	Action enum	Orphan enum
irrelevant	[E]{i}	EXCLUDE	IRRELEVANT
relevant	[E]{r}	EXCLUDE	RELEVANT
orphaned	[E]{o}	EXCLUDE	IS_LAST
attached	[E]{a}	EXCLUDE	IS_NOT_LAST
to delete	[D]	DELETE	n/a
to include	[I]	INCLUDE	n/a
outside	[O]	OUTSIDE	n/a

“enum” refers to the enumerations defined in GraphPolicy.java. Note also that, as for the introduction to Deleting in OMERO, “links” are simply edges in the graph, distinct from the classes implementing ILink.java which themselves have several links, not least to their parent and child objects.

When traversal begins, the target objects are to be included (e.g., for Chgrp2) or deleted (e.g., for Delete2) and other objects are irrelevant.

A list of transition rules is associated with the requested operation. Each of the target objects is examined in turn and the rules matched against the state of that object and of those directly linked to it in either direction. If a rule matches, it may either abort the operation with an error condition or, more usually, change the state of any of the objects it matches. Changed objects are themselves queued for examination and rule matching. The traversal is complete when all queued objects have been examined with no further transition rule matches. Rules that can abort the operation are checked only after the other rules have completed processing. GraphTraversal.java’s planOperation method is at the heart of this matching process.

Further graph node states

Usual behavior is for orphaned objects related to the target objects to be included in the operation, but not the otherwise-attached objects, the non-orphans who have excluded parents that are to be neither deleted nor included. The related children that may be orphans are exactly those identified as being relevant. Transition rules match these against excluded parents to discover if the relevant objects do have any qualifying parents, changing them to be attached objects. If no further rules match and some objects remain as relevant, then they are automatically changed to orphans and examined for further rule matches. After that processing completes, attached objects are changed back to being relevant to confirm that excluded qualifying parents still exist to change them to being attached: this is necessary in case, after an object was considered attached, other rules changed all those qualifying parents from being excluded so that the object is now an orphan.

Objects that are changed to be outside are effectively rendered invisible, outside consideration in the execution phase. In the execution of an operation the graph traversal code removes links between included and excluded objects, but it allows links to remain between outside objects and other objects. Outside objects typically implement IGlobal.java and have no owner or group.

An additional aspect of objects’ state is if permissions are to be checked for them. For instance, typically I may move only my own objects to a different group, but if another user tags my image with my tag, then I may still move my image and tag to a different group, also moving that link even though it is not my own object: in that case, permissions checking is disabled for that ImageAnnotationLink. All objects initially have permissions checking enabled, but the consequence of a rule may be to disable permissions checking, and if an object with permissions checking disabled matches a further rule, the objects changed by that rule also have permissions checking disabled.

Configuration

Defining the model graph transition rules

To reduce its complexity, GraphTraversal.java does not include specific detail of how to traverse the graph of OME-Remote Objects model objects. Instead, subclasses of GraphPolicy.java guide the traversal of the model object graph, configured by blitz-graph-rules.xml which names and defines the lists of transition rules. The named lists of rules are associated with request object classes by the definition of the graphRequestFactory bean in blitz-servantDefinitions.xml, which also specifies which model object properties may never be set to null in executing any requested operation.

blitz-graph-rules.xml begins with a comment that provides a key to the notation used for transition rules. The rules name and match model objects based on the state of the graph nodes, the types of the corresponding objects, the permissions the user has on those objects, and the names of the properties linking the objects. To illustrate this, the following sections briefly describe some different kinds of rule from the deleteRules list.

Propagating deletion

p:matches="L:ILink.parent = [D], L.child = C:[E]{o}/d"
p:changes="C:[D]"

If an ILink’s parent (e.g., a dataset) is to be deleted, and its child (e.g., an image) is orphaned and deletable by the user, then delete the child also.

p:matches="PlateAcquisition[D].wellSample = WS:WellSample[E]"
p:changes="WS:[D]"

If a plate acquisition (run) is to be deleted, also delete its well samples (fields).

p:matches="Fileset[D] = I:Image[E].fileset"
p:changes="I:[D]"

If a fileset is to be deleted, then also delete its images.

p:matches="T:Thumbnail[E].pixels =/!o [D]"
p:changes="T:[D]/n"

If the pixels of a thumbnail are to be deleted, and are owned by a different user, then delete the thumbnail regardless of permissions on it.

Curtailing deletion

p:matches="Well[D].plate = C:[E]{!a}"
p:changes="C:{a}"

If a well is to be deleted but its plate is excluded and not attached, regard the plate as attached.

p:matches="C:Channel[E]{r}.pixels = Pixels[E]{i}"
p:changes="C:{a}"

If an irrelevant pixels object has a relevant channel, then regard the channel as attached.

p:matches="Pixels[D].relatedTo = P:[E]{!a}"
p:changes="P:{a}"

If a pixels object is to be deleted, regard any related, excluded pixels objects as attached. Because the pixels of an image are related to the pixels of a projection of that image, this rule prevents the deletion of an image from causing inadvertent deletion of the image’s projections.

p:matches="L:ILink[!D].parent = [E]/d, L.child = C:[E]{r}"
p:changes="C:{a}"

If an ILink that is not to be deleted itself has a deletable, excluded parent and a relevant child, regard that child as attached.

Other kinds of transition rule

p:matches="E:IEnum[E]"
p:changes="E:[O]"

Regard excluded IEnum objects as being outside the operation. (Rules do not need to match on links among multiple objects.)

p:matches="F:Fileset[!D].images = [D], F.images = [!D]"
p:error="may not split {F}"

Throw an error if a fileset that is not to be deleted includes an image that is to be deleted and an image that is not to be deleted.

In reviewing the chgrpRules list, one sees conditions that require matching $to_private or !$to_private. A request, in this case Chgrp2I.java, may set arbitrary conditions upon which rules may be predicated. The to_private condition, or its absence, is used to cause different behavior when the objects are being moved into a private group.

Logging

Changing the log level

It is informative to observe the sequence of rule applications as the graph is traversed and decisions about model objects are made. To do so requires configuring Omero logging for the server, specifically https://github.com/ome/openmicroscopy/blob/develop/etc/logback.xml. To activate graph traversal debug logging, adjust the ends of the lines,

<logger name="omero.cmd.graphs" level="INFO"/>
<logger name="ome.services.graphs" level="INFO"/>

such that they instead read,

<logger name="omero.cmd.graphs" level="DEBUG"/>
<logger name="ome.services.graphs" level="DEBUG"/>

The resulting extra information in var/log/Blitz-0.log is of particular assistance in debugging: it pinpoints the rule applications that caused incorrect determinations of what action to take with model objects. Note that a * suffix on a model object referenced in the logs indicates that permissions are not to be checked for it.

Expanding the reports of transition rule matches

In the previous section, it can be seen that model objects that match rule conditions may be named. For example, in,

p:matches="Fileset[D] = I:Image[E].fileset"
p:changes="I:[D]"

the image is named I. When a rule matches, the debug logging reports which model object matched each name. If it remains unclear why a rule matched, further objects may be named. For example, changing the first line to name the fileset,

p:matches="F:Fileset[D] = I:Image[E].fileset"

would also report in the log which fileset matched the rule.

Encouragement

On first reading, the above may feel daunting. If model object graph traversal is not working as desired, thus requires adjustment, review of debug logs from var/log/Blitz-0.log typically pinpoints the cause and a minor adjustment to blitz-graph-rules.xml often suffices as the fix, with integration tests providing reassurance that the adjustment was acceptable. Sometimes it can take time and thought to devise that fix, but one can expect small changes to suffice to fix most bugs. In getting this new graph traversal implementation to initially pass integration testing, no test failures required a substantial rethink of the basic approach and GraphTraversal.java itself did not require a significant rewrite.

The actual lists of transition rules arose in part as a way to achieve the desired behavior and are not yet as simple and comprehensive as they could be. While they necessarily reflect the inherent complexity of the object model of OME-Remote Objects, there is potential for reviewing the rule lists and, perhaps with some additional marker interfaces, making them more succinct and regular. Incremental movements toward this goal are worth pursuing.

Options

Every one of the request object classes introduced in the new implementation of graph traversal is a derived class of GraphModify2 and inherits data members that configure its operation. Each request may define additional data members for options specific to it, for instance Chgrp2 requires the ID of the target group to be specified. The data members offered by all of the new requests are,

targetObjects

specifies which model objects the operation is to target

childOptions

specifies types of model objects (and, for annotations, namespaces) that should always or never be included in the operation (i.e. always considered to be orphans, or attached, regardless of excluded parents)

dryRun

specifies if the request is to determine which model objects would be included in and deleted by the operation, without actually executing the operation.

SkipHead

The SkipHead request allows specification of the target objects with reference to a common parent. It wraps an inner request data member that starts acting only after graph traversal reaches types listed in startFrom. For example, to target the images of a specific plate, give the plate in targetObjects and name Image in startFrom.

This feature is achieved by running the initial request with dryRun set to true and the graph traversal policy modified so as to not examine included nodes of types listed in startFrom. A subsequent request then runs, targeting the startFrom model objects that were included in the first request.