Deleting in OMERO

Deleting data in OMERO is complex due to the highly linked nature of data in the database. For example, an Image has links to Datasets, Comments, Tags, Instrument, Acquisition metadata etc. If the image is deleted, some of this other data should remain and some should be deleted with the image (since it has no other relevance).

In the 4.2.1 release of OMERO, an improved deleting service was introduced to fix several problems or requirements related to the delete functionality (see #2615 for tickets):

  • Need a better way to define what gets deleted when certain data gets deleted (e.g. Image case above)

  • Need to be able to configure this definition, since different users have different needs

  • Deleting large amounts of data (e.g. Plate of HCS data) was too memory-intensive (data was loaded from the database during delete)

  • Poor logging of deletes

  • Large deletes (e.g. screen data) take time: Clients need to be able to keep working while deletes run ‘in the background’

  • Binary data (pixels, thumbnails, files etc) was not removed at delete time - required sysadmin to clean up later

Future releases will continue this work (see #2911) and the 5.1.0 release of OMERO offers a new implementation of deletion.

Finality of deletion

Import in OMERO 5.x uploads the image and companion files intact and stores them within subdirectories of the directory configured by the value of omero.managed.dir, typically ManagedRepository. The files relating to a specific fileset are stored together on the server’s filesystem and they are read by Bio-Formats when images are viewed in OMERO clients. If any of a fileset’s files, or the corresponding entries for them in the database, are deleted, then the fileset may no longer be readable. If all the fileset’s files are deleted, the fileset will certainly be unreadable, and there is no ‘undo’ that will bring it back.

Delete behavior (technical)

Configuring what gets deleted is done using XML files. Since OMERO 5.1, the delete behavior defaults to a Model graph operations implementation that is configured by https://github.com/ome/omero-blitz/blob/v5.8.0/src/main/resources/ome/services/blitz-graph-rules.xml.

Delete Image

The general delete behavior for deleting an Image is to remove every piece of data from the database that was added when the image was imported, removing pixel data and thumbnails from disk. In addition, the following data is deleted:

  • Comments on the image

  • Rating of the image

  • ROIs for this image (see below)

  • Image Rendering settings for yourself and other users

Optional - In OMERO.web and OMERO.insight, you will be asked whether you also want to delete:

  • Files attached to the image (if not linked elsewhere). In that case, the binary data will be removed from disk too.

  • Your own tags on the image (if not used elsewhere)

The same option is available when deleting dataset, project, plate, screen.

Delete Dataset or Project

When deleting a Project or Dataset, you have the option to also delete tags and annotations (as for Image above). You also can choose whether to ‘delete contents’. This will delete any Datasets (or Images) that are contained in the Project (or Dataset). However, Datasets and Images will not get deleted if they are also contained in other Projects or Datasets respectively.

If a user decides to delete/keep the annotations (see Optional above) when deleting a Project (or Dataset) and its contents, the rule associated to the annotation will be apply to all objects.

Delete Screen, Plate or Plate Acquisition

When deleting a Screen, you have the option to also delete tags and annotations. You also can choose whether to ‘delete contents’. This will delete any Plates that are contained in the Screen. However, Plates will not get deleted if they are also contained in other Screen.

When deleting a Plate, you have the option to also delete tags and annotations but NOT the option to ‘delete contents’.

If the Plate has Plate Acquisitions, you can delete one or more Plate Acquisition at once.

Delete Tag/Attachment

You can delete a Tag/Attachment, and it will be removed from all images. However you cannot delete a Tag/Attachment if it has been used by another user in the same collaborative group. This is to prevent potential loss of significant amount of annotation effort by other users. You will need to get the other users to first remove your Tag/Attachment where they have used it, before you can delete it.

Known Issue: if the owner of the Tag/Attachment is also an owner of the group (e.g. PI), they will be able to delete their Tag/Attachment, even if others have used it.

Delete multi-file Images and Image sets

An Image, or a set of Images, may come from a single file or a set of dependent files. For instance, a single Leica LIF file may contain many Images, as may a Zeiss mdb file with lsm files. On the other hand, some file formats, like Deltavision with log file, or the original ICS file format, use multiple files to represent a single Image. At import time, these groups of related files and Images are organized into Filesets: a Fileset is a set of files that encode a set of Images. The simplest case where there is one file per Image still has a corresponding Fileset.

Even if many Images come from the same file, they may be separately selected and viewed in client software. However, at least at present, a Fileset may not be partially deleted: either all the files and Images from it are deleted, or none are. So, for instance, the Images from the same Leica LIF file may be deleted only all at once, and the Deltavision log file is not deleted separately from the main file. The same applies to high-content screening data: a Plate with its Wells and Images are all stored in one Fileset and may be deleted only together.

Each Fileset has a corresponding directory on the server in which, perhaps in subdirectories, all its files are stored. All the file paths for an Image’s Fileset can be accessed from the tool-bar at the top of the right-hand panel.

Delete in collaborative group

Some more discussion of delete issues in a collaborative group, where your data are linked to data of other users, can be found on the Groups and permissions system page.

  • A user cannot remove Images from another user’s Dataset, or remove Datasets (or Plates) from Projects (or Screens).

  • A user cannot delete anything that belongs to another user.

Group owner rights

An owner of the group, usually a PI, can delete anything that belongs to other members of the group.

Edge cases

These are ‘known issues’ that may cause problems for some users (not for most). These will be resolved in future depending on priority.

  • Other users’ ROIs (and associated measurements) are deleted from images.

  • Multiply-linked objects are unlinked and not deleted e.g. Project p1 contains two Datasets d1 and d2, Project p2 contains Dataset d1. If the Project p1 is deleted, the Dataset d1 is only unlinked from p1 and not completely deleted.

Binary data

When Images, Plates or File Annotations have been successfully deleted from the database the corresponding binary data is deleted from the binary repository. It is possible that some files may not be successfully deleted if they are locked for any reason. In this case, the undeleted files can be removed manually via omero admin cleanse. This also deletes any empty directories left behind after the binary data that they contained has been deleted.