Search and indexing configuration

How Indexing works

Indexing is not driven by the user, but happens automatically in the background and can be controlled by a number of settings listed under Search. The indexer runs periodically as defined by omero.search.cron and parses the latest batch of new or modified objects in the database.

Upon successful completion, the persistent count in the configuration table will be incremented.

omero=# select value from configuration where name = 'PersistentEventLogLoader.v2.current_id';
 value
-------
 30983
(1 row)

Note

Presence of more than one PersistentEventLogLoader.* value in your database indicates that you have run indexing with multiple versions of the server. This is fine. To allow a new server version to force an update, the configuration key may be changed. For example, PersistentEventLogLoader.current_id became PersistentEventLogLoader.v2.current_id in https://github.com/ome/openmicroscopy/commit/a5cb64a.

Missing search results

If you are having any difficult with search results not appearing timely, first you should start by checking the health of the Indexer-0 process:

Check the server’s log directory for a file named Indexer-0.log and monitor its progress (e.g. using tail or similar). If messages of the format:
```
INFO  [   ome.services.fulltext.FullTextIndexer ] (3-thread-2) INDEXED 2 objects in 1 batch(es) [2483 ms.]
```
are periodically being appended to the log file, then your indexer process may be running behind. You can either wait for it to catch up, or try increasing the search batch size in order to speed processing. See the section on the omero.search.batch setting for more information.
If there are no updates to the Indexer-0.log file even when new images, tags, files, etc. are added to the server, then it is possible that the Indexer process has become stuck. It is possible to force a restart of the indexer using the IceGrid Tools like so:
```
> omero admin ice
Ice 3.6.3   Copyright (c) 2003-2016 ZeroC, Inc.
>>> server list
Blitz-0
DropBox
FileServer
Indexer-0
...
>>> server stop Indexer-0
```
You do not need to manually re-start the Indexer, as IceGrid will handle the creation of a new Indexer process automatically.

In case neither of the above seems to be the case, then your indexer is running normally and more likely your index has been corrupted. You will need to re-index OMERO. Reasons why this might have occurred include:

Missing search terms are part of a very large text file. If the indexer’s maximum file size limit is reached, a file may not be indexed. See the section on the omero.search.max_file_size setting for more information on increasing this limit.
A bug in Lucene prior to OMERO 5.0.1 caused some documents to be “sealed” in that old search terms would return the document, but newer terms would not.

Re-indexing

Background re-indexing

Under most circumstances, you should be able to re-index the database while the server is still running. If you need to make any adjustments to the server configuration or the process heap size, first shut the server down and make these changes before restarting the server. Use the following steps to initiate a re-indexing.

Disable the search indexer process and stop any currently running indexer processes:
```
$ omero admin reindex --prepare
```
Remove the existing search Indexes by deleting the contents of the FullText subdirectory of your omero.data.dir:
```
$ omero admin reindex --wipe
```
Reset the indexer’s progress counter in the database:
```
$ omero admin reindex --reset 0
```
Re-enable/restart the indexer process:
```
$ omero admin reindex --finish
```

Depending on the size of your database, it may take the indexer some time to finish re-indexing. During this time, your OMERO server will remain available for use, however the search functionality will be degraded until the re-indexing is finished. See Monitoring re-indexing for information on how long this should take.

Note

Once you wipe your full-text directory, searches will return fewer or no results until re-indexing is complete.

Monitoring re-indexing

During re-indexing, it is possible to estimate the percent indexed using the following SQL command:

omero=> select 'At ' ||  current_timestamp(0) || ', Percent indexed: ' || trunc(((select count(*) from eventlog el, configuration c where el.id < cast(c.value as int) and (c.name like 'PersistentEventLogLoader%')) * 1.0) / (select count(*) from eventlog) * 100, 2) || '%';
                  ?column?
----------------------------------------------------
 At 2014-06-14 07:54:37+00, Percent indexed: 70.90%
(1 row)

This value is also logged periodically when re-indexing in the background and is available via JMX. See Metrics for more information.