Search functionality

Ditto provides a search functionality as one of the services around its managed digital twins. The functionality is available for the following APIs.

Search index

Ditto’s microservice things-search automatically consumes all events which are emitted for changes to Things and Policies and updates an for search optimized representation of the Thing data into its own database.

No custom indexes have to be defined as the structure in the database is “flattened” so that all data contained in Things can be searched for efficiently.

Consistency

Ditto’s search index provides eventual consistency.

In order to reduce load to the database when processing updates in a high frequency, the search index is updated in small batches with a default interval of 1 second (configurable via environment variable THINGS_SEARCH_UPDATER_STREAM_WRITE_INTERVAL).

That means that when a thing is updated and the API (e.g. the HTTP endpoint) returns a success response, the search index will not reflect that change in that instant. The change will most likely be reflected in the search index within 1-2 seconds. In rare cases the duration until consistency is reached again might be higher.

Search queries

Queries can be made via Ditto’s APIs (HTTP or Ditto Protocol e.g. via WebSocket).

Example: Search for all things located in “living-room”, reorder the list to start with the lowest thing ID as the first element, and return the first 5 results:

Filter:     eq(attributes/location,"living-room")
Sorting:    sort(+thingId)
Paging:     size(5),cursor(CURSOR_ID)

Search count queries

The same syntax applies for search count queries - only the sorting and paging makes no sense here, so there are not necessary to specify.

Namespaces

The Search supports specifying in which namespaces it should be searched. This may significantly improve the search performance when many Things of different namespaces are managed in Ditto’s search index.

RQL

In order to apply queries when searching, Ditto uses the RQL notation which is also applied for other scenarios (e.g. filtering notifications).

Sorting and paging options

The sort option governs the order of search results.

sort(<+|-><property1>,<+|-><property2>,...)

If not given, search results are listed in the ascending order of thing IDs, namely sort(+thingId).

The size option

size(<count>)

limits the search results delivered in one HTTP response or one Ditto protocol message to <count> items.

If the paging option is not explicitly specified a default value of 25 is used. The maximum allowed count is 200.

cursor(<cursor-id>)

Starts the search at the position of the cursor with ID <cursor-id>. The cursor ID is obtained from the field cursor of a previous response and marks the position after the last entry of the previous search. A response includes no cursor if there are no more results.

If a request has a cursor option, then any included filter or sort option may not differ from the original request of the cursor. Otherwise, the request is rejected.

Example - return ten items with a cursor

option=size(10),cursor(<cursor-from-previous-result>)

RQL paging (deprecated)

The RQL limiting part specifies which part (or “page”) should be returned of a large search result set.

limit(<offset>,<count>)

Limits the search results to <count> items, starting with the item at index <offset>.

• if the paging option is not explicitly specified, the default value limit(0,25) is used, i.e. the first 25 results are returned.
• the maximum allowed count is 200.

Example - return the first ten items

limit(0,10)

Example - return the items 11 to 20

limit(10,10)

i.e. Return the next ten items (from index 11 to 20)