Things - Search protocol specification

The search aspect of the Ditto protocol consists of 3 commands and 4 events that together implement the reactive-streams protocol over any duplex transport layer. For each search request, Ditto acts as the reactive-streams publisher of pages of search results, and the client acts as the subscriber. By reactive-streams means, the client controls how fast pages are delivered to it and may cancel a search request before all results are sent.

While connections do not expose or require a duplex transport layer, the search protocol is available for them as well: Send commands from client to Ditto via any connection source. For each command, 0 or more events from Ditto to client are published to the reply-target of the source.

For reactive-streams on the JVM, a publisher-subscriber pair is identified by a Subscription object according to reference equality. Similarly, the search protocol commands and events of one search query are identified by a subscription ID.

Each search protocol command or event corresponds to a reactive-streams signal and are bound by the same rules in the reactive-streams specification.

Interaction pattern

For one search query, the commands from client to Ditto should follow this protocol:

subscribe request* cancel?

The client should send one “subscribe” command, followed by multiple “request” commands and an optional “cancel” command.

In response to a “subscribe” command and after each “request” command, Ditto will send 0 or more events to the client according to the following protocol:

created next* (complete | failed)?

A “created” event bearing the subscription ID is always sent. 0 or more “next” events are sent according to the amount of search results and the number of pages requested by the client. A “complete” or “failed” event comes at the end unless the client sends a “cancel” command before the search results are exhausted.

There is no special event in response to a “cancel” command. The client may continue to receive buffered “next”, “complete” or “failed” events after sending a “cancel” command.

In addition to the rules of reactive-streams, Ditto guarantees that no “complete” or “failed” event will arrive before the client expresses its readiness by a first “request” command. The reason is to facilitate concurrency at the client side. Without the extra guarantee, a multi-threaded client would have to process a “complete” or “failed” event in parallel of the preceding “created” event. It would put the burden of sequentialization at the client side and complicate the programming there.

Commands from Client to Ditto

Subscribe

Sent a “subscribe” command to Ditto to start receiving search results. Ditto will always respond with a “created” event.

The standard filter, options and the namespaces can be specified in the value field of a “subscribe” command. They have identical semantics and default values as other search APIs. In particular:

• When given in the options field, size(<count>) limits the number of search results delivered in one “next” event to <count> items. The default value of <count> is 25 and the maximum value is 200.
• When given in the options field, sort(<+"-><property1>, ...) sets the order of search results. If not given, the default sort(+thingId) is used.

The paging options cursor and limit of the HTTP-API are not supported here, because they are not meaningful for the search protocol. For the HTTP-API, those options are for iterating through large numbers of search results over many HTTP requests in a stateless manner. The search protocol is not stateless and does not require the client to keep track of any cursor or offset; results of any size are streamed over an arbitrarily long period of time.

Request

After obtaining a subscription ID from a “created” event, use “request” commands to tell Ditto how many pages of search results you are prepared to receive. Ditto will send “next” events until all requested pages are fulfilled, the search results are exhausted, or an error occurred.

Cancel

After obtaining a subscription ID from a “created” event, use a “cancel” command to stop Ditto from sending more pages of the search results. Pages in-flight may yet arrive, but the client will eventually stop receiving events of the same subscription ID.

Events from Ditto to Client

Created

To any “subscribe” command, Ditto will always respond with a “created” event.

Next

Each “next” event contains one page of the search results. Ditto will not send more “next” events for a given subscription ID than the total number of pages requested by previous “request” commands.

Complete

A search subscription ends with a “complete” or a “failed” event from Ditto, or with an “cancel” command from the client. Ditto sends a “complete” event when all pages of the search results are delivered to the client.

Failed

A search subscription ends with a “complete” or a “failed” event from Ditto, or with an “cancel” command from the client. Ditto sends a “failed” event when an internal error occurred, or when the client breaches the reactive-streams specification. It is not possible to “request” more pages of the search results after a “failed” event.