Re: [ecf-dev] Remote service question

[Scott Lewis <slewis@xxxxxxxxxxxxx>]

On 10/19/2014 11:18 AM, Wim Jongman wrote:

I guess I forgot what the question was ;)

On Sun, Oct 19, 2014 at 3:42 PM, Wim Jongman <wim.jongman@xxxxxxxxx> wrote:

Hi,

Suppose I want to stream data from a remote service to myself. I am now thinking that this is polling only.  I call the remote service to get any available data. However, if the remote service does not have data this leads to either locking (the remote service waits until data comes available) or unwanted network traffic (the remote service says "I have nothing"). I can use Async calls but this still requires me to poll the Future to find out if there is data available.

To continuously push data from the remote server when it becomes available I need to "reverse" e.g. I have to make a service available that the data provider can consume and it can then push whenever new data is ready without me having to poll (either through a Future or any other primitive way).

I wish I could consume a remote service and then set a callback that it could call whenever data is available.

Yes.   A few thoughts/ideas here below.   I wish I could say to you that your use case is trivially supported with OSGi Remote Services (any implementation...or specifically ECF's), but in truth I think that at the root is a slight mismatch between the 'remote procedure call' (RPC) model underneath OSGi Remote Services and what you are looking to do. 

First, with ECF's support for CompletableFuture, it is possible to easy to define an asynchronous/non-blocking remote method invocation, e.g. see [1].  But this isn't exactly what you are looking for, at least partially because it only completes *once*.  But it does provide what I think of as a very nice way to express asynchronous/non-blocking remote services.

Let's say we have declared a service API that looks like this:

void myMethodCall(String s, SomeCallback callback)

with the behavior being that 'callback can/is'  called potentially many times.  An example is all the Eclipse methods that have IProgressMonitor as the final argument in the method call.

The question is:  How to implement this remotely/in a distributed app?   As you know, it's possible to pass method arguments and return values either by reference or by value.   For most remote method calls passing by value is the norm (e.g. 's' argument in the above), but it is possible to implement remote references, allowing 'callback' instances/object references to be referred to/called from out of process (e.g. a remote service host).  

For example, Java's RMI is an implementation that has call by reference.   And it would be possible using RMI and/or other call-by-reference distribution to implement a provider that supports the above callback structure.  For example, an RMI-based provider could (pretty) easily be created for OSGi remote services.

But call-by-reference can/does end up being very complex in the real (distributed) world...for a number of reasons.  One reason is that unlike local/memory object references, remote references frequently fail and when they do it's often unclear what should happen.   Another reason is that remote references make garbage collection particularly complex (since not all object references are in local memory).   A third reason is that the network can introduce blocking that is problematic (for example, with Eclipse EFS if a remote file system is used the calls to the IProgressMonitor can block/be very slow...meaning that Eclipse can/does just 'freeze up' for long periods).   So in any event, the distributed systems world has kind of moved away from fully pass-by-reference for RPC systems,  because of these and other complexities.

Another common way to implement this sort of desired behavior is with event-driven approaches/asynchronous messaging.   One example is the ECF filetransfer API...another is the ECF datashare and shared object APIs, as well as JMS, vert.x, akka, zeromq, mqtt, javagroups, etc...all of the asynchronous/messaging-based middleware.

More than an implementation difference, I think what you are asking for implies a different sort of API than the 'normal' RPC, which I summarize as:    A thread calls a method synchronously and method returns a single value to same thread.   What I think you are really asking for is a simple way to express a behavior that isn't clearly expressed with a single method call (RPC), e.g. send a message and some data, return from this immediately/non-blocking, and receive one or more (remote) callbacks (with arbitrary data).    IMHO this is awkward for the remote procedure call/OSGi remote service model in general, and it gets really complicated when you factor in the distributed systems issues of (e.g.) partial failure (what should happen if one of n callbacks fails?).  And then things get *really* complex when we are talking about > 2 group members.

There are of course other ways to deal with this sort of behavior e.g. callbacks can be 'client-exported' remote services., the 'remote whiteboard pattern' like the chat example that you and Markus created, etc.   IMHO these approaches have their own complexities (e.g. how to dynamically associate the callback remote service instance with the service interface, how to do reliable discovery of the 'whole/associated thing', what happens when things fail because of network unreliability, etc.).  But even so, there could probably be more/better standardized API and/or meta-data specifically for doing the 'remote whiteboard pattern'.

It's my hope that we and others can/could eventually create standardized API (e.g. MultiCompletableFuture?, remote streams, standardized event/messaging models, replicated data structures, or collectors, etc  :) that would help deal with this and similar use cases.  ECF has much of the underlying support for different approaches (e.g. asynchronous messaging, replicated data models, etc), meaning that we can easily try out/implement a variety of approaches.

My $0.03.

Scott