[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [january-dev] remote services with IDatasets
|
On 6/8/2017 8:00 AM, Matt.Gerring@xxxxxxxxxxxxx wrote:
Hello,
I would like to replace some of the services which we hand code using json over activemq with remote services one day. A limitation is that it must be possible to runs scans from python, which is why we went that way. Therefore this might be helpful to us too :)
Yes I think it might. FWIW, these days it's not a big deal to create
new distribution providers with ECF's APIs [1], so if an existing
provider (e.g. the Py4j, protobuf one I describe in my note) is not
suitable, or you have a significant legacy codebase based upon an
existing transport+serialization, it would be possible to turn that
codebase into remote services by creating a new distribution
provider...or use multiple distribution providers.
There are many benefits of using remote services...e.g. service
dynamics, integration with DS, support for service versioning, fewer
worries about osgi classloading and serialization, async as well as sync
service invocation, and others. For more about this see [2].
Scott
[1]
https://wiki.eclipse.org/Tutorial:_Creating_Custom_Distribution_Providers
[2] https://wiki.eclipse.org/OSGi_Remote_Services_and_ECF
Matt
-----Original Message-----
From: january-dev-bounces@xxxxxxxxxxx [mailto:january-dev-bounces@xxxxxxxxxxx] On Behalf Of Peter.Chang@xxxxxxxxxxxxx
Sent: 08 June 2017 15:55
To: january-dev@xxxxxxxxxxx
Subject: Re: [january-dev] remote services with IDatasets
Hi Scott,
This is interesting to us as we are encountering some difficulties with RMI for Java-to-Java communications and OSGi class loading. Remote services look like a promising way to replace RMI.
Regards,
Peter
-----Original Message-----
From: january-dev-bounces@xxxxxxxxxxx [mailto:january-dev-bounces@xxxxxxxxxxx] On Behalf Of Scott Lewis
Sent: 07 June 2017 18:28
To: january-dev@xxxxxxxxxxx
Subject: [january-dev] remote services with IDatasets
Hi Folks,
Some of you may be familiar with ECF's impl of OSGi remote services
[1]. Remote services allows OSGi services (defined as one or more java
interfaces) to be made available/proxied outside of process. I believe
ICE is using it currently.
ECF's implementation of remote services is pluggable and allows different transports to be used (at service registration time) to remote
a service. We call these 'distribution providers' and now have quite
a few of them [2], from rest/jax-rs, to rosgi, to jms, to xmlrpc, to mqtt, to plain 'ol tcp and others. These distribution providers encapsulate both the wire protocol/transport (e.g. http) as well as the serialization scheme (e.g. json).
We have recently completed and are testing a distribution provider that
uses Py4j + Google's Protocol buffers [3]. After some experimentation,
I've found that protocol buffers (binary mode) are fairly performant and relatively space efficient on serialization and deserialization, and Py4j is fairly performant *if* parameters and return values are serialized to byte[] (and therefore passed by value) rather than passed by reference...reference is the default for py4j...except for byte
[]s. Pass-by-reference often causes many round trips between
(Python<->Java) in Py4j and this can quickly become a major performance problem with large amounts of data being exchanged (our observation).
So what's the point of this? Some of us are using [3] to provide a
modular, performant localhost interaction between OSGi runtimes and Python code...by using OSGi remote services and ECF's Py4j-based and protocol buffers-based distribution provider [3]. This interaction is bi-directional as OSGi services are bi-directional, but for our use case we have been focusing on java code calling into data analysis code implemented in Python.
An example using [4] is provided here [4]. This example has the OSGi service interface in java, and this service is *implemented* by the python code in the python-src directory [5]. The consumer bundle shows how the service gets injected (by DS) at runtime and then is used as with any other OSGi service.
One of the things I've been contemplating is to use protobuf to define the serialization/deserialization of January IDatasets...to and from numpy Datasets (or perhaps some subclass or metatype). This would allow performant and easy exchange of Dataset/IDatasets between Python and Java...which is what we (and perhaps others) are interested in.
I wanted to explain this work publicly to this list, as I won't be able
to attend the upcoming summit. I am, however, looking for possible
collaboration on parts of things and willing to make contributions to January if they are desired.
Thanks,
Scott
[1] https://wiki.eclipse.org/Eclipse_Communication_Framework_Project
[2] https://wiki.eclipse.org/Distribution_Providers
[3] https://github.com/ECF/Py4j-RemoteServicesProvider
[4] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/examples
[5]
https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/examples/org.eclipse.ecf.examples.protobuf.hello
_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/january-dev
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/january-dev
_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/january-dev
