Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [science-iwg] About java-python integration
One other technology to point out for Python/Java (and other language) integration:   ECF Remote Services [1]

ECF Remote Services implements the OSGi Remote Services and Remote Service Admin specifications.

ECF has a pluggable distribution system, allowing remote services to be defined independently of serialization + transport and even programming language.   There are multiple distribution providers already implemented (e.g. tcp/java serialization, Google rpc/protocol buffers, xmlrpc, websockets, jax-rs impls, hazelcast, jms, xmpp, mqtt, others), and we've recently made it much easier to create new providers.  For example, I expect that it would be very simple to introduce a thrift-based provider and/or providers based upon analysisrpc or other rpc frameworks.    With sufficient resources, I can/would be willing to create new distribution providers as needed by this working group.

Remote services may be dynamically declared, implemented, discovered, and consumed/used without knowledge of or concern for the underlying serialization or networking protocols.  For an example of this please see this tutorial [1], and these articles [2,3,4].

For those already using OSGI services (in Eclipse, in servers, in devices, etc), remote services makes things very easy, and ECF's pluggable distribution providers allow the service declaration to be cleanly separated not only from the implementation, but also from any particular rpc/distribution system.   Since it extends the OSGi service registry in a standardized/specified manner, it interoperates seemlessly with DS/blueprint/injection frameworks [1].

Just to be clear, I don't see ECF Remote Services as competing with thrift, google rpc, xmlrpc, rest frameworks, mqtt, py4j, xml vs. json vs other serialization or any  inter-process communication system, but rather as a dynamic service layer above and independent of any particular rpc/distribution [5].   

Also, FWIW, I've been personally contributing to a Python-based implementation of OSGi + dependency injection + Remote Services + Remote Service Admin (soon) via the iPopo project [6].   It might also be useful for some on this list for Python/Java/other runtime integration.


[1] https://wiki.eclipse.org/Tutorial:_Using_Google_RPC/ProtocolBuffers_for_Remote_Services
[2] https://dzone.com/articles/transport-independent-remote-services  
[3] http://eclipseecf.blogspot.com/2016/06/polyglot-remote-services.html
[4] http://eclipseecf.blogspot.com/
[5] https://wiki.eclipse.org/OSGi_Remote_Services_and_ECF  and https://wiki.eclipse.org/Eclipse_Communication_Framework_Project
[6] http://ipopo.coderxpress.net/wiki/doku.php


On 6/13/2016 10:24 AM, Tracy Miranda wrote:
Hi Michele,

As you've observed, jpy was not a library we were aware of when we did the evaluation of java-python solutions. I'd be interested in any insights you have with regard to its approach. We have settled on Py4J as the java-python bridge which we are settling down with, building a house & having babies... :-)

Here is the criteria we used to evaluate the bridges at the time, it might be useful to you should you chose to do your own evaluation for specific use-cases.

Technical merit:
--------------------
What is overall architecture, who manages the interpreter lifecycle? (e.g. Py4J has separate Python/Java processes, uses sockets)
Does the implementation provide full support for C-libraries or conditional/partial support? (Py4J works with an independent Python process so full numpy support)
Does the library handle memory management (e.g. garbage collection on objects shared between Python and Java processes)?
What is the threading model between Java/Python
Related to threading, Is there support for indirect recursive calls between Java & Python? (Our recent work involved adding this feature to Py4J)
Is remote support possible or on the roadmap?
Does the library pass data and objects by call-by-ref or call-by-value (i.e. copying) or a combination of both.
Is there evidence of a good testsuite for the functionality?
Performance!!!! (this is a whole huge area in itself, so depends what you are optimizing for e.g individual calls vs batch, in due course we would like to formally measure & publish some figures for the solutions we are working on)

Non-technical considerations:
-------------------------------------
What is the project licensing?
Are there any platform support limitations (We need to support Windows, Mac, Linux)
What evidence is there the project is alive and thriving (look for releases in last 6 months, activity on forums)
Look for evidence of adoption (e.g Py4J is used by Apache Spark for PySpark, they also make contributions)
Look for other evidence of sustainability (any funding, companies, institutions involved)
Is the project open to external contributions?
Is there a roadmap?
Is there sufficient documentation?

There was no one perfect solution, but Py4J fulfilled many of the requirements and we are working on filling in the gaps. 
Let me know if any questions, otherwise I will (separately) send out an update on our Py4J activities especially after our productive week at EclipseCon France last week. 

Regards,
Tracy




On Mon, Jun 13, 2016 at 4:17 PM, Michele Gabusi <michele.gabusi@xxxxxx> wrote:

Hi,

thank you all for your answer. They have been very instructive indeed.

jpy is distributed with an Apache License from 0.8 version (on github); however, the previous release - which is available on pip right now - is GPL. Hopefully in future license should not be a serious problems. Thank you for having clarified the other ways to integrate java and python. Actually I was partially aware of them (thanks to the EclipseCon presentations), but now the picture is clearer.

Cheers,

Michele.   

Il 11/06/2016 09:41, Christian Pontesegger ha scritto:
It seems jpy (at least now) is using an Apache 2.0 license, so there would not be any license problems.

Jonah is currently implementing python support using Py4j and we already have a well working prototype. If jpy would have benefits over Py4j it would be worth considering an implementation for that, too.

Christian

On 06/10/2016 05:33 PM, Michele Gabusi wrote:
Hi,

recently I was looking around for a handy solution to setup bidirectional communication between Java and Python. I know that this is not a trivial issue. However, looking at the past EclipseCon presentations, I noticed that jpy libraries (licensed under GPL3) have not been mentioned.

http://jpy.readthedocs.io/en/latest/intro.html
https://pypi.python.org/pypi/jpy/

Do you know any reason that might discourage to implement them?
Do anyone has any experience with these libraries?

Cheers,

Michele.


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg


--
Logo Engineering

Michele Gabusi
Data Mining and Business Analytics
Big Data Competency Center
michele.gabusi@xxxxxx

Engineering Ingegneria Informatica
Corso Stati Uniti, 23/C - 35127 Padova - Italy
Tel. +39-049.8283549
Fax +39-049.8283569
www.eng.it


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg



_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg


Back to the top