Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Gemini » Dynamic persistence units(Any ideas?)
Dynamic persistence units [message #890509] Fri, 22 June 2012 09:10 Go to next message
Thomas Gillet is currently offline Thomas Gillet
Messages: 14
Registered: May 2011
Junior Member
Hi everybody,

I'm looking for ways to make persistence units more dynamic for a while now, and I must say I didn't find anything satisfying. I sincerely hope it's only because I missed the good things.
So meanwhile I post some thoughts here. (Maybe not the best place though, feel free to indicate places more suitable for general topics like this.)
EDIT: cross-posted on the osgi-dev mailing list with the same subject (see mail-archive.com).

Note: I use the word "module" bellow to reference any bundle or group of bundles implementing a given feature.

1. Multiple PUs

That's what we do now. Each module embeds a PU with the entities it needs. Easy. But:

- This leads to a lot of different PUs, sometime with very little data in it (some of them contains only a single entity with only one single field).

- No mapping or query across different PUs. So when several modules need to add properties on the same entity, each PU must handle its own "foreign key" column, stay synchronized when an entity is removed, etc... Also there is no way to query entities using properties from different modules.

I though for a while that EclipseLink's composite persistent units could save me, but the use of static <jar-file> tags in the persistence.xml make this feature unusable in this case.

2. Extending PU at runtime / deployment-time

The simple and obvious approach (well obvious for me at least) is to add a fragment with new entities to a PU bundle.
This would be a great feature and the exclude-unlisted-classes tag seems to be the solution, but it is not clear to me if it is supported or not by Gemini JPA. It's not working for me, and the only bug I found about this is bug #356509 about a missing test case (sorry I cannot post links), which just confused me more.

An even greater feature would be to add entities at runtime (for example using a white-board pattern to register new entities). But I don't see how to do it with EclipseLink (and I didn't check any other implementation for now).
Anyway, this approach would probably cause problems with load-time weaving.

3. Dynamic entities

As a last resort, I gave a shot to the dynamic entities available in EclipseLink.
I managed to create some kind of "dictionary entity", which is basically a map where you can add any property. So there is one entity type for each "important" object in my application, and then every module is free to add the properties it needs. A new column is created for each new property.

This approach could bring a lot of flexibility to my app, resolving the issues about cross-module mapping and query. But I'm still struggling with 2 main issues there:

- The persistence unit must be restarted after adding a property (no way to use properties added after the dynamic entity has been added to the session).

- The ALTER TABLE statement to add the new columns must be generated manually, including the column data definition (found no way to get it from EclipseLink, even though it does generate it at table creation time). Meaning I have an ugly switch to map each java class to the appropriate SQL statement. At this point I'm wondering if I still need to use JPA at all...

Then resulting tables are likely to be big and messy. Also only basic data types are handled. I guess support for collections or even relational mappings is possible but would require a fair amount of code. Unless maybe there is a way to construct a dynamic entity from an existing (annotated) class, so modules could provide entities or embeddables as property values.

4. JDO

Drastic change. But after reading some documentation, I think the JDO API is far more comprehensive than the poor JPA one.
Especially, it seems that support for programmatic persistence unit creation and for dynamic weaving is part of the standard API. Both features I would think essentials for a portable integration into an OSGi environment.

But I didn't try any actual implementation, so maybe it's only a good idea on paper.
If someone could shed some light on JDO dynamic capabilities compared to JPA, I would be eager to read it.


And that's it. End of the long speech.
Hope it could bring pertinent comments about all that. Maybe even solutions, who knows?

-- Thomas

[Updated on: Thu, 28 June 2012 11:49]

Report message to a moderator

Re: Dynamic persistence units [message #890902 is a reply to message #890509] Sat, 23 June 2012 07:27 Go to previous messageGo to next message
Thorsten Schlathölter is currently offline Thorsten Schlathölter
Messages: 208
Registered: February 2012
Location: Düsseldorf
Senior Member
Hi Thomas,
I don't know EclipseLink and its capability to deal with dynamics so I am not sure if the following will help. The following describes how I am currently dealing with dynamics in hibernate. (Actually the truth is that I am planning to use this approach. I have just implemented it in some test bundles to look if it really works - which it does. But I have no further experience with it. Comments are therefore welcome.)

Here is in brief what I do in order to deal with persistence/bundle dynamics in hibernate.
I have defined an interface IDomainClassContributor:

public interface IDomainClassContributor {
	List<Class> getDomainClasses();
}


This interface can be used by different bundles to provide a service that contributes persistent classes to my hibernate configuration. These services are injected into my SessionFactoryBean:

<bean id="sessionFactory" class="xxx.spring.ExtendableLocalSessionFactoryBean">
	   
   <property name="domainClassContributors">
      <osgi:set id="domainContributors" interface="xxx.spring.IDomainClassContributor" cardinality="0..N">
      </osgi:set>
   </property> 
		
   <property name="dataSource" ref="MYDataSource"/>
</bean>


Before a new hibernate sessionFactory is created, I add the contributed classes to the hibernate configuration. Care must be taken with respect to the classloader because hibernate will try to create these persistent classes by name in the course of sessionFactory creation. Since classes are contributed from different bundles hibernate will not be able to load all classes by default. I provide a special contextClassLoader to deal with this issue.

private synchronized SessionFactory newExtendedSessionFactory(Configuration config) {
	ClassLoader cl = Thread.currentThread().getContextClassLoader();
	LoadContextClassLoaderProxy lccl = new LoadContextClassLoaderProxy(cl);
	Thread.currentThread().setContextClassLoader(lccl);
	if (domainContributors!=null)
	{
		for (IDomainClassContributor contributor: domainContributors)
		{
			List<Class> classes = contributor.getDomainClasses();
			for (Class clss: classes)
			{
				lccl.addLookupEntry(clss);
				config.addClass(clss);
			}
		}
	}
			
	SessionFactory factory = super.newSessionFactory(config);

	Thread.currentThread().setContextClassLoader(cl);
	
	return factory;
}


I add a ServiceListener to deal with changes of the IDomainClassContributor services:

FrameworkUtil.getBundle(this.getClass()).getBundleContext().addServiceListener(new ServiceListener() {
				
   @Override
   public void serviceChanged(ServiceEvent arg0) {
   try {
      // This creates a new Configuration and a new SessionFacade
      afterPropertiesSet();
   } catch (Exception e) {
      e.printStackTrace();
   }
}
}, "("+Constants.OBJECTCLASS+"="+IDomainClassContributor.class.getName()+")");


Whenever the listener detects a change in IDomainClassContributor services it recreates the extendedSessionFactory.

The sessionFactory described above is injected into DAOs. In order to make sure that they always use the most current SessionFactory, I use a proxy.

@Override
protected synchronized SessionFactory newSessionFactory(Configuration config)
		throws HibernateException {

	extendedSessionFactory = newExtendedSessionFactory(config);
			
	/*
	 * Add a proxy to ensure that all accesses to the SessionFactory use the current extendedSessionFactory
	 */
	SessionFactory proxiedFactory = (SessionFactory) Proxy.newProxyInstance(getClass().getClassLoader(), 
			new Class[]{SessionFactory.class}, new InvocationHandler() {
				
				@Override
				public Object invoke(Object proxy, Method method, Object[] args)
						throws Throwable {
					return method.invoke(getExtendedSessionFactory(), args);
				}
			});
	return proxiedFactory;
}


Thats it. This way I can have different bundles which contribute to my persistence model. Due to the lister, this approach is also capable to handle the dynamics of bundle installation/start/stop.

Maybe this gives some new ideas.

Regards,
Thorsten
Re: Dynamic persistence units [message #890912 is a reply to message #890509] Sat, 23 June 2012 08:13 Go to previous messageGo to next message
Cristiano Gaviao is currently offline Cristiano Gaviao
Messages: 253
Registered: July 2009
Senior Member
Hi Thomas,

I think the main barrier for what you search for is the OSGi JPA Spec. It doesn't contain anything about persistence unit modularization, what I think it is sad. I think that Aries JPA has the same fault.

There is a bug opened in Gemini for this subject: 363195.

I've forked gemini jpa code, to make some experiments in contemplate this need, but the time that I've spent was not enough.

regards,

Cristiano
Re: Dynamic persistence units [message #891957 is a reply to message #890509] Tue, 26 June 2012 12:42 Go to previous messageGo to next message
Mike Keith is currently offline Mike Keith
Messages: 211
Registered: July 2009
Senior Member
Hi Thomas,

Perhaps you could be more specific about your use case? My biggest question is "At what stage are the entities known?" I assume that the persistence bundle is already created, but not necessarily installed or started, and you find out that you need to add some additional entities to the p-unit? Is it safe to assume the p-unit bundle is not yet active? Has it been installed/resolved? Is the plan to add the entities and then refresh the persistence bundle? Is it that you don't want to break open the bundle and add more classes and modify the persistence descriptor?

-Mike
Re: Dynamic persistence units [message #892509 is a reply to message #891957] Thu, 28 June 2012 09:43 Go to previous messageGo to next message
Thomas Gillet is currently offline Thomas Gillet
Messages: 14
Registered: May 2011
Junior Member
Mike, here is more details about my use case:

Our application is composed of basic bundles - generic and not likely to change often - and feature bundles - which provide more specific functionalities.
Then when a server is deployed for a new project, we install inside it the basic bundles, the feature bundles we need, and possibly create new project-specific bundles.

Ex:
We have a basic bundle with a PU, say, a VehicleManager providing storage for all vehicles handled by the application.
Later we add a feature bundle which needs to store data related to those vehicles, say, a CargoManager keeping trace of what cargo is linked to which vehicle.
=> We must be able to add this new feature without changing or even re-compiling the basic PU bundle.

So the main goal is code modularization. If adding a feature bundle means some PU bundle is restarted, it's ok. (Well, it's ok if the PU restarts by itself, but it's less ok if we had to manually restart it.)

Cristiano:

Yes I totally agree with you, this OSGi JPA spec is merely a bridge to run J2EE apps in OSGi, not something we can use for new developments.
That's why I was almost only speaking about provider specific extensions. Or JDO.

Thorsten:

We are not using Spring at all, only plain OSGi bundles, so your example is a little difficult to understand.
But at least it points out that Hibernate has the dynamic capabilities I'm looking for. So giving up EclipseLink could be an option (but then, I guess I would have to make the OSGi implementation myself...).

-- Thomas
Re: Dynamic persistence units [message #893079 is a reply to message #891957] Mon, 02 July 2012 10:06 Go to previous messageGo to next message
Cristiano Gaviao is currently offline Cristiano Gaviao
Messages: 253
Registered: July 2009
Senior Member
Hi Mike,

My use case is the following:

I have a kind of ERP system composed by lot of business modules: Financial, Human Resources, Sales, etc.

A customer can choose what modules will be used for they installation, and today I have to compound the Punit at development/build time.

What I am looking for is a way that a base Punit be created/extended at runtime based on the bundles and/or fragments installed that is referencing the same punit identification.

So, for example if I had a Financial persistence bundle with a punit named: "punit_financial", I would create another persistent bundle named "Brazilian Financial" that will add new classes to punit "punit_financial" just to fulfill financial localization needs. Being a bundle let it be referenced by other bundles too.
However, I could add a fragment to the main persistent bundle host that could add new queries or even new classes. But being a fragment, It can't be referenced by other bundles by itself, only using the host bundle.

In my view, activating a new persistent bundle which punit ID is already being used must refresh the Punit (I mean, all classes used to compound it).
Re: Dynamic persistence units [message #893556 is a reply to message #890509] Wed, 04 July 2012 10:40 Go to previous messageGo to next message
Mike Keith is currently offline Mike Keith
Messages: 211
Registered: July 2009
Senior Member
Ok, thanks Thomas and Cristiano. I understand where you are coming from and it is different from the way that I had envisioned allowing secondary persistence bundles. It sounds like you both want a way to add one or more bundles (containing additional entity classes) and join them to an existing persistence unit in an existing bundle. In this case the existing punit bundle does not reference the additional bundles, they just kind of add themselves in. Fragments certainly come to mind as being the most practical way to attach these additional bundles.

Cristiano has offered to look into an enhancement to solve this situation, so these additional use cases are certainly helpful. A solution to the fragment case is probably the appropriate direction. Bug 335698 is already open around the use of fragments.

Thanks,
-Mike
Re: Dynamic persistence units [message #893735 is a reply to message #893079] Thu, 05 July 2012 08:12 Go to previous messageGo to next message
Florian Pirchner is currently offline Florian Pirchner
Messages: 87
Registered: July 2009
Member

Hi Cristiano,

thats a perfect description of an ERP usecase. Modular ERP system require some kind of flexibility. Assembling prepared modules and building a new system based on these modules.
To share the same PU seems to be a key concept of assembling different modules.

Sure, the bundles (modules) do not reference each other, but if they are contained in the same PU, the can be treated as one module by the persistence layer. And operations can be processed in a common transaction. And also query issues can be done by using the same PU.

I am really looking forward to that kind of dynamics.

Cheers,
Florian
Re: Dynamic persistence units [message #895890 is a reply to message #893735] Mon, 16 July 2012 08:51 Go to previous messageGo to next message
Thomas Gillet is currently offline Thomas Gillet
Messages: 14
Registered: May 2011
Junior Member
Hello all,

Finally, I have been able to add entities dynamically in a PU, using bundle fragments.
Thanks to James Sutherland for his help (http://www.eclipse.org/forums/index.php/mv/msg/365174/890024/#msg_890024).
It's not a really pretty solution but should do the job until bug #335698 is resolved.

How to

The trick is to use the orm.xml file to add new entities, through the MetadataSource facility of EclipseLink.
This MetadataSource interface takes as inputs the mapping file as well as the class loader used to load it, so the file can be generated programmatically and returned through a custom class loader. The entities are added as (empty) entity tags.

Then to generate the file you have to create a list of all entities. I chose to scan the PU bundle for annotated classes, using bundle.findEntries() to get class files from the bundle and its fragments. Then I load those classes and keep the ones with the Entity annotation.

So now I can add fragments to my PU almost on the fly (the PU still needs to be restarted, but anyway it seems Virgo always restarts the host bundle when adding a fragment).

Drawbacks

This way of generating an XML just to be parsed by EclipseLink doesn't feel right.
There is a lot of methods in the XMLEntityMappings class that made me think the mappings could be created without using any XML, but I didn't manage to do it just by playing around with EntityAccessors and stuff. And I didn't find any documentation.

EDIT: there is indeed a way to bypass the XML file: http://www.eclipse.org/forums/index.php/m/896164/#msg_896164

Then, looking for Entity annotations requires to load ALL classes found in the bundle and its fragments. Maybe not very efficient.
An other solution would be to have fragments provide a list of the entities they contain (manifest header, property file, orm.xml or even persistence.xml file...).

About eclipselink.classloader

While writing the stuff above, I found the "eclipselink.classloader" property is unusable because Gemini internally overrides it (just filled bug #385170 about that).

I guess that if this property was available, entities could be provided by any bundle, not only fragments (unless there is some weaving problem, I don't know if Gemini weaves ALL annotated classes or only classes in PU bundles).

Also, if loading of persistence.xml could be intercepted, maybe this file could be generated instead of the orm.xml, avoiding the use of a MetadataSource. Would be simpler.


And that's it for today. Hope it can help someone.

-- Thomas

MetadataSource wiki: http://wiki.eclipse.org/EclipseLink/Examples/JPA/MetadataSource


[Updated on: Tue, 17 July 2012 09:32]

Report message to a moderator

Re: Dynamic persistence units [message #895962 is a reply to message #895890] Mon, 16 July 2012 12:28 Go to previous message
Mike Keith is currently offline Mike Keith
Messages: 211
Registered: July 2009
Senior Member
Thanks very much for reporting back on your success, Thomas.

We should be able to come up with a simpler solution for you when we address bug #335698.

-Mike
Previous Topic:Gemini DBAccess 1.1.0 Milestone 2
Next Topic:Virgo + Gemini DBAccess + Gemini Blueprint 1 = not possible
Goto Forum:
  


Current Time: Sat Aug 23 07:41:33 EDT 2014

Powered by FUDForum. Page generated in 0.02380 seconds