Home » Modeling » EMF » EMF load resource behavior when the resource is a Database via Teneo
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414650 is a reply to message #414642] |
Thu, 15 November 2007 23:47   |
 Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
--------------090006030203070002050108
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Jason,
Comments below.
jason henriksen wrote:
> Hi All,
>
> I'm posting this here because I needed to make some EMF tweaks in
> order to make my project work the Teneo.
>
> The problem seems to be that EMF treats a Resource as something that
> can be completely loaded into memory. With my 57 million record table
> that plan fails in a spectacular way.
Ouch.
>
> I've built a patch that fixes this behavior by adding a method called
> setSuppressFullResourceLoading(boolean).
That sounds like a Teneo thing...
>
>
> Then I've got code like this in a few places:
>
> public EObject getEObject(URI uri, boolean loadOnDemand)
> {
> //--- JJH
> // if you trim the fragment and try to load this URL
> // you are going to download the entire database table!
> // getResource(uri.trimFragment(), loadOnDemand);
>
> Resource resource =null;
>
> //--- JJH
> if(EcoreUtil.getSuppressFullResourceLoading()){
> resource = getResource(uri, loadOnDemand);
> }
> else{
> resource = getResource(uri.trimFragment(), loadOnDemand);
> }
> //--- JJH
>
> if (resource != null)
> {
> //--- JJH
> if(resource.getContents().size()==1){
> return resource.getContents().get(0);
> }
> //--- JJH
> return resource.getEObject(uri.fragment());
> }
> else
> {
> return null;
> }
> }
>
> I need this on the 2.3 code since that is what our project is based on
> and the company is standardizing on RAD 7 (basically Eclipse 3.2.2)
> If you agree that this is an OK way around this problem, can I use the
> ..psf you posted recently to build a patch against the 2.3 branch?
I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
org.eclipse.emf.ecore.util.EcoreUtil or some other one? I assume you
are using ID's because if you are using fragment paths, it would
normally just walk down the branch that's used for that path. You
didn't say, but I assume this in ResourceSetImpl. Normally that looks
like this:
public EObject getEObject(URI uri, boolean loadOnDemand)
{
Resource resource = getResource(uri.trimFragment(), loadOnDemand);
if (resource != null)
{
return resource.getEObject(uri.fragment());
}
else
{
return null;
}
I imagine you could just provide your override in your own derived
ResourceSetImpl without changing the base implementation that everyone
uses. I can't say I really understand what getting a the resource with
the fragment still attached would accomplish. Is this more a question
for Teneo that Martin would normally answer on the EMFT newgroup?
>
> Thanks,
>
> Jason Henriksen
>
--------------090006030203070002050108
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Jason,<br>
<br>
Comments below.<br>
<br>
jason henriksen wrote:
<blockquote cite="mid:fhijbi$9c7$1@build.eclipse.org" type="cite">Hi
All,
<br>
<br>
I'm posting this here because I needed to make some EMF tweaks in order
to make my project work the Teneo.
<br>
<br>
The problem seems to be that EMF treats a Resource as something that
can be completely loaded into memory. With my 57 million record table
that plan fails in a spectacular way.
<br>
</blockquote>
Ouch.<br>
<blockquote cite="mid:fhijbi$9c7$1@build.eclipse.org" type="cite"><br>
I've built a patch that fixes this behavior by adding a method called
setSuppressFullResourceLoading(boolean).
<br>
</blockquote>
That sounds like a Teneo thing...<br>
<blockquote cite="mid:fhijbi$9c7$1@build.eclipse.org" type="cite"><br>
<br>
Then I've got code like this in a few places:
<br>
<br>
public EObject getEObject(URI uri, boolean loadOnDemand)
<br>
{
<br>
//--- JJH
<br>
// if you trim the fragment and try to load this URL
<br>
// you are going to download the entire database table!
<br>
// getResource(uri.trimFragment(), loadOnDemand);
<br>
<br>
Resource resource =null;
<br>
<br>
//--- JJH
<br>
if(EcoreUtil.getSuppressFullResourceLoading()){
<br>
resource = getResource(uri, loadOnDemand);
<br>
}
<br>
else{
<br>
resource = getResource(uri.trimFragment(), loadOnDemand);
<br>
}
<br>
//--- JJH
<br>
<br>
if (resource != null)
<br>
{
<br>
//--- JJH
<br>
if(resource.getContents().size()==1){
<br>
return resource.getContents().get(0);
<br>
}
<br>
//--- JJH
<br>
return resource.getEObject(uri.fragment());
<br>
}
<br>
else
<br>
{
<br>
return null;
<br>
}
<br>
}
<br>
<br>
I need this on the 2.3 code since that is what our project is based on
and the company is standardizing on RAD 7 (basically Eclipse 3.2.2)
<br>
If you agree that this is an OK way around this problem, can I use the
...psf you posted recently to build a patch against the 2.3 branch?
<br>
</blockquote>
I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
org.eclipse.emf.ecore.util.EcoreUtil or some other one? I assume you
are using ID's because if you are using fragment paths, it would
normally just walk down the branch that's used for that path. You
didn't say, but I assume this in ResourceSetImpl. Normally that looks
like this:<br>
<small><br>
</small>
<blockquote><small> public EObject getEObject(URI uri, boolean
loadOnDemand)</small><br>
<small> {</small><br>
<small> Resource resource = getResource(uri.trimFragment(),
loadOnDemand);</small><br>
<small> if (resource != null)</small><br>
<small> {</small><br>
<small> return resource.getEObject(uri.fragment());</small><br>
<small> }</small><br>
<small> else</small><br>
<small> {</small><br>
<small> return null;</small><br>
<small> }</small><br>
</blockquote>
I imagine you could just provide your override in your own derived
ResourceSetImpl without changing the base implementation that everyone
uses. I can't say I really understand what getting a the resource with
the fragment still attached would accomplish. Is this more a question
for Teneo that Martin would normally answer on the EMFT newgroup?<br>
<br>
<blockquote cite="mid:fhijbi$9c7$1@build.eclipse.org" type="cite"><br>
Thanks,
<br>
<br>
Jason Henriksen
<br>
<br>
</blockquote>
<br>
</body>
</html>
--------------090006030203070002050108--
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414652 is a reply to message #414650] |
Fri, 16 November 2007 00:19 |
Jason Henriksen
Messages: 231
Registered: July 2009 |
Senior Member |
|
|
Hi Ed,
> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
> org.eclipse.emf.ecore.util.EcoreUtil?
Yes, I've got my own patched version that I use for my project that adds
this method.
However, running my own patched version of things is not a fun and makes
my client a little nervous. I'd rather be running directly on your
releases and not have to manage my own special sauce if I can reasonably
avoid it.
> I assume you
> are using ID's because if you are using fragment paths, it would
> normally just walk down the branch that's used for that path.
The url's end up looking something like this:
hibernate://Consumer/12345 (pseudo code)
Teneo has protocol handlers that get the URL above and build a SQL query
out of it. I'm not fully clear on all the teneo aspects of this but
what I found from looking at the database is that if you clear the
fragment the query gets turned from
"Select * from Consumer where id=12345" // return 1 or 0 records.
into
"Select * from Consumer" // return 57 million records.
Or to put my own comments into the source:
public EObject getEObject(URI uri, boolean loadOnDemand)
{
//--- Load the entire resource (i.e. the whole table)
Resource resource = getResource(uri.trimFragment(), loadOnDemand);
if (resource != null){
//--- Out of the resource, get the item I asked for.
// (But never get here because I ran out of memory last line)
return resource.getEObject(uri.fragment());
}
else{
return null;
}
> I imagine you could just provide your override in your own derived
> ResourceSetImpl without changing the base implementation that everyone
> uses.
In fact, that is what I'm doing now. But Teneo can't really be used
against very large databases if EMF behaves this way, which is a bummer.
I can publish this as something for people to download and use as a
patch, but setting a flag to control the behavior in ECoreUtil seemed
like the better plan. If you don't ask for the load suppression,
everything acts as it does today. But if you run into the problem, you
can just set the flag and be happy.
> I can't say I really understand what getting a the resource with
> the fragment still attached would accomplish. Is this more a question
> for Teneo that Martin would normally answer on the EMFT newgroup?
I hope that makes sense. If I'm mis-understanding the behavior of that
code, I'd love to be brought up to speed.
Thanks,
Jason
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414674 is a reply to message #414652] |
Fri, 16 November 2007 11:37 |
Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
Jason,
Comments below.
jason henriksen wrote:
>
> Hi Ed,
>
>
>> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
>> org.eclipse.emf.ecore.util.EcoreUtil?
>
> Yes, I've got my own patched version that I use for my project that
> adds this method.
That explains it.
>
> However, running my own patched version of things is not a fun and
> makes my client a little nervous. I'd rather be running directly on
> your releases and not have to manage my own special sauce if I can
> reasonably avoid it.
Yep, it would make me nervous. I'm not sure why you can't just
specialize your own derived ResourceSetImpl.
>
>> I assume you are using ID's because if you are using fragment paths,
>> it would normally just walk down the branch that's used for that path.
>
> The url's end up looking something like this:
> hibernate://Consumer/12345 (pseudo code)
>
> Teneo has protocol handlers that get the URL above and build a SQL
> query out of it. I'm not fully clear on all the teneo aspects of this
> but what I found from looking at the database is that if you clear the
> fragment the query gets turned from
> "Select * from Consumer where id=12345" // return 1 or 0 records.
> into
> "Select * from Consumer" // return 57 million records.
That would be quite a few!
>
> Or to put my own comments into the source:
>
> public EObject getEObject(URI uri, boolean loadOnDemand)
> {
> //--- Load the entire resource (i.e. the whole table)
> Resource resource = getResource(uri.trimFragment(), loadOnDemand);
I guess because the objects are all a huge list in resource.getContents().
>
> if (resource != null){
> //--- Out of the resource, get the item I asked for.
> // (But never get here because I ran out of memory last line)
And Java only lets you have about 1.5G max for the heap. It's not very
scalable. :-P
> return resource.getEObject(uri.fragment());
> }
> else{
> return null;
> }
>
>> I imagine you could just provide your override in your own derived
>> ResourceSetImpl without changing the base implementation that
>> everyone uses.
>
> In fact, that is what I'm doing now. But Teneo can't really be used
> against very large databases if EMF behaves this way, which is a bummer.
It's not the size of the database I suppose, but rather the height of
the tables in it. Does Hibernate address these kinds of issues?
> I can publish this as something for people to download and use as a
> patch, but setting a flag to control the behavior in ECoreUtil seemed
> like the better plan. If you don't ask for the load suppression,
> everything acts as it does today. But if you run into the problem,
> you can just set the flag and be happy.
I would imagine that most applications have complete control over what
resource set is used, so I'm still not sure why providing an alternative
resource set implementation wouldn't suffice.
>
>> I can't say I really understand what getting a the resource with the
>> fragment still attached would accomplish. Is this more a question
>> for Teneo that Martin would normally answer on the EMFT newgroup?
>
> I hope that makes sense. If I'm mis-understanding the behavior of
> that code, I'd love to be brought up to speed.
It does, but it also seems to me that JPA/JDO/Hibernate would all suffer
from these same fundamental problems so I'm curious how they deal with
it. Do they somehow produce a query that returns a much smaller set of
the objects you're actually likely to use? I'll add the EMFT newsgroup
to the reply so Martin is more likely to see this and comment.
>
> Thanks,
>
> Jason
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414694 is a reply to message #414674] |
Fri, 16 November 2007 14:17 |
Martin Taal
Messages: 5468
Registered: July 2009 |
Senior Member |
|
|
Hi Jason,
I think I am confused....
Can you describe what how the system gets from the url:
hibernate://Consumer/12345 to the query and also for this one:
hibernate://Consumer to the query?
As far as I know the urifragment should be something like this:
Consumer|id=12345
gr. Martin
Ed Merks wrote:
> Jason,
>
> Comments below.
> jason henriksen wrote:
>>
>> Hi Ed,
>>
>>
>>> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
>>> org.eclipse.emf.ecore.util.EcoreUtil?
>>
>> Yes, I've got my own patched version that I use for my project that
>> adds this method.
> That explains it.
>>
>> However, running my own patched version of things is not a fun and
>> makes my client a little nervous. I'd rather be running directly on
>> your releases and not have to manage my own special sauce if I can
>> reasonably avoid it.
> Yep, it would make me nervous. I'm not sure why you can't just
> specialize your own derived ResourceSetImpl.
>>
>>> I assume you are using ID's because if you are using fragment paths,
>>> it would normally just walk down the branch that's used for that path.
>>
>> The url's end up looking something like this:
>> hibernate://Consumer/12345 (pseudo code)
>>
>> Teneo has protocol handlers that get the URL above and build a SQL
>> query out of it. I'm not fully clear on all the teneo aspects of this
>> but what I found from looking at the database is that if you clear the
>> fragment the query gets turned from
>> "Select * from Consumer where id=12345" // return 1 or 0 records.
>> into
>> "Select * from Consumer" // return 57 million records.
> That would be quite a few!
>>
>> Or to put my own comments into the source:
>>
>> public EObject getEObject(URI uri, boolean loadOnDemand)
>> {
>> //--- Load the entire resource (i.e. the whole table)
>> Resource resource = getResource(uri.trimFragment(), loadOnDemand);
> I guess because the objects are all a huge list in resource.getContents().
>>
>> if (resource != null){
>> //--- Out of the resource, get the item I asked for.
>> // (But never get here because I ran out of memory last line)
> And Java only lets you have about 1.5G max for the heap. It's not very
> scalable. :-P
>> return resource.getEObject(uri.fragment());
>> }
>> else{
>> return null;
>> }
>>
>>> I imagine you could just provide your override in your own derived
>>> ResourceSetImpl without changing the base implementation that
>>> everyone uses.
>>
>> In fact, that is what I'm doing now. But Teneo can't really be used
>> against very large databases if EMF behaves this way, which is a bummer.
> It's not the size of the database I suppose, but rather the height of
> the tables in it. Does Hibernate address these kinds of issues?
>> I can publish this as something for people to download and use as a
>> patch, but setting a flag to control the behavior in ECoreUtil seemed
>> like the better plan. If you don't ask for the load suppression,
>> everything acts as it does today. But if you run into the problem,
>> you can just set the flag and be happy.
> I would imagine that most applications have complete control over what
> resource set is used, so I'm still not sure why providing an alternative
> resource set implementation wouldn't suffice.
>>
>>> I can't say I really understand what getting a the resource with the
>>> fragment still attached would accomplish. Is this more a question
>>> for Teneo that Martin would normally answer on the EMFT newgroup?
>>
>> I hope that makes sense. If I'm mis-understanding the behavior of
>> that code, I'd love to be brought up to speed.
> It does, but it also seems to me that JPA/JDO/Hibernate would all suffer
> from these same fundamental problems so I'm curious how they deal with
> it. Do they somehow produce a query that returns a much smaller set of
> the objects you're actually likely to use? I'll add the EMFT newsgroup
> to the reply so Martin is more likely to see this and comment.
>>
>> Thanks,
>>
>> Jason
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414715 is a reply to message #414694] |
Mon, 19 November 2007 22:24 |
Jason Henriksen
Messages: 231
Registered: July 2009 |
Senior Member |
|
|
Hi Guys,
I'm slowly going back through my notes and figuring out why I did some
of this stuff. (I've been heads down for months and so I'm finally
coming up for air and post-mortems)
So the uri fragmen Martin gave was like this:
Consumer|id=12345
or in generic terms:
<object-type>|<primary-key-column>=<primary-key-id>
The problem with this is that my tables are legacy and do not have a
consistent primary key column. Also, my client did not allow me to make
the primary key column visible in the URI in case they decide to change
the name of that column later. (Changing that column strikes me as a
singularly bad but they're paying the bill so what they say goes)
The URI Style they wanted is
hibernate://#<object-type>.<primary-key-id>
This is enough information because I runtime I can inspect the object
type and determine which column is the primary-key-column
To do this I ended up making a couple of changes. Specifically I made a
URIMapper that would create a URI for a given Database object and when I
came across a URI of that type I could turn it back into a query.
Here's my code:
public URI getURIFor(EObject input)
{
HbDataStore dataStore = HbHelper.INSTANCE.getDataStore(
this.defaultDataStoreName );
//--- get the type name for the URI
String typeName = dataStore.getEntityNameStrategy().
toEntityName(input.eClass());
//--- figure out the field and value for the id
PersistentClass pc = dataStore.getPersistentClass( typeName );
Property p = pc.getIdentifierProperty();
String value = p.getGetter(input.getClass()).get(input).toString();
//--- return the abbreviated URI
URI result = URI.createURI("hibernate://#"+typeName+"."+value);
return result;
}
//--- Map my abbreviated URI into a more standard API for Teneo
public URI mapURI(URI input) {
HbDataStore dataStore =
HbHelper.INSTANCE.getDataStore(defaultDataStoreName );
String frag = input.fragment();
if(frag!=null){
String[] parts = frag.split("\\.");
if(parts.length==3){
return input;
}
if(parts.length==2){
PersistentClass pc = dataStore.getPersistentClass(parts[0]);
Property p = pc.getIdentifierProperty();
Column c = (Column)p.getPersistentClass().
getIdentifier().getColumnIterator().next();
// important use this for rehydrating the query SQL
String name = c.getCanonicalName();
URI result = input.trimFragment().
appendFragment(parts[0]+"."+name+"."+parts[1]);
return result;
}
else{
return input;
}
}
return input;
}
Does that make any sense? I then use the methods when I serialize and
de-serialize the objects so that the URI's look right. The problem for
me seemed to be that the URIs ran afoul of the trimFragment() code.
If you guys have a better solution to how I could get this kind of
effect without running into the fragment URI problem I'd love to hear
your thoughts. In the mean time I'm going to try building a separate
ResourceSetImpl as Ed suggested and see if I can get this to work more
cleanly that way.
The client wants to move away from these abbreviated URLs into REST
urls, but we need to stay in the 2.3 EMF branch. This code will get
re-implemented and I'd like to solve the problem cleanly and in a way
that minimizes me hack around with you guys' code.
Thanks for any thoughts,
Jason
PS: My client is successfully using EMF and Teneo to manage their large
consumer database and are very happy with the results and performance.
They're happy enough to expand this technique to other projects and
start slowly deploying it enterprise wide. The company Directors are
mentioning EMF and Teneo in very positive tones. Thanks guys!
Martin Taal wrote:
> Hi Jason,
> I think I am confused....
> Can you describe what how the system gets from the url:
> hibernate://Consumer/12345 to the query and also for this one:
> hibernate://Consumer to the query?
>
> As far as I know the urifragment should be something like this:
> Consumer|id=12345
>
> gr. Martin
>
> Ed Merks wrote:
>> Jason,
>>
>> Comments below.
>> jason henriksen wrote:
>>>
>>> Hi Ed,
>>>
>>>
>>>> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is that
>>>> org.eclipse.emf.ecore.util.EcoreUtil?
>>>
>>> Yes, I've got my own patched version that I use for my project that
>>> adds this method.
>> That explains it.
>>>
>>> However, running my own patched version of things is not a fun and
>>> makes my client a little nervous. I'd rather be running directly on
>>> your releases and not have to manage my own special sauce if I can
>>> reasonably avoid it.
>> Yep, it would make me nervous. I'm not sure why you can't just
>> specialize your own derived ResourceSetImpl.
>>>
>>>> I assume you are using ID's because if you are using fragment paths,
>>>> it would normally just walk down the branch that's used for that path.
>>>
>>> The url's end up looking something like this:
>>> hibernate://Consumer/12345 (pseudo code)
>>>
>>> Teneo has protocol handlers that get the URL above and build a SQL
>>> query out of it. I'm not fully clear on all the teneo aspects of
>>> this but what I found from looking at the database is that if you
>>> clear the fragment the query gets turned from
>>> "Select * from Consumer where id=12345" // return 1 or 0 records.
>>> into
>>> "Select * from Consumer" // return 57 million records.
>> That would be quite a few!
>>>
>>> Or to put my own comments into the source:
>>>
>>> public EObject getEObject(URI uri, boolean loadOnDemand)
>>> {
>>> //--- Load the entire resource (i.e. the whole table)
>>> Resource resource = getResource(uri.trimFragment(), loadOnDemand);
>> I guess because the objects are all a huge list in
>> resource.getContents().
>>>
>>> if (resource != null){
>>> //--- Out of the resource, get the item I asked for.
>>> // (But never get here because I ran out of memory last line)
>> And Java only lets you have about 1.5G max for the heap. It's not
>> very scalable. :-P
>>> return resource.getEObject(uri.fragment());
>>> }
>>> else{
>>> return null;
>>> }
>>>
>>>> I imagine you could just provide your override in your own derived
>>>> ResourceSetImpl without changing the base implementation that
>>>> everyone uses.
>>>
>>> In fact, that is what I'm doing now. But Teneo can't really be used
>>> against very large databases if EMF behaves this way, which is a bummer.
>> It's not the size of the database I suppose, but rather the height of
>> the tables in it. Does Hibernate address these kinds of issues?
>>> I can publish this as something for people to download and use as a
>>> patch, but setting a flag to control the behavior in ECoreUtil seemed
>>> like the better plan. If you don't ask for the load suppression,
>>> everything acts as it does today. But if you run into the problem,
>>> you can just set the flag and be happy.
>> I would imagine that most applications have complete control over what
>> resource set is used, so I'm still not sure why providing an
>> alternative resource set implementation wouldn't suffice.
>>>
>>>> I can't say I really understand what getting a the resource with the
>>>> fragment still attached would accomplish. Is this more a question
>>>> for Teneo that Martin would normally answer on the EMFT newgroup?
>>>
>>> I hope that makes sense. If I'm mis-understanding the behavior of
>>> that code, I'd love to be brought up to speed.
>> It does, but it also seems to me that JPA/JDO/Hibernate would all
>> suffer from these same fundamental problems so I'm curious how they
>> deal with it. Do they somehow produce a query that returns a much
>> smaller set of the objects you're actually likely to use? I'll add
>> the EMFT newsgroup to the reply so Martin is more likely to see this
>> and comment.
>>>
>>> Thanks,
>>>
>>> Jason
>
>
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414718 is a reply to message #414715] |
Tue, 20 November 2007 00:12 |
Ed Merks
Messages: 33140
Registered: July 2009 |
Senior Member |
|
|
Jason,
Commetns below.
jason henriksen wrote:
>
> Hi Guys,
>
> I'm slowly going back through my notes and figuring out why I did some
> of this stuff. (I've been heads down for months and so I'm finally
> coming up for air and post-mortems)
>
> So the uri fragmen Martin gave was like this:
> Consumer|id=12345
> or in generic terms:
> <object-type>|<primary-key-column>=<primary-key-id>
>
> The problem with this is that my tables are legacy and do not have a
> consistent primary key column. Also, my client did not allow me to
> make the primary key column visible in the URI in case they decide to
> change the name of that column later. (Changing that column strikes
> me as a singularly bad but they're paying the bill so what they say goes)
You can specialize a resource's getURIFragment and getEObject methods to
produce an consume any form of fragment you desire. I suppose there
might be limitation placed by the fact it's in a data base though...
>
> The URI Style they wanted is
> hibernate://#<object-type>.<primary-key-id>
> This is enough information because I runtime I can inspect the object
> type and determine which column is the primary-key-column
>
> To do this I ended up making a couple of changes. Specifically I made
> a URIMapper that would create a URI for a given Database object and
> when I came across a URI of that type I could turn it back into a query.
I would expect changes localized to the above to methods in a derived
ResourceImpl. URIMappings are for redirecting the non-fragment part of
the URIs.
> Here's my code:
>
>
> public URI getURIFor(EObject input)
> {
> HbDataStore dataStore = HbHelper.INSTANCE.getDataStore(
> this.defaultDataStoreName );
> //--- get the type name for the URI
> String typeName = dataStore.getEntityNameStrategy().
> toEntityName(input.eClass());
> //--- figure out the field and value for the id
> PersistentClass pc = dataStore.getPersistentClass( typeName );
> Property p = pc.getIdentifierProperty();
> String value = p.getGetter(input.getClass()).get(input).toString();
>
> //--- return the abbreviated URI
> URI result = URI.createURI("hibernate://#"+typeName+"."+value);
> return result;
> }
>
>
> //--- Map my abbreviated URI into a more standard API for Teneo
> public URI mapURI(URI input) {
> HbDataStore dataStore =
> HbHelper.INSTANCE.getDataStore(defaultDataStoreName );
>
> String frag = input.fragment();
> if(frag!=null){
> String[] parts = frag.split("\\.");
> if(parts.length==3){
> return input;
> }
> if(parts.length==2){
> PersistentClass pc = dataStore.getPersistentClass(parts[0]);
> Property p = pc.getIdentifierProperty();
> Column c = (Column)p.getPersistentClass().
> getIdentifier().getColumnIterator().next();
> // important use this for rehydrating the query SQL
> String name = c.getCanonicalName();
> URI result = input.trimFragment().
> appendFragment(parts[0]+"."+name+"."+parts[1]);
> return result;
> }
> else{
> return input;
> }
> }
> return input;
> }
>
> Does that make any sense? I then use the methods when I serialize and
> de-serialize the objects so that the URI's look right. The problem
> for me seemed to be that the URIs ran afoul of the trimFragment() code.
I think if you'd done it in the resource you'd have run into less problems.
>
> If you guys have a better solution to how I could get this kind of
> effect without running into the fragment URI problem I'd love to hear
> your thoughts. In the mean time I'm going to try building a separate
> ResourceSetImpl as Ed suggested and see if I can get this to work more
> cleanly that way.
>
> The client wants to move away from these abbreviated URLs into REST
> urls, but we need to stay in the 2.3 EMF branch. This code will get
> re-implemented and I'd like to solve the problem cleanly and in a way
> that minimizes me hack around with you guys' code.
>
> Thanks for any thoughts,
>
> Jason
>
> PS: My client is successfully using EMF and Teneo to manage their
> large consumer database and are very happy with the results and
> performance. They're happy enough to expand this technique to other
> projects and start slowly deploying it enterprise wide. The company
> Directors are mentioning EMF and Teneo in very positive tones. Thanks
> guys!
Hopefully by specializing the resource set's fragmentless-URI lookup and
the resource's fragment lookup you can get a solution that fits better
and still has the same characteristics that you seek.
>
>
>
>
> Martin Taal wrote:
>> Hi Jason,
>> I think I am confused....
>> Can you describe what how the system gets from the url:
>> hibernate://Consumer/12345 to the query and also for this one:
>> hibernate://Consumer to the query?
>>
>> As far as I know the urifragment should be something like this:
>> Consumer|id=12345
>>
>> gr. Martin
>>
>> Ed Merks wrote:
>>> Jason,
>>>
>>> Comments below.
>>> jason henriksen wrote:
>>>>
>>>> Hi Ed,
>>>>
>>>>
>>>>> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is
>>>>> that org.eclipse.emf.ecore.util.EcoreUtil?
>>>>
>>>> Yes, I've got my own patched version that I use for my project that
>>>> adds this method.
>>> That explains it.
>>>>
>>>> However, running my own patched version of things is not a fun and
>>>> makes my client a little nervous. I'd rather be running directly
>>>> on your releases and not have to manage my own special sauce if I
>>>> can reasonably avoid it.
>>> Yep, it would make me nervous. I'm not sure why you can't just
>>> specialize your own derived ResourceSetImpl.
>>>>
>>>>> I assume you are using ID's because if you are using fragment
>>>>> paths, it would normally just walk down the branch that's used for
>>>>> that path.
>>>>
>>>> The url's end up looking something like this:
>>>> hibernate://Consumer/12345 (pseudo code)
>>>>
>>>> Teneo has protocol handlers that get the URL above and build a SQL
>>>> query out of it. I'm not fully clear on all the teneo aspects of
>>>> this but what I found from looking at the database is that if you
>>>> clear the fragment the query gets turned from
>>>> "Select * from Consumer where id=12345" // return 1 or 0 records.
>>>> into
>>>> "Select * from Consumer" // return 57 million records.
>>> That would be quite a few!
>>>>
>>>> Or to put my own comments into the source:
>>>>
>>>> public EObject getEObject(URI uri, boolean loadOnDemand)
>>>> {
>>>> //--- Load the entire resource (i.e. the whole table)
>>>> Resource resource = getResource(uri.trimFragment(), loadOnDemand);
>>> I guess because the objects are all a huge list in
>>> resource.getContents().
>>>>
>>>> if (resource != null){
>>>> //--- Out of the resource, get the item I asked for.
>>>> // (But never get here because I ran out of memory last line)
>>> And Java only lets you have about 1.5G max for the heap. It's not
>>> very scalable. :-P
>>>> return resource.getEObject(uri.fragment());
>>>> }
>>>> else{
>>>> return null;
>>>> }
>>>>
>>>>> I imagine you could just provide your override in your own
>>>>> derived ResourceSetImpl without changing the base implementation
>>>>> that everyone uses.
>>>>
>>>> In fact, that is what I'm doing now. But Teneo can't really be
>>>> used against very large databases if EMF behaves this way, which is
>>>> a bummer.
>>> It's not the size of the database I suppose, but rather the height
>>> of the tables in it. Does Hibernate address these kinds of issues?
>>>> I can publish this as something for people to download and use as a
>>>> patch, but setting a flag to control the behavior in ECoreUtil
>>>> seemed like the better plan. If you don't ask for the load
>>>> suppression, everything acts as it does today. But if you run into
>>>> the problem, you can just set the flag and be happy.
>>> I would imagine that most applications have complete control over
>>> what resource set is used, so I'm still not sure why providing an
>>> alternative resource set implementation wouldn't suffice.
>>>>
>>>>> I can't say I really understand what getting a the resource with
>>>>> the fragment still attached would accomplish. Is this more a
>>>>> question for Teneo that Martin would normally answer on the EMFT
>>>>> newgroup?
>>>>
>>>> I hope that makes sense. If I'm mis-understanding the behavior of
>>>> that code, I'd love to be brought up to speed.
>>> It does, but it also seems to me that JPA/JDO/Hibernate would all
>>> suffer from these same fundamental problems so I'm curious how they
>>> deal with it. Do they somehow produce a query that returns a much
>>> smaller set of the objects you're actually likely to use? I'll add
>>> the EMFT newsgroup to the reply so Martin is more likely to see this
>>> and comment.
>>>>
>>>> Thanks,
>>>>
>>>> Jason
>>
>>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: EMF load resource behavior when the resource is a Database via Teneo [message #414722 is a reply to message #414718] |
Tue, 20 November 2007 09:10 |
Martin Taal
Messages: 5468
Registered: July 2009 |
Senior Member |
|
|
Hi Jason,
As Ed suggests I would override the HibernateResource and the getEObjectById/getURIFragment methods.
This seems also easier to me.
gr. Martin
Ed Merks wrote:
> Jason,
>
> Commetns below.
>
> jason henriksen wrote:
>>
>> Hi Guys,
>>
>> I'm slowly going back through my notes and figuring out why I did some
>> of this stuff. (I've been heads down for months and so I'm finally
>> coming up for air and post-mortems)
>>
>> So the uri fragmen Martin gave was like this:
>> Consumer|id=12345
>> or in generic terms:
>> <object-type>|<primary-key-column>=<primary-key-id>
>>
>> The problem with this is that my tables are legacy and do not have a
>> consistent primary key column. Also, my client did not allow me to
>> make the primary key column visible in the URI in case they decide to
>> change the name of that column later. (Changing that column strikes
>> me as a singularly bad but they're paying the bill so what they say goes)
> You can specialize a resource's getURIFragment and getEObject methods to
> produce an consume any form of fragment you desire. I suppose there
> might be limitation placed by the fact it's in a data base though...
>>
>> The URI Style they wanted is
>> hibernate://#<object-type>.<primary-key-id>
>> This is enough information because I runtime I can inspect the object
>> type and determine which column is the primary-key-column
>>
>> To do this I ended up making a couple of changes. Specifically I made
>> a URIMapper that would create a URI for a given Database object and
>> when I came across a URI of that type I could turn it back into a query.
> I would expect changes localized to the above to methods in a derived
> ResourceImpl. URIMappings are for redirecting the non-fragment part of
> the URIs.
>> Here's my code:
>>
>>
>> public URI getURIFor(EObject input)
>> {
>> HbDataStore dataStore = HbHelper.INSTANCE.getDataStore(
>> this.defaultDataStoreName );
>> //--- get the type name for the URI
>> String typeName = dataStore.getEntityNameStrategy().
>> toEntityName(input.eClass());
>> //--- figure out the field and value for the id
>> PersistentClass pc = dataStore.getPersistentClass( typeName );
>> Property p = pc.getIdentifierProperty();
>> String value = p.getGetter(input.getClass()).get(input).toString();
>>
>> //--- return the abbreviated URI
>> URI result = URI.createURI("hibernate://#"+typeName+"."+value);
>> return result;
>> }
>>
>>
>> //--- Map my abbreviated URI into a more standard API for Teneo
>> public URI mapURI(URI input) {
>> HbDataStore dataStore =
>> HbHelper.INSTANCE.getDataStore(defaultDataStoreName );
>>
>> String frag = input.fragment();
>> if(frag!=null){
>> String[] parts = frag.split("\\.");
>> if(parts.length==3){
>> return input;
>> }
>> if(parts.length==2){
>> PersistentClass pc = dataStore.getPersistentClass(parts[0]);
>> Property p = pc.getIdentifierProperty();
>> Column c = (Column)p.getPersistentClass().
>> getIdentifier().getColumnIterator().next();
>> // important use this for rehydrating the query SQL
>> String name = c.getCanonicalName();
>> URI result = input.trimFragment().
>> appendFragment(parts[0]+"."+name+"."+parts[1]);
>> return result;
>> }
>> else{
>> return input;
>> }
>> }
>> return input;
>> }
>>
>> Does that make any sense? I then use the methods when I serialize and
>> de-serialize the objects so that the URI's look right. The problem
>> for me seemed to be that the URIs ran afoul of the trimFragment() code.
> I think if you'd done it in the resource you'd have run into less problems.
>>
>> If you guys have a better solution to how I could get this kind of
>> effect without running into the fragment URI problem I'd love to hear
>> your thoughts. In the mean time I'm going to try building a separate
>> ResourceSetImpl as Ed suggested and see if I can get this to work more
>> cleanly that way.
>>
>> The client wants to move away from these abbreviated URLs into REST
>> urls, but we need to stay in the 2.3 EMF branch. This code will get
>> re-implemented and I'd like to solve the problem cleanly and in a way
>> that minimizes me hack around with you guys' code.
>>
>> Thanks for any thoughts,
>>
>> Jason
>>
>> PS: My client is successfully using EMF and Teneo to manage their
>> large consumer database and are very happy with the results and
>> performance. They're happy enough to expand this technique to other
>> projects and start slowly deploying it enterprise wide. The company
>> Directors are mentioning EMF and Teneo in very positive tones. Thanks
>> guys!
> Hopefully by specializing the resource set's fragmentless-URI lookup and
> the resource's fragment lookup you can get a solution that fits better
> and still has the same characteristics that you seek.
>>
>>
>>
>>
>> Martin Taal wrote:
>>> Hi Jason,
>>> I think I am confused....
>>> Can you describe what how the system gets from the url:
>>> hibernate://Consumer/12345 to the query and also for this one:
>>> hibernate://Consumer to the query?
>>>
>>> As far as I know the urifragment should be something like this:
>>> Consumer|id=12345
>>>
>>> gr. Martin
>>>
>>> Ed Merks wrote:
>>>> Jason,
>>>>
>>>> Comments below.
>>>> jason henriksen wrote:
>>>>>
>>>>> Hi Ed,
>>>>>
>>>>>
>>>>>> I don't recognize EcoreUtil.getSuppressFullResourceLoading() is
>>>>>> that org.eclipse.emf.ecore.util.EcoreUtil?
>>>>>
>>>>> Yes, I've got my own patched version that I use for my project that
>>>>> adds this method.
>>>> That explains it.
>>>>>
>>>>> However, running my own patched version of things is not a fun and
>>>>> makes my client a little nervous. I'd rather be running directly
>>>>> on your releases and not have to manage my own special sauce if I
>>>>> can reasonably avoid it.
>>>> Yep, it would make me nervous. I'm not sure why you can't just
>>>> specialize your own derived ResourceSetImpl.
>>>>>
>>>>>> I assume you are using ID's because if you are using fragment
>>>>>> paths, it would normally just walk down the branch that's used for
>>>>>> that path.
>>>>>
>>>>> The url's end up looking something like this:
>>>>> hibernate://Consumer/12345 (pseudo code)
>>>>>
>>>>> Teneo has protocol handlers that get the URL above and build a SQL
>>>>> query out of it. I'm not fully clear on all the teneo aspects of
>>>>> this but what I found from looking at the database is that if you
>>>>> clear the fragment the query gets turned from
>>>>> "Select * from Consumer where id=12345" // return 1 or 0 records.
>>>>> into
>>>>> "Select * from Consumer" // return 57 million records.
>>>> That would be quite a few!
>>>>>
>>>>> Or to put my own comments into the source:
>>>>>
>>>>> public EObject getEObject(URI uri, boolean loadOnDemand)
>>>>> {
>>>>> //--- Load the entire resource (i.e. the whole table)
>>>>> Resource resource = getResource(uri.trimFragment(), loadOnDemand);
>>>> I guess because the objects are all a huge list in
>>>> resource.getContents().
>>>>>
>>>>> if (resource != null){
>>>>> //--- Out of the resource, get the item I asked for.
>>>>> // (But never get here because I ran out of memory last line)
>>>> And Java only lets you have about 1.5G max for the heap. It's not
>>>> very scalable. :-P
>>>>> return resource.getEObject(uri.fragment());
>>>>> }
>>>>> else{
>>>>> return null;
>>>>> }
>>>>>
>>>>>> I imagine you could just provide your override in your own
>>>>>> derived ResourceSetImpl without changing the base implementation
>>>>>> that everyone uses.
>>>>>
>>>>> In fact, that is what I'm doing now. But Teneo can't really be
>>>>> used against very large databases if EMF behaves this way, which is
>>>>> a bummer.
>>>> It's not the size of the database I suppose, but rather the height
>>>> of the tables in it. Does Hibernate address these kinds of issues?
>>>>> I can publish this as something for people to download and use as a
>>>>> patch, but setting a flag to control the behavior in ECoreUtil
>>>>> seemed like the better plan. If you don't ask for the load
>>>>> suppression, everything acts as it does today. But if you run into
>>>>> the problem, you can just set the flag and be happy.
>>>> I would imagine that most applications have complete control over
>>>> what resource set is used, so I'm still not sure why providing an
>>>> alternative resource set implementation wouldn't suffice.
>>>>>
>>>>>> I can't say I really understand what getting a the resource with
>>>>>> the fragment still attached would accomplish. Is this more a
>>>>>> question for Teneo that Martin would normally answer on the EMFT
>>>>>> newgroup?
>>>>>
>>>>> I hope that makes sense. If I'm mis-understanding the behavior of
>>>>> that code, I'd love to be brought up to speed.
>>>> It does, but it also seems to me that JPA/JDO/Hibernate would all
>>>> suffer from these same fundamental problems so I'm curious how they
>>>> deal with it. Do they somehow produce a query that returns a much
>>>> smaller set of the objects you're actually likely to use? I'll add
>>>> the EMFT newsgroup to the reply so Martin is more likely to see this
>>>> and comment.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jason
>>>
>>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
Goto Forum:
Current Time: Thu Apr 25 12:03:44 GMT 2024
Powered by FUDForum. Page generated in 0.04244 seconds |
] 
Search
Help
Register
Login
Home
