Home » Modeling » EMF » Auto unload - memory management
| Auto unload - memory management [message #425264] |
Fri, 21 November 2008 10:37  |
Eclipse User |
|
|
|
Originally posted by: wesendon.in.tum.de
Hi,
I'm working on a server whith a emf based versioning system (analog to
svn). Every version contains the changes to come to the next version and
every 10th version has the complete project state for safety reasons.
These project states aren't needed often (except the state of the head
version).
At the moment we have about 1000 versions and the server's memory need
lies at 1gb (main part of this are the project states). What I did is,
to put every projecstate in its own resource, which allows me to unload
them or not loading them in the first place. A thread wakes up every 10
minutes and unloads every unneeded loaded resource containing a project
state. But this isn't the nicest solution.
Now my question, is there any automatic mechanism to auto unload rarely
used resources in emf? Or can you somehow define emf objects as weak
referenced, so that java's gc can unload them when memory is needed?
Greetings,
Otto
|
|
|
| Re: Auto unload - memory management [message #425278 is a reply to message #425264] |
Fri, 21 November 2008 12:55  |
|
Otto,
Comment below...
Otto schrieb:
> Hi,
>
> I'm working on a server whith a emf based versioning system (analog to
> svn). Every version contains the changes to come to the next version
> and every 10th version has the complete project state for safety reasons.
Funny, I spent some time on this years's EclipseCon to discuss a similar
design with some guys (Max and Jonas) from the TU Munich. Are you
working on the same project?
> These project states aren't needed often (except the state of the head
> version).
>
> At the moment we have about 1000 versions and the server's memory need
> lies at 1gb (main part of this are the project states). What I did is,
> to put every projecstate in its own resource, which allows me to
> unload them or not loading them in the first place. A thread wakes up
> every 10 minutes and unloads every unneeded loaded resource containing
> a project state. But this isn't the nicest solution.
>
> Now my question, is there any automatic mechanism to auto unload
> rarely used resources in emf? Or can you somehow define emf objects as
> weak referenced, so that java's gc can unload them when memory is needed?
Let me copy a recent answer to a similar question:
CDO is a good choice to transparently address scalability issues with
EMF models. CDO internally represents your model instances without
having strong Java references. Combined with some smart caching
techniques on multiple levels there is nothing (including containment
and cross references) anymore that prevents single objects from being
reclaimed by the garbage collector when they are no longer referenced.
Lazy loading and unloading of objects happens completely automatic and
transparent. In addition CDO offers some features like partial list
loading and adaptive pre-fetching to further enhance the user
experience. We have evidence of models with > 4 GB sizes which can be
easily traversed on client machines with 512 MB main memory.
Cheers
/Eike
----
http://thegordian.blogspot.com
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: Auto unload - memory management [message #425280 is a reply to message #425278] |
Fri, 21 November 2008 14:11 |
| Eclipse User |
|
|
|
Originally posted by: wesendon.in.tum.de
Hi,
I'm from TU München too and working on their team ;). We can't use CDO
because we want to support disconnected operations. So, is there any
possibility inside emf to handle this issue? Any hints?
greetings,
Otto
Eike Stepper schrieb:
> Otto,
>
> Comment below...
>
>
>
> Otto schrieb:
>> Hi,
>>
>> I'm working on a server whith a emf based versioning system (analog to
>> svn). Every version contains the changes to come to the next version
>> and every 10th version has the complete project state for safety reasons.
> Funny, I spent some time on this years's EclipseCon to discuss a similar
> design with some guys (Max and Jonas) from the TU Munich. Are you
> working on the same project?
>
>> These project states aren't needed often (except the state of the head
>> version).
>>
>> At the moment we have about 1000 versions and the server's memory need
>> lies at 1gb (main part of this are the project states). What I did is,
>> to put every projecstate in its own resource, which allows me to
>> unload them or not loading them in the first place. A thread wakes up
>> every 10 minutes and unloads every unneeded loaded resource containing
>> a project state. But this isn't the nicest solution.
>>
>> Now my question, is there any automatic mechanism to auto unload
>> rarely used resources in emf? Or can you somehow define emf objects as
>> weak referenced, so that java's gc can unload them when memory is needed?
> Let me copy a recent answer to a similar question:
>
> CDO is a good choice to transparently address scalability issues with
> EMF models. CDO internally represents your model instances without
> having strong Java references. Combined with some smart caching
> techniques on multiple levels there is nothing (including containment
> and cross references) anymore that prevents single objects from being
> reclaimed by the garbage collector when they are no longer referenced.
> Lazy loading and unloading of objects happens completely automatic and
> transparent. In addition CDO offers some features like partial list
> loading and adaptive pre-fetching to further enhance the user
> experience. We have evidence of models with > 4 GB sizes which can be
> easily traversed on client machines with 512 MB main memory.
>
> Cheers
> /Eike
>
> ----
> http://thegordian.blogspot.com
>
|
|
|
| Re: Auto unload - memory management [message #425281 is a reply to message #425280] |
Fri, 21 November 2008 14:13 |
|
Otto schrieb:
> Hi,
>
> I'm from TU München too and working on their team ;). We can't use CDO
> because we want to support disconnected operations. So, is there any
> possibility inside emf to handle this issue? Any hints?
You should definitively look at EStore and EStoreEObjectImpl.
Cheers
/Eike
----
http://thegordian.blogspot.com
>
> greetings,
> Otto
>
> Eike Stepper schrieb:
>> Otto,
>>
>> Comment below...
>>
>>
>>
>> Otto schrieb:
>>> Hi,
>>>
>>> I'm working on a server whith a emf based versioning system (analog
>>> to svn). Every version contains the changes to come to the next
>>> version and every 10th version has the complete project state for
>>> safety reasons.
>> Funny, I spent some time on this years's EclipseCon to discuss a
>> similar design with some guys (Max and Jonas) from the TU Munich. Are
>> you working on the same project?
>>
>>> These project states aren't needed often (except the state of the
>>> head version).
>>>
>>> At the moment we have about 1000 versions and the server's memory
>>> need lies at 1gb (main part of this are the project states). What I
>>> did is, to put every projecstate in its own resource, which allows
>>> me to unload them or not loading them in the first place. A thread
>>> wakes up every 10 minutes and unloads every unneeded loaded resource
>>> containing a project state. But this isn't the nicest solution.
>>>
>>> Now my question, is there any automatic mechanism to auto unload
>>> rarely used resources in emf? Or can you somehow define emf objects
>>> as weak referenced, so that java's gc can unload them when memory is
>>> needed?
>> Let me copy a recent answer to a similar question:
>>
>> CDO is a good choice to transparently address scalability issues with
>> EMF models. CDO internally represents your model instances without
>> having strong Java references. Combined with some smart caching
>> techniques on multiple levels there is nothing (including containment
>> and cross references) anymore that prevents single objects from being
>> reclaimed by the garbage collector when they are no longer
>> referenced. Lazy loading and unloading of objects happens completely
>> automatic and transparent. In addition CDO offers some features like
>> partial list loading and adaptive pre-fetching to further enhance the
>> user experience. We have evidence of models with > 4 GB sizes which
>> can be easily traversed on client machines with 512 MB main memory.
>>
>> Cheers
>> /Eike
>>
>> ----
>> http://thegordian.blogspot.com
>>
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: Auto unload - memory management [message #425285 is a reply to message #425281] |
Fri, 21 November 2008 14:43 |
| Eclipse User |
|
|
|
Originally posted by: wesendon.in.tum.de
Hi,
the problem is we can't change the genmodel property feature delegation
because we use an existing model. And also it seems to me that
implementing an own eStore isn't an easy but error-prone task. Isn't
there a intelligent way to use the unload method with our existing project?
greetings,
Otto
Eike Stepper schrieb:
> Otto schrieb:
>> Hi,
>>
>> I'm from TU München too and working on their team ;). We can't use CDO
>> because we want to support disconnected operations. So, is there any
>> possibility inside emf to handle this issue? Any hints?
> You should definitively look at EStore and EStoreEObjectImpl.
>
> Cheers
> /Eike
>
> ----
> http://thegordian.blogspot.com
>
>
>>
>> greetings,
>> Otto
>>
>> Eike Stepper schrieb:
>>> Otto,
>>>
>>> Comment below...
>>>
>>>
>>>
>>> Otto schrieb:
>>>> Hi,
>>>>
>>>> I'm working on a server whith a emf based versioning system (analog
>>>> to svn). Every version contains the changes to come to the next
>>>> version and every 10th version has the complete project state for
>>>> safety reasons.
>>> Funny, I spent some time on this years's EclipseCon to discuss a
>>> similar design with some guys (Max and Jonas) from the TU Munich. Are
>>> you working on the same project?
>>>
>>>> These project states aren't needed often (except the state of the
>>>> head version).
>>>>
>>>> At the moment we have about 1000 versions and the server's memory
>>>> need lies at 1gb (main part of this are the project states). What I
>>>> did is, to put every projecstate in its own resource, which allows
>>>> me to unload them or not loading them in the first place. A thread
>>>> wakes up every 10 minutes and unloads every unneeded loaded resource
>>>> containing a project state. But this isn't the nicest solution.
>>>>
>>>> Now my question, is there any automatic mechanism to auto unload
>>>> rarely used resources in emf? Or can you somehow define emf objects
>>>> as weak referenced, so that java's gc can unload them when memory is
>>>> needed?
>>> Let me copy a recent answer to a similar question:
>>>
>>> CDO is a good choice to transparently address scalability issues with
>>> EMF models. CDO internally represents your model instances without
>>> having strong Java references. Combined with some smart caching
>>> techniques on multiple levels there is nothing (including containment
>>> and cross references) anymore that prevents single objects from being
>>> reclaimed by the garbage collector when they are no longer
>>> referenced. Lazy loading and unloading of objects happens completely
>>> automatic and transparent. In addition CDO offers some features like
>>> partial list loading and adaptive pre-fetching to further enhance the
>>> user experience. We have evidence of models with > 4 GB sizes which
>>> can be easily traversed on client machines with 512 MB main memory.
>>>
>>> Cheers
>>> /Eike
>>>
>>> ----
>>> http://thegordian.blogspot.com
>>>
|
|
|
| Re: Auto unload - memory management [message #425286 is a reply to message #425285] |
Fri, 21 November 2008 14:54 |
|
Otto schrieb:
> Hi,
>
> the problem is we can't change the genmodel property feature
> delegation because we use an existing model.
Then I don't see a way to achieve it transparently ;-(
With https://bugs.eclipse.org/bugs/show_bug.cgi?id=247130 there might
come a chance, though.
> And also it seems to me that implementing an own eStore isn't an easy
> but error-prone task.
Agreed. Usually the major part of overall complexity is caused by
optimizing the characteristics rather than adding functions.
> Isn't there a intelligent way to use the unload method with our
> existing project?
While I'm not sure whether it is intelligent or less error-prone than
the other way you could try to use Resource::unload() in combination
with unsetting all features (excluding inter-resource cross references
with inverses)...
Cheers
/Eike
----
http://thegordian.blogspot.com
>
> greetings,
> Otto
>
>
>
> Eike Stepper schrieb:
>> Otto schrieb:
>>> Hi,
>>>
>>> I'm from TU München too and working on their team ;). We can't use
>>> CDO because we want to support disconnected operations. So, is there
>>> any possibility inside emf to handle this issue? Any hints?
>> You should definitively look at EStore and EStoreEObjectImpl.
>>
>> Cheers
>> /Eike
>>
>> ----
>> http://thegordian.blogspot.com
>>
>>
>>>
>>> greetings,
>>> Otto
>>>
>>> Eike Stepper schrieb:
>>>> Otto,
>>>>
>>>> Comment below...
>>>>
>>>>
>>>>
>>>> Otto schrieb:
>>>>> Hi,
>>>>>
>>>>> I'm working on a server whith a emf based versioning system
>>>>> (analog to svn). Every version contains the changes to come to the
>>>>> next version and every 10th version has the complete project state
>>>>> for safety reasons.
>>>> Funny, I spent some time on this years's EclipseCon to discuss a
>>>> similar design with some guys (Max and Jonas) from the TU Munich.
>>>> Are you working on the same project?
>>>>
>>>>> These project states aren't needed often (except the state of the
>>>>> head version).
>>>>>
>>>>> At the moment we have about 1000 versions and the server's memory
>>>>> need lies at 1gb (main part of this are the project states). What
>>>>> I did is, to put every projecstate in its own resource, which
>>>>> allows me to unload them or not loading them in the first place. A
>>>>> thread wakes up every 10 minutes and unloads every unneeded loaded
>>>>> resource containing a project state. But this isn't the nicest
>>>>> solution.
>>>>>
>>>>> Now my question, is there any automatic mechanism to auto unload
>>>>> rarely used resources in emf? Or can you somehow define emf
>>>>> objects as weak referenced, so that java's gc can unload them when
>>>>> memory is needed?
>>>> Let me copy a recent answer to a similar question:
>>>>
>>>> CDO is a good choice to transparently address scalability issues
>>>> with EMF models. CDO internally represents your model instances
>>>> without having strong Java references. Combined with some smart
>>>> caching techniques on multiple levels there is nothing (including
>>>> containment and cross references) anymore that prevents single
>>>> objects from being reclaimed by the garbage collector when they are
>>>> no longer referenced. Lazy loading and unloading of objects happens
>>>> completely automatic and transparent. In addition CDO offers some
>>>> features like partial list loading and adaptive pre-fetching to
>>>> further enhance the user experience. We have evidence of models
>>>> with > 4 GB sizes which can be easily traversed on client machines
>>>> with 512 MB main memory.
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>> ----
>>>> http://thegordian.blogspot.com
>>>>
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Goto Forum:
Current Time: Fri Apr 19 23:52:24 GMT 2024
Powered by FUDForum. Page generated in 0.03267 seconds |
] 
Search
Help
Register
Login
Home
