Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Way to open and close repository thanks to an admin repository
[CDO] Way to open and close repository thanks to an admin repository [message #1780883] Mon, 29 January 2018 14:42 Go to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hello,

I am working on a project and we use CDO to store data.
In fact, we can have a lot of H2 repositories, managed dynamically thanks to an admin repository.

At one moment, we don't need to have all the repositories in memory in the CDO server. So, we would like to close the unused repositories to avoid consuming memory, but we don't want to delete them because they can be used later.
Is there a way to close the repository on the CDO server from the client ?

Moreover, to save time at the CDO server start, we would like to load the repositories to the CDO server memory only on user demand. Is there a way to do that ?

Thanks in advance.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1780886 is a reply to message #1780883] Mon, 29 January 2018 15:07 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Salut Cyril,

I'm not so familiar with admin repos because Christian Damus invented and implemented them. I'll ask him if he can remember these details...

Cheers
Eike

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1780912 is a reply to message #1780883] Mon, 29 January 2018 22:27 Go to previous messageGo to next message
Christian Damus is currently offline Christian DamusFriend
Messages: 1270
Registered: July 2009
Location: Canada
Senior Member
Hi, Cyril,

My memory of how the remote administration of CDO repositories works is faint, and it has been a long time since I did any work on the implementation. Happily, I did document it in some degree on my blog:

CDO Repository Administration

This includes a video demonstration of how the feature works in a Papyrus context from back in the Luna release, plus a link to the Bugzilla work item that has further references to the specific commits in Git that implement the feature.

I'm not sure whether the repository administration scenarios implemented in Luna support your needs. We provide for creation and deletion of repositories, but it sounds like you need additionally to be able to publish or hide repositories from users. That could be a new requirement that we don't support (yet). With luck, this reply provides enough pointers to glean the answers you need.

HTH,

Christian

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1780970 is a reply to message #1780912] Tue, 30 January 2018 16:11 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hi Eike and Christian,

Thanks for your answers.

Unfortunately, what I have read doesn't help me.
So I will try to modify directly the code to implement these new functionalities.
Would you have any idea/suggestion to help me ?
Would you be interested by the result ?

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1781064 is a reply to message #1780970] Thu, 01 February 2018 07:07 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Yes, please feel free to provide a patch or, even better, a Gerrit review. I'll be happy to review it. In both cases we'll need a bugzilla ;-)

I don't think it's an easy thing to achieve, though. You'll need to work on the CDOAdmin framework and protocol because it currently only works with (already) active repositories on the server. You'll need to introduce kind of placeholders for inactive repositories. And you'll probably need to listen to user sessions coming and going. And all this is not even the result of an in-depth analysis; more challenges are likely to come up, once you've started...

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1781172 is a reply to message #1781064] Fri, 02 February 2018 09:42 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hello,

I am new on contributing on eclipse projects. So I have followed this tuto :
http://www.vogella.com/tutorials/EclipseTargetPlatform/article.html
But I have always some issues on my dev environment. I think they comes from missing plugins.
Maybe there are some web pages that describe the way to configure the dev environment to contribute to CDO but I have not found them.
Have you some hints, please ?
Thanks in advance.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1781175 is a reply to message #1781172] Fri, 02 February 2018 10:09 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Hi Cyril,

I've updated our Contribution Guide to explain how we provision a complete CDO development environment: https://wiki.eclipse.org/CDO_Source_Installation

I hope that helps ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1782054 is a reply to message #1781175] Fri, 16 February 2018 14:10 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hi Eike,

Thanks for your answer. That helped me a lot. :)

For the moment, I have implemented the following methods in the CDOAdmin framework :
• public boolean activateRepository(String name, String type);
• public boolean deactivateRepository(String name, String type);
• public boolean isActivatedRepository(String name, String type);
For the inactive repositories, I get them from the catalog in DefaultCDORepositoryConfigurationManager.java.
For me, the implemented deactivation consists in calling the method LifecycleUtil.deactivate() on a IRepository.

I have made some tests and it seems to work.

But I have some more questions :
• One of my tests consists in doing sequentially 100 times: 1/ create a repository, 2/ add 100000 objects, 3/ deactivate the repository. Regarding the used heap memory of the CDO server, I expected that it would be stable after each deactivation but the amount of used memory is continiously growing until the heap max. Moreover, when the used heap memory is just below the max, the CDO server seems to slow down and there are more TimeoutException during commit or deactivation. Does it seem ok for you ? Are there some references or a cache to empty after one deactivation ? Do you have any idea about the elements that can stay in memory after the deactivation ?
• Do you have some advices regarding the modification of the API interfaces and plugin versions ?

Thanks in advance.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1782056 is a reply to message #1782054] Fri, 16 February 2018 14:56 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Cyril Chevé wrote on Fri, 16 February 2018 15:10

• One of my tests consists in doing sequentially 100 times: 1/ create a repository, 2/ add 100000 objects, 3/ deactivate the repository. Regarding the used heap memory of the CDO server, I expected that it would be stable after each deactivation but the amount of used memory is continiously growing until the heap max. Moreover, when the used heap memory is just below the max, the CDO server seems to slow down and there are more TimeoutException during commit or deactivation. Does it seem ok for you ? Are there some references or a cache to empty after one deactivation ? Do you have any idea about the elements that can stay in memory after the deactivation ?


To me that sounds like normal Java garbage collection characteristics. Please try to call java.lang.System.gc() after the deactivation of a repository. Does that change the characteristics?

Cyril Chevé wrote on Fri, 16 February 2018 15:10

• Do you have some advices regarding the modification of the API interfaces and plugin versions ?


The rules of https://wiki.eclipse.org/Evolving_Java-based_APIs apply to all public APIs and are automatically enforced by PDE API Tools (e.g., if you installed your workspace with Oomph). We're not planning for a major release, so API changes that require a major version increment are prohibited. In particular that means you must not add methods to public interfaces. Instead you should introduce extension interfaces, let the default implementations implement these extension interfaces, and let the client code do instanceof checks for (and downcasts to) these extension interfaces. With this trick you're only required to increment the minor versions of the affected API plugins (if that didn't already happen in the current release cycle). Makes sense?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1782126 is a reply to message #1782056] Mon, 19 February 2018 09:53 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hi,

Thanks for the information about the plugin versions. I will modify the code that I have done to follow these rules.

Regarding the consumption of the memory, I have tested using the GC command.
I add the result screenshots: with and without the GC command.

For me, it seems that some elements remain in memory even after each deactivation. Do you have any idea ?

Thanks.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1782132 is a reply to message #1782126] Mon, 19 February 2018 11:03 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Unfortunately the screenshots don't tell me a lot. I think you'd need to take a memory snapshot after the deactivation and inspect the GC roots. You'd need to find out whether the repository or any objects contained in it stay strongly reachable, and if so, through what exact reference path. Please note that the deactivation of a component doesn't necessarily free up all its memory (although in most cases that will happen). Make sure that you don't keep any string references to the repo or any of its contained components.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783164 is a reply to message #1782132] Thu, 08 March 2018 16:03 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hello,

I have done a memory snapshot as you said. I think I may have a lead on the reference on the deactivated repository.
In fact, it seems that there are references of my repository, through the DBStore and the DBStoreAccessor, in HorizontalNonAuditClassMapping$FeatureDeltaWriter. I have checked that the repository, the DBStore and the DBStoreAccesor are all in the INATIVE state.
What I understand is that when accessing to the data of the repository before it is deactivated, HorizontalNonAuditClassMapping classes are created and so its attribute deltaWriter which is of type ThreadLocal. But I don't find how this attribute is freed. Do you have any idea ?

Thanks.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783224 is a reply to message #1783164] Fri, 09 March 2018 11:22 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
A silly question: Do you release the handle to yourrepository (i.e., set it to null) after deactivating the repo?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783247 is a reply to message #1783224] Fri, 09 March 2018 15:08 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hi,

I am not sure to understand what you mean by "the handle".
Here is some code that I have added in CDOAdminServer :

  CDOAdminServerRepository repository = (CDOAdminServerRepository)getRepository(name);
    IRepository delegate = repository.getDelegate();

    // Do we have any connected users?
    if (hasConnections(delegate))
    {
      OM.LOG.warn("Impossible to deactivate repository " + name + " because of existing connections ");
      return false;
    }

    LifecycleUtil.deactivate(delegate);
    removeElement(repository);
    delegate = null;

But the hprof file (opened with Eclipse Memory Analyzer) seems to indicate that there are always references on the deactivated repositories from ThreadLocal objects. I cannot add the hprof file in attachment because of its size but it is similar to the screenshot of my precedent reply.

Cyril

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783725 is a reply to message #1783247] Fri, 16 March 2018 10:02 Go to previous messageGo to next message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hello,

I have done some modifications in the class HorizontalNonAuditClassMapping.java in order to be sure that this class keeps the references to the deactivated repositories.
So I have added:
• in HorizontalNonAuditClassMapping$FeatureDeltaWriter inner class, a method to set the field accessor to null,
• in the finally block in HorizontalNonAuditClassMapping.writeRevisionDelta() method, a call to this added method and a call to deltaWriter.remove().
With these modifications, the used heap memory is quite stable (see screenshot).

But I am not sure about the side effects of these modifications because accessor is also used in the visit() method. Does it seem good for you ?
Could the fact to remove the ThreadLocal have an impact on performances ?

Moreover, I have done the same kind of test but, instead of deactivating the repositories, I remove the repositories with the existing api. The heap memory behaves in the same way.

Thanks.

Cyril

[Updated on: Fri, 16 March 2018 16:18]

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783784 is a reply to message #1783725] Sat, 17 March 2018 07:29 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Hi Cyril,

It's hard to comment on code changes that I can't apply to my code base. What about submitting a Gerrit review with your proposed changes?! In general all heap space should be released after the following piece of code:

IRepository repository = createAndConfigureRepo();
repository.activate();

while (System.in.available() == 0) Thread.sleep(100);

repository.deactivate();
repository = null; // <== Give the garbage collector a chance to collect the repo and its associated structures.


Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Way to open and close repository thanks to an admin repository [message #1783853 is a reply to message #1783784] Mon, 19 March 2018 14:17 Go to previous message
Cyril Chevé is currently offline Cyril ChevéFriend
Messages: 14
Registered: October 2016
Junior Member
Hello Eike,

As it seems to be the same behavior with the deletion of a repository, I have logged an issue on bugzilla:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=532600

I have commited modifications on gerrit:
https://git.eclipse.org/r/119666

Tell me if all seem ok for you.

Thanks.

Cyril

Report message to a moderator

Previous Topic:EFactoryImpl.create query
Next Topic:EMF OCL Example cannot find org.eclipse.emf.ecore.codegen.ecore templates
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Fri Apr 19 21:13:49 GMT 2024

Powered by FUDForum. Page generated in 0.03954 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top

Sign up to our Newsletter

A fresh new issue delivered monthly