Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Caching and explicit garbage collection
[CDO] Caching and explicit garbage collection [message #1799463] Fri, 07 December 2018 08:59 Go to next message
Robert Schulk is currently offline Robert SchulkFriend
Messages: 51
Registered: July 2015
Member
Our situation:
* Very large CDO archive which does not fit into RAM
* We want to recursively access every object

We access the objects recursively via the method eobject.eContents().
Even though we do not keep references to any objects and also tried using separate cdoviews which are closed from time to time, the RAM builds up.

What did work for us is opening new cdoSessions for larger chunks of data and then close the session again. But this does not seem a nice way to do it.

Is there any way to explicitly clear the cache of a session?
Please note: we are still using CDO 4.5.
Re: [CDO] Caching and explicit garbage collection [message #1799487 is a reply to message #1799463] Fri, 07 December 2018 16:16 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6444
Registered: July 2009
Senior Member
The CDOView caches CDOObjects, wich are generally small and cheap. The CDOSession caches CDORevisions, which are often big and expensive. Both caches are soft caches by default (the type of the revision cache can not be changed). That means, they try to keep their elements as long as possible. I'm not sure that an explicit call to gc() is supposed to impact the cache contents. But as soon as heap space is needed AND free heap space is not sufficient AND cache contents are reclaimable (i.e., ONLY softly referenced), then those cache contents should be garbage-collected. If you're saying that that's not the case for you then you must have seen an OutOfMemoryError. Right?

Re: [CDO] Caching and explicit garbage collection [message #1800146 is a reply to message #1799487] Thu, 20 December 2018 12:52 Go to previous messageGo to next message
Robert Schulk is currently offline Robert SchulkFriend
Messages: 51
Registered: July 2015
Member
So, I tried my best to get the out of memory error, but I could not ;).

The only problem that I see is, that the freed memory is always just a small chunk. This causes constant calls to the garbage collector and therefore very high cpu usage.

Is it somehow possible to increase the memory which is freed?
Re: [CDO] Caching and explicit garbage collection [message #1800853 is a reply to message #1800146] Wed, 09 January 2019 06:42 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6444
Registered: July 2009
Senior Member
I doubt that you could do better than a garbage collector. Perhaps you could try a different one? I have absolutely no experience with that, though.

There's also code at the beginning of CDOSessionImpl.refresh(Provider), which you could try to mimic:

    Map<CDOBranch, List<InternalCDOView>> views = new HashMap<CDOBranch, List<InternalCDOView>>();
    Map<CDOBranch, Map<CDOID, InternalCDORevision>> viewedRevisions = new HashMap<CDOBranch, Map<CDOID, InternalCDORevision>>();
    collectViewedRevisions(views, viewedRevisions);
    cleanupRevisionCache(viewedRevisions);


And:

  private void collectViewedRevisions(Map<CDOBranch, List<InternalCDOView>> views, Map<CDOBranch, Map<CDOID, InternalCDORevision>> viewedRevisions)
  {
    for (InternalCDOView view : getViews())
    {
      if (view.getTimeStamp() == CDOView.UNSPECIFIED_DATE)
      {
        CDOBranch branch = view.getBranch();
        Map<CDOID, InternalCDORevision> revisions = viewedRevisions.get(branch);
        boolean needNewMap = revisions == null;
        if (needNewMap)
        {
          revisions = CDOIDUtil.createMap();
        }

        view.collectViewedRevisions(revisions);
        if (!revisions.isEmpty())
        {
          List<InternalCDOView> list = views.get(branch);
          if (list == null)
          {
            list = new ArrayList<InternalCDOView>();
            views.put(branch, list);
          }

          list.add(view);

          if (needNewMap)
          {
            viewedRevisions.put(branch, revisions);
          }
        }
      }
    }
  }

  private void cleanupRevisionCache(Map<CDOBranch, Map<CDOID, InternalCDORevision>> viewedRevisions)
  {
    Set<InternalCDORevision> set = new HashSet<InternalCDORevision>();
    for (Map<CDOID, InternalCDORevision> revisions : viewedRevisions.values())
    {
      for (InternalCDORevision revision : revisions.values())
      {
        set.add(revision);
      }
    }

    InternalCDORevisionCache cache = getRevisionManager().getCache();
    List<CDORevision> currentRevisions = cache.getCurrentRevisions();
    for (CDORevision revision : currentRevisions)
    {
      if (!set.contains(revision))
      {
        cache.removeRevision(revision.getID(), revision);
      }
    }
  }


It's important to NOT remove a revision from the cache if it is currently being referenced from at least one EObject in at least one CDOView. This restriction may not apply if you can guarantee that no user is committing changes to the repository. In that case you could even try to configure a dummy revision cache in your session:

        IConnector connector = Net4jUtil.getConnector(container, connectorType, connectorDescription);

        CDONet4jSessionConfiguration config = CDONet4jUtil.createNet4jSessionConfiguration();
        config.setConnector(connector);
        config.setRepositoryName(repositoryName);
        config.setRevisionManager(CDORevisionUtil.createRevisionManager(CDORevisionCache.NOOP));




Previous Topic:Problem with load
Next Topic:[CDO] Object is unexpectedly non-writable
Goto Forum:
  


Current Time: Mon Sep 16 12:11:49 GMT 2019

Powered by FUDForum. Page generated in 0.02058 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top