Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Listening for Index-Changes
Listening for Index-Changes [message #1818805] Sun, 29 December 2019 14:29 Go to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
Hi,

I would like to build an index-based (with index = IResourceDescriptions) cache, that caches calculations for which I have to access the index (example: checking for global uniqueness of items) and that gets refreshed whenever the index changes. I have a working implementation of such da cache that organizes its contents in buckets with one bucket for each supported EClass. When a new A-item gets added to the index, the A bucket is refreshed but not the B-bucket.

All of this works, but the cache gets notified right now by my ResourceDescriptionStrategy (MyDSLResourceDescriptionStrategy). Obviously, this is not the right place, as descriptions will be created for other reasons than putting them to the index, so the cache currently gets refreshed more often than necessary.

My question: What is the right way to listen to changes in the index?

I assume that I would have to register the cache as an
IResourceDescription.Event.Listener
at an
IResourceDescription.Event.Source
, however I am at a loss at which concrete implementation I would have to register at the moment.

Currently, I use a helper class to access the index that works kinda as proposed in this book: https://www.amazon.de/Implementing-Domain-Specific-Languages-Xtext-Xtend/dp/1786464969/ref=sr_1_1?__mk_de_DE=%EF%BF%BDM%EF%BF%BD%25u017D%EF%BF%BD%EF%BF%BD&keywords=xtext&qid=1577629071&sr=8-1

It makes use of the visible Container-approach in order to implement class-path mechanisms and looks like this:
	@Inject
	var ResourceDescriptionsProvider rdp

	@Inject 
	IContainer.Manager cm

	/** Returns all eObject descriptions of a provided type that are visible from a provided object o. */
	private def searchIndexByName(EObject o, EClass type, QualifiedName fqn) {
		val tmp = o.getVisibleContainers.map [ c |
			c.getExportedObjects(type, fqn, false)
		]
		return tmp.flatten
	}


	/** Returns all containers that are visible from a provided object o. A container is, e.g., an Eclipse project or a class path item. */
	private def getVisibleContainers(EObject o) {
		if (o?.eResource === null)
			return emptyList
		val index = rdp.getResourceDescriptions(o.eResource)
		val rd = index.getResourceDescription(o.eResource.URI)
		if (rd === null) {
			// do something else
			return emptyList
		}
		cm.getVisibleContainers(rd, index)
	}


As you see, I use the RDProvider in order to return a matching IResourceDescriptions instance whenever accessing the index, so I currently have no "fixed" IResourceDescriptions I could register at. The RDProvider actually returns different kinds of IResourceDescriptions instances, so using the same fixed instance would change the behavior.

What am I missing here? Maybe the container-based approach is outdated or am I doing something else wrong? Ideally I would be notified whenever a new item is added to the index and be able to retrieve the concrete EClass of the added item.

Best regards,
Konrad

[Updated on: Sun, 29 December 2019 14:39]

Report message to a moderator

Re: Listening for Index-Changes [message #1818810 is a reply to message #1818805] Sun, 29 December 2019 20:30 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13628
Registered: July 2009
Senior Member
the most important question you did not answer:

eclipse? lsp? web?

in eclipse you should be able to inject IBuilderState/AbstractBuilderState directly or the persistent variant of IResourceDescriptions (ResourceDescriptionsProvider.PERSISTED_DESCRIPTIONS)


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Listening for Index-Changes [message #1818812 is a reply to message #1818810] Sun, 29 December 2019 23:18 Go to previous messageGo to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
Hi Chrstian,

Thanks for your fast reply. :)

MyDSL is running under Eclipse and in a standalone version, in which all files are compiled automatically, triggered by a Jenkins server.

What is the difference (under Eclipse / standalone Setup) for
a) injecting an IBuilderState,
b) using ResourceDescriptionsProvider.PERSISTED_DESCRIPTIONS, and
c) injecting an IResourceDescriptionsProvider and then creating an IResourceDescriptions for a given Resource by calling
rdp.getResourceDescriptions(Resource)

?

It feels as if I am missing a fundamental difference here...
Re: Listening for Index-Changes [message #1818829 is a reply to message #1818812] Mon, 30 December 2019 07:40 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13628
Registered: July 2009
Senior Member
the question is:

where is the code you have. where do you call this code from.
is it about build changes only or about dirty editors etc too.


the builder state is the impl on the eclipse side.
standalone and in eclipse you then have to inject the ResourceDescriptions with named PERSISTED_DESCRIPTIONS


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Listening for Index-Changes [message #1818835 is a reply to message #1818829] Mon, 30 December 2019 09:37 Go to previous messageGo to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
Hi Christian,

some information:
- I use my helper class (IndexHelper) to access the index whenever I need to access the index. This happens mainly under the following conditions:
1. When building a scope (I use a custom scope implementation which basically stores the qualified names and types of the contained items and fetches the corresponding IEObjectDescriptions lazily when it is asked for one by accessing the index.
2. During Validation (e.g. checking for cycles or global uniqueness)
3. During Code generation
- Concerning the dirty editors: I am not sure I follow you completely, but if would be sufficient for me when an editor that contains unsaved changes (= dirty editor, right?) "sees" only the state of saved files when accessing the index.

I tried your approach with the PERSISTED_DESCRIPTIONS (and deactivated my index caching implementation for now). This works if I save the files one by one, but if I clean-build the whole project, now no references get resolved. I assume this might be a problem with my replacement of the default scope implementation, I have not yet overwritten
DefaultResourceDescriptionManager.getImportedNames
, so maybe linked files do not get recompiled yet correctly when compiled by clean build without any specific order. I will check this next.

Is there any document where I could read about the differences between the different kinds of ResourceDescriptions or should I just dig into the code?

Thank you so much for your help!
Re: Listening for Index-Changes [message #1818838 is a reply to message #1818835] Mon, 30 December 2019 09:51 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13628
Registered: July 2009
Senior Member
yes but then again you need the container thing as the index as a hole does not know anything about projects.
and no: there is no documentation. you need to digg / look for examples


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Listening for Index-Changes [message #1818842 is a reply to message #1818838] Mon, 30 December 2019 10:11 Go to previous messageGo to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
OK, great, I will digg into it.

One last question: The problem regarding the clean-build seems to be that the PERSISTED_DESCRIPTIONS-index is initially empty when clean-building, but my scope implementation is backed by the index, so the scopes are empty (although they "know" the qualified names of its entries, the entries are simply not yet part of the index).

Any idea how I could best resolve this remaining issue?
Re: Listening for Index-Changes [message #1818843 is a reply to message #1818842] Mon, 30 December 2019 10:14 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13628
Registered: July 2009
Senior Member
i still have no clue what you actually cache and why you cache it. maybe you are solving the actual problem the wrong way.
besides the PERSISTED_DESCRIPTIONS there is also NAMED_BUILDER_SCOPE which might solve the problem but you are not encouraged to use it.


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Listening for Index-Changes [message #1818845 is a reply to message #1818843] Mon, 30 December 2019 10:35 Go to previous messageGo to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
Hi Christian,

thank you for going into such detail with me.

Caching is deactivated right now, so it is not a caching issue.

I would like to cache computationally expensive operations like, e.g., building a set of a given properties of a certain type of entity which (the property) needs to be unique. This set is checked when a new item is added. Without a cache, all items in the index would have to be traversed every time I need to build this set, as I have no clue whether or not another item has been added to the index. To avoid this recalculation, I would like to cache it, as items are rarely added in general.

The idea to change the scope implementation to one that lazily resolves its contents arose when adding Sirius to my project. Sirius force-resolves *all* references in the model when opening an aird-file, which leads to the creation of a lot of scopes and a lot of accesses to the index - in our case also the containers have to be asked every time. With a large model (~2000 files with a lot of references) this force-resolve alone took over a minute. However, most scopes just refer to the same globally available objects (which they get from the index) under a local name (driven by the local import statements).
It was an enormous performance improvement to thus cache the index-access (as it is the same for almost all scopes in those 2000 files) and just create the aliased IEObjectDescriptions in the lokal scope itself. Resolving went down from 73 s to 3 s.

So, that's where I am coming from. Would you say this is the wrong approach? Surely there is commonly the need to compute something based on the index, which you want to compute again only if the index changes?

[Updated on: Mon, 30 December 2019 10:41]

Report message to a moderator

Re: Listening for Index-Changes [message #1818847 is a reply to message #1818845] Mon, 30 December 2019 10:48 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13628
Registered: July 2009
Senior Member
yes but there is problem that the persistent index is updated after the build.
during the build the NAMED_BUILDER_SCOPE index is used

the question is: what is the operation you do on the index that is expensive e.g. getExportedObjectsByType?
in this case an optimization like this one might help too

https://github.com/eclipse/xtext-core/pull/1253#issuecomment-540941645


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Listening for Index-Changes [message #1818848 is a reply to message #1818847] Mon, 30 December 2019 11:29 Go to previous messageGo to next message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
Hi Christian,

so if I am calling
IResourceDescriptionsProvider.getResourceDescription(Resource)
it will return a NAMED_BUILDER_SCOPE when called during the build and a persistent index when called after the build?

Very intersting link. My cache-implementation looks very similar, but is not bound the the
EObjectDescriptionLookUp
. However, it also binds things to a EClasses by using a Multimap. In my case, the things could be anything computed by using the index, not only the EObjectDescriptions itself. Please find my code below for reference, if it helps you.

I also stumbled upon EObjectDescriptionLookUp during my performance analysis. So far, my understanding is that the performance issues are mainly caused by the following points:
- scopes are repeatatly calculated and always refer to the same IEObjectDescriptions, although they might appear under a different alias from file to file
- scopes are built eagerly, so all items will be retrieved no matter if actually accessed during linking or not
- index access in my implementation is relatively slow (when called repeatatly), mainly because of the Container-thingy I implemented in order to handle classpathes. It is slow, because on every call the following things have to happen:
1. The containers have to be fetched
2. The containers have to be queried by calling
getExportedObjects()
(that's where your EObjectDescriptionLookup2 optimization might be handy)
3. The
StateBasedContainer
implementation calls
isEmpty()
in its
getVisibleContainers
implementation, in my opinion without the need to do so. This is done at every index access in my case and might be a slow operation. I thus extended the StateBasedContainer as shown below (is just does not check for emptyness).

All together: Yes, I think an optimization like the proposed one might help, but I think it would not work as well as caching the whole computation.

I guess you disagree because you expect issues with maintaining the cache?
Do I also assume correctly that there is currently no known "best practise" for caching index-based computations?

Thanks a lot,
Konrad



class MyStateBasedContainerManager extends StateBasedContainerManager {
	
	
	override protected getVisibleContainers(List<String> handles, IResourceDescriptions resourceDescriptions) {
		if (handles.isEmpty())
			return Collections.emptyList();
		val List<IContainer> result = Lists.newArrayListWithExpectedSize(handles.size());
		for(String handle: handles) {
			val IContainer container = createContainer(handle, resourceDescriptions);
//			if (!container.isEmpty() || result.isEmpty())
//			println("Not calling is Empty()...")
			result.add(container);
		}
		return result;
	}
	
}


/** 
 * The default implementation of an IndexBackedCache. 
 */
@Singleton
class IndexBackedCacheImpl implements IndexBackedCache {
	/** Holds a list of all buckets that need to be cleared when an item of a certain type is added to the index. */
	val Multimap<EClass, IndexBackedCacheBucket> refreshMap = LinkedListMultimap.create;
	/** Maps the bucket that holds the cached values to its type. */
	val Map<EClass, IndexBackedCacheBucket> caches = newHashMap
	
	@Inject
	Provider<IndexBackedCacheBucket> cacheProv
	
	
	override void onNewElement(EClass typeOfNewElement) {
		val cs = getAffectedBuckets(typeOfNewElement)
		
		for (c : cs) {
			c.clear
		}
	}
	
	override <T> T get(EClass type, Object key, Provider<T> provider) {
		var cache = caches.get(type)
		if (cache === null) {
			cache = create(type);
		}
		
		cache.get(key, provider)
	}
	
	/** Creates a new Cache bucket of the provided type. */
	private def create(EClass type) {
		val rslt = cacheProv.get
		rslt.type = type
		
		caches.put(type, rslt)
		refreshMap.clear
		
		return rslt
	}
	
	/** Returns the cache buckets that should be refreshed when a new item of the provided type is added
	 * to the index. The returned list thus returns all cache buckets for any super types of the provided type. */
	private def getAffectedBuckets(EClass type) {
		val rslt = refreshMap.get(type)
		if (!rslt.nullOrEmpty)
			return rslt
			
		rebuildRefreshMapEntry(type)
		
		val r = refreshMap.get(type)
//		if (!r.nullOrEmpty)
//			println('''«type.name» -> Refresh «r.map[it.type.name].join(", ")»''')
		
		return r
	}
	
	/** Rebuilds the entry in the refresh map for the provided type.*/
	private def rebuildRefreshMapEntry(EClass type) {
		for (e : caches.entrySet) {
			val t = e.key
			val c = e.value
			
			if (t.isSuperTypeOf(type)) {
				refreshMap.put(type, c)
			}
		}
	}
	
	/** An inner bucket, which holds the actual cached values. */
	static class IndexBackedCacheBucket {
		val Map<Object, Object> items = newHashMap;
		
		@Accessors(#[PUBLIC_GETTER])
		var EClass type;
		
		protected new() {
			// do nothing
		}
	
		def <T> T get(Object key, Provider<T> provider) {
			var rslt = items.get(key)
			if (rslt !== null || items.containsKey(key)) {
				// cache hit
				return rslt as T
			} else {
				// cache miss
				val tmp = provider.get
				items.put(key, tmp)
				return tmp
			}
		}
		
		def clear() {
//			println('''Flushing Bucket «type.name»''')
			items.clear
		}
	}
}
Re: Listening for Index-Changes [message #1818882 is a reply to message #1818848] Tue, 31 December 2019 15:22 Go to previous message
Konrad Jünemann is currently offline Konrad JünemannFriend
Messages: 55
Registered: December 2018
Member
I guess that might have been a little bit too detailed for the forum.

Anyway, thanks again for your outstanding support and advice. I will dig a bit deeper and write again, if I find an interesting solution.

Best regards,
Konrad
Previous Topic:The future of Xtend ?
Next Topic:Building the index in Eclipse for non-Java projects?
Goto Forum:
  


Current Time: Tue Oct 20 01:24:34 GMT 2020

Powered by FUDForum. Page generated in 0.02098 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top