Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Performance issue when doing index of resourceset
Performance issue when doing index of resourceset [message #1843126] Fri, 16 July 2021 10:28 Go to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
Hi,
I'm facing a performance issue with a single DSL file. We have added a second DSL, one of this DSL file contains around 3300 lines.
Example with reduced content:
DQRules-configuration test {
	
	generateDqRulesOnlyForTargetEntites

	DqRules-Id-Mapping {

		DqRules-Id-Mapping-From-Legacy-Slf {
			"MONITOR.xxx.001" --> "MONITOR.yyy.001"
			"MONITOR.xxx.002" --> "MONITOR.yyy.002"

		}
		
		DqRules-Id-Mapping-New{
			
		}
	}

	DQRules-Deactivation-Data_Exists {
	}
	
	DQRules-Deactivation-Validity{
	}
	
	DQRules-Deactivation-Null_Check {
        "model.abc.test1"
        "model.abc.test2"

	}

	DQRules-Deactivation-Uniqueness {


	}

	DQRules-Deactivation-Ref_Integrity {
        "model.cde.test1"
        "model.cde.test2"

	}

}

The DSL grammar for this file see below.
Now we are facing the performance issue when we doing the indexing (see the code below) with this new DSL file. It took now much more time to do it. Even the lines in this file were commented out, no change with the performance. Only when the content of this file be reduced, then the indexing performance get improved.
But we don't have issue with the file of another DSL, which also contains a lot of lines.
Is this performance because of bad grammar design? Or what else can be the reason?
		
List<IResourceDescription> descriptions = newArrayList();
		for (Resource resource : resourceSet.getResources()) {
			IResourceServiceProvider serviceProvider = serviceRegistry.getResourceServiceProvider(resource.getURI());
			IResourceDescription description = serviceProvider.getResourceDescriptionManager()
					.getResourceDescription(resource);
			descriptions.add(description);
		}
		index = new ResourceDescriptionsData(descriptions);
		EcoreUtil.resolvAll(resourceSet)
 ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(resourceSet, index);


class ConfigurationDqRules extends Configuration {
	Boolean generateDqRulesOnlyForTargetEntites
	contains DqRulesIdMappingList dqRulesIdMappingList
	contains DataExistsDqRuleDeactivation dataExistsDqRuleDeactivation
	contains ValidityDqRuleDeactivation validityDqRuleDeactivation
	contains NullCheckDqRuleDeactivation nullCheckDqRuleDeactivation
	contains UniquenessDqRuleDeactivation uniquenessDqRuleDeactivation
	contains RefIntegrityDqRuleDeactivation refIntegrityDqRuleDeactivation
}


class DqRulesIdMappingList {
	contains DqRulesIdMappingFromLegacySlf dqRulesIdMappingFromLegacySlf
	contains DqRulesIdMappingNew dqRulesIdMappingNew
}

class DqRulesIdMappingFromLegacySlf {
	contains DqRulesIdMapping [] dqRulesIdMapping
}

class DqRulesIdMappingNew {
	contains DqRulesIdMapping [] dqRulesIdMapping
}

class DqRulesIdMapping {
	String dqRuleId
	String mapTo
}

class DataExistsDqRuleDeactivation {
	contains  DataExistsDqRule [] dataExistsDqRule
}

class DataExistsDqRule {
	String entity
}

class ValidityDqRuleDeactivation{
	contains ValidityDqRule [] ValidityDqRule
}

class ValidityDqRule {
	String entityProperty
}

class NullCheckDqRuleDeactivation {
	contains NullCheckDqRule [] nullCheckDqRule
}

class NullCheckDqRule {
	String entityProperty
}

class UniquenessDqRuleDeactivation {
	contains UniquenessDqRule [] uniquenessDqRule
}

class UniquenessDqRule {
	String entity
}


[Updated on: Fri, 16 July 2021 11:02]

Report message to a moderator

Re: Performance issue when doing index of resourceset [message #1843127 is a reply to message #1843126] Fri, 16 July 2021 10:30 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14026
Registered: July 2009
Senior Member
hi, i recommend you to profile and share the results.

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Performance issue when doing index of resourceset [message #1843131 is a reply to message #1843127] Fri, 16 July 2021 11:04 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
hi Christian,

thanks for the quick reply.
What do you mean exactly use the profile and share the result?
Re: Performance issue when doing index of resourceset [message #1843132 is a reply to message #1843131] Fri, 16 July 2021 11:13 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
Here is the performance compare: we have around 5000 files of DSL1 + 1 file of DSL2 (which cause the performance issue)
Situation 1: DSL2 file with less than 50 lines, finish indexing with all files took 71445ms
Situation 2: DSL2 file with 3300 lines, finish indexing with all files took 265773ms
Situation 3: DSL2 file with 3300 lines but comment out 3250 lines, finish indexing with all files took 161888ms

and main part of the code for the standalone mode look like below
		logger.info('''init model''')
		val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
		logger.info('''done.''')
		
		logger.info('''init conf''')
		new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
		logger.info('''done.''')
		
		val standalone = modelInjector.getInstance(StandaloneExecutor)
		
		logger.info('''run standalone''')
		
		logger.info('''load model resources into resource set''')

		val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
		logger.info('''model folder: «modelFolder»''')

		val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
		modelEnvironment.setProjectLocation(mainArgs.modelDir)

		val resourceSet = new ResourceSetImpl()
		resourceSet.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
		
		val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
		modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
		
		val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model"); add("conf");], true)

		logger.info('''[loading] «modelFiles.length» files ...''')
		
		var returnCode = -1

		try {
			val standaloneResourceAdapter = new StandaloneResourceAdapter
			
			for (f : modelFiles) {
				val uri = URI.createFileURI(f.absolutePath)
				logger.debug("[load] " + uri.toString)
				val resource = resourceSet.getResource(uri, true)
				resource.eAdapters.add(standaloneResourceAdapter)
			}
			
			resourceSet.eAdapters.add(standaloneResourceAdapter)
			
			
			returnCode = standalone.doGenerate(mainArgs, modelInjector, resourceSet, CompilationMode.SINGLE_AND_BATCH)
		} catch (Exception e) {
			logger.error("message: " + e.message)
			logger.error("stacktrace: " + e.stackTrace.toString)
		}

[Updated on: Fri, 16 July 2021 11:16]

Report message to a moderator

Re: Performance issue when doing index of resourceset [message #1843134 is a reply to message #1843132] Fri, 16 July 2021 11:38 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7220
Registered: July 2009
Senior Member
Hi

I don't understand the semantics of "contains" in your DSL, but ...

It looks as if your model may be very flat and so you may violate the EMF expectation that your model is a tree. EMF does linear search at each level which is fine for the siblings in a typical tree but can be disastrous for a large flat file if any algorithm is quadratic.

Regards

Ed Willink
Re: Performance issue when doing index of resourceset [message #1843135 is a reply to message #1843134] Fri, 16 July 2021 11:53 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14026
Registered: July 2009
Senior Member
i mean yourkit, jvisualvm etc.
they will show you where cpu times is spent


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Performance issue when doing index of resourceset [message #1843136 is a reply to message #1843134] Fri, 16 July 2021 11:59 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
Hi Ed,

we use xcore to define the meta model. "contains" is the keyword in xcore.

Another interesting thing is, we execute the xtext generator both in Eclipse IDE mode and Standalone mode. With the same DSL files, the execution time in Eclipse IDE is much more faster than the standalone mode.
The only different part in the code is in standalone mode we createInjectorAndDoEMFRegistration and load the resources to resourceset explicitly in the code. Then both modes use the same indexing code. I don't know what happen behind the Eclipse IDE mode. But it seams that the Eclipse IDE mode can treat the DSL elements more effcient.
		logger.info('''init model''')
		val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
		logger.info('''done.''')
		
		logger.info('''init conf''')
		new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
		logger.info('''done.''')
		
		val standalone = modelInjector.getInstance(StandaloneExecutor)
		
		logger.info('''run standalone''')
		
		logger.info('''load model resources into resource set''')

		val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
		logger.info('''model folder: «modelFolder»''')

		val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
		modelEnvironment.setProjectLocation(mainArgs.modelDir)

		val resourceSet = new ResourceSetImpl()
		resourceSet.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
		
		val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
		modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
		
		val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model"); add("conf");], true)

		logger.info('''[loading] «modelFiles.length» files ...''')
		
		var returnCode = -1

		try {
			val standaloneResourceAdapter = new StandaloneResourceAdapter
			
			for (f : modelFiles) {
				val uri = URI.createFileURI(f.absolutePath)
				logger.debug("[load] " + uri.toString)
				val resource = resourceSet.getResource(uri, true)
				resource.eAdapters.add(standaloneResourceAdapter)
			}
			
			resourceSet.eAdapters.add(standaloneResourceAdapter)
			
			// the indexing happened in the doGenerate method
			returnCode = standalone.doGenerate(mainArgs, modelInjector, resourceSet, CompilationMode.SINGLE_AND_BATCH)
		} catch (Exception e) {
			logger.error("message: " + e.message)
			logger.error("stacktrace: " + e.stackTrace.toString)
		}
Re: Performance issue when doing index of resourceset [message #1843137 is a reply to message #1843136] Fri, 16 July 2021 12:05 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14026
Registered: July 2009
Senior Member
please measure. and maybe you can provide a complete reproducible hello world example

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Performance issue when doing index of resourceset [message #1843138 is a reply to message #1843137] Fri, 16 July 2021 12:06 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14026
Registered: July 2009
Senior Member
btw this call order seems wrong

EcoreUtil.resolvAll(resourceSet)
ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(resourceSet, index);

if you install index after resolving,
then resolving wont profit from indexing


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Performance issue when doing index of resourceset [message #1843141 is a reply to message #1843138] Fri, 16 July 2021 12:36 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
hi Christian,

sorry, that was my copy paste error, the resolveAll is after the indexing in the code.
Re: Performance issue when doing index of resourceset [message #1843323 is a reply to message #1843141] Fri, 23 July 2021 13:25 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
I tried to process two DSL model files separately. and even not load the DSL2 model and not get index of DSL2 model (see the codes are commented out). The indexing execution time has no change still very long. How can this happen? I tried several times, only difference is the size of DSL2 model file.
I'm not familiar with EMF. If I am wrong, please correct me. So following is my guess:
when we call "createInjectorAndDoEMFRegistration()", all DSL1 and DSL2 models are parsed into Antlr trees (or xtext nodemodel) in memory. And then even the DSL2 files are not loaded to the XtextResourceSet. The Antlr tree still contains the big DSL2 part. It slows down to create index for DSL1 models.

		logger.info('''init model''')
		val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
		logger.info('''done.''')
		
		logger.info('''init conf''')
		val configurationInjector = new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
		logger.info('''done.''')
	
		logger.info('''init GitManager''')
		val gitManager = modelInjector.getInstance(GitManager) as StandaloneGitManager
		gitManager.userName = mainArgs.gitUserName
		gitManager.password = mainArgs.gitPassword
		logger.info('''done.''')
		
		val standalone = modelInjector.getInstance(StandaloneExecutor)
		
		logger.info('''run standalone''')
		
		logger.info('''load model resources into resource set''')

		val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
		logger.info('''model folder: «modelFolder»''')
		
//		val configurationEnvironment = (configurationInjector.getProvider(IEnvironment).get as AbstractEnvironment)
//		configurationEnvironment.setProjectLocation(mainArgs.modelDir)
		
		val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
		modelEnvironment.setProjectLocation(mainArgs.modelDir)

//		val resourceSet = new ResourceSetImpl()
		val modelResourceset = modelInjector.getInstance(XtextResourceSet)
		modelResourceset.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
		
		val configurationResourceset = configurationInjector.getInstance(XtextResourceSet)
		configurationResourceset.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
		
		val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
		modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
		
		val configurationFileSystemAccess = configurationInjector.getInstance(JavaIoFileSystemAccess)
		configurationFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
		
		val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model");], true)
		
		val configurationFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("conf");], true)

		logger.info('''[loading] «modelFiles.length + configurationFiles.length» files ...''')
		
		var returnCode = -1

		try {
			val standaloneResourceAdapter = new StandaloneResourceAdapter
			
			for (f : modelFiles) {
				val uri = URI.createFileURI(f.absolutePath)
				logger.debug("[load] " + uri.toString)
				val resource = modelResourceset.getResource(uri, true)
				resource.eAdapters.add(standaloneResourceAdapter)
			}
			
			modelResourceset.eAdapters.add(standaloneResourceAdapter)
			
//			for (f : configurationFiles) {
//				val uri = URI.createFileURI(f.absolutePath)
//				logger.debug("[load] " + uri.toString)
//				val resource = configurationResourceset.getResource(uri, true)
//				resource.eAdapters.add(standaloneResourceAdapter)
//			}
//			
//			configurationResourceset.eAdapters.add(standaloneResourceAdapter)
			
			returnCode = standalone.doGenerate(mainArgs, modelInjector, modelResourceset, configurationResourceset, CompilationMode.SINGLE_AND_BATCH)
		} catch (Exception e) {
			logger.error("message: " + e.message)
			logger.error("stacktrace: " + e.stackTrace.toString)
		}


.......

protected void doGenerate(......) {
		doIndex();
.....

}

	protected void doIndex() {
			long startTimeIndexing = System.nanoTime();

			logger.info("[indexing]");
			// indexing
			List<IResourceDescription> modelDescriptions = newArrayList();
			List<IResourceDescription> configurationDescriptions = newArrayList();
			for (Resource resource : modelResourceSet.getResources()) {
				IResourceServiceProvider serviceProvider = serviceRegistry
						.getResourceServiceProvider(resource.getURI());
				IResourceDescription description = serviceProvider.getResourceDescriptionManager()
						.getResourceDescription(resource);
				modelDescriptions.add(description);
			}
			modelIndex = new ResourceDescriptionsData(modelDescriptions);

//			for (Resource resource : configurationResourceSet.getResources()) {
//				IResourceServiceProvider serviceProvider = serviceRegistry.getResourceServiceProvider(resource.getURI());
//				IResourceDescription description = serviceProvider.getResourceDescriptionManager()
//						.getResourceDescription(resource);
//				configurationDescriptions.add(description);
//			}
//			configurationIndex = new ResourceDescriptionsData(configurationDescriptions);

			ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(modelResourceSet, modelIndex);
			ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(configurationResourceSet,
					configurationIndex);

			long endTime = System.nanoTime();
			long durationNanoSecond = (endTime - startTimeIndexing);
			logger.info("[indexing] finished in " + durationNanoSecond / 1000000 + "ms.");
		}


Re: Performance issue when doing index of resourceset [message #1843324 is a reply to message #1843323] Fri, 23 July 2021 13:26 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14026
Registered: July 2009
Senior Member
again. please provide a yourkit or similar measurement or a complete example

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Previous Topic:Deadlock between Job and event listener on Display thread
Next Topic:Umsetzung von Xtend-Funktionen basierend auf einem UML-Klassendiagramm
Goto Forum:
  


Current Time: Wed Jul 28 22:36:22 GMT 2021

Powered by FUDForum. Page generated in 0.02857 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top