| Performance issue when doing index of resourceset [message #1843126] |
Fri, 16 July 2021 10:28  |
 Juewei Jin
Messages: 17
Registered: July 2009 |
Junior Member |
|
|
Hi,
I'm facing a performance issue with a single DSL file. We have added a second DSL, one of this DSL file contains around 3300 lines.
Example with reduced content:
DQRules-configuration test {
generateDqRulesOnlyForTargetEntites
DqRules-Id-Mapping {
DqRules-Id-Mapping-From-Legacy-Slf {
"MONITOR.xxx.001" --> "MONITOR.yyy.001"
"MONITOR.xxx.002" --> "MONITOR.yyy.002"
}
DqRules-Id-Mapping-New{
}
}
DQRules-Deactivation-Data_Exists {
}
DQRules-Deactivation-Validity{
}
DQRules-Deactivation-Null_Check {
"model.abc.test1"
"model.abc.test2"
}
DQRules-Deactivation-Uniqueness {
}
DQRules-Deactivation-Ref_Integrity {
"model.cde.test1"
"model.cde.test2"
}
}
The DSL grammar for this file see below.
Now we are facing the performance issue when we doing the indexing (see the code below) with this new DSL file. It took now much more time to do it. Even the lines in this file were commented out, no change with the performance. Only when the content of this file be reduced, then the indexing performance get improved.
But we don't have issue with the file of another DSL, which also contains a lot of lines.
Is this performance because of bad grammar design? Or what else can be the reason?
List<IResourceDescription> descriptions = newArrayList();
for (Resource resource : resourceSet.getResources()) {
IResourceServiceProvider serviceProvider = serviceRegistry.getResourceServiceProvider(resource.getURI());
IResourceDescription description = serviceProvider.getResourceDescriptionManager()
.getResourceDescription(resource);
descriptions.add(description);
}
index = new ResourceDescriptionsData(descriptions);
EcoreUtil.resolvAll(resourceSet)
ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(resourceSet, index);
class ConfigurationDqRules extends Configuration {
Boolean generateDqRulesOnlyForTargetEntites
contains DqRulesIdMappingList dqRulesIdMappingList
contains DataExistsDqRuleDeactivation dataExistsDqRuleDeactivation
contains ValidityDqRuleDeactivation validityDqRuleDeactivation
contains NullCheckDqRuleDeactivation nullCheckDqRuleDeactivation
contains UniquenessDqRuleDeactivation uniquenessDqRuleDeactivation
contains RefIntegrityDqRuleDeactivation refIntegrityDqRuleDeactivation
}
class DqRulesIdMappingList {
contains DqRulesIdMappingFromLegacySlf dqRulesIdMappingFromLegacySlf
contains DqRulesIdMappingNew dqRulesIdMappingNew
}
class DqRulesIdMappingFromLegacySlf {
contains DqRulesIdMapping [] dqRulesIdMapping
}
class DqRulesIdMappingNew {
contains DqRulesIdMapping [] dqRulesIdMapping
}
class DqRulesIdMapping {
String dqRuleId
String mapTo
}
class DataExistsDqRuleDeactivation {
contains DataExistsDqRule [] dataExistsDqRule
}
class DataExistsDqRule {
String entity
}
class ValidityDqRuleDeactivation{
contains ValidityDqRule [] ValidityDqRule
}
class ValidityDqRule {
String entityProperty
}
class NullCheckDqRuleDeactivation {
contains NullCheckDqRule [] nullCheckDqRule
}
class NullCheckDqRule {
String entityProperty
}
class UniquenessDqRuleDeactivation {
contains UniquenessDqRule [] uniquenessDqRule
}
class UniquenessDqRule {
String entity
}
[Updated on: Fri, 16 July 2021 11:02] Report message to a moderator |
|
|
|
|
| Re: Performance issue when doing index of resourceset [message #1843132 is a reply to message #1843131] |
Fri, 16 July 2021 11:13  |
Juewei Jin
Messages: 17
Registered: July 2009 |
Junior Member |
|
|
Here is the performance compare: we have around 5000 files of DSL1 + 1 file of DSL2 (which cause the performance issue)
Situation 1: DSL2 file with less than 50 lines, finish indexing with all files took 71445ms
Situation 2: DSL2 file with 3300 lines, finish indexing with all files took 265773ms
Situation 3: DSL2 file with 3300 lines but comment out 3250 lines, finish indexing with all files took 161888ms
and main part of the code for the standalone mode look like below
logger.info('''init model''')
val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
logger.info('''done.''')
logger.info('''init conf''')
new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
logger.info('''done.''')
val standalone = modelInjector.getInstance(StandaloneExecutor)
logger.info('''run standalone''')
logger.info('''load model resources into resource set''')
val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
logger.info('''model folder: «modelFolder»''')
val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
modelEnvironment.setProjectLocation(mainArgs.modelDir)
val resourceSet = new ResourceSetImpl()
resourceSet.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model"); add("conf");], true)
logger.info('''[loading] «modelFiles.length» files ...''')
var returnCode = -1
try {
val standaloneResourceAdapter = new StandaloneResourceAdapter
for (f : modelFiles) {
val uri = URI.createFileURI(f.absolutePath)
logger.debug("[load] " + uri.toString)
val resource = resourceSet.getResource(uri, true)
resource.eAdapters.add(standaloneResourceAdapter)
}
resourceSet.eAdapters.add(standaloneResourceAdapter)
returnCode = standalone.doGenerate(mainArgs, modelInjector, resourceSet, CompilationMode.SINGLE_AND_BATCH)
} catch (Exception e) {
logger.error("message: " + e.message)
logger.error("stacktrace: " + e.stackTrace.toString)
}
[Updated on: Fri, 16 July 2021 11:16] Report message to a moderator |
|
|
|
|
| Re: Performance issue when doing index of resourceset [message #1843136 is a reply to message #1843134] |
Fri, 16 July 2021 11:59 |
Juewei Jin
Messages: 17
Registered: July 2009 |
Junior Member |
|
|
Hi Ed,
we use xcore to define the meta model. "contains" is the keyword in xcore.
Another interesting thing is, we execute the xtext generator both in Eclipse IDE mode and Standalone mode. With the same DSL files, the execution time in Eclipse IDE is much more faster than the standalone mode.
The only different part in the code is in standalone mode we createInjectorAndDoEMFRegistration and load the resources to resourceset explicitly in the code. Then both modes use the same indexing code. I don't know what happen behind the Eclipse IDE mode. But it seams that the Eclipse IDE mode can treat the DSL elements more effcient.
logger.info('''init model''')
val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
logger.info('''done.''')
logger.info('''init conf''')
new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
logger.info('''done.''')
val standalone = modelInjector.getInstance(StandaloneExecutor)
logger.info('''run standalone''')
logger.info('''load model resources into resource set''')
val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
logger.info('''model folder: «modelFolder»''')
val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
modelEnvironment.setProjectLocation(mainArgs.modelDir)
val resourceSet = new ResourceSetImpl()
resourceSet.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model"); add("conf");], true)
logger.info('''[loading] «modelFiles.length» files ...''')
var returnCode = -1
try {
val standaloneResourceAdapter = new StandaloneResourceAdapter
for (f : modelFiles) {
val uri = URI.createFileURI(f.absolutePath)
logger.debug("[load] " + uri.toString)
val resource = resourceSet.getResource(uri, true)
resource.eAdapters.add(standaloneResourceAdapter)
}
resourceSet.eAdapters.add(standaloneResourceAdapter)
// the indexing happened in the doGenerate method
returnCode = standalone.doGenerate(mainArgs, modelInjector, resourceSet, CompilationMode.SINGLE_AND_BATCH)
} catch (Exception e) {
logger.error("message: " + e.message)
logger.error("stacktrace: " + e.stackTrace.toString)
}
|
|
|
|
|
|
| Re: Performance issue when doing index of resourceset [message #1843323 is a reply to message #1843141] |
Fri, 23 July 2021 13:25 |
Juewei Jin
Messages: 17
Registered: July 2009 |
Junior Member |
|
|
I tried to process two DSL model files separately. and even not load the DSL2 model and not get index of DSL2 model (see the codes are commented out). The indexing execution time has no change still very long. How can this happen? I tried several times, only difference is the size of DSL2 model file.
I'm not familiar with EMF. If I am wrong, please correct me. So following is my guess:
when we call "createInjectorAndDoEMFRegistration()", all DSL1 and DSL2 models are parsed into Antlr trees (or xtext nodemodel) in memory. And then even the DSL2 files are not loaded to the XtextResourceSet. The Antlr tree still contains the big DSL2 part. It slows down to create index for DSL1 models.
logger.info('''init model''')
val modelInjector = new StandaloneModelStandaloneSetup().createInjectorAndDoEMFRegistration()
logger.info('''done.''')
logger.info('''init conf''')
val configurationInjector = new ConfigurationStandaloneSetup().createInjectorAndDoEMFRegistration
logger.info('''done.''')
logger.info('''init GitManager''')
val gitManager = modelInjector.getInstance(GitManager) as StandaloneGitManager
gitManager.userName = mainArgs.gitUserName
gitManager.password = mainArgs.gitPassword
logger.info('''done.''')
val standalone = modelInjector.getInstance(StandaloneExecutor)
logger.info('''run standalone''')
logger.info('''load model resources into resource set''')
val modelFolder = new File(URI.createURI(mainArgs.modelDir + "/model").toFileString)
logger.info('''model folder: «modelFolder»''')
// val configurationEnvironment = (configurationInjector.getProvider(IEnvironment).get as AbstractEnvironment)
// configurationEnvironment.setProjectLocation(mainArgs.modelDir)
val modelEnvironment = (modelInjector.getProvider(IEnvironment).get as AbstractEnvironment)
modelEnvironment.setProjectLocation(mainArgs.modelDir)
// val resourceSet = new ResourceSetImpl()
val modelResourceset = modelInjector.getInstance(XtextResourceSet)
modelResourceset.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
val configurationResourceset = configurationInjector.getInstance(XtextResourceSet)
configurationResourceset.loadOptions.put(XtextResource.OPTION_ENCODING, StandardCharsets.UTF_8.name());
val modelFileSystemAccess = modelInjector.getInstance(JavaIoFileSystemAccess)
modelFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
val configurationFileSystemAccess = configurationInjector.getInstance(JavaIoFileSystemAccess)
configurationFileSystemAccess.setOutputPath(URI.createURI(mainArgs.outputDir).toFileString)
val modelFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("model");], true)
val configurationFiles = FileUtils.listFiles(modelFolder, new ArrayList<String> => [add("conf");], true)
logger.info('''[loading] «modelFiles.length + configurationFiles.length» files ...''')
var returnCode = -1
try {
val standaloneResourceAdapter = new StandaloneResourceAdapter
for (f : modelFiles) {
val uri = URI.createFileURI(f.absolutePath)
logger.debug("[load] " + uri.toString)
val resource = modelResourceset.getResource(uri, true)
resource.eAdapters.add(standaloneResourceAdapter)
}
modelResourceset.eAdapters.add(standaloneResourceAdapter)
// for (f : configurationFiles) {
// val uri = URI.createFileURI(f.absolutePath)
// logger.debug("[load] " + uri.toString)
// val resource = configurationResourceset.getResource(uri, true)
// resource.eAdapters.add(standaloneResourceAdapter)
// }
//
// configurationResourceset.eAdapters.add(standaloneResourceAdapter)
returnCode = standalone.doGenerate(mainArgs, modelInjector, modelResourceset, configurationResourceset, CompilationMode.SINGLE_AND_BATCH)
} catch (Exception e) {
logger.error("message: " + e.message)
logger.error("stacktrace: " + e.stackTrace.toString)
}
.......
protected void doGenerate(......) {
doIndex();
.....
}
protected void doIndex() {
long startTimeIndexing = System.nanoTime();
logger.info("[indexing]");
// indexing
List<IResourceDescription> modelDescriptions = newArrayList();
List<IResourceDescription> configurationDescriptions = newArrayList();
for (Resource resource : modelResourceSet.getResources()) {
IResourceServiceProvider serviceProvider = serviceRegistry
.getResourceServiceProvider(resource.getURI());
IResourceDescription description = serviceProvider.getResourceDescriptionManager()
.getResourceDescription(resource);
modelDescriptions.add(description);
}
modelIndex = new ResourceDescriptionsData(modelDescriptions);
// for (Resource resource : configurationResourceSet.getResources()) {
// IResourceServiceProvider serviceProvider = serviceRegistry.getResourceServiceProvider(resource.getURI());
// IResourceDescription description = serviceProvider.getResourceDescriptionManager()
// .getResourceDescription(resource);
// configurationDescriptions.add(description);
// }
// configurationIndex = new ResourceDescriptionsData(configurationDescriptions);
ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(modelResourceSet, modelIndex);
ResourceDescriptionsData.ResourceSetAdapter.installResourceDescriptionsData(configurationResourceSet,
configurationIndex);
long endTime = System.nanoTime();
long durationNanoSecond = (endTime - startTimeIndexing);
logger.info("[indexing] finished in " + durationNanoSecond / 1000000 + "ms.");
}
|
|
|
|
Powered by
FUDForum. Page generated in 0.02047 seconds
] 
Search
Help
Register
Login
Home
