Home » Modeling » Epsilon » Persisting Model repository over the boundary of the ant run(workflow shows poor performance if used in loops)
| Persisting Model repository over the boundary of the ant run [message #1838708] |
Wed, 03 March 2021 23:00  |
|
As part of our work, we are designing Epsilon workflows in Ant that run in externally controlled loops. Each workflow instantiation results in the creation of a model repository and load of the requisite models.
We are aware that the project repository spans the ant run. We require a means to provide the model repository as a parameter to the AntRunner, however it only provides User Properties as a parameter.
This is a major performance issue for us, as it adds about 1 second (0.78) to each learning loop invocation.
Do you have a recommendation to overcome this issue? It seems that these performance issues in using workflow were important in your design, as in quote below.
We are thinking the following design could help:
* User starts the ant-run and gives a user-property identifying a repository by name.
* As soon as an EpsilonTask is executed, the property is evaluated. A registry is queried using the repository name. If the name exists, the repository becomes the current project repository. Otherwise it is created, stored and made the current repository.
* Optionally, a new task offers purging a repository
* Optionally, a new api offers purging a repository
* Optionally, a repository can be locked to a thread and not allow multiple access.
Quote:
An ANT task can access the project in which it is contained by invoking the Task.getProject() method. To facilitate sharing of arbitrary information between tasks, ANT projects provide two convenience methods, namely addReference(String key, Object ref) and getReference(String key) : Object. The former is used to add key-value pairs, which are then accessible using the latter from other tasks of the project.
To avoid loading models multiple times and to enable on-the-fly management of models from different Epsilon modules without needing to store and re-load the models after each task, a reference to a project-wide model repository has been added to the current ANT project using the addReference method discussed above. In this way, all the subclasses of the abstract EpsilonTask can invoke the getProjectRepository() method to access the project model repository.
|
|
| |
| Re: Persisting Model repository over the boundary of the ant run [message #1838713 is a reply to message #1838710] |
Thu, 04 March 2021 02:52  |
|
|
That is a good idea, but in Eclipse the runner is on a separate classloader. Which means I can see the outside classes from the inside, but not the other way around. So I can receive the reference but I am not sure how I would capture it. Also, I do not know the distinguished name. I assume there is some sort of collision avoidance when the name of the reference is assigned in ant?
|
|
| |
| Re: Persisting Model repository over the boundary of the ant run [message #1838880 is a reply to message #1838730] |
Mon, 08 March 2021 01:19 |
|
I have looked at you suggested alternatives, and believe they do not help based on the following:
(suggestion ==> blocking factor)
* use an org.apache.tools.ant.Project instead of AntRunner to run your workflow as shown in [1] ==> This will do away with installing Tasks via Eclipse task extension point and will remove the classpath protection that you get with the runner (classloader is scoped to the thread)
* use Epsilon's "for" task if you need to iterate over files [2] ==> We are not iterating over files, but have a controller that uses a metric. It is a machine learning system that compares solutions.
* use AntContrib's "foreach" task [3], which provides additional options. ==> Foreach ' iterates over a list, a list of paths, or both', which, as above, is not what we are doing.
We have moved the machine learning control out of ant deliberately, because ant is not designed to be Turing-complete. Yes, there are probably some sub-call hacks, but it is not intended to be used that way.
If we keep the while-loop outside of Ant, as we should, we get hit by the re-instantiation cost of the Epsilon model repo. This is significant, so it means Epsilon workflow is not a usable component for us. If we code in Java the cost of implementation is higher and the flexibility lower. So this also reduces attractiveness of applying Epsilon.
I should probably mention that our company has been at this cross-roads before. Because of the time cost of this issue, the developers then chose to discard workflow and write code in Java based on all-in-one. This leads to a fast solution, but the flexibility is low. I presume that potential industry adopters of Epsilon will come up against this repeatedly.
If you are interested in seeing the performance of workflow, we have a Docker container definition that creates an Antrunner application which includes Epsilon (refer my ticket about the dependencies). This is a flexible alternative to using the all-in-one, which allows the use of other MDE-based technologies via ant. You are welcome to this code. It will also allow you to verify the time cost for the instantiation of the project model repositories.
|
|
| | | |
Goto Forum:
Current Time: Sat Apr 27 08:38:57 GMT 2024
Powered by FUDForum. Page generated in 0.03374 seconds |
] 
Search
Help
Register
Login
Home
