Hi again, Just an update on the testing front: it appears macOS users will have trouble running the plugged-in tests via mvn -T 1C clean install, and given that we now have multiple test options (ci, unit, plugged) in Maven, I have disabled the plugged-in tests as part of the build. This allows cleaner separation and reporting between building and testing, especially at ci.eclipse.org/epsilon, where we can monitor each stage. So if you’re relying on mvn clean install, you may also want to run the following commands locally after the build is successful: mvn -f tests/org.eclipse.epsilon.test install -P plugged mvn -f tests/org.eclipse.epsilon.test install -P unit I’m thinking of adding a .cmd and .sh file to the root of the repo so that this can be automated and won’t need to remember the commands. Also if anyone with a Mac can get the plugged-in tests to work via Maven (i.e. you know how to append -XstartOnFirstThread to the arguments conditionally or via -D property) please reply or message Dimitris. Thanks, Sina Hi everyone, An update on this: I have managed to get ALL tests working via Maven (except for Simulink, haven’t tried that) and at least on Windows and Fedora, they’re all passing! However, as is evident by the presence of “EpsilonJenkinsTestSuite”, there is still some trouble with the CI. For some strange reason, it appears either the version of Mockito on Jenkins is not being updated (I “recently” bumped it up to 2.23 from 2.13, and before that it was 1.8.4 or something) and the errors are to do with the fact that some of the methods are not visible. This means some of the unit tests that rely on package-private classes and methods fail – see [1] for the 3 suites (at time of writing) that are failing on Jenkins but passing locally (both from Eclipse and Maven). If anyone knows why these tests are failing on the CI and how to fix it (without changing the code), please let me know. Also it’d be great if you could test on your machine and see if the following commands are both successful (from root of org.eclipse.epsilon repo): mvn -T 1C clean install mvn -f tests/org.eclipse.epsilon.test surefire:test -P "ci,-plugged" mvn -f tests/org.eclipse.epsilon.test surefire:test -P "unit,-plugged" Note that the last command requires at least 8 GB RAM (at least, if it manages to pick up the EvlAdvancedTestSuite, I’m having trouble with it not being picked up by Surefire). Also if anyone encounters failures when running EpsilonEclipseTestSuite (previously known as EpsilonTestSuite) from Eclipse please let me know. We don’t want any failures before a major release ;) Thanks, Sina [1] https://git.eclipse.org/c/epsilon/org.eclipse.epsilon.git/tree/tests/org.eclipse.epsilon.test/src/org/eclipse/epsilon/test/EpsilonSurefireTestSuite.java I have worked on this some more, tidying up some regressions and places I had missed, and merged into master. Note that there are a few tests that are a number of commented out tests in EpsilonSurefireTestSuite. Specifically, anything involving HUTN or EUnit is out of action: can't work out why, but unfortunately there's quite a lot of tests for HUTN (both direct and indirect), including a few in EGX and probably a good chunk of Flock. EUnit is the top priority to fix, given that some of the most important tests are part of EolAcceptanceTestSuite. @Antonio, I would appreciate any guidance on this. Aside from that and Simulink (which is a whole 'nother can of worms, not worth bothering with given how much of a pain it is to get the test working locally from within Eclipse on Windows due to the strict JRE version requirement, environment variables etc., let alone Maven!), everything else seems to be working.
Any pointers on how to fix HUTN would be appreciated. Currently I've converted everything to ise the new JAR-aware FileUtil.getFileStandalone method (as opposed to class.getResource) and it works from Eclipse but not Maven. From what I can work out, there's a lot of reliance on EMF and registration, but I don't know enough about that. Hi everyone, I’ve been working on trying to get the test suite working via Maven (following on from Horacio’s work on this) on the smadani/unittests branch and I’ve had some success in fixing some of the issues, though not everything is working flawlessly. As a reminder, the main difference is that we write all resources to a temporary directory so that we’re not relying on having the src folder available, meaning tests can be run from outside of Eclipse. Unfortunately the temp folder isn’t cleaned up properly and there area few failures but it’s an improvement on the state of things before. I’ve managed to get rid of Hamcrest altogether since in pretty much every case it was being used like assertEquals / assertTrue anyway and it caused more problems than it solved (deprecations, an unnecessary dependency to manage etc.), so that helped. The current setup in the branch is to do mvn surefire:test (as Horacio suggested) on the org.eclipse.epsilon.test plugin once everything’s built. As explained there’s some issues with HUTN but that’s about it as far as I can see on the Maven side of things. Due to the refactoring to creating temp files, there is only one annoying test that I’ve spent almost a full day trying to figure out why it won’t work and that’s Traceability in EGL (the Hierarchy one, though I suspect only Louis would know how all this traceability stuff in EGL works). There’s also this weird issue with ETL where it seems that the transformation is in-place so when testing with multiple IEtlModule implementations / parameters the oracle needs to be copied back over (no other reason I can think of for failures) and sometimes, only for the Tree2Graph test, sometimes it fails all the tests and sometimes it doesn’t, even though the exact same methodology is used as the OO2DB test. Definitely something dodgy going on there, perhaps with EMF resource registration or something. Even if we don’t fix these problems, that’s still the vast majority of the tests working, so I suppose it’s better than nothing. We could comment out these tests that aren’t working for the Maven build (i.e. create a separate suite) and only run them locally (i.e. from within Eclipse) as a workaround, which we need to do for some tests anyway (for example the Simulink tests won’t work on CI if Simulink isn’t set up). We could then re-use the old way of running the tests (i.e. relying on being able to navigate to the src folder using X.class.getResource(“”).toString().replace(“bin”, “src”) or whatever) for these troublesome tests rather than temp files. Thanks, Sina Hi Horacio, Why do we have multiple hamcrest versions? Would it be possible to unify to a specific hamcrest version? Pure Maven won't care about the Eclipse metadata: the build will be based entirely on dependencies. Surefire can exclude dependencies if you have a problem with those: Hi all, I have been working on this problem and there are several issues and challenges associated with making this work. I’ll try to point them out individually, if I can untangle the mess in my head. 1. Tycho surefire plugin. It seems that Tycho takes over the tests, which is good to run the plug-in tests. However, running the unit tests from within a nested eclipse fails due to discrepancies between versions of the compiled classes. - A workaround is to split the tests plugin into two, and have a separate tests.unit plugin which we use to run the tests. For pros/cons see points: 2, Junit vs Maven. Making JUnit directly run the tests (i.e. mimicking what we do from Eclipse run configurations) is a PITA because it requires to setup the classpath. However, this has the advantage that the file access (for test models, metamodels and scripts) as currently done would work. Running the tests via maven we no longer have to worry about the class path. However, maven runs the tests from the compiled classes, which means that file access has to be changed to access resources within classes. In my proposed attempt I went with the mvn path (but has some side effects too). 3. The current file resolution approach woks as expected with scripts/models that import other models within JUnit because since we access files in the sources folder the paths to imported files work. When reading resources from classes this does not work. 4. Some HUTN tests expect the Family.ecore metramodel to be registered (I think), but it gets registered with the class resource path so then it can not be used… or something like that. Needs more investigation. - As I said before I went via the maven approach and created a separate plugin for the unit tests. - The FileUitl class has been modified to make use of temp files to create temp versions of the required models and scripts; it also respects the folder hierarchy so relative paths work . I have changed a lot of the tests to play nicely with this change. But the results tell me that there are still a couple of places where some missing imported or references models are missing. - The tests can be invoked using mvn surefire:test. Explicit call to maven’s surefire needs to be done to avoid Tycho’s surefire. This command has to be executed after the main maven build has finished. - There are several test that fail atm. Some because of assertions, other for missing files and other for the missing Families meta model. Further, I am always getting out of memory errors when some of the new tests try to start separate threads. I am using Xmx6166M and still getting this errors. More investigation needs to be done to see if the failing tests are because of actual errors or because as a result of issues on creating the temp files. These errors needs further investigation because I think the results from the mvn run are different from the ones I get from running within eclipse. - One of the effects of the change is that some tests fail from within eclipse with a Security exception caused by conflicts between hamcrest versions. I have tried to solve this issue by changing the import order of dependencies, changing where dependencies are imported, etc. and can’t figure out what else to do. I am not happy that we would always need to run the tests from the command line. Any ideas or help on this is welcome. I have pushed this idea to hoyos/unittests I think this is a great idea, and only discovered that EpsilonTestSuite was not run as part of mvn clean install recently. If we can guarantee that the tests will be run in a Unix environment then this would work. Currently there are a number of tests that fail on Windows and even under Linux due to the aforementioned file locations, permissions etc. Perhaps we could start with a smaller subset and gradually add the tests which are currently failing in certain circumstances once fixed? Also the Xmx needs to be adjusted to at least 4G (I think I may have already done this) to accommodate the full test suite. _______________________________________________
epsilon-dev mailing list
epsilon-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/epsilon-dev
_______________________________________________
epsilon-dev mailing list
epsilon-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/epsilon-dev
-- Antonio Garcia-Dominguez
|
