Hi Thierry,
In general canceling indexing operation doesn't work for both indexers (current implementation for DLTK indexer doesn't care too much about cancel - see AbstractJob#isCancelled and ProgressJob). So while indexing source modules from huge project you can
try to cancel it but it won't work as expected. The difference is only while closing IDE - as indexing thread is daemon, when you close IDE while being in the middle of indexing then in case of H2 everything will be just dropped when thread dies. In case of
Lucene we wait for all indexing jobs to be finished as we must close index writers gracefully.
Anyway we can fix it of course :-). In general i plan to add patch proposal to DLTK that will enable canceling. It will give a possibility to cancel indexing in "Progress" view as well as while closing IDE ("Saving workspace state..." window where you can
cancel ongoing operations). Anyway there might be some side-effect when canceling in "Progress" view as index might not be complete in that case and it will be rebuild/updated after next clean or IDE restart (but I guess that user should be aware of it while
pressing "Stop" :-).
What do you think about it?
Greetings,
Bart
On 5/12/2016 10:32 PM, thierry blind wrote:
Hi Bart,
I have another question concerning the lucene indexer:
I was wondering why it's not possible to interrupt an indexing operation (for example to close eclipse asap and restart the indexing operation later).
With the H2 indexer I could stop the indexing (after some clicks on the stop button in the progress panel), or even close eclipse and then hit the indexing stop button.
Now you have no more control on the indexing process and the progress bar (with its stop button) becomes totally useless, and even worse, when you close eclipse, it continues the indexing as a background process, taking a lot of memory (and time)! Or you can
kill eclipse...
When you take the big projects I've sent you, indexing takes time and you are "forced" to wait that indexing ends before being able to close eclipse, which is not the case with classic java projects and their build process. I see that as a regression, but maybe
I'm too rude :S
Thierry.
From: thierryblind@xxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 16:23:07 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Thank you ! I'm glad to help ;) And thank you for your patience ;)
Thierry.
From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 16:03:01 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Thierry, your help is invaluable :) You pointed me into the right direction. I didn't check the scenario with dependent projects but it looks like it is the root cause of the problems.
Thanks again for your help and have a nice holiday leave! :)
On 5/4/2016 5:53 PM, thierry blind wrote:
Maybe in my case I simply though it does loop forever, because I had a looooooot of big projects depending on same big project "A" (by doing Properties -> PHP -> Include Path -> Projects -> Add -> projet A).
And it took forever with a big amount of place taken by org.eclipse.dltk.core.index.lucene
Strangely it was much faster on a new and clean workspace :S
If I have for example 5 projects that depends on a common project "A", the project "A" will then be indexed 5 times.
In my case with H2 indexer it took 1.1 GB, with the Lucene indexer it takes more than 3GB on a safe and new workspace!
So really sorry for pointing in the wrong direction :/
And yes there is still room for improvements ;)
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 15:39:01 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
OK, great. For now I have found something else while using your workspace that is not OK. Even if indexing finishes OK there are some source modules being re-indexed while closing/restarting IDE. At least it might give me some clues what might happened as I
think that the problem is related somehow to storing/reading source files timestamps as this is the place where some redundant indexing might be triggered.
On 5/4/2016 5:07 PM, thierry blind wrote:
I'll repackage another workspace and tell you when it's available ok ? Maybe I removed too much...
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 15:00:53 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
I used your workspace in my dev env but unfortunately indexer finished without a problem. It only took some time to close the PDT running threads (after closing IDE) as Lucene was merging and closing huge indexes (for about 45-60 seconds) but it finished it
gracefully. Anyway I will try to do my best to reproduce it ;)
On 5/4/2016 4:13 PM, thierry blind wrote:
With pleasure ;)
You can send it at my email address:
thierryblind@xxxxxxx
Kaloyan, if you're also interested, tell me ok?
I really hope you will also reproduce this bug on your eclipse, I will go crazy if I'm the only one to have it! lol
Of course each time you start your eclipse you have to delete .metadata/org.eclipse.dltk.core.index.lucene
And some additional infos on what's installed in my eclipse Mars.2 Release:
Eclipse IDE for Java EE Developers 4.5.2.20160218-0600 epp.package.jee null
JavaHL 1.7.9 Win32 Binaries (Optional) 5.0.0.I20150529-1700 org.polarion.eclipse.team.svn.connector.javahl17.win32.feature.group Polarion Software
PDT Extensions 0.22.2.201604260159 org.pdtextensions.feature.feature.group
http://pdt-extensions.org
PHP CodeSniffer 1.3.0.R20111119000000 org.phpsrc.eclipse.pti.tools.codesniffer.feature.group Sven Kiera
PHP Development Tools (PDT) 4.0.0.201605031251 org.eclipse.php.feature.group Eclipse PDT
PHP Development Tools (PDT) Mylyn Integration 4.0.0.201605031251 org.eclipse.php.mylyn.feature.group Eclipse PDT
PHP Development Tools (PDT) Source Code 4.0.0.201605031251 org.eclipse.php.source.feature.group Eclipse PDT
PHP Tool Integration Core 1.6.0.R20111119000000 org.phpsrc.eclipse.pti.core.feature.group Sven Kiera
PHPUnit 0.8.0.R20111119000000 org.phpsrc.eclipse.pti.tools.phpunit.feature.group Sven Kiera
PMD Plug-in 4.0.8.v20151204-2156 net.sourceforge.pmd.eclipse.feature.group PMD Project
Subversive SVN Team Provider 3.0.4.I20160131-1700 org.eclipse.team.svn.feature.group Eclipse.org
Thierry.
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 14:02:11 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
That would be great! Thank you one more time for all the feedback! This zipped workspace would be very appreciated, I will send you soon my private address that you can send the google drive link at.
On 5/4/2016 3:50 PM, thierry blind wrote:
It won't be possible at work for Skype sorry :S
And I followed progress bar since first try ;)
Here is what happen : I have PHP projects A, B, C and D. Projects B, C and D depend on project A.
So indexer starts with project A, continues with project B ( only few files), goes back to A, does a little bit from C, goes back to A, does a little bit from D, and then
it seems it only loops inside of A. It seems that if I only have project A open in my workspace, indexer works correctly and then I can open project B and indexer still works correctly, and so on...
I'll try to exclude most of the projects of my workspace , zip my workspace and share it on my google drive, would it be ok?
The workspace will look awful with lot of broken things, but mainly it will continue to loop forever (hopefully also for you!), it's the most important no ? ;)
Of course there will still be private stuff inside, please give me a private email address, so I can send you the google drive link ;)
Thierry.
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 11:59:33 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Unfortunately nothing special in the thread dumps. Could you possibly take a look at the indexing progress bar in "Progress" view while having this infinite indexing? Does the names of the files being indexed change or progress stuck only on a one particular
file? Maybe it would be possible for you to arrange some short Skype call so you can show me what is going on?
On 5/4/2016 1:38 PM, thierry blind wrote:
Here you have :)
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 09:39:25 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Thierry, could you possibly make some thread dump in a time when there is this infinite looping? I think that it might tell me something more.
On 5/4/2016 11:32 AM, thierry blind wrote:
Yes ;)
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 09:29:49 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Did you delete "org.eclipse.dltk.core.index.lucene" from .metadata before second run?
On 5/4/2016 11:15 AM, thierry blind wrote:
Second run, same loop problem, I had to kill eclipse, all errors that you see in log files happen when closing eclipse.
I'll create a fresh new workspace, copy a set of my PHP projects, import them and see what indexer does...
Thierry.
From:
thierryblind@xxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 08:29:03 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
I'll give a try and tell you ;)
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 08:24:21 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Is it still the same after closing PDT, deleting "org.eclipse.dltk.core.index.lucene" and starting IDE once again?
On 5/4/2016 10:17 AM, thierry blind wrote:
Hi Bart,
I killed my eclipse session yesterday evening, it was looping ;)
I've attached my logs ! There are full of errors :S
Thierry.
From:
bartlomiej.l@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 18:01:36 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Hi Guys,
First of all thank you Thierry for spending some time on testing new indexer. The problem that you found doesn't look good as you already said. I will try to find the root cause as fast as possible. Could you possibly attach your entire .log file?
Greetings,
Bart
On 5/3/2016 7:23 PM, thierry blind wrote:
Hi Kaloyan,
I've restarted everything (no more errors for now), first time it was probably my fault (I had an eclipse zombie process when I started a new eclipse instance).
I'm working on an existing workspace for now.
But there is still something very very wrong here.
I have very big projects (mainly because I use ORMs like Propel to generate a lot of PHP classes), and it seems that the indexer loops over same (Propel) files again and again.
The size of the "org.eclipse.dltk.core.index.lucene" directory is much bigger than the "org.eclipse.dltk.core.index.sql.h2 directory" !!!!
And indexing is still not ended... I will probably stop it before it fills my hard drive.
I will continue tests tomorrow, sadly tomorrow will be the last day I can do some tests before next 2-3 weeks :/
Thierry.
From:
kaloyan.r@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 16:44:15 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Hi Thierry,
Does the same happen in a fresh new workspace?
Kaloyan
Hi Kaloyan,
I just updated to latest PDT snapshot 4.0.0.201605031251 (from
http://download.eclipse.org/tools/pdt/updates/latest-nightly),
but I can't get the Lucene indexer to work.
Before filling a bug report I wanted to know what I do wrong:
1) I deleted .metadata\.plugins\org.eclipse.dltk.core.index.sql.h2 from my workspace
2) I started eclipse, now I get an infinite loop of following errors:
eclipse.buildId=4.5.2.M20160212-1500
java.version=1.8.0_66
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=fr_FR
Framework arguments: -product org.eclipse.epp.package.jee.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product org.eclipse.epp.package.jee.product
org.eclipse.dltk.core.index.lucene
Error
Tue May 03 18:16:00 CEST 2016
Lock obtain timed out:
NativeFSLock@C:\workspace\.metadata\.plugins\org.eclipse.dltk.core.index.lucene\index\bc981875-9762-4936-b1d8-fbd9b4d8d2a3\timestamps\write.lock
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
NativeFSLock@C:\regroup\workspace10\.metadata\.plugins\org.eclipse.dltk.core.index.lucene\index\bc981875-9762-4936-b1d8-fbd9b4d8d2a3\timestamps\write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:89)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.getTimestampsWriter(IndexContainer.java:128)
at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.delete(IndexContainer.java:193)
at org.eclipse.dltk.internal.core.index.lucene.LuceneManager.delete(LuceneManager.java:225)
at org.eclipse.dltk.internal.core.index.lucene.LuceneIndexer.indexDocument(LuceneIndexer.java:161)
at org.eclipse.dltk.internal.core.index2.SourceModulesRequest.run(SourceModulesRequest.java:81)
at org.eclipse.dltk.core.search.indexing.AbstractJob.execute(AbstractJob.java:80)
at org.eclipse.dltk.internal.core.search.processing.JobManager.run(JobManager.java:473)
at java.lang.Thread.run(Thread.java:745)
-------------------------------------------------------------------------------------------------------------
eclipse.buildId=4.5.2.M20160212-1500
java.version=1.8.0_66
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=fr_FR
Framework arguments: -product org.eclipse.epp.package.jee.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product org.eclipse.epp.package.jee.product
org.eclipse.dltk.core.index.lucene
Error
Tue May 03 18:15:42 CEST 2016
null
java.lang.NullPointerException
at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.delete(IndexContainer.java:193)
at org.eclipse.dltk.internal.core.index.lucene.LuceneManager.delete(LuceneManager.java:225)
at org.eclipse.dltk.internal.core.index.lucene.LuceneIndexer.indexDocument(LuceneIndexer.java:161)
at org.eclipse.dltk.internal.core.index2.SourceModulesRequest.run(SourceModulesRequest.java:81)
at org.eclipse.dltk.core.search.indexing.AbstractJob.execute(AbstractJob.java:80)
at org.eclipse.dltk.internal.core.search.processing.JobManager.run(JobManager.java:473)
at java.lang.Thread.run(Thread.java:745)
-------------------------------------------------------------------------------------------------------------
Thierry.
From:
thierryblind@xxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 15:17:41 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Hope they enjoy their holidays, I will have mine next week ;)
I will test the Lucene indexer with great curiosity ;)
Thierry.
From:
kaloyan.r@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 15:07:13 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone
Hi,
for me it should be ok ;)
From:
kaloyan.r@xxxxxxxx
To:
pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 13:44:10 +0000
Subject: [pdt-dev] 4.0-M7 Milestone
Hi,
I'd like to declare the 4.0-M7 milestone and go through the SimRel validation to ensure that everything is correct with the new Lucene indexer.
Any last minute changes?
Kaloyan
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________ pdt-dev mailing list
pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev
|