Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [pdt-dev] 4.0-M7 Milestone

Hi Bart,
thank you for the answer.
Yes perfect, you understood well the problem and your solution is a good compromise! ;)

Thierry.


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Fri, 13 May 2016 13:05:04 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Hi Thierry,
In general canceling indexing operation doesn't work for both indexers (current implementation for DLTK indexer doesn't care too much about cancel - see AbstractJob#isCancelled and ProgressJob). So while indexing source modules from huge project you can try to cancel it but it won't work as expected. The difference is only while closing IDE - as indexing thread is daemon, when you close IDE while being in the middle of indexing then in case of H2 everything will be just dropped when thread dies. In case of Lucene we wait for all indexing jobs to be finished as we must close index writers gracefully.
Anyway we can fix it of course :-). In general i plan to add patch proposal to DLTK that will enable canceling. It will give a possibility to cancel indexing in "Progress" view as well as while closing IDE ("Saving workspace state..." window where you can cancel ongoing operations). Anyway there might be some side-effect when canceling in "Progress" view as index might not be complete in that case and it will be rebuild/updated after next clean or IDE restart (but I guess that user should be aware of it while pressing "Stop" :-).
What do you think about it?
Greetings,
Bart

On 5/12/2016 10:32 PM, thierry blind wrote:
Hi Bart,
I have another question concerning the lucene indexer:
I was wondering why it's not possible to interrupt an indexing operation (for example to close eclipse asap and restart the indexing operation later).
With the H2 indexer I could stop the indexing (after some clicks on the stop button in the progress panel), or even close eclipse and then hit the indexing stop button.
Now you have no more control on the indexing process and the progress bar (with its stop button) becomes totally useless, and even worse, when you close eclipse, it continues the indexing as a background process, taking a lot of memory (and time)! Or you can kill eclipse...
When you take the big projects I've sent you, indexing takes time and you are "forced" to wait that indexing ends before being able to close eclipse, which is not the case with classic java projects and their build process. I see that as a regression, but maybe I'm too rude :S

Thierry.


From: thierryblind@xxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 16:23:07 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Thank you ! I'm glad to help ;) And thank you for your patience ;)

Thierry.


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 16:03:01 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Thierry, your help is invaluable :) You pointed me into the right direction. I didn't check the scenario with dependent projects but it looks like it is the root cause of the problems.

Thanks again for your help and have a nice holiday leave! :)

On 5/4/2016 5:53 PM, thierry blind wrote:
Maybe in my case I simply though it does loop forever, because I had a looooooot of big projects depending on same big project "A" (by doing Properties -> PHP -> Include Path -> Projects -> Add -> projet A).
And it took forever with a big amount of place taken by org.eclipse.dltk.core.index.lucene
Strangely it was much faster on a new and clean workspace :S
If I have for example 5 projects that depends on a common project "A", the project "A" will then be indexed 5 times.
In my case with H2 indexer it took 1.1 GB, with the Lucene indexer it takes more than 3GB on a safe and new workspace!
So really sorry for pointing in the wrong direction :/
And yes there is still room for improvements ;)


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 15:39:01 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

OK, great. For now I have found something else while using your workspace that is not OK. Even if indexing finishes OK there are some source modules being re-indexed while closing/restarting IDE. At least it might give me some clues what might happened as I think that the problem is related somehow to storing/reading source files timestamps as this is the place where some redundant indexing might be triggered.

On 5/4/2016 5:07 PM, thierry blind wrote:
I'll repackage another workspace and tell you when it's available ok ? Maybe I removed too much...


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 15:00:53 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

I used your workspace in my dev env but unfortunately indexer finished without a problem. It only took some time to close the PDT running threads (after closing IDE) as Lucene was merging and closing huge indexes (for about 45-60 seconds) but it finished it gracefully. Anyway I will try to do my best to reproduce it ;)

On 5/4/2016 4:13 PM, thierry blind wrote:
With pleasure ;)
You can send it at my email address: thierryblind@xxxxxxx
Kaloyan, if you're also interested, tell me ok?

I really hope you will also reproduce this bug on your eclipse, I will go crazy if I'm the only one to have it! lol
Of course each time you start your eclipse you have to delete .metadata/org.eclipse.dltk.core.index.lucene

And some additional infos on what's installed in my eclipse Mars.2 Release:

  Eclipse IDE for Java EE Developers    4.5.2.20160218-0600    epp.package.jee    null
  JavaHL 1.7.9 Win32 Binaries (Optional)    5.0.0.I20150529-1700    org.polarion.eclipse.team.svn.connector.javahl17.win32.feature.group    Polarion Software
  PDT Extensions    0.22.2.201604260159    org.pdtextensions.feature.feature.group    http://pdt-extensions.org
  PHP CodeSniffer    1.3.0.R20111119000000    org.phpsrc.eclipse.pti.tools.codesniffer.feature.group    Sven Kiera
  PHP Development Tools (PDT)    4.0.0.201605031251    org.eclipse.php.feature.group    Eclipse PDT
  PHP Development Tools (PDT) Mylyn Integration    4.0.0.201605031251    org.eclipse.php.mylyn.feature.group    Eclipse PDT
  PHP Development Tools (PDT) Source Code    4.0.0.201605031251    org.eclipse.php.source.feature.group    Eclipse PDT
  PHP Tool Integration Core    1.6.0.R20111119000000    org.phpsrc.eclipse.pti.core.feature.group    Sven Kiera
  PHPUnit    0.8.0.R20111119000000    org.phpsrc.eclipse.pti.tools.phpunit.feature.group    Sven Kiera
  PMD Plug-in    4.0.8.v20151204-2156    net.sourceforge.pmd.eclipse.feature.group    PMD Project
  Subversive SVN Team Provider    3.0.4.I20160131-1700    org.eclipse.team.svn.feature.group    Eclipse.org

Thierry.


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 14:02:11 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

That would be great! Thank you one more time for all the feedback! This zipped workspace would be very appreciated, I will send you soon my private address that you can send the google drive link at.

On 5/4/2016 3:50 PM, thierry blind wrote:
It won't be possible at work for Skype sorry :S
And I followed progress bar since first try ;)
Here is what happen : I have PHP projects A, B, C and D. Projects B, C and D depend on project A.
So indexer starts with project A, continues with project B (only few files), goes back to A, does a little bit from C, goes back to A, does a little bit from D, and then it seems it only loops inside of A. It seems that if I only have project A open in my workspace, indexer works correctly and then I can open project B and indexer still works correctly, and so on...

I'll try to exclude most of the projects of my workspace , zip my workspace and share it on my google drive, would it be ok?
The workspace will look awful with lot of broken things, but mainly it will continue to loop forever (hopefully also for you!), it's the most important no ? ;)
Of course there will still be private stuff inside, please give me a private email address, so I can send you the google drive link ;)

Thierry.


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 11:59:33 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Unfortunately nothing special in the thread dumps. Could you possibly take a look at the indexing progress bar in "Progress" view while having this infinite indexing? Does the names of the files being indexed change or progress stuck only on a one particular file? Maybe it would be possible for you to arrange some short Skype call so you can show me what is going on?

On 5/4/2016 1:38 PM, thierry blind wrote:
Here you have :)


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 09:39:25 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Thierry, could you possibly make some thread dump in a time when there is this infinite looping? I think that it might tell me something more.

On 5/4/2016 11:32 AM, thierry blind wrote:
Yes ;)


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 09:29:49 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Did you delete "org.eclipse.dltk.core.index.lucene" from .metadata before second run?

On 5/4/2016 11:15 AM, thierry blind wrote:
Second run, same loop problem, I had to kill eclipse, all errors that you see in log files happen when closing eclipse.

I'll create a fresh new workspace, copy a set of my PHP projects, import them and see what indexer does...

Thierry.


From: thierryblind@xxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 08:29:03 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

I'll give a try and tell you ;)


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Wed, 4 May 2016 08:24:21 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Is it still the same after closing PDT, deleting "org.eclipse.dltk.core.index.lucene" and starting IDE once again?

On 5/4/2016 10:17 AM, thierry blind wrote:
Hi Bart,
I killed my eclipse session yesterday evening, it was looping ;)
I've attached my logs ! There are full of errors :S

Thierry.


From: bartlomiej.l@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 18:01:36 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Hi Guys,

First of all thank you Thierry for spending some time on testing new indexer. The problem that you found doesn't look good as you already said. I will try to find the root cause as fast as possible. Could you possibly attach your entire .log file?

Greetings,
Bart

On 5/3/2016 7:23 PM, thierry blind wrote:
Hi Kaloyan,
I've restarted everything (no more errors for now), first time it was probably my fault (I had an eclipse zombie process when I started a new eclipse instance).
I'm working on an existing workspace for now.
But there is still something very very wrong here.
I have very big projects (mainly because I use ORMs like Propel to generate a lot of PHP classes), and it seems that the indexer loops over same (Propel) files again and again.
The size of the "org.eclipse.dltk.core.index.lucene" directory is much bigger than the "org.eclipse.dltk.core.index.sql.h2 directory" !!!!
And indexing is still not ended... I will probably stop it before it fills my hard drive.
I will continue tests tomorrow, sadly tomorrow will be the last day I can do some tests before next 2-3 weeks :/

Thierry.


From: kaloyan.r@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 16:44:15 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Hi Thierry,

Does the same happen in a fresh new workspace?

Kaloyan



From: pdt-dev-bounces@xxxxxxxxxxx <pdt-dev-bounces@xxxxxxxxxxx> on behalf of thierry blind <thierryblind@xxxxxxx>
Sent: Tuesday, May 3, 2016 7:29 PM
To: PDT Developers
Subject: Re: [pdt-dev] 4.0-M7 Milestone
 
Hi Kaloyan,
I just updated to latest PDT snapshot 4.0.0.201605031251 (from http://download.eclipse.org/tools/pdt/updates/latest-nightly), but I can't get the Lucene indexer to work.
Before filling a bug report I wanted to know what I do wrong:
1) I deleted .metadata\.plugins\org.eclipse.dltk.core.index.sql.h2 from my workspace
2) I started eclipse, now I get an infinite loop of following errors:

eclipse.buildId=4.5.2.M20160212-1500
java.version=1.8.0_66
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=fr_FR
Framework arguments:  -product org.eclipse.epp.package.jee.product
Command-line arguments:  -os win32 -ws win32 -arch x86 -product org.eclipse.epp.package.jee.product

org.eclipse.dltk.core.index.lucene
Error
Tue May 03 18:16:00 CEST 2016
Lock obtain timed out: NativeFSLock@C:\workspace\.metadata\.plugins\org.eclipse.dltk.core.index.lucene\index\bc981875-9762-4936-b1d8-fbd9b4d8d2a3\timestamps\write.lock

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@C:\regroup\workspace10\.metadata\.plugins\org.eclipse.dltk.core.index.lucene\index\bc981875-9762-4936-b1d8-fbd9b4d8d2a3\timestamps\write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
    at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.getTimestampsWriter(IndexContainer.java:128)
    at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.delete(IndexContainer.java:193)
    at org.eclipse.dltk.internal.core.index.lucene.LuceneManager.delete(LuceneManager.java:225)
    at org.eclipse.dltk.internal.core.index.lucene.LuceneIndexer.indexDocument(LuceneIndexer.java:161)
    at org.eclipse.dltk.internal.core.index2.SourceModulesRequest.run(SourceModulesRequest.java:81)
    at org.eclipse.dltk.core.search.indexing.AbstractJob.execute(AbstractJob.java:80)
    at org.eclipse.dltk.internal.core.search.processing.JobManager.run(JobManager.java:473)
    at java.lang.Thread.run(Thread.java:745)

-------------------------------------------------------------------------------------------------------------

eclipse.buildId=4.5.2.M20160212-1500
java.version=1.8.0_66
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=fr_FR
Framework arguments:  -product org.eclipse.epp.package.jee.product
Command-line arguments:  -os win32 -ws win32 -arch x86 -product org.eclipse.epp.package.jee.product

org.eclipse.dltk.core.index.lucene
Error
Tue May 03 18:15:42 CEST 2016
null

java.lang.NullPointerException
    at org.eclipse.dltk.internal.core.index.lucene.IndexContainer.delete(IndexContainer.java:193)
    at org.eclipse.dltk.internal.core.index.lucene.LuceneManager.delete(LuceneManager.java:225)
    at org.eclipse.dltk.internal.core.index.lucene.LuceneIndexer.indexDocument(LuceneIndexer.java:161)
    at org.eclipse.dltk.internal.core.index2.SourceModulesRequest.run(SourceModulesRequest.java:81)
    at org.eclipse.dltk.core.search.indexing.AbstractJob.execute(AbstractJob.java:80)
    at org.eclipse.dltk.internal.core.search.processing.JobManager.run(JobManager.java:473)
    at java.lang.Thread.run(Thread.java:745)

-------------------------------------------------------------------------------------------------------------

Thierry.


From: thierryblind@xxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 15:17:41 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Hope they enjoy their holidays, I will have mine next week ;)
I will test the Lucene indexer with great curiosity ;)

Thierry.


From: kaloyan.r@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 15:07:13 +0000
Subject: Re: [pdt-dev] 4.0-M7 Milestone

Thanks, Thierry!


I guess the rest are enjoying the public holiday in Poland and are not working hard on PDT ??


I went ahead and copied the latest nightly to this location:
http://download.eclipse.org/tools/pdt/updates/4.0-m7



I am pushed it to the SimRel: https://git.eclipse.org/r/71914/


Kaloyan



From: pdt-dev-bounces@xxxxxxxxxxx <pdt-dev-bounces@xxxxxxxxxxx> on behalf of thierry blind <thierryblind@xxxxxxx>
Sent: Tuesday, May 3, 2016 5:07:26 PM
To: PDT Developers
Subject: Re: [pdt-dev] 4.0-M7 Milestone
 
Hi,
for me it should be ok ;)


From: kaloyan.r@xxxxxxxx
To: pdt-dev@xxxxxxxxxxx
Date: Tue, 3 May 2016 13:44:10 +0000
Subject: [pdt-dev] 4.0-M7 Milestone

Hi,


I'd like to declare the 4.0-M7 milestone and go through the SimRel validation to ensure that everything is correct with the new Lucene indexer.


Any last minute changes?


Kaloyan

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________
pdt-dev mailing list
pdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/pdt-dev


_______________________________________________ pdt-dev mailing list pdt-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/pdt-dev

Back to the top