[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [cdt-dev] Help with Indexer Bug
|
What you've described is what I tried in https://bugs.eclipse.org/bugs/show_bug.cgi?id=413768#c11. I had three
moderately sized sample projects. In two of those projects the indexer took twice as long to complete (and created a
noticably larger .pdom file). In the third project indexing took seven times longer (and also created a noticably
larger .pdom file).
I agree that having a fully accurate indexer is a key feature. Especially now that the indexer is use for more
user-visible things like the Codan checkers. However, it is also important to do so without destroying performance.
I would love to be proven wrong, but from my investigations I don't think there is a simple change that can be made to
resolve this issue. There is alot of history in the path to the current implementation. Accepting inaccurate results
seems to have been one of the key compromises that was made in order to achieve the current speed and size results.
-Andrew
On 13-10-02 10:23 AM, Joseph Henry wrote:
> So what is the solution here?
>
>
>
> Pragma once semantics are only valid within the bounds of the current compilation unit.
>
>
>
> I can see that when processing a.c, that it is a good assumption that all instances of a include guarded header are
> going to be the same, but it is not a good assumption that all the headers included in a.c are going to be exactly the
> same as the headers included in b.c.
>
>
>
> To me it seems like the index should store version based on the top most level file, so common.h from the a.c tree would
> be the same, but then when common.h is encountered from the b.c tree, it would be different.
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] *On Behalf Of *Schorn, Markus
> *Sent:* Wednesday, October 02, 2013 9:54 AM
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> I tried to give an explanation in my previous email:
>
> The attempt to correctly generate multiple versions resulted in way to many header versions, such that the indexing was
> too slow and the database grew to unreasonable sizes.
>
> Therefore CDT does make the assumption that a header file with pragma once semantics produces the same macro-definitions
> and declarations whenever it is included within one project.
>
> Markus.
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx] *On Behalf
> Of *Joseph Henry
> *Sent:* Wed, 02. 10. 2013 15:40
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> So here is what I have found:
>
>
>
> In CPreprocessor.detectIncludeGuard if an include guard is found, then pragmaOnce semantics are turned on, but what is
> causing this bug, is that trackSignificantMacros is never turned on.
>
>
>
> Why are significant macros not turned on just because an include guard is found? There are thousands of ways that this
> could result in the index being wrong.
>
>
>
> In my opinion, track significant macros should always be turned on.
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx] *On Behalf
> Of *Schorn, Markus
> *Sent:* Wednesday, October 02, 2013 2:15 AM
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> I guess common.h is identified as a file with pragma once semantics, and therefore the index will maintain only a single
> version of it. To verify this, you can put some useless code before the include-guard. This should cause the indexer to
> create the two different versions for common.h
>
>
>
> Why does this work like this?
>
> When implementing the support for multiple versions we ended up with the following situation:
>
> The attempt to correctly generate multiple versions resulted in way to many header versions, such that the indexing was
> too slow and the database grew to unreasonable sizes.
>
> Therefore CDT does make the assumption that a header file with pragma once semantics produces the same macro-definitions
> and declarations whenever it is included within one project.
>
>
>
> So, to overcome the indexing problem you need to find a way that prevents CDT from assigning the pragma once semantics
> property to this file.
>
>
>
> Markus.
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx] *On Behalf
> Of *Joseph Henry
> *Sent:* Tue, 01. 10. 2013 23:37
> *To:* CDT General developers list.
> *Cc:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> When processing b.c common.h is read from the index, and whatever version is read does not include the definition for
> SYMBOL.
>
> Sent from my iPhone
>
>
> On Oct 1, 2013, at 5:14 PM, "Sergey Prigogin" <eclipse.sprigogin@xxxxxxxxx <mailto:eclipse.sprigogin@xxxxxxxxx>> wrote:
>
> The next step is to verify that two variants of common.h are written to the index. These two variants should be
> produced while indexing a.c and b.c respectively due to SYMBOL being defined in one case and undefined in the other.
>
>
>
> -sergey
>
>
>
> On Tue, Oct 1, 2013 at 1:50 PM, Joseph Henry <Joseph.Henry@xxxxxxx <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Actually it does seem to be added.
>
>
>
> In the function CPrepocessor.executeIfdef
>
>
>
> There is a block of code:
>
>
>
> macro= fMacroDictionary.get(namechars);
>
> isTaken= (macro == *null*) == isIfndef;
>
> *if* (macro == *null*) {
>
> macro = *new* UndefinedMacro(namechars);
>
> fCurrentContext.significantMacroUndefined(namechars);
>
> } *else* {
>
> fCurrentContext.significantMacroDefined(namechars);
>
> }
>
>
>
>
>
> It gets added in the else.
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx
> <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of *Sergey Prigogin
> *Sent:* Tuesday, October 01, 2013 4:37 PM
>
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> Do you know if SYMBOL is added to significant macros of common.h? If not, what condition prevents it from being added?
>
>
>
> -sergey
>
>
>
> On Tue, Oct 1, 2013 at 1:15 PM, Joseph Henry <Joseph.Henry@xxxxxxx <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Ok,
>
>
>
> So I have (mostly) solved the problem with the #pragma once. I will be pushing the code change soon, but I have
> found another indexer issue.
>
>
>
> Lets say I have file a.h
>
>
>
> *#ifndef*A_H_
>
> *#define*A_H_
>
>
>
> *#define*SYMBOL 1
>
>
>
> *#endif*
>
>
>
> And file common.h
>
>
>
> *#ifndef*COMMON_H_
>
> *#define*COMMON_H_
>
>
>
> *#if*!defined(SYMBOL)
>
> *#define*SYMBOL 2
>
> *#endif*
>
>
>
> *#define*COMMON
>
>
>
> *#endif*
>
> * *
>
> If I have file a.c:
>
>
>
> *#include*"a.h"
>
>
>
> *#include*"common.h"
>
>
>
> *_int_*_* a = SYMBOL;_
>
>
>
> and b.c:
>
>
>
> *#include*"common.h"
>
>
>
> *int** b = _SYMBOL_;
>
>
>
> b.c will say the SYMBOL is undefined.
>
>
>
> This does not matter whether or not it is in the form of #ifndef SYMBOL or #if !defined(SYMBOL)
>
>
>
>
>
> I will be filling a bug for this shortly, but in the meantime, do you have any ideas what could be causing this?
>
>
>
> Obviously the indexer thinks that symbol has already been defined, but from b.c, it clearly has not.
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx
> <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of *Sergey Prigogin
> *Sent:* Monday, September 30, 2013 3:25 PM
>
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> I agree that presence of any conditions around #pragma once should not prevent detection of #ifdef/#defne/#endif
> include guard. It's hard for me to give any practical suggesting regarding the code, since I haven't seen it.
>
>
>
> -sergey
>
>
>
> On Mon, Sep 30, 2013 at 12:09 PM, Joseph Henry <Joseph.Henry@xxxxxxx <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Well I did find one case in which this will not work, maybe you can help me with this.
>
>
>
> So lets say that pragmaInclude.h was changed to this:
>
>
>
> *#if* __MSC_VER_ > 1000
>
> *#pragma*once
>
> *#endif*
>
>
>
> *#ifndef*PRAGMAINCLUDE_H_
>
> *#define*PRAGMAINCLUDE_H_
>
>
>
> *#define*SYMBOL 0
>
>
>
> *#endif*
>
>
>
> The code that I wrote will not skip over the #pragma once in this case. The only variables that I have in
> this function is the Lexer of the code, so I am not really sure how I can ascertain that I need to process
> this line or not.
>
>
>
> Any Ideas?
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx> [mailto:cdt-dev-bounces@xxxxxxxxxxx
> <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of *Sergey Prigogin
> *Sent:* Monday, September 30, 2013 3:01 PM
>
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
>
>
>
>
> On Mon, Sep 30, 2013 at 11:37 AM, Joseph Henry <Joseph.Henry@xxxxxxx <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Ok,
>
>
>
> So I have implemented a solution that seems to solve this problem.
>
>
>
> What I did was in IncludeGuardDetection.findIncludeGuard I added a some logic to skip the first #pragma
> once (if it exists).
>
>
>
> This seems to work without messing up anything.
>
>
>
> What is the next step? How would I go about possibly contributing this fix?
>
>
>
> This is great that you were able to implement a solution to the problem. Could you please push the patch to
> Gerrit (http://wiki.eclipse.org/CDT/git#Using_Gerrit_for_CDT) for review. Thanks.
>
>
>
> Thanks,
>
> Joseph Henry.
>
> -sergey
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>
> [mailto:cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of *Sergey Prigogin
>
>
> *Sent:* Sunday, September 22, 2013 1:42 AM
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> The attached project is quite more complex than what was discussed in this email thread so far. Is the
> problem reproducible with the code quoted in this email thread?
>
>
>
> -sergey
>
>
>
> On Fri, Sep 20, 2013 at 8:40 PM, Joseph Henry <Joseph.Henry@xxxxxxx <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> It is. I have attached a project that reproduces this case in both bugs named in this thread.
>
> Sent from my iPhone
>
>
> On Sep 20, 2013, at 5:02 PM, "Sergey Prigogin" <eclipse.sprigogin@xxxxxxxxx
> <mailto:eclipse.sprigogin@xxxxxxxxx>> wrote:
>
>
>
>
>
> On Fri, Sep 20, 2013 at 1:47 PM, Joseph Henry <Joseph.Henry@xxxxxxx
> <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> When CPreprocessor encounters #include "header.h" in b.c it should include content from
> both, header.h and stddef.h. header.h may be included from the index or as a source.
>
>
>
> But it does not. When header.h is read from the index, it contains nothing, except what is
> defined in header.h
>
>
>
> Lets say that header.h looks like this.
>
>
>
> #ifndef _HEADER
>
> #define _HEADER
>
>
>
> #include “stddef.h”
>
>
>
> #endif
>
>
>
> And lets say stddef.h looks like this
>
>
>
> #pragma once
>
> #define NULL 0
>
>
>
> In the function processInclusionFromIndex in CPrepocessor, the InternalFileContent is read
> from the index. When it calls getMacroDefinitions(), this returns only 1 definition which is
> _HEADER. If the #pragma once is not there this returns 2 definitions which are _HEADER and NULL
>
>
>
> There seems to be a gap between discussion in this email and the actual reproducible case. Is
> the problem reproducible with stddef.h qouted in this email?
>
>
>
> -sergey
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>
> [mailto:cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of
> *Sergey Prigogin
> *Sent:* Friday, September 20, 2013 4:07 PM
>
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
>
>
>
>
> On Fri, Sep 20, 2013 at 12:44 PM, Joseph Henry <Joseph.Henry@xxxxxxx
> <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Honestly I am not really sure.
>
>
>
> This is what I saw when stepping though the code in the debugger.
>
>
>
> File stddef.h has this structure:
>
>
>
> #pragma once
>
> #ifndef __STD
>
> #define __STD
>
>
>
> …..
>
>
>
> #endif
>
>
>
> I have a File with the following structure:
>
> a.c
> |_ stddef.h
> |_ header.h
> |_ stddef.h
>
>
>
> and there is another File with the following structure:
>
>
>
> b.c
> |_ header.h
> | |_ stddef.h
>
>
>
>
>
> Now this is what I know happens:
>
>
>
> When b.c is parsed, it finds the #include “header.h” and executes that statement.
>
> When it looks for header.h it finds it, and its location is in the index.
>
> Now header.h is included from the index, but it does not contain any instance of stddef.h.
>
>
>
> A header stored in the index never contains content from another header. Each header is
> stored separately. When CPreprocessor encounters #include "header.h" in b.c it should
> include content from both, header.h and stddef.h. header.h may be included from the index or
> as a source. This depends on significant macros of header.h.
>
>
>
> -sergey
>
>
>
> What I don’t understand is why (or where) header.h is stored in the index without any
> instance of stddef.h. It seems like a #pragma once should be treated just like a #ifndef
> #define #endif block. So from b.c since this file has not been encountered, it should be
> included.
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>
> [mailto:cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf Of
> *Sergey Prigogin
> *Sent:* Friday, September 20, 2013 3:12 PM
>
>
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
>
>
>
>
> On Fri, Sep 20, 2013 at 11:22 AM, Joseph Henry <Joseph.Henry@xxxxxxx
> <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Ok,
>
>
>
> After further looking into this, is seems that these are 2 separate issues. If you
> look at https://bugs.eclipse.org/bugs/show_bug.cgi?id=413768#c2 it shows a use case
> that supposedly should work, but that is the exact use case that is broken for me
> (with one minor change). I will explain.
>
>
>
> Basically I have this situation:
>
>
>
> a.c
>
> |_ stddef.h (version S1)
>
> |_ header.h
>
> |_ stddef.h (version S2)
>
> |_ use declaration found in S1
>
>
>
> And these another source file that contains header.h
>
>
>
> b.c
>
> |_ header.h
>
> | |_ stddef.h (version S1)
>
> |_ use declaration found in S1
>
>
>
>
>
> The problem comes into play where stddef.h has a #pragma once in it at the top of
> the file like:
>
>
>
> If stddef.h had #pragma once, there would not be S2, would it?
>
>
>
> Did you mean something like
>
>
>
> a.c
> |_ stddef.h (version S1)
> |_ header.h
> |_ stddef.h (version S1)
> |_ use declaration found in S1
>
> b.c
> |_ header.h
> | |_ stddef.h (version S2)
> |_ use declaration found in S2
>
>
>
> ?
>
>
>
> #pragma once
>
> #ifndef __STDDEF__
>
> #define __STDDEF__
>
>
>
> …….
>
> #define NULL 0
>
> ……
>
>
>
> #endif
>
>
>
> The #pragma once is what is causing the problem. Since it is outside of the #ifndef
> block, it seems like the version of stddef.h that is included in header.h is just
> simply not put in the index. So from b.c, the definition of NULL cannot be found.
>
>
>
> Maybe this explains my problem a bit more.
>
>
>
> Any help would be great.
>
>
>
> Thanks,
>
> Joseph Henry.
>
>
>
> -sergey
>
>
>
> *From:*cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>
> [mailto:cdt-dev-bounces@xxxxxxxxxxx <mailto:cdt-dev-bounces@xxxxxxxxxxx>] *On Behalf
> Of *Sergey Prigogin
> *Sent:* Friday, September 20, 2013 1:08 PM
> *To:* CDT General developers list.
> *Subject:* Re: [cdt-dev] Help with Indexer Bug
>
>
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=413768#c11 contains analysis of the
> problem and suggests a potential solution. I'm not sure whether that solution would
> not break other use cases, but it's at least something to try. I would start with
> diagnosing the problem described in bug 417193
> <https://bugs.eclipse.org/bugs/show_bug.cgi?id=417193> similar to how it wasdone for
> bug 413768 <https://bugs.eclipse.org/bugs/show_bug.cgi?id=413768>.
>
>
>
> -sergey
>
>
>
> On Fri, Sep 20, 2013 at 9:50 AM, Joseph Henry <Joseph.Henry@xxxxxxx
> <mailto:Joseph.Henry@xxxxxxx>> wrote:
>
> Hi all,
>
>
>
> I have filed bug 417193 <https://bugs.eclipse.org/bugs/show_bug.cgi?id=417193>,
> which seems to be related to bug 413768
> <https://bugs.eclipse.org/bugs/show_bug.cgi?id=413768>.
>
>
>
> I have not gotten much response on either of these bugs and was wondering if
> someone could point me in the right direction to fix this. I really need this
> bug fixed so I have been looking into it, but cant seem to make any headway.
>
>
>
> If anyone could point me in the right direction that would be great.
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx <mailto:cdt-dev@xxxxxxxxxxx>
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
>
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>