Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Dynamic Languages Toolkit (DLTK) » parsers, builders, build types, and build participants
parsers, builders, build types, and build participants [message #32883] Tue, 07 October 2008 00:09 Go to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
I am trying to figure out how to reorganize my sources to properly build and
run in the new 1.0M2 version of DLTK.

One important issue is that we build our own custom model of
a) our language's source code
b) the run-time-library that it can call

We have
1) a parser
2) a custom model builder (depends on AST)
3) a semantic checker (depends on AST and our custom model)

1) The parser wrapper class, VScriptSourceParser, implements
ISourceParser.pase.
That method also calls (2), our custom model builder.

2) The semantic checker is invoked from VScriptSourceElementParser, which
extends AbstractSourceElementParser (which uses SourceElementRequestVisitor
to build the DLTK model) and implements ISourceElementParser. My parse
method also builds my custom model.

This structure seems to mostly work correctly when invoked dynamically from
the editor.
In addition, I have built a custom builder which
1) Pass 1 - for each module:
a) attempts to fetch the AST tree from the DLTK module cache by calling
SourceParserUtils.getModuleDeclaration. If the cache is empty or
out-of-date, it
i) reparses the compilation unit
ii) calls your SourceElementVisitor to rebuild your DLTK model
iii) calls my custom visitor to rebuild my custom model
2) Pass 2 - for all modules
a) calls my semantic checker (after your DLTK model and my custom model
are built for each module)

You've informed us via this newsgroup of the new interface
IBuildParticipant.
These are fetched in the IScriptBuilder.initialize method.
They are used in ValidatorBuilder.buildModule
and ValidatorStructureBuilder.buildStructure.

My questions are:

1) Does the structure I have described above still make sense in the new
version, i.e.
a) adding a custom model builder to my class which implements
ISourceParser and
b) adding a semantic checker which uses the custom model to
ISourceElementParser?
2) What is the difference between ValidatorBuilder and
ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
3) Even though I want to use the new IBuildParticipant interfaces that
you have described,
don't I have to write my own custom builder to make sure I have built
my own custom model for each
module before I attempt to do semantic checking on any module (which
uses the custom model)?
4) Can you suggest other/better ways of doing what I want to do that fit
into your architecture?

Thanks,
Chuck
Re: parsers, builders, build types, and build participants [message #33059 is a reply to message #32883] Tue, 07 October 2008 07:13 Go to previous messageGo to next message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Chuck,

Just after M2 we have generalized the builder infrastructure, please see
this message http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html

I would recommend you to use DLTK from CVS or fresh integration builds
so you can more closely follow DLTK changes.

Now the builder configuration is simpler - you provide multiple
implementations of the IBuildParticipant and that's all.
SourceParser/SourceElementParser should not be affected - they should
perform only their primary task.

If you need to know type of the build operation
(full/incremental/reconcile) or perform additional actions after all
source modules are processed - you should implement
org.eclipse.dltk.core.builder.IBuildParticipantExtension

Regards,
Alex

Chuck Doucette wrote:
> I am trying to figure out how to reorganize my sources to properly build and
> run in the new 1.0M2 version of DLTK.
>
> One important issue is that we build our own custom model of
> a) our language's source code
> b) the run-time-library that it can call
>
> We have
> 1) a parser
> 2) a custom model builder (depends on AST)
> 3) a semantic checker (depends on AST and our custom model)
>
> 1) The parser wrapper class, VScriptSourceParser, implements
> ISourceParser.pase.
> That method also calls (2), our custom model builder.
>
> 2) The semantic checker is invoked from VScriptSourceElementParser, which
> extends AbstractSourceElementParser (which uses SourceElementRequestVisitor
> to build the DLTK model) and implements ISourceElementParser. My parse
> method also builds my custom model.
>
> This structure seems to mostly work correctly when invoked dynamically from
> the editor.
> In addition, I have built a custom builder which
> 1) Pass 1 - for each module:
> a) attempts to fetch the AST tree from the DLTK module cache by calling
> SourceParserUtils.getModuleDeclaration. If the cache is empty or
> out-of-date, it
> i) reparses the compilation unit
> ii) calls your SourceElementVisitor to rebuild your DLTK model
> iii) calls my custom visitor to rebuild my custom model
> 2) Pass 2 - for all modules
> a) calls my semantic checker (after your DLTK model and my custom model
> are built for each module)
>
> You've informed us via this newsgroup of the new interface
> IBuildParticipant.
> These are fetched in the IScriptBuilder.initialize method.
> They are used in ValidatorBuilder.buildModule
> and ValidatorStructureBuilder.buildStructure.
>
> My questions are:
>
> 1) Does the structure I have described above still make sense in the new
> version, i.e.
> a) adding a custom model builder to my class which implements
> ISourceParser and
> b) adding a semantic checker which uses the custom model to
> ISourceElementParser?
> 2) What is the difference between ValidatorBuilder and
> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
> 3) Even though I want to use the new IBuildParticipant interfaces that
> you have described,
> don't I have to write my own custom builder to make sure I have built
> my own custom model for each
> module before I attempt to do semantic checking on any module (which
> uses the custom model)?
> 4) Can you suggest other/better ways of doing what I want to do that fit
> into your architecture?
>
> Thanks,
> Chuck
>
>
Re: parsers, builders, build types, and build participants [message #33197 is a reply to message #33059] Tue, 07 October 2008 21:42 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
Indeed - I would prefer to build my model in one place.
The builder is preferable over the indexer.
However, I would like to reflect the changes in the editor (i.e. a working
copy) in an alternate version of my custom model - not just when the file is
saved and built as a compilation unit.

However, right now it appears as if the DLTK model is built in the method
AbstractSourceElementParser.parseSourceModule via a call to
SourceElementRequestVisitor.traverse.

I assume I should build my custom model in my class which extends
AbstractSourceElementParser.

I'm not sure if the parser and DLTK model builder is invoked from a builder
like ValidatorBuilder or ValidatorStructureBuilder.

Chuck

"Alex Panchenko" <alex@xored.com> wrote in message
news:gcf26r$6br$1@build.eclipse.org...
> Chuck,
>
> Just after M2 we have generalized the builder infrastructure, please see
> this message http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>
> I would recommend you to use DLTK from CVS or fresh integration builds so
> you can more closely follow DLTK changes.
>
> Now the builder configuration is simpler - you provide multiple
> implementations of the IBuildParticipant and that's all.
> SourceParser/SourceElementParser should not be affected - they should
> perform only their primary task.
>
> If you need to know type of the build operation
> (full/incremental/reconcile) or perform additional actions after all
> source modules are processed - you should implement
> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>
> Regards,
> Alex
>
> Chuck Doucette wrote:
>> I am trying to figure out how to reorganize my sources to properly build
>> and run in the new 1.0M2 version of DLTK.
>>
>> One important issue is that we build our own custom model of
>> a) our language's source code
>> b) the run-time-library that it can call
>>
>> We have
>> 1) a parser
>> 2) a custom model builder (depends on AST)
>> 3) a semantic checker (depends on AST and our custom model)
>>
>> 1) The parser wrapper class, VScriptSourceParser, implements
>> ISourceParser.pase.
>> That method also calls (2), our custom model builder.
>>
>> 2) The semantic checker is invoked from VScriptSourceElementParser,
>> which extends AbstractSourceElementParser (which uses
>> SourceElementRequestVisitor to build the DLTK model) and implements
>> ISourceElementParser. My parse method also builds my custom model.
>>
>> This structure seems to mostly work correctly when invoked dynamically
>> from the editor.
>> In addition, I have built a custom builder which
>> 1) Pass 1 - for each module:
>> a) attempts to fetch the AST tree from the DLTK module cache by calling
>> SourceParserUtils.getModuleDeclaration. If the cache is empty or
>> out-of-date, it
>> i) reparses the compilation unit
>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>> iii) calls my custom visitor to rebuild my custom model
>> 2) Pass 2 - for all modules
>> a) calls my semantic checker (after your DLTK model and my custom
>> model are built for each module)
>>
>> You've informed us via this newsgroup of the new interface
>> IBuildParticipant.
>> These are fetched in the IScriptBuilder.initialize method.
>> They are used in ValidatorBuilder.buildModule
>> and ValidatorStructureBuilder.buildStructure.
>>
>> My questions are:
>>
>> 1) Does the structure I have described above still make sense in the
>> new version, i.e.
>> a) adding a custom model builder to my class which implements
>> ISourceParser and
>> b) adding a semantic checker which uses the custom model to
>> ISourceElementParser?
>> 2) What is the difference between ValidatorBuilder and
>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>> 3) Even though I want to use the new IBuildParticipant interfaces that
>> you have described,
>> don't I have to write my own custom builder to make sure I have
>> built my own custom model for each
>> module before I attempt to do semantic checking on any module
>> (which uses the custom model)?
>> 4) Can you suggest other/better ways of doing what I want to do that
>> fit into your architecture?
>>
>> Thanks,
>> Chuck
>>
Re: parsers, builders, build types, and build participants [message #33233 is a reply to message #33197] Wed, 08 October 2008 00:56 Go to previous messageGo to next message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Chuck,

In the past (0.95) the best place to add additional code was source
element parser, because there were no builder implementations provided
by the DLTK.

Now (1.0) there is a builder, so you should move you code out of the
source element parser to the separate classes.

In current CVS HEAD (and last integration builds) all builder elements
are sorted out to the correct places, though several iterations during
1.0 life-cycle were required to make general enough solution. To avoid
any possible confusion I will describe only the current (post M2) state.

org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
contributed IBuildParticipants and that's all.

If you want standard parser - you should specify it as the first step in
the sequence - see the ruby example.

The goals of the builder are
a) report errors
b) if applicable, compile and generate compiled data structures or files

DLTK model is not the essential part of these goals, so it is not called
automatically. Actually, now nothing is called automatically - only the
specified IBuildParticipants are executed. So application could use
different version of the parser in builder, if needed.

Also, IBuildParticipants are called during reconcile operation to
process not saved files in the editor.

It should be simpler now.

Regards,
Alex


Chuck Doucette wrote:
> Indeed - I would prefer to build my model in one place.
> The builder is preferable over the indexer.
> However, I would like to reflect the changes in the editor (i.e. a working
> copy) in an alternate version of my custom model - not just when the file is
> saved and built as a compilation unit.
>
> However, right now it appears as if the DLTK model is built in the method
> AbstractSourceElementParser.parseSourceModule via a call to
> SourceElementRequestVisitor.traverse.
>
> I assume I should build my custom model in my class which extends
> AbstractSourceElementParser.
>
> I'm not sure if the parser and DLTK model builder is invoked from a builder
> like ValidatorBuilder or ValidatorStructureBuilder.
>
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:gcf26r$6br$1@build.eclipse.org...
>> Chuck,
>>
>> Just after M2 we have generalized the builder infrastructure, please see
>> this message http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>
>> I would recommend you to use DLTK from CVS or fresh integration builds so
>> you can more closely follow DLTK changes.
>>
>> Now the builder configuration is simpler - you provide multiple
>> implementations of the IBuildParticipant and that's all.
>> SourceParser/SourceElementParser should not be affected - they should
>> perform only their primary task.
>>
>> If you need to know type of the build operation
>> (full/incremental/reconcile) or perform additional actions after all
>> source modules are processed - you should implement
>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>
>> Regards,
>> Alex
>>
>> Chuck Doucette wrote:
>>> I am trying to figure out how to reorganize my sources to properly build
>>> and run in the new 1.0M2 version of DLTK.
>>>
>>> One important issue is that we build our own custom model of
>>> a) our language's source code
>>> b) the run-time-library that it can call
>>>
>>> We have
>>> 1) a parser
>>> 2) a custom model builder (depends on AST)
>>> 3) a semantic checker (depends on AST and our custom model)
>>>
>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>> ISourceParser.pase.
>>> That method also calls (2), our custom model builder.
>>>
>>> 2) The semantic checker is invoked from VScriptSourceElementParser,
>>> which extends AbstractSourceElementParser (which uses
>>> SourceElementRequestVisitor to build the DLTK model) and implements
>>> ISourceElementParser. My parse method also builds my custom model.
>>>
>>> This structure seems to mostly work correctly when invoked dynamically
>>> from the editor.
>>> In addition, I have built a custom builder which
>>> 1) Pass 1 - for each module:
>>> a) attempts to fetch the AST tree from the DLTK module cache by calling
>>> SourceParserUtils.getModuleDeclaration. If the cache is empty or
>>> out-of-date, it
>>> i) reparses the compilation unit
>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>> iii) calls my custom visitor to rebuild my custom model
>>> 2) Pass 2 - for all modules
>>> a) calls my semantic checker (after your DLTK model and my custom
>>> model are built for each module)
>>>
>>> You've informed us via this newsgroup of the new interface
>>> IBuildParticipant.
>>> These are fetched in the IScriptBuilder.initialize method.
>>> They are used in ValidatorBuilder.buildModule
>>> and ValidatorStructureBuilder.buildStructure.
>>>
>>> My questions are:
>>>
>>> 1) Does the structure I have described above still make sense in the
>>> new version, i.e.
>>> a) adding a custom model builder to my class which implements
>>> ISourceParser and
>>> b) adding a semantic checker which uses the custom model to
>>> ISourceElementParser?
>>> 2) What is the difference between ValidatorBuilder and
>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>> 3) Even though I want to use the new IBuildParticipant interfaces that
>>> you have described,
>>> don't I have to write my own custom builder to make sure I have
>>> built my own custom model for each
>>> module before I attempt to do semantic checking on any module
>>> (which uses the custom model)?
>>> 4) Can you suggest other/better ways of doing what I want to do that
>>> fit into your architecture?
>>>
>>> Thanks,
>>> Chuck
>>>
>
Re: parsers, builders, build types, and build participants [message #33268 is a reply to message #33233] Wed, 08 October 2008 22:38 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
Thank you for your response.
In general - that sounds great.
I just wrap all of my build participants in classes which implement
IBuildParticipant,
and then they are called as part of a build and/or from a reconcile
operation.

Should that include the parser itself - or is that called separately?

One of my build participants builds my own custom model (rather than a DLTK
model).
The other build participant is a semantic checker which depends on the
existence of that custom model - both for the current module and for any
other referenced modules.

Is there any way I can organize the way that my build partipants can be
called other than writing a custom builder (which I had already done)?

The real issue is that my custom model is not serialized, so it must be
recreated after reparsing each file. Is your model serialized into a build
State object?

Thanks,
Chuck

"Alex Panchenko" <alex@xored.com> wrote in message
news:gch0fa$b71$1@build.eclipse.org...
> Chuck,
>
> In the past (0.95) the best place to add additional code was source
> element parser, because there were no builder implementations provided by
> the DLTK.
>
> Now (1.0) there is a builder, so you should move you code out of the
> source element parser to the separate classes.
>
> In current CVS HEAD (and last integration builds) all builder elements are
> sorted out to the correct places, though several iterations during 1.0
> life-cycle were required to make general enough solution. To avoid any
> possible confusion I will describe only the current (post M2) state.
>
> org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
> contributed IBuildParticipants and that's all.
>
> If you want standard parser - you should specify it as the first step in
> the sequence - see the ruby example.
>
> The goals of the builder are
> a) report errors
> b) if applicable, compile and generate compiled data structures or files
>
> DLTK model is not the essential part of these goals, so it is not called
> automatically. Actually, now nothing is called automatically - only the
> specified IBuildParticipants are executed. So application could use
> different version of the parser in builder, if needed.
>
> Also, IBuildParticipants are called during reconcile operation to process
> not saved files in the editor.
>
> It should be simpler now.
>
> Regards,
> Alex
>
>
> Chuck Doucette wrote:
>> Indeed - I would prefer to build my model in one place.
>> The builder is preferable over the indexer.
>> However, I would like to reflect the changes in the editor (i.e. a
>> working copy) in an alternate version of my custom model - not just when
>> the file is saved and built as a compilation unit.
>>
>> However, right now it appears as if the DLTK model is built in the method
>> AbstractSourceElementParser.parseSourceModule via a call to
>> SourceElementRequestVisitor.traverse.
>>
>> I assume I should build my custom model in my class which extends
>> AbstractSourceElementParser.
>>
>> I'm not sure if the parser and DLTK model builder is invoked from a
>> builder like ValidatorBuilder or ValidatorStructureBuilder.
>>
>> Chuck
>>
>> "Alex Panchenko" <alex@xored.com> wrote in message
>> news:gcf26r$6br$1@build.eclipse.org...
>>> Chuck,
>>>
>>> Just after M2 we have generalized the builder infrastructure, please see
>>> this message http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>>
>>> I would recommend you to use DLTK from CVS or fresh integration builds
>>> so you can more closely follow DLTK changes.
>>>
>>> Now the builder configuration is simpler - you provide multiple
>>> implementations of the IBuildParticipant and that's all.
>>> SourceParser/SourceElementParser should not be affected - they should
>>> perform only their primary task.
>>>
>>> If you need to know type of the build operation
>>> (full/incremental/reconcile) or perform additional actions after all
>>> source modules are processed - you should implement
>>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>>
>>> Regards,
>>> Alex
>>>
>>> Chuck Doucette wrote:
>>>> I am trying to figure out how to reorganize my sources to properly
>>>> build and run in the new 1.0M2 version of DLTK.
>>>>
>>>> One important issue is that we build our own custom model of
>>>> a) our language's source code
>>>> b) the run-time-library that it can call
>>>>
>>>> We have
>>>> 1) a parser
>>>> 2) a custom model builder (depends on AST)
>>>> 3) a semantic checker (depends on AST and our custom model)
>>>>
>>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>>> ISourceParser.pase.
>>>> That method also calls (2), our custom model builder.
>>>>
>>>> 2) The semantic checker is invoked from VScriptSourceElementParser,
>>>> which extends AbstractSourceElementParser (which uses
>>>> SourceElementRequestVisitor to build the DLTK model) and implements
>>>> ISourceElementParser. My parse method also builds my custom model.
>>>>
>>>> This structure seems to mostly work correctly when invoked dynamically
>>>> from the editor.
>>>> In addition, I have built a custom builder which
>>>> 1) Pass 1 - for each module:
>>>> a) attempts to fetch the AST tree from the DLTK module cache by calling
>>>> SourceParserUtils.getModuleDeclaration. If the cache is empty or
>>>> out-of-date, it
>>>> i) reparses the compilation unit
>>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>>> iii) calls my custom visitor to rebuild my custom model
>>>> 2) Pass 2 - for all modules
>>>> a) calls my semantic checker (after your DLTK model and my custom
>>>> model are built for each module)
>>>>
>>>> You've informed us via this newsgroup of the new interface
>>>> IBuildParticipant.
>>>> These are fetched in the IScriptBuilder.initialize method.
>>>> They are used in ValidatorBuilder.buildModule
>>>> and ValidatorStructureBuilder.buildStructure.
>>>>
>>>> My questions are:
>>>>
>>>> 1) Does the structure I have described above still make sense in the
>>>> new version, i.e.
>>>> a) adding a custom model builder to my class which implements
>>>> ISourceParser and
>>>> b) adding a semantic checker which uses the custom model to
>>>> ISourceElementParser?
>>>> 2) What is the difference between ValidatorBuilder and
>>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>>> 3) Even though I want to use the new IBuildParticipant interfaces
>>>> that you have described,
>>>> don't I have to write my own custom builder to make sure I have
>>>> built my own custom model for each
>>>> module before I attempt to do semantic checking on any module
>>>> (which uses the custom model)?
>>>> 4) Can you suggest other/better ways of doing what I want to do that
>>>> fit into your architecture?
>>>>
>>>> Thanks,
>>>> Chuck
>>>>
>>
Re: parsers, builders, build types, and build participants [message #33373 is a reply to message #33268] Mon, 13 October 2008 07:03 Go to previous messageGo to next message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Chuck Doucette wrote:
> Thank you for your response.
> In general - that sounds great.
> I just wrap all of my build participants in classes which implement
> IBuildParticipant,
> and then they are called as part of a build and/or from a reconcile
> operation.
>
> Should that include the parser itself - or is that called separately?

During reconcile operation first the DLTK model is updated.
The update consists of the call to the source element parser and most of
the languages call source parser in it the implementation of source
element parser.

During build step if the parser is required you should explicitly
specify it as the build participant. The explicit configuration allows
greater flexibility. There is standard implementation provided in the
class org.eclipse.dltk.core.builder.ParserBuildParticipantFactory (it is
listed first in the ruby example). Or you can make you own implementation.

> One of my build participants builds my own custom model (rather than a DLTK
> model).
> The other build participant is a semantic checker which depends on the
> existence of that custom model - both for the current module and for any
> other referenced modules.
>
> Is there any way I can organize the way that my build partipants can be
> called other than writing a custom builder (which I had already done)?

You should use
<extension point="org.eclipse.dltk.core.buildParticipant"> and specify
all the steps as buildParticipant elements. The subsequent steps could
have the <requires/> element, if they require the results created by the
previous ones.

Implementation of the IScriptBuilder interface is a good way too, the
main difference is that IBuildParticipants are called during reconcile
operations and they designed to operate at the finer level - per source
module.

> The real issue is that my custom model is not serialized, so it must be
> recreated after reparsing each file. Is your model serialized into a build
> State object?

DLTK model is not serializable and it is built by demand. It is not
built by the builder for the future use - it is built only when required.

>
> Thanks,
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:gch0fa$b71$1@build.eclipse.org...
>> Chuck,
>>
>> In the past (0.95) the best place to add additional code was source
>> element parser, because there were no builder implementations provided by
>> the DLTK.
>>
>> Now (1.0) there is a builder, so you should move you code out of the
>> source element parser to the separate classes.
>>
>> In current CVS HEAD (and last integration builds) all builder elements are
>> sorted out to the correct places, though several iterations during 1.0
>> life-cycle were required to make general enough solution. To avoid any
>> possible confusion I will describe only the current (post M2) state.
>>
>> org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
>> contributed IBuildParticipants and that's all.
>>
>> If you want standard parser - you should specify it as the first step in
>> the sequence - see the ruby example.
>>
>> The goals of the builder are
>> a) report errors
>> b) if applicable, compile and generate compiled data structures or files
>>
>> DLTK model is not the essential part of these goals, so it is not called
>> automatically. Actually, now nothing is called automatically - only the
>> specified IBuildParticipants are executed. So application could use
>> different version of the parser in builder, if needed.
>>
>> Also, IBuildParticipants are called during reconcile operation to process
>> not saved files in the editor.
>>
>> It should be simpler now.
>>
>> Regards,
>> Alex
>>
>>
>> Chuck Doucette wrote:
>>> Indeed - I would prefer to build my model in one place.
>>> The builder is preferable over the indexer.
>>> However, I would like to reflect the changes in the editor (i.e. a
>>> working copy) in an alternate version of my custom model - not just when
>>> the file is saved and built as a compilation unit.
>>>
>>> However, right now it appears as if the DLTK model is built in the method
>>> AbstractSourceElementParser.parseSourceModule via a call to
>>> SourceElementRequestVisitor.traverse.
>>>
>>> I assume I should build my custom model in my class which extends
>>> AbstractSourceElementParser.
>>>
>>> I'm not sure if the parser and DLTK model builder is invoked from a
>>> builder like ValidatorBuilder or ValidatorStructureBuilder.
>>>
>>> Chuck
>>>
>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>> news:gcf26r$6br$1@build.eclipse.org...
>>>> Chuck,
>>>>
>>>> Just after M2 we have generalized the builder infrastructure, please see
>>>> this message http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>>>
>>>> I would recommend you to use DLTK from CVS or fresh integration builds
>>>> so you can more closely follow DLTK changes.
>>>>
>>>> Now the builder configuration is simpler - you provide multiple
>>>> implementations of the IBuildParticipant and that's all.
>>>> SourceParser/SourceElementParser should not be affected - they should
>>>> perform only their primary task.
>>>>
>>>> If you need to know type of the build operation
>>>> (full/incremental/reconcile) or perform additional actions after all
>>>> source modules are processed - you should implement
>>>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>>>
>>>> Regards,
>>>> Alex
>>>>
>>>> Chuck Doucette wrote:
>>>>> I am trying to figure out how to reorganize my sources to properly
>>>>> build and run in the new 1.0M2 version of DLTK.
>>>>>
>>>>> One important issue is that we build our own custom model of
>>>>> a) our language's source code
>>>>> b) the run-time-library that it can call
>>>>>
>>>>> We have
>>>>> 1) a parser
>>>>> 2) a custom model builder (depends on AST)
>>>>> 3) a semantic checker (depends on AST and our custom model)
>>>>>
>>>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>>>> ISourceParser.pase.
>>>>> That method also calls (2), our custom model builder.
>>>>>
>>>>> 2) The semantic checker is invoked from VScriptSourceElementParser,
>>>>> which extends AbstractSourceElementParser (which uses
>>>>> SourceElementRequestVisitor to build the DLTK model) and implements
>>>>> ISourceElementParser. My parse method also builds my custom model.
>>>>>
>>>>> This structure seems to mostly work correctly when invoked dynamically
>>>>> from the editor.
>>>>> In addition, I have built a custom builder which
>>>>> 1) Pass 1 - for each module:
>>>>> a) attempts to fetch the AST tree from the DLTK module cache by calling
>>>>> SourceParserUtils.getModuleDeclaration. If the cache is empty or
>>>>> out-of-date, it
>>>>> i) reparses the compilation unit
>>>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>>>> iii) calls my custom visitor to rebuild my custom model
>>>>> 2) Pass 2 - for all modules
>>>>> a) calls my semantic checker (after your DLTK model and my custom
>>>>> model are built for each module)
>>>>>
>>>>> You've informed us via this newsgroup of the new interface
>>>>> IBuildParticipant.
>>>>> These are fetched in the IScriptBuilder.initialize method.
>>>>> They are used in ValidatorBuilder.buildModule
>>>>> and ValidatorStructureBuilder.buildStructure.
>>>>>
>>>>> My questions are:
>>>>>
>>>>> 1) Does the structure I have described above still make sense in the
>>>>> new version, i.e.
>>>>> a) adding a custom model builder to my class which implements
>>>>> ISourceParser and
>>>>> b) adding a semantic checker which uses the custom model to
>>>>> ISourceElementParser?
>>>>> 2) What is the difference between ValidatorBuilder and
>>>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>>>> 3) Even though I want to use the new IBuildParticipant interfaces
>>>>> that you have described,
>>>>> don't I have to write my own custom builder to make sure I have
>>>>> built my own custom model for each
>>>>> module before I attempt to do semantic checking on any module
>>>>> (which uses the custom model)?
>>>>> 4) Can you suggest other/better ways of doing what I want to do that
>>>>> fit into your architecture?
>>>>>
>>>>> Thanks,
>>>>> Chuck
>>>>>
>
>
Re: parsers, builders, build types, and build participants [message #33562 is a reply to message #33373] Thu, 23 October 2008 18:28 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
I am still a bit confused.

Since my project nature class extends ScriptNature, your builder,
ScriptBuilder, is automatically added to each of our DLTK projects' build
specs. Thus, when Eclipse wants to build a DLTK project, it calls your DLTK
builder class, ScriptBuilder, which implements IncrementalProjectBuilder.
That defers most of its work to any user's custom DLTK builder(s), which
implement IScriptBuilder.

My builder class, VScriptBuilder, which implements IScriptBuilder, calls
each of my build participants in order.
It currently calls them directly - without them implementing any
IBuildParticipant interface.
1. Phase 1 - parse to create AST tree - use AST tree to create custom model
(at least of declarations)
a) module's source contents -> parsing -> produces AST tree
b) AST tree -> custom model builder (extends ASTVisitor) -> custom model
2. Phase 2 - semantic check modules based on AST tree and custom model
a) AST tree -> semantic checker (extends AST Visitor - reports most
problems)

You say I should use IBuildParticipant - at least for 1b (my custom model
builder) and 2a (my semantic checker),
and perhaps also for my parser (but perhaps not if I split off the custom
model builder).
Apparently, IBuildParticipant is used by:
a) ValidatorBuilder
b) ValidatorStructureBuilder

ValidatorStructureBuilder implements IStructureBuilder, which is used by
AbstractSourceModule.buildStructure and
ReconcileWorkingCopyOperation.makeConsistent (one or the other used during
reconcilation).
I can't see where/how ValidatorBuilder is used at all - but perhaps that
doesn't matter.

So, do I need to
a) add extension wrapper classes around my builder components which
implement IBuildParticipant
b) keep my existing builder class, VScriptBuilder, since it implements
IScriptBuilder and will ultimately be called by the Eclipse build manager.
c) use your existing ValidatorStructureBuilder which will call my build
participants during reconciliation
d) remove calls to my semantic checker from my class
VScriptSourceElementParser, which extends ISourceElementParser (this is
where you had told me to put my semantic checker since it was getting called
during reconciliation)
e) remove calls to my custom model builder in my class VScriptSourceParser,
which extends ISourceParser.

Thanks,
Chuck

"Alex Panchenko" <alex@xored.com> wrote in message
news:gcurs7$61p$1@build.eclipse.org...
> Chuck Doucette wrote:
>> Thank you for your response.
>> In general - that sounds great.
>> I just wrap all of my build participants in classes which implement
>> IBuildParticipant,
>> and then they are called as part of a build and/or from a reconcile
>> operation.
>>
>> Should that include the parser itself - or is that called separately?
>
> During reconcile operation first the DLTK model is updated.
> The update consists of the call to the source element parser and most of
> the languages call source parser in it the implementation of source
> element parser.
>
> During build step if the parser is required you should explicitly specify
> it as the build participant. The explicit configuration allows greater
> flexibility. There is standard implementation provided in the class
> org.eclipse.dltk.core.builder.ParserBuildParticipantFactory (it is listed
> first in the ruby example). Or you can make you own implementation.
>
>> One of my build participants builds my own custom model (rather than a
>> DLTK model).
>> The other build participant is a semantic checker which depends on the
>> existence of that custom model - both for the current module and for any
>> other referenced modules.
>>
>> Is there any way I can organize the way that my build partipants can be
>> called other than writing a custom builder (which I had already done)?
>
> You should use
> <extension point="org.eclipse.dltk.core.buildParticipant"> and specify all
> the steps as buildParticipant elements. The subsequent steps could have
> the <requires/> element, if they require the results created by the
> previous ones.
>
> Implementation of the IScriptBuilder interface is a good way too, the main
> difference is that IBuildParticipants are called during reconcile
> operations and they designed to operate at the finer level - per source
> module.
>
>> The real issue is that my custom model is not serialized, so it must be
>> recreated after reparsing each file. Is your model serialized into a
>> build State object?
>
> DLTK model is not serializable and it is built by demand. It is not built
> by the builder for the future use - it is built only when required.
>
>>
>> Thanks,
>> Chuck
>>
>> "Alex Panchenko" <alex@xored.com> wrote in message
>> news:gch0fa$b71$1@build.eclipse.org...
>>> Chuck,
>>>
>>> In the past (0.95) the best place to add additional code was source
>>> element parser, because there were no builder implementations provided
>>> by the DLTK.
>>>
>>> Now (1.0) there is a builder, so you should move you code out of the
>>> source element parser to the separate classes.
>>>
>>> In current CVS HEAD (and last integration builds) all builder elements
>>> are sorted out to the correct places, though several iterations during
>>> 1.0 life-cycle were required to make general enough solution. To avoid
>>> any possible confusion I will describe only the current (post M2) state.
>>>
>>> org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
>>> contributed IBuildParticipants and that's all.
>>>
>>> If you want standard parser - you should specify it as the first step in
>>> the sequence - see the ruby example.
>>>
>>> The goals of the builder are
>>> a) report errors
>>> b) if applicable, compile and generate compiled data structures or files
>>>
>>> DLTK model is not the essential part of these goals, so it is not called
>>> automatically. Actually, now nothing is called automatically - only the
>>> specified IBuildParticipants are executed. So application could use
>>> different version of the parser in builder, if needed.
>>>
>>> Also, IBuildParticipants are called during reconcile operation to
>>> process not saved files in the editor.
>>>
>>> It should be simpler now.
>>>
>>> Regards,
>>> Alex
>>>
>>>
>>> Chuck Doucette wrote:
>>>> Indeed - I would prefer to build my model in one place.
>>>> The builder is preferable over the indexer.
>>>> However, I would like to reflect the changes in the editor (i.e. a
>>>> working copy) in an alternate version of my custom model - not just
>>>> when the file is saved and built as a compilation unit.
>>>>
>>>> However, right now it appears as if the DLTK model is built in the
>>>> method AbstractSourceElementParser.parseSourceModule via a call to
>>>> SourceElementRequestVisitor.traverse.
>>>>
>>>> I assume I should build my custom model in my class which extends
>>>> AbstractSourceElementParser.
>>>>
>>>> I'm not sure if the parser and DLTK model builder is invoked from a
>>>> builder like ValidatorBuilder or ValidatorStructureBuilder.
>>>>
>>>> Chuck
>>>>
>>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>>> news:gcf26r$6br$1@build.eclipse.org...
>>>>> Chuck,
>>>>>
>>>>> Just after M2 we have generalized the builder infrastructure, please
>>>>> see this message
>>>>> http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>>>>
>>>>> I would recommend you to use DLTK from CVS or fresh integration builds
>>>>> so you can more closely follow DLTK changes.
>>>>>
>>>>> Now the builder configuration is simpler - you provide multiple
>>>>> implementations of the IBuildParticipant and that's all.
>>>>> SourceParser/SourceElementParser should not be affected - they should
>>>>> perform only their primary task.
>>>>>
>>>>> If you need to know type of the build operation
>>>>> (full/incremental/reconcile) or perform additional actions after all
>>>>> source modules are processed - you should implement
>>>>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>>>>
>>>>> Regards,
>>>>> Alex
>>>>>
>>>>> Chuck Doucette wrote:
>>>>>> I am trying to figure out how to reorganize my sources to properly
>>>>>> build and run in the new 1.0M2 version of DLTK.
>>>>>>
>>>>>> One important issue is that we build our own custom model of
>>>>>> a) our language's source code
>>>>>> b) the run-time-library that it can call
>>>>>>
>>>>>> We have
>>>>>> 1) a parser
>>>>>> 2) a custom model builder (depends on AST)
>>>>>> 3) a semantic checker (depends on AST and our custom model)
>>>>>>
>>>>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>>>>> ISourceParser.pase.
>>>>>> That method also calls (2), our custom model builder.
>>>>>>
>>>>>> 2) The semantic checker is invoked from
>>>>>> VScriptSourceElementParser, which extends AbstractSourceElementParser
>>>>>> (which uses SourceElementRequestVisitor to build the DLTK model) and
>>>>>> implements ISourceElementParser. My parse method also builds my
>>>>>> custom model.
>>>>>>
>>>>>> This structure seems to mostly work correctly when invoked
>>>>>> dynamically from the editor.
>>>>>> In addition, I have built a custom builder which
>>>>>> 1) Pass 1 - for each module:
>>>>>> a) attempts to fetch the AST tree from the DLTK module cache by
>>>>>> calling SourceParserUtils.getModuleDeclaration. If the cache is
>>>>>> empty or out-of-date, it
>>>>>> i) reparses the compilation unit
>>>>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>>>>> iii) calls my custom visitor to rebuild my custom model
>>>>>> 2) Pass 2 - for all modules
>>>>>> a) calls my semantic checker (after your DLTK model and my custom
>>>>>> model are built for each module)
>>>>>>
>>>>>> You've informed us via this newsgroup of the new interface
>>>>>> IBuildParticipant.
>>>>>> These are fetched in the IScriptBuilder.initialize method.
>>>>>> They are used in ValidatorBuilder.buildModule
>>>>>> and ValidatorStructureBuilder.buildStructure.
>>>>>>
>>>>>> My questions are:
>>>>>>
>>>>>> 1) Does the structure I have described above still make sense in
>>>>>> the new version, i.e.
>>>>>> a) adding a custom model builder to my class which implements
>>>>>> ISourceParser and
>>>>>> b) adding a semantic checker which uses the custom model to
>>>>>> ISourceElementParser?
>>>>>> 2) What is the difference between ValidatorBuilder and
>>>>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>>>>> 3) Even though I want to use the new IBuildParticipant interfaces
>>>>>> that you have described,
>>>>>> don't I have to write my own custom builder to make sure I
>>>>>> have built my own custom model for each
>>>>>> module before I attempt to do semantic checking on any module
>>>>>> (which uses the custom model)?
>>>>>> 4) Can you suggest other/better ways of doing what I want to do
>>>>>> that fit into your architecture?
>>>>>>
>>>>>> Thanks,
>>>>>> Chuck
>>>>>>
>>
Re: parsers, builders, build types, and build participants [message #33596 is a reply to message #33562] Thu, 23 October 2008 22:37 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
FYI:
>> You should use
>> <extension point="org.eclipse.dltk.core.buildParticipant"> and specify
>> all the steps as buildParticipant elements. The subsequent steps could
>> have the <requires/> element, if they require the results created by the
>> previous ones.

This extension is not defined in your plugin - at least not in the stable
build 1.0M2.
I am downloading the Integration build now.

Chuck

"Chuck Doucette" <cdoucette@vaultus.com> wrote in message
news:gdqfs8$96j$1@build.eclipse.org...
>I am still a bit confused.
>
> Since my project nature class extends ScriptNature, your builder,
> ScriptBuilder, is automatically added to each of our DLTK projects' build
> specs. Thus, when Eclipse wants to build a DLTK project, it calls your
> DLTK builder class, ScriptBuilder, which implements
> IncrementalProjectBuilder.
> That defers most of its work to any user's custom DLTK builder(s), which
> implement IScriptBuilder.
>
> My builder class, VScriptBuilder, which implements IScriptBuilder, calls
> each of my build participants in order.
> It currently calls them directly - without them implementing any
> IBuildParticipant interface.
> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
> model (at least of declarations)
> a) module's source contents -> parsing -> produces AST tree
> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
> model
> 2. Phase 2 - semantic check modules based on AST tree and custom model
> a) AST tree -> semantic checker (extends AST Visitor - reports most
> problems)
>
> You say I should use IBuildParticipant - at least for 1b (my custom model
> builder) and 2a (my semantic checker),
> and perhaps also for my parser (but perhaps not if I split off the custom
> model builder).
> Apparently, IBuildParticipant is used by:
> a) ValidatorBuilder
> b) ValidatorStructureBuilder
>
> ValidatorStructureBuilder implements IStructureBuilder, which is used by
> AbstractSourceModule.buildStructure and
> ReconcileWorkingCopyOperation.makeConsistent (one or the other used during
> reconcilation).
> I can't see where/how ValidatorBuilder is used at all - but perhaps that
> doesn't matter.
>
> So, do I need to
> a) add extension wrapper classes around my builder components which
> implement IBuildParticipant
> b) keep my existing builder class, VScriptBuilder, since it implements
> IScriptBuilder and will ultimately be called by the Eclipse build manager.
> c) use your existing ValidatorStructureBuilder which will call my build
> participants during reconciliation
> d) remove calls to my semantic checker from my class
> VScriptSourceElementParser, which extends ISourceElementParser (this is
> where you had told me to put my semantic checker since it was getting
> called during reconciliation)
> e) remove calls to my custom model builder in my class
> VScriptSourceParser, which extends ISourceParser.
>
> Thanks,
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:gcurs7$61p$1@build.eclipse.org...
>> Chuck Doucette wrote:
>>> Thank you for your response.
>>> In general - that sounds great.
>>> I just wrap all of my build participants in classes which implement
>>> IBuildParticipant,
>>> and then they are called as part of a build and/or from a reconcile
>>> operation.
>>>
>>> Should that include the parser itself - or is that called separately?
>>
>> During reconcile operation first the DLTK model is updated.
>> The update consists of the call to the source element parser and most of
>> the languages call source parser in it the implementation of source
>> element parser.
>>
>> During build step if the parser is required you should explicitly specify
>> it as the build participant. The explicit configuration allows greater
>> flexibility. There is standard implementation provided in the class
>> org.eclipse.dltk.core.builder.ParserBuildParticipantFactory (it is listed
>> first in the ruby example). Or you can make you own implementation.
>>
>>> One of my build participants builds my own custom model (rather than a
>>> DLTK model).
>>> The other build participant is a semantic checker which depends on the
>>> existence of that custom model - both for the current module and for any
>>> other referenced modules.
>>>
>>> Is there any way I can organize the way that my build partipants can be
>>> called other than writing a custom builder (which I had already done)?
>>
>> You should use
>> <extension point="org.eclipse.dltk.core.buildParticipant"> and specify
>> all the steps as buildParticipant elements. The subsequent steps could
>> have the <requires/> element, if they require the results created by the
>> previous ones.
>>
>> Implementation of the IScriptBuilder interface is a good way too, the
>> main difference is that IBuildParticipants are called during reconcile
>> operations and they designed to operate at the finer level - per source
>> module.
>>
>>> The real issue is that my custom model is not serialized, so it must be
>>> recreated after reparsing each file. Is your model serialized into a
>>> build State object?
>>
>> DLTK model is not serializable and it is built by demand. It is not built
>> by the builder for the future use - it is built only when required.
>>
>>>
>>> Thanks,
>>> Chuck
>>>
>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>> news:gch0fa$b71$1@build.eclipse.org...
>>>> Chuck,
>>>>
>>>> In the past (0.95) the best place to add additional code was source
>>>> element parser, because there were no builder implementations provided
>>>> by the DLTK.
>>>>
>>>> Now (1.0) there is a builder, so you should move you code out of the
>>>> source element parser to the separate classes.
>>>>
>>>> In current CVS HEAD (and last integration builds) all builder elements
>>>> are sorted out to the correct places, though several iterations during
>>>> 1.0 life-cycle were required to make general enough solution. To avoid
>>>> any possible confusion I will describe only the current (post M2)
>>>> state.
>>>>
>>>> org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
>>>> contributed IBuildParticipants and that's all.
>>>>
>>>> If you want standard parser - you should specify it as the first step
>>>> in the sequence - see the ruby example.
>>>>
>>>> The goals of the builder are
>>>> a) report errors
>>>> b) if applicable, compile and generate compiled data structures or
>>>> files
>>>>
>>>> DLTK model is not the essential part of these goals, so it is not
>>>> called automatically. Actually, now nothing is called automatically -
>>>> only the specified IBuildParticipants are executed. So application
>>>> could use different version of the parser in builder, if needed.
>>>>
>>>> Also, IBuildParticipants are called during reconcile operation to
>>>> process not saved files in the editor.
>>>>
>>>> It should be simpler now.
>>>>
>>>> Regards,
>>>> Alex
>>>>
>>>>
>>>> Chuck Doucette wrote:
>>>>> Indeed - I would prefer to build my model in one place.
>>>>> The builder is preferable over the indexer.
>>>>> However, I would like to reflect the changes in the editor (i.e. a
>>>>> working copy) in an alternate version of my custom model - not just
>>>>> when the file is saved and built as a compilation unit.
>>>>>
>>>>> However, right now it appears as if the DLTK model is built in the
>>>>> method AbstractSourceElementParser.parseSourceModule via a call to
>>>>> SourceElementRequestVisitor.traverse.
>>>>>
>>>>> I assume I should build my custom model in my class which extends
>>>>> AbstractSourceElementParser.
>>>>>
>>>>> I'm not sure if the parser and DLTK model builder is invoked from a
>>>>> builder like ValidatorBuilder or ValidatorStructureBuilder.
>>>>>
>>>>> Chuck
>>>>>
>>>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>>>> news:gcf26r$6br$1@build.eclipse.org...
>>>>>> Chuck,
>>>>>>
>>>>>> Just after M2 we have generalized the builder infrastructure, please
>>>>>> see this message
>>>>>> http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>>>>>
>>>>>> I would recommend you to use DLTK from CVS or fresh integration
>>>>>> builds so you can more closely follow DLTK changes.
>>>>>>
>>>>>> Now the builder configuration is simpler - you provide multiple
>>>>>> implementations of the IBuildParticipant and that's all.
>>>>>> SourceParser/SourceElementParser should not be affected - they should
>>>>>> perform only their primary task.
>>>>>>
>>>>>> If you need to know type of the build operation
>>>>>> (full/incremental/reconcile) or perform additional actions after all
>>>>>> source modules are processed - you should implement
>>>>>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>>>>>
>>>>>> Regards,
>>>>>> Alex
>>>>>>
>>>>>> Chuck Doucette wrote:
>>>>>>> I am trying to figure out how to reorganize my sources to properly
>>>>>>> build and run in the new 1.0M2 version of DLTK.
>>>>>>>
>>>>>>> One important issue is that we build our own custom model of
>>>>>>> a) our language's source code
>>>>>>> b) the run-time-library that it can call
>>>>>>>
>>>>>>> We have
>>>>>>> 1) a parser
>>>>>>> 2) a custom model builder (depends on AST)
>>>>>>> 3) a semantic checker (depends on AST and our custom model)
>>>>>>>
>>>>>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>>>>>> ISourceParser.pase.
>>>>>>> That method also calls (2), our custom model builder.
>>>>>>>
>>>>>>> 2) The semantic checker is invoked from
>>>>>>> VScriptSourceElementParser, which extends
>>>>>>> AbstractSourceElementParser (which uses SourceElementRequestVisitor
>>>>>>> to build the DLTK model) and implements ISourceElementParser. My
>>>>>>> parse method also builds my custom model.
>>>>>>>
>>>>>>> This structure seems to mostly work correctly when invoked
>>>>>>> dynamically from the editor.
>>>>>>> In addition, I have built a custom builder which
>>>>>>> 1) Pass 1 - for each module:
>>>>>>> a) attempts to fetch the AST tree from the DLTK module cache by
>>>>>>> calling SourceParserUtils.getModuleDeclaration. If the cache is
>>>>>>> empty or out-of-date, it
>>>>>>> i) reparses the compilation unit
>>>>>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>>>>>> iii) calls my custom visitor to rebuild my custom model
>>>>>>> 2) Pass 2 - for all modules
>>>>>>> a) calls my semantic checker (after your DLTK model and my
>>>>>>> custom model are built for each module)
>>>>>>>
>>>>>>> You've informed us via this newsgroup of the new interface
>>>>>>> IBuildParticipant.
>>>>>>> These are fetched in the IScriptBuilder.initialize method.
>>>>>>> They are used in ValidatorBuilder.buildModule
>>>>>>> and ValidatorStructureBuilder.buildStructure.
>>>>>>>
>>>>>>> My questions are:
>>>>>>>
>>>>>>> 1) Does the structure I have described above still make sense in
>>>>>>> the new version, i.e.
>>>>>>> a) adding a custom model builder to my class which
>>>>>>> implements ISourceParser and
>>>>>>> b) adding a semantic checker which uses the custom model to
>>>>>>> ISourceElementParser?
>>>>>>> 2) What is the difference between ValidatorBuilder and
>>>>>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>>>>>> 3) Even though I want to use the new IBuildParticipant interfaces
>>>>>>> that you have described,
>>>>>>> don't I have to write my own custom builder to make sure I
>>>>>>> have built my own custom model for each
>>>>>>> module before I attempt to do semantic checking on any module
>>>>>>> (which uses the custom model)?
>>>>>>> 4) Can you suggest other/better ways of doing what I want to do
>>>>>>> that fit into your architecture?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Chuck
>>>>>>>
>>>
>
Re: parsers, builders, build types, and build participants [message #33666 is a reply to message #33596] Thu, 23 October 2008 23:19 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
buildParticipant is in the latest integration build.

Chuck

"Chuck Doucette" <cdoucette@vaultus.com> wrote in message
news:gdquf7$qk0$1@build.eclipse.org...
> FYI:
>>> You should use
>>> <extension point="org.eclipse.dltk.core.buildParticipant"> and specify
>>> all the steps as buildParticipant elements. The subsequent steps could
>>> have the <requires/> element, if they require the results created by the
>>> previous ones.
>
> This extension is not defined in your plugin - at least not in the stable
> build 1.0M2.
> I am downloading the Integration build now.
>
> Chuck
>
> "Chuck Doucette" <cdoucette@vaultus.com> wrote in message
> news:gdqfs8$96j$1@build.eclipse.org...
>>I am still a bit confused.
>>
>> Since my project nature class extends ScriptNature, your builder,
>> ScriptBuilder, is automatically added to each of our DLTK projects' build
>> specs. Thus, when Eclipse wants to build a DLTK project, it calls your
>> DLTK builder class, ScriptBuilder, which implements
>> IncrementalProjectBuilder.
>> That defers most of its work to any user's custom DLTK builder(s), which
>> implement IScriptBuilder.
>>
>> My builder class, VScriptBuilder, which implements IScriptBuilder, calls
>> each of my build participants in order.
>> It currently calls them directly - without them implementing any
>> IBuildParticipant interface.
>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>> model (at least of declarations)
>> a) module's source contents -> parsing -> produces AST tree
>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>> model
>> 2. Phase 2 - semantic check modules based on AST tree and custom model
>> a) AST tree -> semantic checker (extends AST Visitor - reports most
>> problems)
>>
>> You say I should use IBuildParticipant - at least for 1b (my custom model
>> builder) and 2a (my semantic checker),
>> and perhaps also for my parser (but perhaps not if I split off the custom
>> model builder).
>> Apparently, IBuildParticipant is used by:
>> a) ValidatorBuilder
>> b) ValidatorStructureBuilder
>>
>> ValidatorStructureBuilder implements IStructureBuilder, which is used by
>> AbstractSourceModule.buildStructure and
>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>> during reconcilation).
>> I can't see where/how ValidatorBuilder is used at all - but perhaps that
>> doesn't matter.
>>
>> So, do I need to
>> a) add extension wrapper classes around my builder components which
>> implement IBuildParticipant
>> b) keep my existing builder class, VScriptBuilder, since it implements
>> IScriptBuilder and will ultimately be called by the Eclipse build
>> manager.
>> c) use your existing ValidatorStructureBuilder which will call my build
>> participants during reconciliation
>> d) remove calls to my semantic checker from my class
>> VScriptSourceElementParser, which extends ISourceElementParser (this is
>> where you had told me to put my semantic checker since it was getting
>> called during reconciliation)
>> e) remove calls to my custom model builder in my class
>> VScriptSourceParser, which extends ISourceParser.
>>
>> Thanks,
>> Chuck
>>
>> "Alex Panchenko" <alex@xored.com> wrote in message
>> news:gcurs7$61p$1@build.eclipse.org...
>>> Chuck Doucette wrote:
>>>> Thank you for your response.
>>>> In general - that sounds great.
>>>> I just wrap all of my build participants in classes which implement
>>>> IBuildParticipant,
>>>> and then they are called as part of a build and/or from a reconcile
>>>> operation.
>>>>
>>>> Should that include the parser itself - or is that called separately?
>>>
>>> During reconcile operation first the DLTK model is updated.
>>> The update consists of the call to the source element parser and most of
>>> the languages call source parser in it the implementation of source
>>> element parser.
>>>
>>> During build step if the parser is required you should explicitly
>>> specify it as the build participant. The explicit configuration allows
>>> greater flexibility. There is standard implementation provided in the
>>> class org.eclipse.dltk.core.builder.ParserBuildParticipantFactory (it is
>>> listed first in the ruby example). Or you can make you own
>>> implementation.
>>>
>>>> One of my build participants builds my own custom model (rather than a
>>>> DLTK model).
>>>> The other build participant is a semantic checker which depends on the
>>>> existence of that custom model - both for the current module and for
>>>> any other referenced modules.
>>>>
>>>> Is there any way I can organize the way that my build partipants can be
>>>> called other than writing a custom builder (which I had already done)?
>>>
>>> You should use
>>> <extension point="org.eclipse.dltk.core.buildParticipant"> and specify
>>> all the steps as buildParticipant elements. The subsequent steps could
>>> have the <requires/> element, if they require the results created by the
>>> previous ones.
>>>
>>> Implementation of the IScriptBuilder interface is a good way too, the
>>> main difference is that IBuildParticipants are called during reconcile
>>> operations and they designed to operate at the finer level - per source
>>> module.
>>>
>>>> The real issue is that my custom model is not serialized, so it must be
>>>> recreated after reparsing each file. Is your model serialized into a
>>>> build State object?
>>>
>>> DLTK model is not serializable and it is built by demand. It is not
>>> built by the builder for the future use - it is built only when
>>> required.
>>>
>>>>
>>>> Thanks,
>>>> Chuck
>>>>
>>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>>> news:gch0fa$b71$1@build.eclipse.org...
>>>>> Chuck,
>>>>>
>>>>> In the past (0.95) the best place to add additional code was source
>>>>> element parser, because there were no builder implementations provided
>>>>> by the DLTK.
>>>>>
>>>>> Now (1.0) there is a builder, so you should move you code out of the
>>>>> source element parser to the separate classes.
>>>>>
>>>>> In current CVS HEAD (and last integration builds) all builder elements
>>>>> are sorted out to the correct places, though several iterations during
>>>>> 1.0 life-cycle were required to make general enough solution. To avoid
>>>>> any possible confusion I will describe only the current (post M2)
>>>>> state.
>>>>>
>>>>> org.eclipse.dltk.internal.core.builder.StandardScriptBuilder calls
>>>>> contributed IBuildParticipants and that's all.
>>>>>
>>>>> If you want standard parser - you should specify it as the first step
>>>>> in the sequence - see the ruby example.
>>>>>
>>>>> The goals of the builder are
>>>>> a) report errors
>>>>> b) if applicable, compile and generate compiled data structures or
>>>>> files
>>>>>
>>>>> DLTK model is not the essential part of these goals, so it is not
>>>>> called automatically. Actually, now nothing is called automatically -
>>>>> only the specified IBuildParticipants are executed. So application
>>>>> could use different version of the parser in builder, if needed.
>>>>>
>>>>> Also, IBuildParticipants are called during reconcile operation to
>>>>> process not saved files in the editor.
>>>>>
>>>>> It should be simpler now.
>>>>>
>>>>> Regards,
>>>>> Alex
>>>>>
>>>>>
>>>>> Chuck Doucette wrote:
>>>>>> Indeed - I would prefer to build my model in one place.
>>>>>> The builder is preferable over the indexer.
>>>>>> However, I would like to reflect the changes in the editor (i.e. a
>>>>>> working copy) in an alternate version of my custom model - not just
>>>>>> when the file is saved and built as a compilation unit.
>>>>>>
>>>>>> However, right now it appears as if the DLTK model is built in the
>>>>>> method AbstractSourceElementParser.parseSourceModule via a call to
>>>>>> SourceElementRequestVisitor.traverse.
>>>>>>
>>>>>> I assume I should build my custom model in my class which extends
>>>>>> AbstractSourceElementParser.
>>>>>>
>>>>>> I'm not sure if the parser and DLTK model builder is invoked from a
>>>>>> builder like ValidatorBuilder or ValidatorStructureBuilder.
>>>>>>
>>>>>> Chuck
>>>>>>
>>>>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>>>>> news:gcf26r$6br$1@build.eclipse.org...
>>>>>>> Chuck,
>>>>>>>
>>>>>>> Just after M2 we have generalized the builder infrastructure, please
>>>>>>> see this message
>>>>>>> http://dev.eclipse.org/mhonarc/lists/dltk-dev/msg01142.html
>>>>>>>
>>>>>>> I would recommend you to use DLTK from CVS or fresh integration
>>>>>>> builds so you can more closely follow DLTK changes.
>>>>>>>
>>>>>>> Now the builder configuration is simpler - you provide multiple
>>>>>>> implementations of the IBuildParticipant and that's all.
>>>>>>> SourceParser/SourceElementParser should not be affected - they
>>>>>>> should perform only their primary task.
>>>>>>>
>>>>>>> If you need to know type of the build operation
>>>>>>> (full/incremental/reconcile) or perform additional actions after all
>>>>>>> source modules are processed - you should implement
>>>>>>> org.eclipse.dltk.core.builder.IBuildParticipantExtension
>>>>>>>
>>>>>>> Regards,
>>>>>>> Alex
>>>>>>>
>>>>>>> Chuck Doucette wrote:
>>>>>>>> I am trying to figure out how to reorganize my sources to properly
>>>>>>>> build and run in the new 1.0M2 version of DLTK.
>>>>>>>>
>>>>>>>> One important issue is that we build our own custom model of
>>>>>>>> a) our language's source code
>>>>>>>> b) the run-time-library that it can call
>>>>>>>>
>>>>>>>> We have
>>>>>>>> 1) a parser
>>>>>>>> 2) a custom model builder (depends on AST)
>>>>>>>> 3) a semantic checker (depends on AST and our custom model)
>>>>>>>>
>>>>>>>> 1) The parser wrapper class, VScriptSourceParser, implements
>>>>>>>> ISourceParser.pase.
>>>>>>>> That method also calls (2), our custom model builder.
>>>>>>>>
>>>>>>>> 2) The semantic checker is invoked from
>>>>>>>> VScriptSourceElementParser, which extends
>>>>>>>> AbstractSourceElementParser (which uses SourceElementRequestVisitor
>>>>>>>> to build the DLTK model) and implements ISourceElementParser. My
>>>>>>>> parse method also builds my custom model.
>>>>>>>>
>>>>>>>> This structure seems to mostly work correctly when invoked
>>>>>>>> dynamically from the editor.
>>>>>>>> In addition, I have built a custom builder which
>>>>>>>> 1) Pass 1 - for each module:
>>>>>>>> a) attempts to fetch the AST tree from the DLTK module cache by
>>>>>>>> calling SourceParserUtils.getModuleDeclaration. If the cache is
>>>>>>>> empty or out-of-date, it
>>>>>>>> i) reparses the compilation unit
>>>>>>>> ii) calls your SourceElementVisitor to rebuild your DLTK model
>>>>>>>> iii) calls my custom visitor to rebuild my custom model
>>>>>>>> 2) Pass 2 - for all modules
>>>>>>>> a) calls my semantic checker (after your DLTK model and my
>>>>>>>> custom model are built for each module)
>>>>>>>>
>>>>>>>> You've informed us via this newsgroup of the new interface
>>>>>>>> IBuildParticipant.
>>>>>>>> These are fetched in the IScriptBuilder.initialize method.
>>>>>>>> They are used in ValidatorBuilder.buildModule
>>>>>>>> and ValidatorStructureBuilder.buildStructure.
>>>>>>>>
>>>>>>>> My questions are:
>>>>>>>>
>>>>>>>> 1) Does the structure I have described above still make sense in
>>>>>>>> the new version, i.e.
>>>>>>>> a) adding a custom model builder to my class which
>>>>>>>> implements ISourceParser and
>>>>>>>> b) adding a semantic checker which uses the custom model to
>>>>>>>> ISourceElementParser?
>>>>>>>> 2) What is the difference between ValidatorBuilder and
>>>>>>>> ValidatorStructureBuilder - what is a STRUCTURE_BUILD type?
>>>>>>>> 3) Even though I want to use the new IBuildParticipant
>>>>>>>> interfaces that you have described,
>>>>>>>> don't I have to write my own custom builder to make sure I
>>>>>>>> have built my own custom model for each
>>>>>>>> module before I attempt to do semantic checking on any
>>>>>>>> module (which uses the custom model)?
>>>>>>>> 4) Can you suggest other/better ways of doing what I want to do
>>>>>>>> that fit into your architecture?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Chuck
>>>>>>>>
>>>>
>>
>
>
Re: parsers, builders, build types, and build participants [message #33702 is a reply to message #33562] Fri, 24 October 2008 06:45 Go to previous messageGo to next message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Hi Chuck,

Please see the answers inline.

Chuck Doucette wrote:
> I am still a bit confused.
>
> Since my project nature class extends ScriptNature, your builder,
> ScriptBuilder, is automatically added to each of our DLTK projects' build
> specs. Thus, when Eclipse wants to build a DLTK project, it calls your DLTK
> builder class, ScriptBuilder, which implements IncrementalProjectBuilder.
> That defers most of its work to any user's custom DLTK builder(s), which
> implement IScriptBuilder.
>
> My builder class, VScriptBuilder, which implements IScriptBuilder, calls
> each of my build participants in order.
> It currently calls them directly - without them implementing any
> IBuildParticipant interface.
> 1. Phase 1 - parse to create AST tree - use AST tree to create custom model
> (at least of declarations)
> a) module's source contents -> parsing -> produces AST tree
> b) AST tree -> custom model builder (extends ASTVisitor) -> custom model
> 2. Phase 2 - semantic check modules based on AST tree and custom model
> a) AST tree -> semantic checker (extends AST Visitor - reports most
> problems)
>
> You say I should use IBuildParticipant - at least for 1b (my custom model
> builder) and 2a (my semantic checker),
> and perhaps also for my parser (but perhaps not if I split off the custom
> model builder).
> Apparently, IBuildParticipant is used by:
> a) ValidatorBuilder
> b) ValidatorStructureBuilder
>
> ValidatorStructureBuilder implements IStructureBuilder, which is used by
> AbstractSourceModule.buildStructure and
> ReconcileWorkingCopyOperation.makeConsistent (one or the other used during
> reconcilation).
> I can't see where/how ValidatorBuilder is used at all - but perhaps that
> doesn't matter.
>
> So, do I need to
> a) add extension wrapper classes around my builder components which
> implement IBuildParticipant

If you classes are provided as IBuildParticipants they will be called by
the build and reconcile operations.

For scripting languages tasks like parsing and error reporting operate
on the single source module usually, so the implementation could be a
bit simpler if compared with IScriptBuilder way.

> b) keep my existing builder class, VScriptBuilder, since it implements
> IScriptBuilder and will ultimately be called by the Eclipse build manager.

Some builder parts fit well into the IBuildParticipant way, so you
should wrap them as this extension.

On the other hand, there could be builder parts that do not fit well
into the IBuildParticipant, so you should keep them in ScriptBuilder. If
all of the operations could be wrapped as IBuildParticipant, then it
looks like you don't need your own ScriptBuilder implementation.

> c) use your existing ValidatorStructureBuilder which will call my build
> participants during reconciliation

it was a temporary internal class and it is removed now. If you want to
your code to be called from the reconcile - you should declare it as
IBuildParticipants.

> d) remove calls to my semantic checker from my class
> VScriptSourceElementParser, which extends ISourceElementParser (this is
> where you had told me to put my semantic checker since it was getting called
> during reconciliation)

In the DLTK 0.95 it was the acceptable way to add additional code, since
the builder infrastructure was not so comprehensive like it is now.
Though in general it was not the best way, since they are quite
different tasks and most of the time they should be executed in
different moments of time.

Taking all these issues into account the builder infrastructure was
extended to simplify implementation of additional semantic checkers, etc.

So I recommend to implement it is as IBuildParticipant if you decide to
go this way.

> e) remove calls to my custom model builder in my class VScriptSourceParser,
> which extends ISourceParser.

The correct place to update your model depends on how you use it.
Unfortunately I don't know the details of your custom model, so I will
post some thoughts and you could select the appropriate solution yourself.

If you model could be built on demand, you don't need to prepare it for
future use and it could be treated like inherent part of AST - you can
put it into your SourceParser, also you can extend ModelDeclaration
class and add additional fields with your custom model of that source
module.

In contrast, if you need to prepare your custom model beforehand for all
modules (for example, type/method indexing works that way) and it is an
independent part from the AST - it is better to build it separately, so
you don't need that code in SourceParser, and it could be implemented as
IBuildParticipant.

Hope that helps.

Regards,
Alex
Re: parsers, builders, build types, and build participants [message #33837 is a reply to message #33702] Fri, 24 October 2008 23:41 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
Thanks for your suggestions.

Our custom modlel building is basically equivalent to indexing.
We want to know where all the type/variable/method declarations are being
made so we can resolve
references to them from other modules.
We want to index everything (all modules) before we do any semantic
checking.
We keep this information in our own hierarchy - similar to your model - but
completely
distinct from the AST tree.

Thus, if one build participant was our indexer/custom-model-builder, it
doesn't work
in the current architecture, since it the builder that would iterate through
each module
calling the build participants in order. Instead, we want to call the
indexer/custom-model builder for all of the modules first - and then when
that is done go ahead and call the semantic checker for all modules.

Our custom builder which implements IScriptBuilder does what we want, if it
is invoked.

The problem is that
a) you have strongly suggested that we use the IBuildParticipant
interface (and we would prefer to since it is the recommended way to
integrate with your architecture and gives us more information about what we
are building)
b) If we use IBuildParticipant, and disable our customer builder, then
our build participants are auomatically invoked:
i) during building via ScriptBuilder and StandardScriptBuilder
[ this causes semantic errors from unresolved inter-module
references ]
ii) during reconciling via StructureBuilder
c) If we don't use IBuildParticipant and use our own custom builder
instead
i) we can tightly control the order our participants are invoked in
the build process and thus resolve the inter-module references
ii) we have no well defined way to integrate our participants into the
reconciliation process other than perhaps reverting to using the
SourceElementParser.
d) It may be possible to both enable our custom build and use build
participants.
My boss suggested that if I could determine whether my build participants
were being invoked in the builder or the reconciler, I could proceed with
their execution or exit permaturely. I thought perhaps make that
determination based on the type of IBuildContext - but all of the classes
that implement IBuildContext are in internal packages which are not
exported.

Perhaps you could add a member to IBuildContext like:

int getBuildContextType();

It could return something like:

EXTERNAL_MODULE_BUILD_CONTEXT
RECONCILE_BUILD_CONTEXT
SOURCE_MODULE_BUILD_CONTEXT

Chuck

"Alex Panchenko" <alex@xored.com> wrote in message
news:gdrqu9$rna$1@build.eclipse.org...
> Hi Chuck,
>
> Please see the answers inline.
>
> Chuck Doucette wrote:
>> I am still a bit confused.
>>
>> Since my project nature class extends ScriptNature, your builder,
>> ScriptBuilder, is automatically added to each of our DLTK projects' build
>> specs. Thus, when Eclipse wants to build a DLTK project, it calls your
>> DLTK builder class, ScriptBuilder, which implements
>> IncrementalProjectBuilder.
>> That defers most of its work to any user's custom DLTK builder(s), which
>> implement IScriptBuilder.
>>
>> My builder class, VScriptBuilder, which implements IScriptBuilder, calls
>> each of my build participants in order.
>> It currently calls them directly - without them implementing any
>> IBuildParticipant interface.
>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>> model (at least of declarations)
>> a) module's source contents -> parsing -> produces AST tree
>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>> model
>> 2. Phase 2 - semantic check modules based on AST tree and custom model
>> a) AST tree -> semantic checker (extends AST Visitor - reports most
>> problems)
>>
>> You say I should use IBuildParticipant - at least for 1b (my custom model
>> builder) and 2a (my semantic checker),
>> and perhaps also for my parser (but perhaps not if I split off the custom
>> model builder).
>> Apparently, IBuildParticipant is used by:
>> a) ValidatorBuilder
>> b) ValidatorStructureBuilder
>>
>> ValidatorStructureBuilder implements IStructureBuilder, which is used by
>> AbstractSourceModule.buildStructure and
>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>> during reconcilation).
>> I can't see where/how ValidatorBuilder is used at all - but perhaps that
>> doesn't matter.
>>
>> So, do I need to
>> a) add extension wrapper classes around my builder components which
>> implement IBuildParticipant
>
> If you classes are provided as IBuildParticipants they will be called by
> the build and reconcile operations.
>
> For scripting languages tasks like parsing and error reporting operate on
> the single source module usually, so the implementation could be a bit
> simpler if compared with IScriptBuilder way.
>
>> b) keep my existing builder class, VScriptBuilder, since it implements
>> IScriptBuilder and will ultimately be called by the Eclipse build
>> manager.
>
> Some builder parts fit well into the IBuildParticipant way, so you should
> wrap them as this extension.
>
> On the other hand, there could be builder parts that do not fit well into
> the IBuildParticipant, so you should keep them in ScriptBuilder. If all of
> the operations could be wrapped as IBuildParticipant, then it looks like
> you don't need your own ScriptBuilder implementation.
>
>> c) use your existing ValidatorStructureBuilder which will call my build
>> participants during reconciliation
>
> it was a temporary internal class and it is removed now. If you want to
> your code to be called from the reconcile - you should declare it as
> IBuildParticipants.
>
>> d) remove calls to my semantic checker from my class
>> VScriptSourceElementParser, which extends ISourceElementParser (this is
>> where you had told me to put my semantic checker since it was getting
>> called during reconciliation)
>
> In the DLTK 0.95 it was the acceptable way to add additional code, since
> the builder infrastructure was not so comprehensive like it is now. Though
> in general it was not the best way, since they are quite different tasks
> and most of the time they should be executed in different moments of time.
>
> Taking all these issues into account the builder infrastructure was
> extended to simplify implementation of additional semantic checkers, etc.
>
> So I recommend to implement it is as IBuildParticipant if you decide to go
> this way.
>
>> e) remove calls to my custom model builder in my class
>> VScriptSourceParser, which extends ISourceParser.
>
> The correct place to update your model depends on how you use it.
> Unfortunately I don't know the details of your custom model, so I will
> post some thoughts and you could select the appropriate solution yourself.
>
> If you model could be built on demand, you don't need to prepare it for
> future use and it could be treated like inherent part of AST - you can put
> it into your SourceParser, also you can extend ModelDeclaration class and
> add additional fields with your custom model of that source module.
>
> In contrast, if you need to prepare your custom model beforehand for all
> modules (for example, type/method indexing works that way) and it is an
> independent part from the AST - it is better to build it separately, so
> you don't need that code in SourceParser, and it could be implemented as
> IBuildParticipant.
>
> Hope that helps.
>
> Regards,
> Alex
Re: parsers, builders, build types, and build participants [message #33886 is a reply to message #33837] Sun, 26 October 2008 04:35 Go to previous messageGo to next message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Hi Chuck,

You can use different combinations of the current APIs if it fits your
usecase.

At the moment you can acquire the type of the build operation if your
build participant implements IBuildParticipantExtension interface.

public interface IBuildParticipantExtension {
int FULL_BUILD = IScriptBuilder.FULL_BUILD;
int INCREMENTAL_BUILD=IScriptBuilder.INCREMENTAL_BUILD;
int RECONCILE_BUILD = 10;

void beginBuild(int buildType);
void endBuild(IProgressMonitor monitor);
}

Thanks for the suggestion to add method to the IBuildContext, I have
just committed it so it will be available in the next integration build.

Back to the type checking in builder. You can implement the specified
functionality thru the IBuildParticipant way too, for example please see
the
org.eclipse.dltk.tcl.internal.validators.packages.PackageReq uireChecker
class in the org.eclipse.dltk.tcl.validators plugin. It checks the
"package require" statements and verifies that the required packages are
implemented somewhere. In short words - the information is collected
during build() method calls, and all the checks are performed in the
endBuild() method.

Anyway, you should use the APIs that suits you well. If something does
not suit you - you are not required to use it. Or you can suggest
changes to the API to make it more usable for you.

Regards,
Alex


Chuck Doucette wrote:
> Thanks for your suggestions.
>
> Our custom modlel building is basically equivalent to indexing.
> We want to know where all the type/variable/method declarations are being
> made so we can resolve
> references to them from other modules.
> We want to index everything (all modules) before we do any semantic
> checking.
> We keep this information in our own hierarchy - similar to your model - but
> completely
> distinct from the AST tree.
>
> Thus, if one build participant was our indexer/custom-model-builder, it
> doesn't work
> in the current architecture, since it the builder that would iterate through
> each module
> calling the build participants in order. Instead, we want to call the
> indexer/custom-model builder for all of the modules first - and then when
> that is done go ahead and call the semantic checker for all modules.
>
> Our custom builder which implements IScriptBuilder does what we want, if it
> is invoked.
>
> The problem is that
> a) you have strongly suggested that we use the IBuildParticipant
> interface (and we would prefer to since it is the recommended way to
> integrate with your architecture and gives us more information about what we
> are building)
> b) If we use IBuildParticipant, and disable our customer builder, then
> our build participants are auomatically invoked:
> i) during building via ScriptBuilder and StandardScriptBuilder
> [ this causes semantic errors from unresolved inter-module
> references ]
> ii) during reconciling via StructureBuilder
> c) If we don't use IBuildParticipant and use our own custom builder
> instead
> i) we can tightly control the order our participants are invoked in
> the build process and thus resolve the inter-module references
> ii) we have no well defined way to integrate our participants into the
> reconciliation process other than perhaps reverting to using the
> SourceElementParser.
> d) It may be possible to both enable our custom build and use build
> participants.
> My boss suggested that if I could determine whether my build participants
> were being invoked in the builder or the reconciler, I could proceed with
> their execution or exit permaturely. I thought perhaps make that
> determination based on the type of IBuildContext - but all of the classes
> that implement IBuildContext are in internal packages which are not
> exported.
>
> Perhaps you could add a member to IBuildContext like:
>
> int getBuildContextType();
>
> It could return something like:
>
> EXTERNAL_MODULE_BUILD_CONTEXT
> RECONCILE_BUILD_CONTEXT
> SOURCE_MODULE_BUILD_CONTEXT
>
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:gdrqu9$rna$1@build.eclipse.org...
>> Hi Chuck,
>>
>> Please see the answers inline.
>>
>> Chuck Doucette wrote:
>>> I am still a bit confused.
>>>
>>> Since my project nature class extends ScriptNature, your builder,
>>> ScriptBuilder, is automatically added to each of our DLTK projects' build
>>> specs. Thus, when Eclipse wants to build a DLTK project, it calls your
>>> DLTK builder class, ScriptBuilder, which implements
>>> IncrementalProjectBuilder.
>>> That defers most of its work to any user's custom DLTK builder(s), which
>>> implement IScriptBuilder.
>>>
>>> My builder class, VScriptBuilder, which implements IScriptBuilder, calls
>>> each of my build participants in order.
>>> It currently calls them directly - without them implementing any
>>> IBuildParticipant interface.
>>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>>> model (at least of declarations)
>>> a) module's source contents -> parsing -> produces AST tree
>>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>>> model
>>> 2. Phase 2 - semantic check modules based on AST tree and custom model
>>> a) AST tree -> semantic checker (extends AST Visitor - reports most
>>> problems)
>>>
>>> You say I should use IBuildParticipant - at least for 1b (my custom model
>>> builder) and 2a (my semantic checker),
>>> and perhaps also for my parser (but perhaps not if I split off the custom
>>> model builder).
>>> Apparently, IBuildParticipant is used by:
>>> a) ValidatorBuilder
>>> b) ValidatorStructureBuilder
>>>
>>> ValidatorStructureBuilder implements IStructureBuilder, which is used by
>>> AbstractSourceModule.buildStructure and
>>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>>> during reconcilation).
>>> I can't see where/how ValidatorBuilder is used at all - but perhaps that
>>> doesn't matter.
>>>
>>> So, do I need to
>>> a) add extension wrapper classes around my builder components which
>>> implement IBuildParticipant
>> If you classes are provided as IBuildParticipants they will be called by
>> the build and reconcile operations.
>>
>> For scripting languages tasks like parsing and error reporting operate on
>> the single source module usually, so the implementation could be a bit
>> simpler if compared with IScriptBuilder way.
>>
>>> b) keep my existing builder class, VScriptBuilder, since it implements
>>> IScriptBuilder and will ultimately be called by the Eclipse build
>>> manager.
>> Some builder parts fit well into the IBuildParticipant way, so you should
>> wrap them as this extension.
>>
>> On the other hand, there could be builder parts that do not fit well into
>> the IBuildParticipant, so you should keep them in ScriptBuilder. If all of
>> the operations could be wrapped as IBuildParticipant, then it looks like
>> you don't need your own ScriptBuilder implementation.
>>
>>> c) use your existing ValidatorStructureBuilder which will call my build
>>> participants during reconciliation
>> it was a temporary internal class and it is removed now. If you want to
>> your code to be called from the reconcile - you should declare it as
>> IBuildParticipants.
>>
>>> d) remove calls to my semantic checker from my class
>>> VScriptSourceElementParser, which extends ISourceElementParser (this is
>>> where you had told me to put my semantic checker since it was getting
>>> called during reconciliation)
>> In the DLTK 0.95 it was the acceptable way to add additional code, since
>> the builder infrastructure was not so comprehensive like it is now. Though
>> in general it was not the best way, since they are quite different tasks
>> and most of the time they should be executed in different moments of time.
>>
>> Taking all these issues into account the builder infrastructure was
>> extended to simplify implementation of additional semantic checkers, etc.
>>
>> So I recommend to implement it is as IBuildParticipant if you decide to go
>> this way.
>>
>>> e) remove calls to my custom model builder in my class
>>> VScriptSourceParser, which extends ISourceParser.
>> The correct place to update your model depends on how you use it.
>> Unfortunately I don't know the details of your custom model, so I will
>> post some thoughts and you could select the appropriate solution yourself.
>>
>> If you model could be built on demand, you don't need to prepare it for
>> future use and it could be treated like inherent part of AST - you can put
>> it into your SourceParser, also you can extend ModelDeclaration class and
>> add additional fields with your custom model of that source module.
>>
>> In contrast, if you need to prepare your custom model beforehand for all
>> modules (for example, type/method indexing works that way) and it is an
>> independent part from the AST - it is better to build it separately, so
>> you don't need that code in SourceParser, and it could be implemented as
>> IBuildParticipant.
>>
>> Hope that helps.
>>
>> Regards,
>> Alex
>
>
Re: parsers, builders, build types, and build participants [message #33932 is a reply to message #33886] Thu, 30 October 2008 20:56 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
I still can't seem to find an architecture that works well with the latest
DLTK build.

Although I liked your suggestion to:
a) implement IBuildParticipantExtension so beginBuild and endBuild methods
would be called
b) postpone some processing until endBuild

This doesn't quite work.

The issue is that if I declare a method "a" in module A, and I refer to
method A.a in module B,
I don't want to flag that method reference if it is actually defined
elsewhere.
Thus, I must
a) process all of the external declarations of *all* modules to be built
(not just the current module)
b) semantic check all references (etc.) of each module

Your build participant structure calls each build participant in order on
each module.

My overall builder which implements IScriptBuilder does what I want.
In order to avoid redundant work, I attempted to disable my build
participants from building
(but allow them to be used for reconciliation); however, this didn't work
since your
StandardScriptBuilder clears all of the problems from my builder.
I believe this is a bug.

At the moment, I have enabled my builder and your builder, and even though
they
are doing redundant work - at least I can see the problems being generated
in the problems view.

Chuck

"Alex Panchenko" <alex@xored.com> wrote in message
news:ge0s2i$i2t$1@build.eclipse.org...
> Hi Chuck,
>
> You can use different combinations of the current APIs if it fits your
> usecase.
>
> At the moment you can acquire the type of the build operation if your
> build participant implements IBuildParticipantExtension interface.
>
> public interface IBuildParticipantExtension {
> int FULL_BUILD = IScriptBuilder.FULL_BUILD;
> int INCREMENTAL_BUILD=IScriptBuilder.INCREMENTAL_BUILD;
> int RECONCILE_BUILD = 10;
>
> void beginBuild(int buildType);
> void endBuild(IProgressMonitor monitor);
> }
>
> Thanks for the suggestion to add method to the IBuildContext, I have just
> committed it so it will be available in the next integration build.
>
> Back to the type checking in builder. You can implement the specified
> functionality thru the IBuildParticipant way too, for example please see
> the
> org.eclipse.dltk.tcl.internal.validators.packages.PackageReq uireChecker
> class in the org.eclipse.dltk.tcl.validators plugin. It checks the
> "package require" statements and verifies that the required packages are
> implemented somewhere. In short words - the information is collected
> during build() method calls, and all the checks are performed in the
> endBuild() method.
>
> Anyway, you should use the APIs that suits you well. If something does not
> suit you - you are not required to use it. Or you can suggest changes to
> the API to make it more usable for you.
>
> Regards,
> Alex
>
>
> Chuck Doucette wrote:
>> Thanks for your suggestions.
>>
>> Our custom modlel building is basically equivalent to indexing.
>> We want to know where all the type/variable/method declarations are being
>> made so we can resolve
>> references to them from other modules.
>> We want to index everything (all modules) before we do any semantic
>> checking.
>> We keep this information in our own hierarchy - similar to your model -
>> but completely
>> distinct from the AST tree.
>>
>> Thus, if one build participant was our indexer/custom-model-builder, it
>> doesn't work
>> in the current architecture, since it the builder that would iterate
>> through each module
>> calling the build participants in order. Instead, we want to call the
>> indexer/custom-model builder for all of the modules first - and then when
>> that is done go ahead and call the semantic checker for all modules.
>>
>> Our custom builder which implements IScriptBuilder does what we want, if
>> it is invoked.
>>
>> The problem is that
>> a) you have strongly suggested that we use the IBuildParticipant
>> interface (and we would prefer to since it is the recommended way to
>> integrate with your architecture and gives us more information about what
>> we are building)
>> b) If we use IBuildParticipant, and disable our customer builder, then
>> our build participants are auomatically invoked:
>> i) during building via ScriptBuilder and StandardScriptBuilder
>> [ this causes semantic errors from unresolved inter-module
>> references ]
>> ii) during reconciling via StructureBuilder
>> c) If we don't use IBuildParticipant and use our own custom builder
>> instead
>> i) we can tightly control the order our participants are invoked
>> in the build process and thus resolve the inter-module references
>> ii) we have no well defined way to integrate our participants into
>> the reconciliation process other than perhaps reverting to using the
>> SourceElementParser.
>> d) It may be possible to both enable our custom build and use build
>> participants.
>> My boss suggested that if I could determine whether my build participants
>> were being invoked in the builder or the reconciler, I could proceed with
>> their execution or exit permaturely. I thought perhaps make that
>> determination based on the type of IBuildContext - but all of the classes
>> that implement IBuildContext are in internal packages which are not
>> exported.
>>
>> Perhaps you could add a member to IBuildContext like:
>>
>> int getBuildContextType();
>>
>> It could return something like:
>>
>> EXTERNAL_MODULE_BUILD_CONTEXT
>> RECONCILE_BUILD_CONTEXT
>> SOURCE_MODULE_BUILD_CONTEXT
>>
>> Chuck
>>
>> "Alex Panchenko" <alex@xored.com> wrote in message
>> news:gdrqu9$rna$1@build.eclipse.org...
>>> Hi Chuck,
>>>
>>> Please see the answers inline.
>>>
>>> Chuck Doucette wrote:
>>>> I am still a bit confused.
>>>>
>>>> Since my project nature class extends ScriptNature, your builder,
>>>> ScriptBuilder, is automatically added to each of our DLTK projects'
>>>> build specs. Thus, when Eclipse wants to build a DLTK project, it calls
>>>> your DLTK builder class, ScriptBuilder, which implements
>>>> IncrementalProjectBuilder.
>>>> That defers most of its work to any user's custom DLTK builder(s),
>>>> which implement IScriptBuilder.
>>>>
>>>> My builder class, VScriptBuilder, which implements IScriptBuilder,
>>>> calls each of my build participants in order.
>>>> It currently calls them directly - without them implementing any
>>>> IBuildParticipant interface.
>>>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>>>> model (at least of declarations)
>>>> a) module's source contents -> parsing -> produces AST tree
>>>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>>>> model
>>>> 2. Phase 2 - semantic check modules based on AST tree and custom model
>>>> a) AST tree -> semantic checker (extends AST Visitor - reports most
>>>> problems)
>>>>
>>>> You say I should use IBuildParticipant - at least for 1b (my custom
>>>> model builder) and 2a (my semantic checker),
>>>> and perhaps also for my parser (but perhaps not if I split off the
>>>> custom model builder).
>>>> Apparently, IBuildParticipant is used by:
>>>> a) ValidatorBuilder
>>>> b) ValidatorStructureBuilder
>>>>
>>>> ValidatorStructureBuilder implements IStructureBuilder, which is used
>>>> by AbstractSourceModule.buildStructure and
>>>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>>>> during reconcilation).
>>>> I can't see where/how ValidatorBuilder is used at all - but perhaps
>>>> that doesn't matter.
>>>>
>>>> So, do I need to
>>>> a) add extension wrapper classes around my builder components which
>>>> implement IBuildParticipant
>>> If you classes are provided as IBuildParticipants they will be called by
>>> the build and reconcile operations.
>>>
>>> For scripting languages tasks like parsing and error reporting operate
>>> on the single source module usually, so the implementation could be a
>>> bit simpler if compared with IScriptBuilder way.
>>>
>>>> b) keep my existing builder class, VScriptBuilder, since it implements
>>>> IScriptBuilder and will ultimately be called by the Eclipse build
>>>> manager.
>>> Some builder parts fit well into the IBuildParticipant way, so you
>>> should wrap them as this extension.
>>>
>>> On the other hand, there could be builder parts that do not fit well
>>> into the IBuildParticipant, so you should keep them in ScriptBuilder. If
>>> all of the operations could be wrapped as IBuildParticipant, then it
>>> looks like you don't need your own ScriptBuilder implementation.
>>>
>>>> c) use your existing ValidatorStructureBuilder which will call my build
>>>> participants during reconciliation
>>> it was a temporary internal class and it is removed now. If you want to
>>> your code to be called from the reconcile - you should declare it as
>>> IBuildParticipants.
>>>
>>>> d) remove calls to my semantic checker from my class
>>>> VScriptSourceElementParser, which extends ISourceElementParser (this is
>>>> where you had told me to put my semantic checker since it was getting
>>>> called during reconciliation)
>>> In the DLTK 0.95 it was the acceptable way to add additional code, since
>>> the builder infrastructure was not so comprehensive like it is now.
>>> Though in general it was not the best way, since they are quite
>>> different tasks and most of the time they should be executed in
>>> different moments of time.
>>>
>>> Taking all these issues into account the builder infrastructure was
>>> extended to simplify implementation of additional semantic checkers,
>>> etc.
>>>
>>> So I recommend to implement it is as IBuildParticipant if you decide to
>>> go this way.
>>>
>>>> e) remove calls to my custom model builder in my class
>>>> VScriptSourceParser, which extends ISourceParser.
>>> The correct place to update your model depends on how you use it.
>>> Unfortunately I don't know the details of your custom model, so I will
>>> post some thoughts and you could select the appropriate solution
>>> yourself.
>>>
>>> If you model could be built on demand, you don't need to prepare it for
>>> future use and it could be treated like inherent part of AST - you can
>>> put it into your SourceParser, also you can extend ModelDeclaration
>>> class and add additional fields with your custom model of that source
>>> module.
>>>
>>> In contrast, if you need to prepare your custom model beforehand for all
>>> modules (for example, type/method indexing works that way) and it is an
>>> independent part from the AST - it is better to build it separately, so
>>> you don't need that code in SourceParser, and it could be implemented as
>>> IBuildParticipant.
>>>
>>> Hope that helps.
>>>
>>> Regards,
>>> Alex
>>
Re: parsers, builders, build types, and build participants [message #34090 is a reply to message #33932] Tue, 04 November 2008 16:14 Go to previous messageGo to next message
Charles Doucette is currently offline Charles Doucette
Messages: 125
Registered: July 2009
Senior Member
Here's my suggestion.

Since
a) although your build participant structure is fairly flexible - I can't
seem to use it to do what I want for a regular build - although it is great
for reconciliation.
b) Thus, I want to use most/all of my builder for building.

Since your StandardScriptBuilder always runs, and it assumes that if there
are build participants configured that they always run, then it deletes all
problem markers for a file before flushing a new set.
If I want to use a regular builder for building and your
StandardScriptBuilder with my build participants for reconcilation - then
any errors generated from my builder are flushed.

Here is my suggestion:
1) change beginBuild to return a boolean that indicates whether this
build participant will do any work; or, add a new interface and a method -
willDoWork. You can assume if the interface is not implemented that it
returned true.
2) If it returns true, then call build and endBuild and flush/clear
problem markers
3) If it returns false - don't do any of those things - exit prematurely
instead.

Chuck

"Chuck Doucette" <cdoucette@vaultus.com> wrote in message
news:ged751$kgo$1@build.eclipse.org...
>I still can't seem to find an architecture that works well with the latest
>DLTK build.
>
> Although I liked your suggestion to:
> a) implement IBuildParticipantExtension so beginBuild and endBuild methods
> would be called
> b) postpone some processing until endBuild
>
> This doesn't quite work.
>
> The issue is that if I declare a method "a" in module A, and I refer to
> method A.a in module B,
> I don't want to flag that method reference if it is actually defined
> elsewhere.
> Thus, I must
> a) process all of the external declarations of *all* modules to be built
> (not just the current module)
> b) semantic check all references (etc.) of each module
>
> Your build participant structure calls each build participant in order on
> each module.
>
> My overall builder which implements IScriptBuilder does what I want.
> In order to avoid redundant work, I attempted to disable my build
> participants from building
> (but allow them to be used for reconciliation); however, this didn't work
> since your
> StandardScriptBuilder clears all of the problems from my builder.
> I believe this is a bug.
>
> At the moment, I have enabled my builder and your builder, and even though
> they
> are doing redundant work - at least I can see the problems being generated
> in the problems view.
>
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:ge0s2i$i2t$1@build.eclipse.org...
>> Hi Chuck,
>>
>> You can use different combinations of the current APIs if it fits your
>> usecase.
>>
>> At the moment you can acquire the type of the build operation if your
>> build participant implements IBuildParticipantExtension interface.
>>
>> public interface IBuildParticipantExtension {
>> int FULL_BUILD = IScriptBuilder.FULL_BUILD;
>> int INCREMENTAL_BUILD=IScriptBuilder.INCREMENTAL_BUILD;
>> int RECONCILE_BUILD = 10;
>>
>> void beginBuild(int buildType);
>> void endBuild(IProgressMonitor monitor);
>> }
>>
>> Thanks for the suggestion to add method to the IBuildContext, I have just
>> committed it so it will be available in the next integration build.
>>
>> Back to the type checking in builder. You can implement the specified
>> functionality thru the IBuildParticipant way too, for example please see
>> the
>> org.eclipse.dltk.tcl.internal.validators.packages.PackageReq uireChecker
>> class in the org.eclipse.dltk.tcl.validators plugin. It checks the
>> "package require" statements and verifies that the required packages are
>> implemented somewhere. In short words - the information is collected
>> during build() method calls, and all the checks are performed in the
>> endBuild() method.
>>
>> Anyway, you should use the APIs that suits you well. If something does
>> not suit you - you are not required to use it. Or you can suggest changes
>> to the API to make it more usable for you.
>>
>> Regards,
>> Alex
>>
>>
>> Chuck Doucette wrote:
>>> Thanks for your suggestions.
>>>
>>> Our custom modlel building is basically equivalent to indexing.
>>> We want to know where all the type/variable/method declarations are
>>> being made so we can resolve
>>> references to them from other modules.
>>> We want to index everything (all modules) before we do any semantic
>>> checking.
>>> We keep this information in our own hierarchy - similar to your model -
>>> but completely
>>> distinct from the AST tree.
>>>
>>> Thus, if one build participant was our indexer/custom-model-builder, it
>>> doesn't work
>>> in the current architecture, since it the builder that would iterate
>>> through each module
>>> calling the build participants in order. Instead, we want to call the
>>> indexer/custom-model builder for all of the modules first - and then
>>> when that is done go ahead and call the semantic checker for all
>>> modules.
>>>
>>> Our custom builder which implements IScriptBuilder does what we want, if
>>> it is invoked.
>>>
>>> The problem is that
>>> a) you have strongly suggested that we use the IBuildParticipant
>>> interface (and we would prefer to since it is the recommended way to
>>> integrate with your architecture and gives us more information about
>>> what we are building)
>>> b) If we use IBuildParticipant, and disable our customer builder,
>>> then our build participants are auomatically invoked:
>>> i) during building via ScriptBuilder and StandardScriptBuilder
>>> [ this causes semantic errors from unresolved inter-module
>>> references ]
>>> ii) during reconciling via StructureBuilder
>>> c) If we don't use IBuildParticipant and use our own custom builder
>>> instead
>>> i) we can tightly control the order our participants are invoked
>>> in the build process and thus resolve the inter-module references
>>> ii) we have no well defined way to integrate our participants into
>>> the reconciliation process other than perhaps reverting to using the
>>> SourceElementParser.
>>> d) It may be possible to both enable our custom build and use build
>>> participants.
>>> My boss suggested that if I could determine whether my build
>>> participants were being invoked in the builder or the reconciler, I
>>> could proceed with their execution or exit permaturely. I thought
>>> perhaps make that determination based on the type of IBuildContext - but
>>> all of the classes that implement IBuildContext are in internal packages
>>> which are not exported.
>>>
>>> Perhaps you could add a member to IBuildContext like:
>>>
>>> int getBuildContextType();
>>>
>>> It could return something like:
>>>
>>> EXTERNAL_MODULE_BUILD_CONTEXT
>>> RECONCILE_BUILD_CONTEXT
>>> SOURCE_MODULE_BUILD_CONTEXT
>>>
>>> Chuck
>>>
>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>> news:gdrqu9$rna$1@build.eclipse.org...
>>>> Hi Chuck,
>>>>
>>>> Please see the answers inline.
>>>>
>>>> Chuck Doucette wrote:
>>>>> I am still a bit confused.
>>>>>
>>>>> Since my project nature class extends ScriptNature, your builder,
>>>>> ScriptBuilder, is automatically added to each of our DLTK projects'
>>>>> build specs. Thus, when Eclipse wants to build a DLTK project, it
>>>>> calls your DLTK builder class, ScriptBuilder, which implements
>>>>> IncrementalProjectBuilder.
>>>>> That defers most of its work to any user's custom DLTK builder(s),
>>>>> which implement IScriptBuilder.
>>>>>
>>>>> My builder class, VScriptBuilder, which implements IScriptBuilder,
>>>>> calls each of my build participants in order.
>>>>> It currently calls them directly - without them implementing any
>>>>> IBuildParticipant interface.
>>>>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>>>>> model (at least of declarations)
>>>>> a) module's source contents -> parsing -> produces AST tree
>>>>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>>>>> model
>>>>> 2. Phase 2 - semantic check modules based on AST tree and custom
>>>>> model
>>>>> a) AST tree -> semantic checker (extends AST Visitor - reports
>>>>> most problems)
>>>>>
>>>>> You say I should use IBuildParticipant - at least for 1b (my custom
>>>>> model builder) and 2a (my semantic checker),
>>>>> and perhaps also for my parser (but perhaps not if I split off the
>>>>> custom model builder).
>>>>> Apparently, IBuildParticipant is used by:
>>>>> a) ValidatorBuilder
>>>>> b) ValidatorStructureBuilder
>>>>>
>>>>> ValidatorStructureBuilder implements IStructureBuilder, which is used
>>>>> by AbstractSourceModule.buildStructure and
>>>>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>>>>> during reconcilation).
>>>>> I can't see where/how ValidatorBuilder is used at all - but perhaps
>>>>> that doesn't matter.
>>>>>
>>>>> So, do I need to
>>>>> a) add extension wrapper classes around my builder components which
>>>>> implement IBuildParticipant
>>>> If you classes are provided as IBuildParticipants they will be called
>>>> by the build and reconcile operations.
>>>>
>>>> For scripting languages tasks like parsing and error reporting operate
>>>> on the single source module usually, so the implementation could be a
>>>> bit simpler if compared with IScriptBuilder way.
>>>>
>>>>> b) keep my existing builder class, VScriptBuilder, since it implements
>>>>> IScriptBuilder and will ultimately be called by the Eclipse build
>>>>> manager.
>>>> Some builder parts fit well into the IBuildParticipant way, so you
>>>> should wrap them as this extension.
>>>>
>>>> On the other hand, there could be builder parts that do not fit well
>>>> into the IBuildParticipant, so you should keep them in ScriptBuilder.
>>>> If all of the operations could be wrapped as IBuildParticipant, then it
>>>> looks like you don't need your own ScriptBuilder implementation.
>>>>
>>>>> c) use your existing ValidatorStructureBuilder which will call my
>>>>> build participants during reconciliation
>>>> it was a temporary internal class and it is removed now. If you want to
>>>> your code to be called from the reconcile - you should declare it as
>>>> IBuildParticipants.
>>>>
>>>>> d) remove calls to my semantic checker from my class
>>>>> VScriptSourceElementParser, which extends ISourceElementParser (this
>>>>> is where you had told me to put my semantic checker since it was
>>>>> getting called during reconciliation)
>>>> In the DLTK 0.95 it was the acceptable way to add additional code,
>>>> since the builder infrastructure was not so comprehensive like it is
>>>> now. Though in general it was not the best way, since they are quite
>>>> different tasks and most of the time they should be executed in
>>>> different moments of time.
>>>>
>>>> Taking all these issues into account the builder infrastructure was
>>>> extended to simplify implementation of additional semantic checkers,
>>>> etc.
>>>>
>>>> So I recommend to implement it is as IBuildParticipant if you decide to
>>>> go this way.
>>>>
>>>>> e) remove calls to my custom model builder in my class
>>>>> VScriptSourceParser, which extends ISourceParser.
>>>> The correct place to update your model depends on how you use it.
>>>> Unfortunately I don't know the details of your custom model, so I will
>>>> post some thoughts and you could select the appropriate solution
>>>> yourself.
>>>>
>>>> If you model could be built on demand, you don't need to prepare it for
>>>> future use and it could be treated like inherent part of AST - you can
>>>> put it into your SourceParser, also you can extend ModelDeclaration
>>>> class and add additional fields with your custom model of that source
>>>> module.
>>>>
>>>> In contrast, if you need to prepare your custom model beforehand for
>>>> all modules (for example, type/method indexing works that way) and it
>>>> is an independent part from the AST - it is better to build it
>>>> separately, so you don't need that code in SourceParser, and it could
>>>> be implemented as IBuildParticipant.
>>>>
>>>> Hope that helps.
>>>>
>>>> Regards,
>>>> Alex
>>>
>
Re: parsers, builders, build types, and build participants [message #34304 is a reply to message #34090] Mon, 10 November 2008 10:28 Go to previous message
Alex Panchenko is currently offline Alex Panchenko
Messages: 342
Registered: July 2009
Senior Member
Hi Chuck,

I have changed beginBuild() to return boolean.
It will be available in the next integration build.

Regards,
Alex


Chuck Doucette wrote:
> Here's my suggestion.
>
> Since
> a) although your build participant structure is fairly flexible - I can't
> seem to use it to do what I want for a regular build - although it is great
> for reconciliation.
> b) Thus, I want to use most/all of my builder for building.
>
> Since your StandardScriptBuilder always runs, and it assumes that if there
> are build participants configured that they always run, then it deletes all
> problem markers for a file before flushing a new set.
> If I want to use a regular builder for building and your
> StandardScriptBuilder with my build participants for reconcilation - then
> any errors generated from my builder are flushed.
>
> Here is my suggestion:
> 1) change beginBuild to return a boolean that indicates whether this
> build participant will do any work; or, add a new interface and a method -
> willDoWork. You can assume if the interface is not implemented that it
> returned true.
> 2) If it returns true, then call build and endBuild and flush/clear
> problem markers
> 3) If it returns false - don't do any of those things - exit prematurely
> instead.
>
> Chuck
>
> "Chuck Doucette" <cdoucette@vaultus.com> wrote in message
> news:ged751$kgo$1@build.eclipse.org...
>> I still can't seem to find an architecture that works well with the latest
>> DLTK build.
>>
>> Although I liked your suggestion to:
>> a) implement IBuildParticipantExtension so beginBuild and endBuild methods
>> would be called
>> b) postpone some processing until endBuild
>>
>> This doesn't quite work.
>>
>> The issue is that if I declare a method "a" in module A, and I refer to
>> method A.a in module B,
>> I don't want to flag that method reference if it is actually defined
>> elsewhere.
>> Thus, I must
>> a) process all of the external declarations of *all* modules to be built
>> (not just the current module)
>> b) semantic check all references (etc.) of each module
>>
>> Your build participant structure calls each build participant in order on
>> each module.
>>
>> My overall builder which implements IScriptBuilder does what I want.
>> In order to avoid redundant work, I attempted to disable my build
>> participants from building
>> (but allow them to be used for reconciliation); however, this didn't work
>> since your
>> StandardScriptBuilder clears all of the problems from my builder.
>> I believe this is a bug.
>>
>> At the moment, I have enabled my builder and your builder, and even though
>> they
>> are doing redundant work - at least I can see the problems being generated
>> in the problems view.
>>
>> Chuck
>>
>> "Alex Panchenko" <alex@xored.com> wrote in message
>> news:ge0s2i$i2t$1@build.eclipse.org...
>>> Hi Chuck,
>>>
>>> You can use different combinations of the current APIs if it fits your
>>> usecase.
>>>
>>> At the moment you can acquire the type of the build operation if your
>>> build participant implements IBuildParticipantExtension interface.
>>>
>>> public interface IBuildParticipantExtension {
>>> int FULL_BUILD = IScriptBuilder.FULL_BUILD;
>>> int INCREMENTAL_BUILD=IScriptBuilder.INCREMENTAL_BUILD;
>>> int RECONCILE_BUILD = 10;
>>>
>>> void beginBuild(int buildType);
>>> void endBuild(IProgressMonitor monitor);
>>> }
>>>
>>> Thanks for the suggestion to add method to the IBuildContext, I have just
>>> committed it so it will be available in the next integration build.
>>>
>>> Back to the type checking in builder. You can implement the specified
>>> functionality thru the IBuildParticipant way too, for example please see
>>> the
>>> org.eclipse.dltk.tcl.internal.validators.packages.PackageReq uireChecker
>>> class in the org.eclipse.dltk.tcl.validators plugin. It checks the
>>> "package require" statements and verifies that the required packages are
>>> implemented somewhere. In short words - the information is collected
>>> during build() method calls, and all the checks are performed in the
>>> endBuild() method.
>>>
>>> Anyway, you should use the APIs that suits you well. If something does
>>> not suit you - you are not required to use it. Or you can suggest changes
>>> to the API to make it more usable for you.
>>>
>>> Regards,
>>> Alex
>>>
>>>
>>> Chuck Doucette wrote:
>>>> Thanks for your suggestions.
>>>>
>>>> Our custom modlel building is basically equivalent to indexing.
>>>> We want to know where all the type/variable/method declarations are
>>>> being made so we can resolve
>>>> references to them from other modules.
>>>> We want to index everything (all modules) before we do any semantic
>>>> checking.
>>>> We keep this information in our own hierarchy - similar to your model -
>>>> but completely
>>>> distinct from the AST tree.
>>>>
>>>> Thus, if one build participant was our indexer/custom-model-builder, it
>>>> doesn't work
>>>> in the current architecture, since it the builder that would iterate
>>>> through each module
>>>> calling the build participants in order. Instead, we want to call the
>>>> indexer/custom-model builder for all of the modules first - and then
>>>> when that is done go ahead and call the semantic checker for all
>>>> modules.
>>>>
>>>> Our custom builder which implements IScriptBuilder does what we want, if
>>>> it is invoked.
>>>>
>>>> The problem is that
>>>> a) you have strongly suggested that we use the IBuildParticipant
>>>> interface (and we would prefer to since it is the recommended way to
>>>> integrate with your architecture and gives us more information about
>>>> what we are building)
>>>> b) If we use IBuildParticipant, and disable our customer builder,
>>>> then our build participants are auomatically invoked:
>>>> i) during building via ScriptBuilder and StandardScriptBuilder
>>>> [ this causes semantic errors from unresolved inter-module
>>>> references ]
>>>> ii) during reconciling via StructureBuilder
>>>> c) If we don't use IBuildParticipant and use our own custom builder
>>>> instead
>>>> i) we can tightly control the order our participants are invoked
>>>> in the build process and thus resolve the inter-module references
>>>> ii) we have no well defined way to integrate our participants into
>>>> the reconciliation process other than perhaps reverting to using the
>>>> SourceElementParser.
>>>> d) It may be possible to both enable our custom build and use build
>>>> participants.
>>>> My boss suggested that if I could determine whether my build
>>>> participants were being invoked in the builder or the reconciler, I
>>>> could proceed with their execution or exit permaturely. I thought
>>>> perhaps make that determination based on the type of IBuildContext - but
>>>> all of the classes that implement IBuildContext are in internal packages
>>>> which are not exported.
>>>>
>>>> Perhaps you could add a member to IBuildContext like:
>>>>
>>>> int getBuildContextType();
>>>>
>>>> It could return something like:
>>>>
>>>> EXTERNAL_MODULE_BUILD_CONTEXT
>>>> RECONCILE_BUILD_CONTEXT
>>>> SOURCE_MODULE_BUILD_CONTEXT
>>>>
>>>> Chuck
>>>>
>>>> "Alex Panchenko" <alex@xored.com> wrote in message
>>>> news:gdrqu9$rna$1@build.eclipse.org...
>>>>> Hi Chuck,
>>>>>
>>>>> Please see the answers inline.
>>>>>
>>>>> Chuck Doucette wrote:
>>>>>> I am still a bit confused.
>>>>>>
>>>>>> Since my project nature class extends ScriptNature, your builder,
>>>>>> ScriptBuilder, is automatically added to each of our DLTK projects'
>>>>>> build specs. Thus, when Eclipse wants to build a DLTK project, it
>>>>>> calls your DLTK builder class, ScriptBuilder, which implements
>>>>>> IncrementalProjectBuilder.
>>>>>> That defers most of its work to any user's custom DLTK builder(s),
>>>>>> which implement IScriptBuilder.
>>>>>>
>>>>>> My builder class, VScriptBuilder, which implements IScriptBuilder,
>>>>>> calls each of my build participants in order.
>>>>>> It currently calls them directly - without them implementing any
>>>>>> IBuildParticipant interface.
>>>>>> 1. Phase 1 - parse to create AST tree - use AST tree to create custom
>>>>>> model (at least of declarations)
>>>>>> a) module's source contents -> parsing -> produces AST tree
>>>>>> b) AST tree -> custom model builder (extends ASTVisitor) -> custom
>>>>>> model
>>>>>> 2. Phase 2 - semantic check modules based on AST tree and custom
>>>>>> model
>>>>>> a) AST tree -> semantic checker (extends AST Visitor - reports
>>>>>> most problems)
>>>>>>
>>>>>> You say I should use IBuildParticipant - at least for 1b (my custom
>>>>>> model builder) and 2a (my semantic checker),
>>>>>> and perhaps also for my parser (but perhaps not if I split off the
>>>>>> custom model builder).
>>>>>> Apparently, IBuildParticipant is used by:
>>>>>> a) ValidatorBuilder
>>>>>> b) ValidatorStructureBuilder
>>>>>>
>>>>>> ValidatorStructureBuilder implements IStructureBuilder, which is used
>>>>>> by AbstractSourceModule.buildStructure and
>>>>>> ReconcileWorkingCopyOperation.makeConsistent (one or the other used
>>>>>> during reconcilation).
>>>>>> I can't see where/how ValidatorBuilder is used at all - but perhaps
>>>>>> that doesn't matter.
>>>>>>
>>>>>> So, do I need to
>>>>>> a) add extension wrapper classes around my builder components which
>>>>>> implement IBuildParticipant
>>>>> If you classes are provided as IBuildParticipants they will be called
>>>>> by the build and reconcile operations.
>>>>>
>>>>> For scripting languages tasks like parsing and error reporting operate
>>>>> on the single source module usually, so the implementation could be a
>>>>> bit simpler if compared with IScriptBuilder way.
>>>>>
>>>>>> b) keep my existing builder class, VScriptBuilder, since it implements
>>>>>> IScriptBuilder and will ultimately be called by the Eclipse build
>>>>>> manager.
>>>>> Some builder parts fit well into the IBuildParticipant way, so you
>>>>> should wrap them as this extension.
>>>>>
>>>>> On the other hand, there could be builder parts that do not fit well
>>>>> into the IBuildParticipant, so you should keep them in ScriptBuilder.
>>>>> If all of the operations could be wrapped as IBuildParticipant, then it
>>>>> looks like you don't need your own ScriptBuilder implementation.
>>>>>
>>>>>> c) use your existing ValidatorStructureBuilder which will call my
>>>>>> build participants during reconciliation
>>>>> it was a temporary internal class and it is removed now. If you want to
>>>>> your code to be called from the reconcile - you should declare it as
>>>>> IBuildParticipants.
>>>>>
>>>>>> d) remove calls to my semantic checker from my class
>>>>>> VScriptSourceElementParser, which extends ISourceElementParser (this
>>>>>> is where you had told me to put my semantic checker since it was
>>>>>> getting called during reconciliation)
>>>>> In the DLTK 0.95 it was the acceptable way to add additional code,
>>>>> since the builder infrastructure was not so comprehensive like it is
>>>>> now. Though in general it was not the best way, since they are quite
>>>>> different tasks and most of the time they should be executed in
>>>>> different moments of time.
>>>>>
>>>>> Taking all these issues into account the builder infrastructure was
>>>>> extended to simplify implementation of additional semantic checkers,
>>>>> etc.
>>>>>
>>>>> So I recommend to implement it is as IBuildParticipant if you decide to
>>>>> go this way.
>>>>>
>>>>>> e) remove calls to my custom model builder in my class
>>>>>> VScriptSourceParser, which extends ISourceParser.
>>>>> The correct place to update your model depends on how you use it.
>>>>> Unfortunately I don't know the details of your custom model, so I will
>>>>> post some thoughts and you could select the appropriate solution
>>>>> yourself.
>>>>>
>>>>> If you model could be built on demand, you don't need to prepare it for
>>>>> future use and it could be treated like inherent part of AST - you can
>>>>> put it into your SourceParser, also you can extend ModelDeclaration
>>>>> class and add additional fields with your custom model of that source
>>>>> module.
>>>>>
>>>>> In contrast, if you need to prepare your custom model beforehand for
>>>>> all modules (for example, type/method indexing works that way) and it
>>>>> is an independent part from the AST - it is better to build it
>>>>> separately, so you don't need that code in SourceParser, and it could
>>>>> be implemented as IBuildParticipant.
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> Regards,
>>>>> Alex
>
>
Previous Topic:Disable Text hover on comments
Next Topic:[BUILD] I-I200811100931-200811100931
Goto Forum:
  


Current Time: Sun Sep 21 02:05:02 GMT 2014

Powered by FUDForum. Page generated in 0.07543 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software