Home » Eclipse Projects » Dynamic Languages Toolkit (DLTK) » Semantic checking - via validators?
|
| Re: Semantic checking - via validators? [message #23957 is a reply to message #23537] |
Mon, 12 May 2008 15:08   |
 Andrei Sobolev
Messages: 72
Registered: July 2009 |
Member |
|
|
Hi Chuck,
Nice to see, what DLTK help you.
We don't have something special in area of semantic analysis, etc.
Where is some ways to implement such features:
1) Using of ScriptBuilder.
In DLTK we have feature named ScriptBuilder. This is incremental resource builder, used for building Mixin models, etc.
It will give you list of changed resources, and you cold check all this files and set appropriate markers.
All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will contain information about builded resources,
and will be executed on each resource operations, like "resource added", "resource content changed".
2) You could use Validator, Validators framework uses Script Builder for incremental execution of validators. One more
additional point, validators framework manages validator instances and allow unified way to sore configurations, etc.
Also user could turn on/off and execute validators from DLTK UI.
3) If you also plan to make semantic checks for file edited, you need to extend Reconciler from your
ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and do checks, after user will make some changes
in code. By default it is 500 milliseconds delay, so reconciler will be executed only if user not type code 500
milliseconds.
References:
1) Extension point to implement: org.eclipse.dltk.core.builder
2) We have "package require" checker implemented for Tcl. (TclCheckBuilder class).
3) ScriptSourceViewerConfiguration.getReconciler()
Best regards,
Andrei Sobolev.
> Thanks to ANTLR and DLTK, I now have an advanced editor which can
> immediately flag syntactic problems in our language source files.
>
> My question is if/when/how should I find/flag semantic errors in the source
> file?
>
> 1. Is there any part of DLTK framework which could/should be used to
> invoke/perform semantic checks? I assume the answer to this is no. I know
> you support external validators - but we'd like the semantic checking to be
> performed implicitly.
>
> 2. When should semantic checks be performed - whenever the parser is
> invoked?
>
> 3. How should errors be reported - using the parser's error reporter?
> Also, how can we determine the line number for populating IProblem
> in the file if that information is:
> a) not available in DLTKToken
> b) not kept in ASTNode
>
> Thanks,
> Chuck
>
>
|
|
|
| Re: Semantic checking - via validators? [message #26300 is a reply to message #23957] |
Thu, 19 June 2008 19:28 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
I wrote a visitor pattern to do semantic checking.
For the time being, I was attempting to invoke it whenever I invoked the
parser.
It detects problems and tries to report them.
Unfortunately, the problem reporter is not instantiated (i.e. it is null).
Thus, no problems are reported (even though there are some).
The problem appears to in SourceModule.getProblemReporter.
Shouldn't we be able to add problems to the Problems view?
Chuck
"Andrei Sobolev" <haiodo@xored.com> wrote in message
news:g09mhq$9i2$1@build.eclipse.org...
> Hi Chuck,
>
> Nice to see, what DLTK help you.
>
> We don't have something special in area of semantic analysis, etc.
>
> Where is some ways to implement such features:
> 1) Using of ScriptBuilder.
> In DLTK we have feature named ScriptBuilder. This is incremental resource
> builder, used for building Mixin models, etc.
> It will give you list of changed resources, and you cold check all this
> files and set appropriate markers.
> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will
> contain information about builded resources,
> and will be executed on each resource operations, like "resource added",
> "resource content changed".
>
> 2) You could use Validator, Validators framework uses Script Builder for
> incremental execution of validators. One more
> additional point, validators framework manages validator instances and
> allow unified way to sore configurations, etc.
> Also user could turn on/off and execute validators from DLTK UI.
>
> 3) If you also plan to make semantic checks for file edited, you need to
> extend Reconciler from your
> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and
> do checks, after user will make some changes
> in code. By default it is 500 milliseconds delay, so reconciler will be
> executed only if user not type code 500
> milliseconds.
>
>
> References:
> 1) Extension point to implement: org.eclipse.dltk.core.builder
> 2) We have "package require" checker implemented for Tcl. (TclCheckBuilder
> class).
> 3) ScriptSourceViewerConfiguration.getReconciler()
>
> Best regards,
> Andrei Sobolev.
>
>
>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>> immediately flag syntactic problems in our language source files.
>>
>> My question is if/when/how should I find/flag semantic errors in the
>> source
>> file?
>>
>> 1. Is there any part of DLTK framework which could/should be used to
>> invoke/perform semantic checks? I assume the answer to this is no. I
>> know
>> you support external validators - but we'd like the semantic checking to
>> be
>> performed implicitly.
>>
>> 2. When should semantic checks be performed - whenever the parser is
>> invoked?
>>
>> 3. How should errors be reported - using the parser's error reporter?
>> Also, how can we determine the line number for populating
>> IProblem
>> in the file if that information is:
>> a) not available in DLTKToken
>> b) not kept in ASTNode
>>
>> Thanks,
>> Chuck
>>
>>
|
|
|
| Re: Semantic checking - via validators? [message #26340 is a reply to message #26300] |
Fri, 20 June 2008 03:21 |
Alex Panchenko
Messages: 342
Registered: July 2009 |
Senior Member |
|
|
Hi Chuck,
The problem reporter is intentionally disabled (i.e. null) when the
parsing occurs during editor open, because we want to keep current
markers intact to be able to open the file from the problems/tasks view
and navigate to the line the marker is placed on.
In all other situations it should work correctly. Have you tried
stepping through the code in the debugger to see why it is null?
Regards,
Alex
Chuck Doucette wrote:
> I wrote a visitor pattern to do semantic checking.
> For the time being, I was attempting to invoke it whenever I invoked the
> parser.
> It detects problems and tries to report them.
> Unfortunately, the problem reporter is not instantiated (i.e. it is null).
> Thus, no problems are reported (even though there are some).
>
> The problem appears to in SourceModule.getProblemReporter.
>
> Shouldn't we be able to add problems to the Problems view?
>
> Chuck
>
> "Andrei Sobolev" <haiodo@xored.com> wrote in message
> news:g09mhq$9i2$1@build.eclipse.org...
>> Hi Chuck,
>>
>> Nice to see, what DLTK help you.
>>
>> We don't have something special in area of semantic analysis, etc.
>>
>> Where is some ways to implement such features:
>> 1) Using of ScriptBuilder.
>> In DLTK we have feature named ScriptBuilder. This is incremental resource
>> builder, used for building Mixin models, etc.
>> It will give you list of changed resources, and you cold check all this
>> files and set appropriate markers.
>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will
>> contain information about builded resources,
>> and will be executed on each resource operations, like "resource added",
>> "resource content changed".
>>
>> 2) You could use Validator, Validators framework uses Script Builder for
>> incremental execution of validators. One more
>> additional point, validators framework manages validator instances and
>> allow unified way to sore configurations, etc.
>> Also user could turn on/off and execute validators from DLTK UI.
>>
>> 3) If you also plan to make semantic checks for file edited, you need to
>> extend Reconciler from your
>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and
>> do checks, after user will make some changes
>> in code. By default it is 500 milliseconds delay, so reconciler will be
>> executed only if user not type code 500
>> milliseconds.
>>
>>
>> References:
>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>> 2) We have "package require" checker implemented for Tcl. (TclCheckBuilder
>> class).
>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>
>> Best regards,
>> Andrei Sobolev.
>>
>>
>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>> immediately flag syntactic problems in our language source files.
>>>
>>> My question is if/when/how should I find/flag semantic errors in the
>>> source
>>> file?
>>>
>>> 1. Is there any part of DLTK framework which could/should be used to
>>> invoke/perform semantic checks? I assume the answer to this is no. I
>>> know
>>> you support external validators - but we'd like the semantic checking to
>>> be
>>> performed implicitly.
>>>
>>> 2. When should semantic checks be performed - whenever the parser is
>>> invoked?
>>>
>>> 3. How should errors be reported - using the parser's error reporter?
>>> Also, how can we determine the line number for populating
>>> IProblem
>>> in the file if that information is:
>>> a) not available in DLTKToken
>>> b) not kept in ASTNode
>>>
>>> Thanks,
>>> Chuck
>>>
>>>
>
>
|
|
|
| Re: Semantic checking - via validators? [message #26378 is a reply to message #26340] |
Fri, 20 June 2008 16:31 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
But I thought that if your editing created a syntax error, it was
a) flagged in the editor window with a red marker
b) added to the problems window immediately (presumably via the problem
reporter)
So, when is the problem reporter supposed to be set?
Indeed, I did run it in the debugger.
That's where I came up with:
>> The problem appears to in SourceModule.getProblemReporter.
Specifically, the problem reporter is null because noProblemReporter is
true:
protected IProblemReporter getProblemReporter(String natureId) throws
CoreException {
// check for the working copy reporter first
ModelManager.PerWorkingCopyInfo wcInfo = getPerWorkingCopyInfo();
if (wcInfo != null) {
if (wcInfo.noProblemReporter) {
return null;
}
I'm not sure how the field noProblemReporter in
ModelManager.PerWorkingCopyInfo becomes true, since I believe it is
implicitly initialized to false, and I don't see any line of code that
modifies its value.
Chuck
"Alex Panchenko" <alex@xored.com> wrote in message
news:g3f7ob$nab$1@build.eclipse.org...
> Hi Chuck,
>
> The problem reporter is intentionally disabled (i.e. null) when the
> parsing occurs during editor open, because we want to keep current markers
> intact to be able to open the file from the problems/tasks view and
> navigate to the line the marker is placed on.
>
> In all other situations it should work correctly. Have you tried stepping
> through the code in the debugger to see why it is null?
>
> Regards,
> Alex
>
>
> Chuck Doucette wrote:
>> I wrote a visitor pattern to do semantic checking.
>> For the time being, I was attempting to invoke it whenever I invoked the
>> parser.
>> It detects problems and tries to report them.
>> Unfortunately, the problem reporter is not instantiated (i.e. it is
>> null).
>> Thus, no problems are reported (even though there are some).
>>
>> The problem appears to in SourceModule.getProblemReporter.
>>
>> Shouldn't we be able to add problems to the Problems view?
>>
>> Chuck
>>
>> "Andrei Sobolev" <haiodo@xored.com> wrote in message
>> news:g09mhq$9i2$1@build.eclipse.org...
>>> Hi Chuck,
>>>
>>> Nice to see, what DLTK help you.
>>>
>>> We don't have something special in area of semantic analysis, etc.
>>>
>>> Where is some ways to implement such features:
>>> 1) Using of ScriptBuilder.
>>> In DLTK we have feature named ScriptBuilder. This is incremental
>>> resource builder, used for building Mixin models, etc.
>>> It will give you list of changed resources, and you cold check all this
>>> files and set appropriate markers.
>>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder
>>> will contain information about builded resources,
>>> and will be executed on each resource operations, like "resource added",
>>> "resource content changed".
>>>
>>> 2) You could use Validator, Validators framework uses Script Builder for
>>> incremental execution of validators. One more
>>> additional point, validators framework manages validator instances and
>>> allow unified way to sore configurations, etc.
>>> Also user could turn on/off and execute validators from DLTK UI.
>>>
>>> 3) If you also plan to make semantic checks for file edited, you need to
>>> extend Reconciler from your
>>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model,
>>> and do checks, after user will make some changes
>>> in code. By default it is 500 milliseconds delay, so reconciler will be
>>> executed only if user not type code 500
>>> milliseconds.
>>>
>>>
>>> References:
>>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>>> 2) We have "package require" checker implemented for Tcl.
>>> (TclCheckBuilder class).
>>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>>
>>> Best regards,
>>> Andrei Sobolev.
>>>
>>>
>>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>>> immediately flag syntactic problems in our language source files.
>>>>
>>>> My question is if/when/how should I find/flag semantic errors in the
>>>> source
>>>> file?
>>>>
>>>> 1. Is there any part of DLTK framework which could/should be used to
>>>> invoke/perform semantic checks? I assume the answer to this is no. I
>>>> know
>>>> you support external validators - but we'd like the semantic checking
>>>> to be
>>>> performed implicitly.
>>>>
>>>> 2. When should semantic checks be performed - whenever the parser is
>>>> invoked?
>>>>
>>>> 3. How should errors be reported - using the parser's error
>>>> reporter?
>>>> Also, how can we determine the line number for populating
>>>> IProblem
>>>> in the file if that information is:
>>>> a) not available in DLTKToken
>>>> b) not kept in ASTNode
>>>>
>>>> Thanks,
>>>> Chuck
>>>>
>>>>
>>
|
|
|
| Re: Semantic checking - via validators? [message #26451 is a reply to message #26378] |
Mon, 23 June 2008 08:21 |
Alex Panchenko
Messages: 342
Registered: July 2009 |
Senior Member |
|
|
Hi Chuck,
noProblemReporter is true *ONLY* during initial editor open.
After some changes are made in the editor and file is reparsed -
noProblemReporter field will be false and problem reporter will be
available to the parser.
We need this behavior to prevent changes to old markers.
The situation:
- no editors open
- problems (or tasks) view contains some markers
User double clicks on the marker and he/she expects the following actions:
a) the file is opened in the editor
b) cursor is placed on the marker line
If the problem reporter is available to the parser during editor
initialization - markers will be modified and the marker used to open
the editor becomes invalidated and line number could not be retrieved
from it.
JDT contains similar workaround.
Regards,
Alex
Chuck Doucette wrote:
> But I thought that if your editing created a syntax error, it was
> a) flagged in the editor window with a red marker
> b) added to the problems window immediately (presumably via the problem
> reporter)
>
> So, when is the problem reporter supposed to be set?
>
> Indeed, I did run it in the debugger.
> That's where I came up with:
> >> The problem appears to in SourceModule.getProblemReporter.
> Specifically, the problem reporter is null because noProblemReporter is
> true:
>
> protected IProblemReporter getProblemReporter(String natureId) throws
> CoreException {
> // check for the working copy reporter first
> ModelManager.PerWorkingCopyInfo wcInfo = getPerWorkingCopyInfo();
> if (wcInfo != null) {
> if (wcInfo.noProblemReporter) {
> return null;
> }
>
> I'm not sure how the field noProblemReporter in
> ModelManager.PerWorkingCopyInfo becomes true, since I believe it is
> implicitly initialized to false, and I don't see any line of code that
> modifies its value.
>
> Chuck
>
> "Alex Panchenko" <alex@xored.com> wrote in message
> news:g3f7ob$nab$1@build.eclipse.org...
>> Hi Chuck,
>>
>> The problem reporter is intentionally disabled (i.e. null) when the
>> parsing occurs during editor open, because we want to keep current markers
>> intact to be able to open the file from the problems/tasks view and
>> navigate to the line the marker is placed on.
>>
>> In all other situations it should work correctly. Have you tried stepping
>> through the code in the debugger to see why it is null?
>>
>> Regards,
>> Alex
>>
>>
>> Chuck Doucette wrote:
>>> I wrote a visitor pattern to do semantic checking.
>>> For the time being, I was attempting to invoke it whenever I invoked the
>>> parser.
>>> It detects problems and tries to report them.
>>> Unfortunately, the problem reporter is not instantiated (i.e. it is
>>> null).
>>> Thus, no problems are reported (even though there are some).
>>>
>>> The problem appears to in SourceModule.getProblemReporter.
>>>
>>> Shouldn't we be able to add problems to the Problems view?
>>>
>>> Chuck
>>>
>>> "Andrei Sobolev" <haiodo@xored.com> wrote in message
>>> news:g09mhq$9i2$1@build.eclipse.org...
>>>> Hi Chuck,
>>>>
>>>> Nice to see, what DLTK help you.
>>>>
>>>> We don't have something special in area of semantic analysis, etc.
>>>>
>>>> Where is some ways to implement such features:
>>>> 1) Using of ScriptBuilder.
>>>> In DLTK we have feature named ScriptBuilder. This is incremental
>>>> resource builder, used for building Mixin models, etc.
>>>> It will give you list of changed resources, and you cold check all this
>>>> files and set appropriate markers.
>>>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder
>>>> will contain information about builded resources,
>>>> and will be executed on each resource operations, like "resource added",
>>>> "resource content changed".
>>>>
>>>> 2) You could use Validator, Validators framework uses Script Builder for
>>>> incremental execution of validators. One more
>>>> additional point, validators framework manages validator instances and
>>>> allow unified way to sore configurations, etc.
>>>> Also user could turn on/off and execute validators from DLTK UI.
>>>>
>>>> 3) If you also plan to make semantic checks for file edited, you need to
>>>> extend Reconciler from your
>>>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model,
>>>> and do checks, after user will make some changes
>>>> in code. By default it is 500 milliseconds delay, so reconciler will be
>>>> executed only if user not type code 500
>>>> milliseconds.
>>>>
>>>>
>>>> References:
>>>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>>>> 2) We have "package require" checker implemented for Tcl.
>>>> (TclCheckBuilder class).
>>>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>>>
>>>> Best regards,
>>>> Andrei Sobolev.
>>>>
>>>>
>>>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>>>> immediately flag syntactic problems in our language source files.
>>>>>
>>>>> My question is if/when/how should I find/flag semantic errors in the
>>>>> source
>>>>> file?
>>>>>
>>>>> 1. Is there any part of DLTK framework which could/should be used to
>>>>> invoke/perform semantic checks? I assume the answer to this is no. I
>>>>> know
>>>>> you support external validators - but we'd like the semantic checking
>>>>> to be
>>>>> performed implicitly.
>>>>>
>>>>> 2. When should semantic checks be performed - whenever the parser is
>>>>> invoked?
>>>>>
>>>>> 3. How should errors be reported - using the parser's error
>>>>> reporter?
>>>>> Also, how can we determine the line number for populating
>>>>> IProblem
>>>>> in the file if that information is:
>>>>> a) not available in DLTKToken
>>>>> b) not kept in ASTNode
>>>>>
>>>>> Thanks,
>>>>> Chuck
>>>>>
>>>>>
>
|
|
|
| Re: Semantic checking - via validators? [message #26543 is a reply to message #23957] |
Tue, 24 June 2008 02:21 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
As I said in another reply, I have created a semantic checker, and I am
trying to determine where/how to invoke it.
In addition to invoking it in a builder, we also wanted to invoke it
dynamically in the editor, so syntax and semantic errors show up immediately
(just like in JDT). Thus, I believe that we should try to follow your advice
to invoke it via the reconciler.
In the process of trying to determine exactly where/how to do that, I found
out many things that I want to share with other users who may be trying to
do the same thing.
Here's what I have found:
a) My (concrete) class VScriptSourceViewerConfiguration extends the DLTK
abstract class ScriptSourceViewerConfiguration which defines this method:
/**
* Returns the reconciler ready to be used with the given source viewer.
* This implementation always returns <code>null</code>.
*
* @param sourceViewer the source viewer to be configured by this
configuration
* @return a reconciler or <code>null</code> if reconciling should not be
supported
*/
public IReconciler getReconciler(ISourceViewer sourceViewer) {
b) The DLTK class ScriptReconciler implements IReconciler. Its super class,
MonoReconciler depends on an instance of IReconcilingStrategy to do the
reconciliation.
c) The method getReconciler above creates an instance of the DLTK class
ScriptCompositeReconcilingStrategy whose superclass,
CompositeReconcilingStrategy, implements IReconcilingStrategy. It also
depends on an instance of the more straight forward class
ScriptReconcilingStrategy.
d) ScriptReconcilingStrategy depends on ISourceModule.reconcile
e) SourceModule extends AbstractSourceModule (both of which implements
ISourceModule)
The reconcile method calls ReconcileWorkingCopyOperation.runOperation, which
does this:
if (isReadOnly()) {
run(monitor);
} else {
// Use IWorkspace.run(...) to ensure that a build will be done
// in autobuild mode.
// Note that if the tree is locked, this will throw a
// CoreException, but this is ok
// as this operation is modifying the tree (not read-only) and a
// CoreException will be thrown anyway.
IWorkspace wc = ResourcesPlugin.getWorkspace();
wc.run(this, getSchedulingRule(), IWorkspace.AVOID_UPDATE, monitor);
f) IWorkspace.run calls IWorkspaceRunnable.run, and that is defined by
ModelOperation.run, which calls executeOperation.
g) ReconcileWorkingCopyOperation.executeOperation calls makeConsistent,
which calls SourceModule.makeConsistent:
h) SourceModule.makeConsistent does this:
// Remove AST Cache element
ISourceModuleInfoCache sourceModuleInfoCache = ModelManager
..getModelManager().getSourceModuleInfoCache();
sourceModuleInfoCache.remove(this);
openWhenClosed(createElementInfo(), monitor);
i) ModelElement.openWhenClosed calls Openable.generateInfos:
/**
* Generates the element infos for this element, its ancestors (if they are
* not opened) and its children (if it is an Openable). Puts the newly
* created element info in the given map.
*/
protected abstract void generateInfos(Object info, HashMap newElements,
IProgressMonitor pm) throws ModelException;
j) Openable.generateInfos calls AbstractSourceModule.buildStructure:
/**
* Builds this element's structure and properties in the given info object,
* based on this element's current contents (reuse buffer contents if this
* element has an open buffer, or resource contents if this element does not
* have an open buffer). Children are placed in the given newElements table
* (note, this element has already been placed in the newElements table).
* Returns true if successful, or false if an error is encountered while
* determining the structure of this element.
*/
protected abstract boolean buildStructure(OpenableElementInfo info,
IProgressMonitor pm, Map newElements, IResource underlyingResource)
throws ModelException;
k) AbstractSourceModule.buildStructure calls
SourceParserUtil.parseSourceModule
l) SourceParserUtil.parseSourceModule calls
AbstractSourceElementParser.parseSourceModule
m) AbstractSourceElementParser.parseSourceModule calls
SourceParserUtil.getModuleDeclaration:
n) public static ModuleDeclaration getModuleDeclaration(char[] filename,
char[] content, String nature, IProblemReporter reporter,
ISourceModuleInfo mifo, int flags) {
ISourceParser sourceParser;// = new SourceParser(this.fReporter);
sourceParser = DLTKLanguageManager.getSourceParser(nature);
if (sourceParser instanceof ISourceParserExtension) {
((ISourceParserExtension) sourceParser).setFlags(flags);
}
ModuleDeclaration moduleDeclaration = SourceParserUtil
..getModuleFromCache(mifo, flags);
if (moduleDeclaration == null) {
if (reporter != null) {
reporter.clearMarkers();
}
moduleDeclaration = sourceParser.parse(filename, content, reporter);
SourceParserUtil.putModuleToCache(mifo, moduleDeclaration, flags);
}
return moduleDeclaration;
}
o) My class VScriptSourceParser extends AbstractSourceParser which
implements ISourceParser.
The parse method instantiates a vscriptParser and uses it.
NOTES:
o) ISourceParser.parse is also called by:
i) AbstractASTFoldingStructureProvider.getCodeBlocks, which is
ultimately called by:AbstractASTFoldingStructureProvider.initialized and
AbstractASTFoldingStructureProvider.ElementChangedListener.e lementChanged
m) AbstractSourceElementParser.parseSourceModule is also called by:i)
SourceIndexer.indexDocument
n/p) SourceParserUtil.getModuleDeclaration (different from the
getModuleDeclaration above):
public static ModuleDeclaration getModuleDeclaration(ISourceModule module,
IProblemReporter reporter) {
is called from these places:
i) VScriptCompletionEngine.locateVScriptContext(SourceModule, int)
ii) VScriptSelectionEngine.select(ISourceModule, int, int)
iii) MatchLocatorParser.parse(PossibleMatch)
iv) SemanticPositionUpdater.reconcile(ISourceModule sourceModule,
SemanticHighlightingPresenter presenter, Highlighting[] highlightings, List
currentPositions)
"Andrei Sobolev" <haiodo@xored.com> wrote in message
news:g09mhq$9i2$1@build.eclipse.org...
> Hi Chuck,
>
> Nice to see, what DLTK help you.
>
> We don't have something special in area of semantic analysis, etc.
>
> Where is some ways to implement such features:
> 1) Using of ScriptBuilder.
> In DLTK we have feature named ScriptBuilder. This is incremental resource
> builder, used for building Mixin models, etc.
> It will give you list of changed resources, and you cold check all this
> files and set appropriate markers.
> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will
> contain information about builded resources,
> and will be executed on each resource operations, like "resource added",
> "resource content changed".
>
> 2) You could use Validator, Validators framework uses Script Builder for
> incremental execution of validators. One more
> additional point, validators framework manages validator instances and
> allow unified way to sore configurations, etc.
> Also user could turn on/off and execute validators from DLTK UI.
>
> 3) If you also plan to make semantic checks for file edited, you need to
> extend Reconciler from your
> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and
> do checks, after user will make some changes
> in code. By default it is 500 milliseconds delay, so reconciler will be
> executed only if user not type code 500
> milliseconds.
>
>
> References:
> 1) Extension point to implement: org.eclipse.dltk.core.builder
> 2) We have "package require" checker implemented for Tcl. (TclCheckBuilder
> class).
> 3) ScriptSourceViewerConfiguration.getReconciler()
>
> Best regards,
> Andrei Sobolev.
>
>
>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>> immediately flag syntactic problems in our language source files.
>>
>> My question is if/when/how should I find/flag semantic errors in the
>> source
>> file?
>>
>> 1. Is there any part of DLTK framework which could/should be used to
>> invoke/perform semantic checks? I assume the answer to this is no. I
>> know
>> you support external validators - but we'd like the semantic checking to
>> be
>> performed implicitly.
>>
>> 2. When should semantic checks be performed - whenever the parser is
>> invoked?
>>
>> 3. How should errors be reported - using the parser's error reporter?
>> Also, how can we determine the line number for populating
>> IProblem
>> in the file if that information is:
>> a) not available in DLTKToken
>> b) not kept in ASTNode
>>
>> Thanks,
>> Chuck
>>
>>
|
|
|
| Re: Semantic checking - via validators? [message #26601 is a reply to message #26543] |
Tue, 24 June 2008 02:44 |
Charles Doucette
Messages: 125
Registered: July 2009 |
Senior Member |
|
|
Based on what I found below, here are my conclusions:
We should find the smallest/earliest place that we can change that is called
from the editor reconciliation and ideally no place else.
I see two reasonable solutions:
1) Overriding m), i.e. AbstractSourceElementParser.parseSourceModule, since
we already have a class which extends this. It would do the following:
public void parseSourceModule(char[] contents, ISourceModuleInfo astCache,
char[] filename) {
super.parseSourceModule(contents, astCache, filename);
// do semantic checking
}
The only problem with this approach is that it is also called by
SourceIndexer.index.
2) Override VScriptSourceViewerConfiguration.getReconciler, and specifically
to add child class of ScriptReconcilingStrategy which implements reconcile.
It would consist of the following:
protected void reconcile(final boolean initialReconcile) {
super.reconcile(initialReconcile);
// do semantic checking
}
This would not affect SourceIndexer.index, and it is close to the
recommended strategy of implementing my own recognizer, but it builds off of
the existing recognizer instead but is *very* dependent on the current
implementation so as to change/implement as few lines as possible.
Chuck
"Chuck Doucette" <cdoucette@vaultus.com> wrote in message
news:g3plql$hkg$1@build.eclipse.org...
> As I said in another reply, I have created a semantic checker, and I am
> trying to determine where/how to invoke it.
>
> In addition to invoking it in a builder, we also wanted to invoke it
> dynamically in the editor, so syntax and semantic errors show up
> immediately (just like in JDT). Thus, I believe that we should try to
> follow your advice to invoke it via the reconciler.
>
> In the process of trying to determine exactly where/how to do that, I
> found out many things that I want to share with other users who may be
> trying to do the same thing.
>
> Here's what I have found:
>
> a) My (concrete) class VScriptSourceViewerConfiguration extends the DLTK
> abstract class ScriptSourceViewerConfiguration which defines this method:
>
> /**
> * Returns the reconciler ready to be used with the given source viewer.
> * This implementation always returns <code>null</code>.
> *
> * @param sourceViewer the source viewer to be configured by this
> configuration
> * @return a reconciler or <code>null</code> if reconciling should not be
> supported
> */
> public IReconciler getReconciler(ISourceViewer sourceViewer) {
>
> b) The DLTK class ScriptReconciler implements IReconciler. Its super
> class, MonoReconciler depends on an instance of IReconcilingStrategy to do
> the reconciliation.
>
> c) The method getReconciler above creates an instance of the DLTK class
> ScriptCompositeReconcilingStrategy whose superclass,
> CompositeReconcilingStrategy, implements IReconcilingStrategy. It also
> depends on an instance of the more straight forward class
> ScriptReconcilingStrategy.
>
> d) ScriptReconcilingStrategy depends on ISourceModule.reconcile
>
> e) SourceModule extends AbstractSourceModule (both of which implements
> ISourceModule)
> The reconcile method calls ReconcileWorkingCopyOperation.runOperation,
> which does this:
>
> if (isReadOnly()) {
> run(monitor);
> } else {
> // Use IWorkspace.run(...) to ensure that a build will be done
> // in autobuild mode.
> // Note that if the tree is locked, this will throw a
> // CoreException, but this is ok
> // as this operation is modifying the tree (not read-only) and a
> // CoreException will be thrown anyway.
> IWorkspace wc = ResourcesPlugin.getWorkspace();
> wc.run(this, getSchedulingRule(), IWorkspace.AVOID_UPDATE, monitor);
>
> f) IWorkspace.run calls IWorkspaceRunnable.run, and that is defined by
> ModelOperation.run, which calls executeOperation.
>
> g) ReconcileWorkingCopyOperation.executeOperation calls makeConsistent,
> which calls SourceModule.makeConsistent:
>
> h) SourceModule.makeConsistent does this:
>
> // Remove AST Cache element
> ISourceModuleInfoCache sourceModuleInfoCache = ModelManager
> .getModelManager().getSourceModuleInfoCache();
> sourceModuleInfoCache.remove(this);
> openWhenClosed(createElementInfo(), monitor);
>
> i) ModelElement.openWhenClosed calls Openable.generateInfos:
>
> /**
> * Generates the element infos for this element, its ancestors (if they are
> * not opened) and its children (if it is an Openable). Puts the newly
> * created element info in the given map.
> */
> protected abstract void generateInfos(Object info, HashMap newElements,
> IProgressMonitor pm) throws ModelException;
>
> j) Openable.generateInfos calls AbstractSourceModule.buildStructure:
>
> /**
> * Builds this element's structure and properties in the given info object,
> * based on this element's current contents (reuse buffer contents if this
> * element has an open buffer, or resource contents if this element does
> not
> * have an open buffer). Children are placed in the given newElements table
> * (note, this element has already been placed in the newElements table).
> * Returns true if successful, or false if an error is encountered while
> * determining the structure of this element.
> */
> protected abstract boolean buildStructure(OpenableElementInfo info,
> IProgressMonitor pm, Map newElements, IResource underlyingResource)
> throws ModelException;
>
> k) AbstractSourceModule.buildStructure calls
> SourceParserUtil.parseSourceModule
>
> l) SourceParserUtil.parseSourceModule calls
> AbstractSourceElementParser.parseSourceModule
>
> m) AbstractSourceElementParser.parseSourceModule calls
> SourceParserUtil.getModuleDeclaration:
>
> n) public static ModuleDeclaration getModuleDeclaration(char[] filename,
> char[] content, String nature, IProblemReporter reporter,
> ISourceModuleInfo mifo, int flags) {
> ISourceParser sourceParser;// = new SourceParser(this.fReporter);
> sourceParser = DLTKLanguageManager.getSourceParser(nature);
> if (sourceParser instanceof ISourceParserExtension) {
> ((ISourceParserExtension) sourceParser).setFlags(flags);
> }
> ModuleDeclaration moduleDeclaration = SourceParserUtil
> .getModuleFromCache(mifo, flags);
> if (moduleDeclaration == null) {
> if (reporter != null) {
> reporter.clearMarkers();
> }
> moduleDeclaration = sourceParser.parse(filename, content, reporter);
> SourceParserUtil.putModuleToCache(mifo, moduleDeclaration, flags);
> }
> return moduleDeclaration;
> }
>
> o) My class VScriptSourceParser extends AbstractSourceParser which
> implements ISourceParser.
> The parse method instantiates a vscriptParser and uses it.
>
> NOTES:
>
> o) ISourceParser.parse is also called by:
> i) AbstractASTFoldingStructureProvider.getCodeBlocks, which is
> ultimately called by:AbstractASTFoldingStructureProvider.initialized
> and
> AbstractASTFoldingStructureProvider.ElementChangedListener.e lementChanged
>
> m) AbstractSourceElementParser.parseSourceModule is also called by:i)
> SourceIndexer.indexDocument
>
> n/p) SourceParserUtil.getModuleDeclaration (different from the
> getModuleDeclaration above):
> public static ModuleDeclaration getModuleDeclaration(ISourceModule module,
> IProblemReporter reporter) {
> is called from these places:
>
> i) VScriptCompletionEngine.locateVScriptContext(SourceModule, int)
> ii) VScriptSelectionEngine.select(ISourceModule, int, int)
> iii) MatchLocatorParser.parse(PossibleMatch)
> iv) SemanticPositionUpdater.reconcile(ISourceModule sourceModule,
> SemanticHighlightingPresenter presenter, Highlighting[] highlightings,
> List currentPositions)
>
> "Andrei Sobolev" <haiodo@xored.com> wrote in message
> news:g09mhq$9i2$1@build.eclipse.org...
>> Hi Chuck,
>>
>> Nice to see, what DLTK help you.
>>
>> We don't have something special in area of semantic analysis, etc.
>>
>> Where is some ways to implement such features:
>> 1) Using of ScriptBuilder.
>> In DLTK we have feature named ScriptBuilder. This is incremental resource
>> builder, used for building Mixin models, etc.
>> It will give you list of changed resources, and you cold check all this
>> files and set appropriate markers.
>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will
>> contain information about builded resources,
>> and will be executed on each resource operations, like "resource added",
>> "resource content changed".
>>
>> 2) You could use Validator, Validators framework uses Script Builder for
>> incremental execution of validators. One more
>> additional point, validators framework manages validator instances and
>> allow unified way to sore configurations, etc.
>> Also user could turn on/off and execute validators from DLTK UI.
>>
>> 3) If you also plan to make semantic checks for file edited, you need to
>> extend Reconciler from your
>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and
>> do checks, after user will make some changes
>> in code. By default it is 500 milliseconds delay, so reconciler will be
>> executed only if user not type code 500
>> milliseconds.
>>
>>
>> References:
>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>> 2) We have "package require" checker implemented for Tcl.
>> (TclCheckBuilder class).
>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>
>> Best regards,
>> Andrei Sobolev.
>>
>>
>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>> immediately flag syntactic problems in our language source files.
>>>
>>> My question is if/when/how should I find/flag semantic errors in the
>>> source
>>> file?
>>>
>>> 1. Is there any part of DLTK framework which could/should be used to
>>> invoke/perform semantic checks? I assume the answer to this is no. I
>>> know
>>> you support external validators - but we'd like the semantic checking to
>>> be
>>> performed implicitly.
>>>
>>> 2. When should semantic checks be performed - whenever the parser is
>>> invoked?
>>>
>>> 3. How should errors be reported - using the parser's error reporter?
>>> Also, how can we determine the line number for populating
>>> IProblem
>>> in the file if that information is:
>>> a) not available in DLTKToken
>>> b) not kept in ASTNode
>>>
>>> Thanks,
>>> Chuck
>>>
>>>
>
>
|
|
|
| Re: Semantic checking - via validators? [message #26640 is a reply to message #26601] |
Tue, 24 June 2008 05:41 |
Alex Panchenko
Messages: 342
Registered: July 2009 |
Senior Member |
|
|
Hi Chuck,
At the moment the the best way is to override parseSourceModule().
When it is called from Indexer - the problemReporter is null, so you can
skip additional code.
For examples please look at the RubySourceElementParser and
TclSourceElementParser - after standard processing they call TODO task
parser, and that is similar to what you need.
Regards,
Alex
Chuck Doucette wrote:
> Based on what I found below, here are my conclusions:
> We should find the smallest/earliest place that we can change that is called
> from the editor reconciliation and ideally no place else.
> I see two reasonable solutions:
>
> 1) Overriding m), i.e. AbstractSourceElementParser.parseSourceModule, since
> we already have a class which extends this. It would do the following:
>
> public void parseSourceModule(char[] contents, ISourceModuleInfo astCache,
> char[] filename) {
> super.parseSourceModule(contents, astCache, filename);
> // do semantic checking
> }
>
> The only problem with this approach is that it is also called by
> SourceIndexer.index.
>
> 2) Override VScriptSourceViewerConfiguration.getReconciler, and specifically
> to add child class of ScriptReconcilingStrategy which implements reconcile.
> It would consist of the following:
>
> protected void reconcile(final boolean initialReconcile) {
> super.reconcile(initialReconcile);
> // do semantic checking
> }
>
> This would not affect SourceIndexer.index, and it is close to the
> recommended strategy of implementing my own recognizer, but it builds off of
> the existing recognizer instead but is *very* dependent on the current
> implementation so as to change/implement as few lines as possible.
>
> Chuck
>
> "Chuck Doucette" <cdoucette@vaultus.com> wrote in message
> news:g3plql$hkg$1@build.eclipse.org...
>> As I said in another reply, I have created a semantic checker, and I am
>> trying to determine where/how to invoke it.
>>
>> In addition to invoking it in a builder, we also wanted to invoke it
>> dynamically in the editor, so syntax and semantic errors show up
>> immediately (just like in JDT). Thus, I believe that we should try to
>> follow your advice to invoke it via the reconciler.
>>
>> In the process of trying to determine exactly where/how to do that, I
>> found out many things that I want to share with other users who may be
>> trying to do the same thing.
>>
>> Here's what I have found:
>>
>> a) My (concrete) class VScriptSourceViewerConfiguration extends the DLTK
>> abstract class ScriptSourceViewerConfiguration which defines this method:
>>
>> /**
>> * Returns the reconciler ready to be used with the given source viewer.
>> * This implementation always returns <code>null</code>.
>> *
>> * @param sourceViewer the source viewer to be configured by this
>> configuration
>> * @return a reconciler or <code>null</code> if reconciling should not be
>> supported
>> */
>> public IReconciler getReconciler(ISourceViewer sourceViewer) {
>>
>> b) The DLTK class ScriptReconciler implements IReconciler. Its super
>> class, MonoReconciler depends on an instance of IReconcilingStrategy to do
>> the reconciliation.
>>
>> c) The method getReconciler above creates an instance of the DLTK class
>> ScriptCompositeReconcilingStrategy whose superclass,
>> CompositeReconcilingStrategy, implements IReconcilingStrategy. It also
>> depends on an instance of the more straight forward class
>> ScriptReconcilingStrategy.
>>
>> d) ScriptReconcilingStrategy depends on ISourceModule.reconcile
>>
>> e) SourceModule extends AbstractSourceModule (both of which implements
>> ISourceModule)
>> The reconcile method calls ReconcileWorkingCopyOperation.runOperation,
>> which does this:
>>
>> if (isReadOnly()) {
>> run(monitor);
>> } else {
>> // Use IWorkspace.run(...) to ensure that a build will be done
>> // in autobuild mode.
>> // Note that if the tree is locked, this will throw a
>> // CoreException, but this is ok
>> // as this operation is modifying the tree (not read-only) and a
>> // CoreException will be thrown anyway.
>> IWorkspace wc = ResourcesPlugin.getWorkspace();
>> wc.run(this, getSchedulingRule(), IWorkspace.AVOID_UPDATE, monitor);
>>
>> f) IWorkspace.run calls IWorkspaceRunnable.run, and that is defined by
>> ModelOperation.run, which calls executeOperation.
>>
>> g) ReconcileWorkingCopyOperation.executeOperation calls makeConsistent,
>> which calls SourceModule.makeConsistent:
>>
>> h) SourceModule.makeConsistent does this:
>>
>> // Remove AST Cache element
>> ISourceModuleInfoCache sourceModuleInfoCache = ModelManager
>> .getModelManager().getSourceModuleInfoCache();
>> sourceModuleInfoCache.remove(this);
>> openWhenClosed(createElementInfo(), monitor);
>>
>> i) ModelElement.openWhenClosed calls Openable.generateInfos:
>>
>> /**
>> * Generates the element infos for this element, its ancestors (if they are
>> * not opened) and its children (if it is an Openable). Puts the newly
>> * created element info in the given map.
>> */
>> protected abstract void generateInfos(Object info, HashMap newElements,
>> IProgressMonitor pm) throws ModelException;
>>
>> j) Openable.generateInfos calls AbstractSourceModule.buildStructure:
>>
>> /**
>> * Builds this element's structure and properties in the given info object,
>> * based on this element's current contents (reuse buffer contents if this
>> * element has an open buffer, or resource contents if this element does
>> not
>> * have an open buffer). Children are placed in the given newElements table
>> * (note, this element has already been placed in the newElements table).
>> * Returns true if successful, or false if an error is encountered while
>> * determining the structure of this element.
>> */
>> protected abstract boolean buildStructure(OpenableElementInfo info,
>> IProgressMonitor pm, Map newElements, IResource underlyingResource)
>> throws ModelException;
>>
>> k) AbstractSourceModule.buildStructure calls
>> SourceParserUtil.parseSourceModule
>>
>> l) SourceParserUtil.parseSourceModule calls
>> AbstractSourceElementParser.parseSourceModule
>>
>> m) AbstractSourceElementParser.parseSourceModule calls
>> SourceParserUtil.getModuleDeclaration:
>>
>> n) public static ModuleDeclaration getModuleDeclaration(char[] filename,
>> char[] content, String nature, IProblemReporter reporter,
>> ISourceModuleInfo mifo, int flags) {
>> ISourceParser sourceParser;// = new SourceParser(this.fReporter);
>> sourceParser = DLTKLanguageManager.getSourceParser(nature);
>> if (sourceParser instanceof ISourceParserExtension) {
>> ((ISourceParserExtension) sourceParser).setFlags(flags);
>> }
>> ModuleDeclaration moduleDeclaration = SourceParserUtil
>> .getModuleFromCache(mifo, flags);
>> if (moduleDeclaration == null) {
>> if (reporter != null) {
>> reporter.clearMarkers();
>> }
>> moduleDeclaration = sourceParser.parse(filename, content, reporter);
>> SourceParserUtil.putModuleToCache(mifo, moduleDeclaration, flags);
>> }
>> return moduleDeclaration;
>> }
>>
>> o) My class VScriptSourceParser extends AbstractSourceParser which
>> implements ISourceParser.
>> The parse method instantiates a vscriptParser and uses it.
>>
>> NOTES:
>>
>> o) ISourceParser.parse is also called by:
>> i) AbstractASTFoldingStructureProvider.getCodeBlocks, which is
>> ultimately called by:AbstractASTFoldingStructureProvider.initialized
>> and
>> AbstractASTFoldingStructureProvider.ElementChangedListener.e lementChanged
>>
>> m) AbstractSourceElementParser.parseSourceModule is also called by:i)
>> SourceIndexer.indexDocument
>>
>> n/p) SourceParserUtil.getModuleDeclaration (different from the
>> getModuleDeclaration above):
>> public static ModuleDeclaration getModuleDeclaration(ISourceModule module,
>> IProblemReporter reporter) {
>> is called from these places:
>>
>> i) VScriptCompletionEngine.locateVScriptContext(SourceModule, int)
>> ii) VScriptSelectionEngine.select(ISourceModule, int, int)
>> iii) MatchLocatorParser.parse(PossibleMatch)
>> iv) SemanticPositionUpdater.reconcile(ISourceModule sourceModule,
>> SemanticHighlightingPresenter presenter, Highlighting[] highlightings,
>> List currentPositions)
>>
>> "Andrei Sobolev" <haiodo@xored.com> wrote in message
>> news:g09mhq$9i2$1@build.eclipse.org...
>>> Hi Chuck,
>>>
>>> Nice to see, what DLTK help you.
>>>
>>> We don't have something special in area of semantic analysis, etc.
>>>
>>> Where is some ways to implement such features:
>>> 1) Using of ScriptBuilder.
>>> In DLTK we have feature named ScriptBuilder. This is incremental resource
>>> builder, used for building Mixin models, etc.
>>> It will give you list of changed resources, and you cold check all this
>>> files and set appropriate markers.
>>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will
>>> contain information about builded resources,
>>> and will be executed on each resource operations, like "resource added",
>>> "resource content changed".
>>>
>>> 2) You could use Validator, Validators framework uses Script Builder for
>>> incremental execution of validators. One more
>>> additional point, validators framework manages validator instances and
>>> allow unified way to sore configurations, etc.
>>> Also user could turn on/off and execute validators from DLTK UI.
>>>
>>> 3) If you also plan to make semantic checks for file edited, you need to
>>> extend Reconciler from your
>>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and
>>> do checks, after user will make some changes
>>> in code. By default it is 500 milliseconds delay, so reconciler will be
>>> executed only if user not type code 500
>>> milliseconds.
>>>
>>>
>>> References:
>>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>>> 2) We have "package require" checker implemented for Tcl.
>>> (TclCheckBuilder class).
>>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>>
>>> Best regards,
>>> Andrei Sobolev.
>>>
>>>
>>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>>> immediately flag syntactic problems in our language source files.
>>>>
>>>> My question is if/when/how should I find/flag semantic errors in the
>>>> source
>>>> file?
>>>>
>>>> 1. Is there any part of DLTK framework which could/should be used to
>>>> invoke/perform semantic checks? I assume the answer to this is no. I
>>>> know
>>>> you support external validators - but we'd like the semantic checking to
>>>> be
>>>> performed implicitly.
>>>>
>>>> 2. When should semantic checks be performed - whenever the parser is
>>>> invoked?
>>>>
>>>> 3. How should errors be reported - using the parser's error reporter?
>>>> Also, how can we determine the line number for populating
>>>> IProblem
>>>> in the file if that information is:
>>>> a) not available in DLTKToken
>>>> b) not kept in ASTNode
>>>>
>>>> Thanks,
>>>> Chuck
>>>>
>>>>
>>
>
>
|
|
|
Goto Forum:
Current Time: Sat Nov 09 01:40:48 GMT 2024
Powered by FUDForum. Page generated in 0.03800 seconds |
] 
Search
Help
Register
Login
Home
