[ease-dev] Autocompletion feature sum-up

[Martin Kloesch <martin.kloesch@xxxxxxxxx>]

Hi,

since the Google Summer of Code program ends next Friday I decided to freeze the features for auto-completion and use the rest of the time to fix last issues and clean up the code.
This mail should give you a quick overview on what the actual auto-completion features are and what might be some open topics.

General (why I believe my implementation is a good thing):

• EASE now uses a generic interface for completion providers:
• The class CompletionProviderDispatcher handles all registered providers and dispatches the code for completion calculation.
• The CompletionProviderDispatcher is used for both the script shell and the _javascript_ editor (other editors possible but not implemented).
• Different CompletionProviders can be registered using the completionProcessor extension point.
• CompletionProviders can be generic or language specific. Language specific providers must set the language in the extension.
• Each scripting language that wants to have auto-completion must implement the ICompletionAnalyzer interface. This interface is used to parse code to a more easily usable format. 
• New completion providers can be added very fast
• CompletionProviders must implement the ICompletionProvider interface and set up the completionProcessor extension.
• Everything else is handled by the CompletionProviderDispatcher
• Completion providers can be reused
• for different languages (e.g.: available modules completion provider)
• for both script shell and the editor
• Completion for new languages can be implemented fast
• Language must implement ICompletionAnalyzer interface and set extension (script shell only)
  
• Code can be used to detect type of object
• Necessary for completion calculation
• Helpful for other utility functionality (e.g. hover tool-tips)
  

To help everyone understand what is actually happening I will quickly try to outline the flow for completion calculation.

Flow ScriptShell:

• The ScriptShell instantiates the CompletionProviderDispatcher when it starts up (this includes keybindings, etc.)
• The ScriptShell sets the completion providers matching the currently selected scripting language.
• The ScriptShell sets the ICompletionAnalyzer for the currently selected engine.
• The ScriptShell uses the getProviders utility method to query for matching providers.
• The ScriptShell sets the script engine for all providers.
  
• The providers, engine and analyzer are reset when the script engine changes.
• Each line entered in the ScriptShell will be passed to all registered CompletionProviders.
• If a completion provider parses information this information must be stored for later use.
• Completion providers may choose to ignore all given code (e.g.: Keyword completion provider needs no information about code).
• Once the user presses the completion keybinding (default: CTRL+SPACE) the CompletionProviderDispatcher is started.
• The given line will be analyzed using the ICompletionAnalyzer interface.
• All unnecessary code is removed.
• The necessary code will be split into a call chain and a filter.
• A first basic ICompletionContext will be created.
• The created context will be refined using all registered completion providers.
• Context needs refinement if call chain is not empty.
• Only information about first item in call chain necessary (rest are simple class look-ups).
• Refined context also stores information about type of element in call chain (e.g. module method, local variable, ...).
• The refined context is used to calculate the actual proposals.
• All registered completion providers use refined context to calculate matches.
• Since script engine is available completion providers might choose to not calculate proposals (available anyways).
• The matches are calculated based on the filter and call-chain of the given context.
  
• The calculated proposals are transformed to the correct type.
• Since not all completion mechanisms need the same type of proposals a generic intermediate list is calculated.
• This list is then used to calculate the proposals with the corresponding message for the ScriptShell.
• The calculated proposals are displayed using Vidura's updated ContentProposalModifier.
  

The flow for the _javascript_ editor looks very similar:

• The _javascript_ completion provider is instantiated the first time the user presses the completion keybinding (default: CTRL+SPACE)
• Note that this completion provider is of the type IJavaCompletionProposalComputer so it can be used with the existing mechanisms
• Internally the provider creates a CompletionProviderDispatcher and sets the _javascript_CompletionAnalyzer as well as all matching EASE completion providers.
• The _javascript_ completion interface receives an IDocument that will be parsed.
• First the actual content of the document is read.
• Then the completion analyzer creates a string with the document's content as well as the included files.
• Instead of each include the content of the included file will be put into the contents string.
• This will be called until no more includes available.
• Circular imports are detected and ignored.
• The first time a file gets included it will be marked as loaded.
• If at a later point the same file is included again the include statement will be discarded.
• The created code will be given to all completion providers (see Script-Shell flow)
• The code will then be analyzed and the proposals calculated.
• Same flow as in Script-Shell.
  
• This time the other way around, no script engine available so some providers might not calculate proposals while others do now.
• The calculated proposals are again parsed to correct format for _javascript_ editor.
• The proposals are displayed using the _javascript_ editor's internal mechanism.
  

I implemented some ICompletionProviders to have the most important proposals available. The following list will give an overview on what providers are currently implemented:

• LoadedModuleCompletionProvider
• Detects when a module has been loaded.
• Offers information about methods and fields of said module.
• Used for both script shell and editor as well as for all languages.
  
• AvailableModuleCompletionProvider
• Holds list of all available modules (not just loaded)
• Otherwise same as LoadedModuleCompletionProvider.
• ScriptShellCompletionProvider
• Simple completion provider to query given script engine (if available)
• Can be used for local variables, etc.
• Used for all languages but only when script engine available -> ScriptShell only.
  
• SourceStackCompletionProvider
• Used for non-empty call chain.
• Simply follows call chain and calculates all matches based on class of last element and filter.
• Needs a refined context to work.
  
• Used for all languages and both script shell and editor.
  
• KeywordCompletionProvider
• Simple completion provider holding a list of keywords for selected language.
• Used for all languages (with different list of keywords) and both script shell and editor.
  
• CustomAnnotationCompletionProvider
• Used in script only.
• Offers custom annotation to declare type of object.
• Syntax: @type varName varClass
• Will be explained in detail once it is upstream
  

There are still some open points that do not have the highest priority and will (probably) not be implemented as part of the GSOC program:

• All code is single-threaded:
• Calculations might take too long and UI responsiveness could decrease.
• All registered completion providers called sequentially, would be faster if called in parallel.
• At the moment not a problem but might be once more providers are registered.
  
• No caching mechanism:
• Code might be unchanged but will still be parsed.
• Especially bad for editors because lots of code might need to be parsed.
• No mechanism to detect duplicates:
• The same suggestion might be proposed several times (e.g. from loaded module and from available module)
• No scalable recursion mechanism for e.g. included files:
• Each language can define the getIncludedCode method in the ICompletionAnalyzer method.
• Not possible to dynamically add completion providers that also add code during proposal calculation.

I know the e-mail is quite overwhelming but that's probably a good thing because you have lots of cool new features available in EASE ;-)

Best regards,
Martin