Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » junit test formatting removes hidden tokens (i.e. comments)
junit test formatting removes hidden tokens (i.e. comments) [message #996405] Wed, 02 January 2013 01:46 Go to next message
Barrie Treloar is currently offline Barrie TreloarFriend
Messages: 52
Registered: July 2009
Member
How to test a formatting with xtend? says

Christian Dietrich wrote on Thu, 06 September 2012 11:47

simply injecting the serializer and call it doesnt work?


So I've managed to get my formatting tests up and running, but I noticed that my hidden tokens disappear. This isn't the same behaviour as Eclipse XText editor.

Is there someway to not discard the hidden tokens?
I suspect that this needs to be in the parser somewhere.

(Mytest.xtend)

    @Inject extension ParserHelper2<Usrprops>
    @Inject extension ISerializer

    @Test
    def void myFormatTest() {
        val input = '''...'''.toString
        val expectedFormat = '''...'''.toString
        assertEquals(expectedFormat, input.parse.serialize(SaveOptions::newBuilder.format().getOptions()))
   }


(ParserHelper2)
    public T parse(InputStream in, URI uriToUse, Map<?,?> options, ResourceSet resourceSet) {
        Resource resource = resourceFactory.createResource(uriToUse);
        resourceSet.getResources().add(resource);
        try {
            resource.load(in, options);
        } catch (IOException e) {
            throw new WrappedException(e);
        }

        validate(resource);

        @SuppressWarnings("unchecked")
        final T root = (T) (resource.getContents().isEmpty()? null : resource.getContents().get(0));

        return root;
    }



Re: junit test formatting removes hidden tokens (i.e. comments) [message #996907 is a reply to message #996405] Thu, 03 January 2013 06:52 Go to previous messageGo to next message
Barrie Treloar is currently offline Barrie TreloarFriend
Messages: 52
Registered: July 2009
Member
From what I can tell its all in the done in the XtextTokenStream where all the initial hidden tokens are put on the hidden channel.
So its way too late in the formatter to be trying to do anything with the hidden tokens since they have already been stripped.

But what I can't see is how the UI does things differently so that it can format comments...
There is another definition of the ANTLR stuff and Lexer/Parser inside the UI plugin but I can't see anything significantly different yet.

Re: junit test formatting removes hidden tokens (i.e. comments) [message #996914 is a reply to message #996907] Thu, 03 January 2013 07:20 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian ZarnekowFriend
Messages: 2936
Registered: July 2009
Senior Member
Am 03.01.13 07:52, schrieb Barrie Treloar:
> From what I can tell its all in the done in the XtextTokenStream where
> all the initial hidden tokens are put on the hidden channel.
> So its way too late in the formatter to be trying to do anything with
> the hidden tokens since they have already been stripped.
>
> But what I can't see is how the UI does things differently so that it
> can format comments...
> There is another definition of the ANTLR stuff and Lexer/Parser inside
> the UI plugin but I can't see anything significantly different yet.
>
>

The hidden tokens are part of the node model thus they do not fully
disappear. Do you use custom terminal rules to support whitespace and
comments? Do you use hidden tokens per parser rule?

Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com
Re: junit test formatting removes hidden tokens (i.e. comments) [message #997145 is a reply to message #996907] Thu, 03 January 2013 23:46 Go to previous messageGo to next message
Barrie Treloar is currently offline Barrie TreloarFriend
Messages: 52
Registered: July 2009
Member
Well, dont you love debuggers!
A fresh new day gives you lots of new perspective.

Break point inside MyDslFormatterconfigureFormatting(FormattingConfig c) and then running the DSL Eclipse App inside the debugger shows
ContentFormatterFactory$ContentFormatter.format(IDocument document, IRegion region) being invoked.
This is injected with a INodeModelFormatter, by default a DefaultNodeModelFormatter.
DefaultNodeModelFormatter.format(ICompositeNode root, int offset, int length) does the work with a INodeModelStreamer (default NodeModelStreamer)
ITextRegion range = nodeModelStreamer.feedTokenStream(fmt, root, offset, length);

NodeModelStreamer makes sure handle the hidden tokens.

So my question is,
Can I hook into this easily in my unit tests somehow?

Re: junit test formatting removes hidden tokens (i.e. comments) [message #997153 is a reply to message #997145] Fri, 04 January 2013 02:58 Go to previous message
Barrie Treloar is currently offline Barrie TreloarFriend
Messages: 52
Registered: July 2009
Member
Well, looks like I managed to get it working.
For those following the google trail, this is what I am doing.
Note: I've pulled out everything from ParserHelper into a new class ParserHelper2 and added the formatting into this class, since it does similar stuff and needs to parse anyway.

package test;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Map;

import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.common.util.WrappedException;
import org.eclipse.emf.ecore.EObject;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.xtext.Constants;
import org.eclipse.xtext.formatting.INodeModelFormatter;
import org.eclipse.xtext.formatting.INodeModelFormatter.IFormattedRegion;
import org.eclipse.xtext.parser.IParseResult;
import org.eclipse.xtext.resource.IResourceFactory;
import org.eclipse.xtext.resource.XtextResource;
import org.eclipse.xtext.resource.XtextResourceSet;
import org.eclipse.xtext.util.StringInputStream;
import org.eclipse.xtext.validation.CheckMode;
import org.eclipse.xtext.validation.IResourceValidator;
import org.eclipse.xtext.validation.Issue;

import com.google.inject.Inject;
import com.google.inject.Provider;
import com.google.inject.name.Named;

public class ParserHelper2<T extends EObject> {

    @Inject
    protected Provider<XtextResourceSet> resourceSetProvider;

    @Inject
    public LexerAndParserTest lexerTester;

    @Inject
    protected IResourceFactory resourceFactory;

    @Inject
    protected IResourceValidator validator;

    @Inject
    protected INodeModelFormatter formatter;

    @Inject
    @Named(Constants.FILE_EXTENSIONS)
    public String fileExtension;

    public String format(CharSequence text) throws Exception {
        return format(text, resourceSetProvider.get());
    }

    public String format(CharSequence text, ResourceSet resourceSetToUse) throws Exception {
        return format(getAsStream(text), computeUnusedUri(resourceSetToUse), null, resourceSetToUse);
    }

    public String format(CharSequence text, URI uriToUse, ResourceSet resourceSetToUse) throws Exception {
        return format(getAsStream(text), uriToUse, null, resourceSetToUse);
    }

    public String format(InputStream in, URI uriToUse, Map<?, ?> options, ResourceSet resourceSet) {
        XtextResource resource = getResource(in, uriToUse, options, resourceSet);
        IParseResult parseResult = resource.getParseResult();
        if (parseResult == null)
            return null;
        IFormattedRegion r = formatter.format(parseResult.getRootNode(), 0, parseResult.getRootNode().getText().length());
        return r.getFormattedText();
    }

    public T parse(InputStream in, URI uriToUse, Map<?, ?> options, ResourceSet resourceSet) {
        Resource resource = getResource(in, uriToUse, options, resourceSet);

        validate(resource);

        @SuppressWarnings("unchecked")
        final T root = (T) (resource.getContents().isEmpty() ? null : resource.getContents().get(0));

        return root;
    }

    public T parse(CharSequence text) throws Exception {
        return parse(text, resourceSetProvider.get());
    }

    public T parse(CharSequence text, ResourceSet resourceSetToUse) throws Exception {
        return parse(getAsStream(text), computeUnusedUri(resourceSetToUse), null, resourceSetToUse);
    }

    public T parse(CharSequence text, URI uriToUse, ResourceSet resourceSetToUse) throws Exception {
        return parse(getAsStream(text), uriToUse, null, resourceSetToUse);
    }

    protected URI computeUnusedUri(ResourceSet resourceSet) {
        String name = "__synthetic";
        for (int i = 0; i < Integer.MAX_VALUE; i++) {
            URI syntheticUri = URI.createURI(name + i + "." + fileExtension);
            if (resourceSet.getResource(syntheticUri, false) == null)
                return syntheticUri;
        }
        throw new IllegalStateException();
    }

    protected InputStream getAsStream(CharSequence text) {
        return new StringInputStream(text == null ? "" : text.toString());
    }

    public XtextResource getResource(InputStream in, URI uriToUse, Map<?, ?> options, ResourceSet resourceSet) {
        Resource resource = resourceFactory.createResource(uriToUse);

        if (!(resource instanceof XtextResource)) {
            throw new RuntimeException("Can only handle XtextResources and not " + resource.getClass().getCanonicalName());
        }

        resourceSet.getResources().add(resource);
        try {
            resource.load(in, options);
        } catch (IOException e) {
            throw new WrappedException(e);
        }
        return (XtextResource) resource;
    }

    public void validate(T model) {
        validate(model.eResource());
    }

    public void validate(Resource resource) {
        List<Issue> issues = validator.validate(resource, CheckMode.ALL, null);
        if (!issues.isEmpty()) {
            throw new XtextParseError(issues);
        }
    }

}


To use the class in your xtend junit test:
(Mytest.xtend)

    @Inject extension ParserHelper2<Usrprops>

    @Test
    def void myFormatTest() {
        val input = '''...'''.toString
        val expectedFormat = '''...'''.toString
        assertEquals(expectedFormat, input.format())
    }
Previous Topic:Formatter problem
Next Topic:Fix Eclipse XText Site (and include source links)
Goto Forum:
  


Current Time: Fri Nov 21 19:11:04 GMT 2014

Powered by FUDForum. Page generated in 0.01972 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software