Eclipse Community Forums: TMF (Xtext) » Xtext Serialization: how to benchmark?

Home » Modeling » TMF (Xtext) » Xtext Serialization: how to benchmark?

Xtext Serialization: how to benchmark? [message #1790319]

Fri, 08 June 2018 06:59

Eclipse User

Hi all,

what is the best/fair way to benchmark serialization performance of Xtext for a given grammar. Since the results of the benchmark will influence the decision whether we will use Xtext I want to make sure it got a fair chance. Is org.eclipse.xtext.serializer.impl.Serializer#serialize() the right entry point for a JMH benchmark? Is it always doing validation during the serialization process or can that be omitted if the model is known to be valid?

Thanks,
Michael

Re: Xtext Serialization: how to benchmark? [message #1790321 is a reply to message #1790319]

Fri, 08 June 2018 07:16

Eclipse User

Hi, this question is impossible to answer without any context of in which szenarios you plan to serialize models.
so can you give more context
- standalone or eclipse
- with or without formatter
- single files or files referencing each other?
- write of new file or changes to existing ones?
-use of change serializer?

Re: Xtext Serialization: how to benchmark? [message #1790326 is a reply to message #1790321]

Fri, 08 June 2018 07:40

Eclipse User

I'm wondering why this would be such an important point for a decision for or against Xtext. And if you would benchmark on that level, against what would you benchmark other frameworks? Of course, you could use that entry point to benchmark something, but the performance is heavily influenced by the factors that Christian mentioned. I doubt that you would get useful results for your decision that way, but go ahead if you think you would.

In our experience completely different aspects influence the decision on Xtext. Performance is one of them, but usually parsing, validation etc. are more important than serializing. And even more important: Type of language you develop, integration with Eclipse/VSCode etc., code generation support, rich editor support, flexibility, robustness and a long live in past and future.

We would like to understand more your intention and use case to give the right ideas if Xtext could be the right choice for you.

Kind regards,
~Karsten

Re: Xtext Serialization: how to benchmark? [message #1790334 is a reply to message #1790321]

Fri, 08 June 2018 07:57

Eclipse User

Hi Christian,

I am not very familiar with Xtext but I'll try to be as specific as possible:

- standalone
- default (probably no formatter?)
- no references between files
- just one full parse/unparse cycle (no changes)
- default (probably no change serializer?)

Regards
Michael

Re: Xtext Serialization: how to benchmark? [message #1790336 is a reply to message #1790334]

Fri, 08 June 2018 08:00

Eclipse User

if serialization/unparse is no szenario why do you want to benchmark it?

Re: Xtext Serialization: how to benchmark? [message #1790337 is a reply to message #1790334]

Fri, 08 June 2018 08:13

Eclipse User

Hi Karsten,

> And if you would benchmark on that level, against what would you benchmark other frameworks?

Yes.

We don't make decisions only based on performance. That would be very unprofessional and short-sighted. But for the other features/aspects you mentioned we are able to find resources and documentation, e.g. LSP which seems very promising. Regarding performance I couldn't find much though. That's why I specifically asked for that.

Thanks for your help
Michael

Re: Xtext Serialization: how to benchmark? [message #1790340 is a reply to message #1790337]

Fri, 08 June 2018 08:20

Eclipse User

> if serialization/unparse is no szenario why do you want to benchmark it?

Hmm, I don't get that. Never said we don't serialize/unparse. We just do a 'full' serialization after parsing without overwriting an existing file.

Re: Xtext Serialization: how to benchmark? [message #1790341 is a reply to message #1790340]

Fri, 08 June 2018 08:21

Eclipse User

but it makes no sense to serialize a file with no changes.
it is a noop.

Re: Xtext Serialization: how to benchmark? [message #1790357 is a reply to message #1790341]

Fri, 08 June 2018 10:16

Eclipse User

Please keep in mind that we are talking about a benchmark ;) To measure the serialization performance, we need a model instance to serialize. Acquiring this model instance from a textual representation makes sense since we compare different modeling frameworks. I really don't see the problem.

Re: Xtext Serialization: how to benchmark? [message #1790359 is a reply to message #1790357]

Fri, 08 June 2018 10:33

Eclipse User

well you basically tell: year ill test cars soll ill have a look what my porsche boxter does offroad because i am testing cars.

the serialization performance becomes interesting with a full fletched formatter, with a model with many references (e.g. outside of a file) and with no exiting model

Re: Xtext Serialization: how to benchmark? [message #1790362 is a reply to message #1790359]

Fri, 08 June 2018 11:15

Eclipse User

Hi Christian,

the car analogy doesn't get us anywhere.

I already explained that our evaluation is not limited to this benchmark. But for some of our use cases this is relevant. For those performance critical use cases we won't have any references and no special formatter. Why should I care?

Getting back to my original question: is Serializer#serialize() the right entry point for a JMH benchmark?

As I said, I couldn't find much credible information related to performance. If there's documentation that I didn't find, let me know.

Regards,
Michael

Re: Xtext Serialization: how to benchmark? [message #1790363 is a reply to message #1790362]

Fri, 08 June 2018 11:17

Eclipse User

as i said it depends on which usecase you want to benchmark
if you just to test something with no real meaningfulness
calling the serializer this way is fine

[Updated on: Fri, 08 June 2018 11:59] by Moderator

Previous Topic:	Antlr4 to xtext
Next Topic:	The ?= operator

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 16:43:00 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter