Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Xtext Serialization: how to benchmark?
Xtext Serialization: how to benchmark? [message #1790319] Fri, 08 June 2018 10:59 Go to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

Hi all,

what is the best/fair way to benchmark serialization performance of Xtext for a given grammar. Since the results of the benchmark will influence the decision whether we will use Xtext I want to make sure it got a fair chance. Is org.eclipse.xtext.serializer.impl.Serializer#serialize() the right entry point for a JMH benchmark? Is it always doing validation during the serialization process or can that be omitted if the model is known to be valid?

Thanks,
Michael


Re: Xtext Serialization: how to benchmark? [message #1790321 is a reply to message #1790319] Fri, 08 June 2018 11:16 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12229
Registered: July 2009
Senior Member
Hi, this question is impossible to answer without any context of in which szenarios you plan to serialize models.
so can you give more context
- standalone or eclipse
- with or without formatter
- single files or files referencing each other?
- write of new file or changes to existing ones?
-use of change serializer?


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Serialization: how to benchmark? [message #1790326 is a reply to message #1790321] Fri, 08 June 2018 11:40 Go to previous messageGo to next message
Karsten Thoms is currently offline Karsten ThomsFriend
Messages: 612
Registered: July 2009
Location: Dortmund, Germany
Senior Member

I'm wondering why this would be such an important point for a decision for or against Xtext. And if you would benchmark on that level, against what would you benchmark other frameworks? Of course, you could use that entry point to benchmark something, but the performance is heavily influenced by the factors that Christian mentioned. I doubt that you would get useful results for your decision that way, but go ahead if you think you would.

In our experience completely different aspects influence the decision on Xtext. Performance is one of them, but usually parsing, validation etc. are more important than serializing. And even more important: Type of language you develop, integration with Eclipse/VSCode etc., code generation support, rich editor support, flexibility, robustness and a long live in past and future.

We would like to understand more your intention and use case to give the right ideas if Xtext could be the right choice for you.

Kind regards,
~Karsten
Re: Xtext Serialization: how to benchmark? [message #1790334 is a reply to message #1790321] Fri, 08 June 2018 11:57 Go to previous messageGo to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

Hi Christian,

I am not very familiar with Xtext but I'll try to be as specific as possible:

- standalone
- default (probably no formatter?)
- no references between files
- just one full parse/unparse cycle (no changes)
- default (probably no change serializer?)

Regards
Michael



Re: Xtext Serialization: how to benchmark? [message #1790336 is a reply to message #1790334] Fri, 08 June 2018 12:00 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12229
Registered: July 2009
Senior Member
if serialization/unparse is no szenario why do you want to benchmark it?

Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Serialization: how to benchmark? [message #1790337 is a reply to message #1790334] Fri, 08 June 2018 12:13 Go to previous messageGo to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

Hi Karsten,

> And if you would benchmark on that level, against what would you benchmark other frameworks?

Yes.

We don't make decisions only based on performance. That would be very unprofessional and short-sighted. But for the other features/aspects you mentioned we are able to find resources and documentation, e.g. LSP which seems very promising. Regarding performance I couldn't find much though. That's why I specifically asked for that.

Thanks for your help
Michael


Re: Xtext Serialization: how to benchmark? [message #1790340 is a reply to message #1790337] Fri, 08 June 2018 12:20 Go to previous messageGo to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

> if serialization/unparse is no szenario why do you want to benchmark it?

Hmm, I don't get that. Never said we don't serialize/unparse. We just do a 'full' serialization after parsing without overwriting an existing file.


Re: Xtext Serialization: how to benchmark? [message #1790341 is a reply to message #1790340] Fri, 08 June 2018 12:21 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12229
Registered: July 2009
Senior Member
but it makes no sense to serialize a file with no changes.
it is a noop.


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Serialization: how to benchmark? [message #1790357 is a reply to message #1790341] Fri, 08 June 2018 14:16 Go to previous messageGo to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

Please keep in mind that we are talking about a benchmark ;) To measure the serialization performance, we need a model instance to serialize. Acquiring this model instance from a textual representation makes sense since we compare different modeling frameworks. I really don't see the problem.

Re: Xtext Serialization: how to benchmark? [message #1790359 is a reply to message #1790357] Fri, 08 June 2018 14:33 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12229
Registered: July 2009
Senior Member
well you basically tell: year ill test cars soll ill have a look what my porsche boxter does offroad because i am testing cars.

the serialization performance becomes interesting with a full fletched formatter, with a model with many references (e.g. outside of a file) and with no exiting model


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Serialization: how to benchmark? [message #1790362 is a reply to message #1790359] Fri, 08 June 2018 15:15 Go to previous messageGo to next message
Michael Hoffer is currently offline Michael HofferFriend
Messages: 9
Registered: August 2014
Junior Member

Hi Christian,

the car analogy doesn't get us anywhere.

I already explained that our evaluation is not limited to this benchmark. But for some of our use cases this is relevant. For those performance critical use cases we won't have any references and no special formatter. Why should I care?

Getting back to my original question: is Serializer#serialize() the right entry point for a JMH benchmark?

As I said, I couldn't find much credible information related to performance. If there's documentation that I didn't find, let me know.

Regards,
Michael


Re: Xtext Serialization: how to benchmark? [message #1790363 is a reply to message #1790362] Fri, 08 June 2018 15:17 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 12229
Registered: July 2009
Senior Member
as i said it depends on which usecase you want to benchmark
if you just to test something with no real meaningfulness
calling the serializer this way is fine


Need professional support for Xtext, Xpand, EMF?
Go to: https://xtext.itemis.com
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Fri, 08 June 2018 15:59]

Report message to a moderator

Previous Topic:Antlr4 to xtext
Next Topic:The ?= operator
Goto Forum:
  


Current Time: Thu Sep 20 00:05:25 GMT 2018

Powered by FUDForum. Page generated in 0.02408 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top