Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Handly » Thoughts on Handly from the MelnormeEclipse dev.
Thoughts on Handly from the MelnormeEclipse dev. [message #1725812] Mon, 07 March 2016 20:22 Go to next message
Bruno Medeiros is currently offline Bruno MedeirosFriend
Messages: 11
Registered: September 2014
Junior Member
Hi. I'm the main developer of MelnormeEclipse ( https://www.eclipse.org/forums/index.php/m/1725801/#msg_1725801 ), one of the few active projects aiming to tackle the Eclipse language IDE frameworks problem. (Coincidentally, I'm also the BrunoM quoted here http://www.cdtdoug.blogspot.co.uk/2013/08/my-favorite-thing-about-blogging-is.html )

Even though Handly and MelnormeEclipse have both been in development for well over a year, I've only found out about Handly today :( , and only by mere chance - someone mentioned it to me. I wish I had know about it sooner, there is a clear overlap about what our projects aim to do, so maybe we can join forces/cooperate, etc.. It could well be the case that Handly would be a useful library for MelnormeEclipse. And I see from your last blog post that your are eager for feedback from adopters.

The unfortunate bit about all this is that MelnormeEclipse already had some generic code written for it's equivalent of the source model (for the Project Explorer view, and for Outline support), although MelnormeEclipse support is nowhere as comprehensive as Handly's.

Before anything else, are you subscribed to https://dev.eclipse.org/mailman/listinfo/ide-dev ? I think this is the right place to discuss general topics about the issue of IDE development, including the Languages Tools Factorization problem. It would have been good if Handly had been announced there. :'( (out of curiosity, where was Handly announced?)

Another quick question: is Handly being used by JDT? (asking since I saw that you submitted a performance patch to JDT, based on Handly)

I have some questions and comments about Handly usage itself, but I'll leave that for the next post.
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1725827 is a reply to message #1725812] Mon, 07 March 2016 23:17 Go to previous messageGo to next message
Bruno Medeiros is currently offline Bruno MedeirosFriend
Messages: 11
Registered: September 2014
Junior Member
So, first of all let me say, it's really to nice to see a fresh attempt to refactor some of JDT-like functionality into a common library, even if just a subset of JDT. I agree that as things stand currently with JDT, best to start with something smaller, usable, and cleaner, than trying to re-write the whole JDT at once (like the DLTK approach... which quickly stagnated).

As for feedback on Handly itself. I looked into the design document (http://ftp.halifax.rwth-aachen.de/eclipse//handly/docs/handly-overview.pdf), the guide (https://github.com/pisv/gethandly/wiki), some blogs posts, etc.

The main thing I notice is that everywhere you start with the assumption that an IDE wants a JDT-like Java model for use, however, I question that such is actually the best approach. I know JDT is a very advanced IDE, certainly the most advanced Eclipse IDE, but that doesn't mean its design decisions are the best, especially older choices, taken at a time where the Eclipse Platform was vastly different from what it is today. (and indeed, some of these JDT design decisions are quite old, maybe even before there was an "Eclipse Platform")

Indeed, Melnorme took different design decisions with regards to its equivalent of the Java Model ( https://github.com/bruno-medeiros/MelnormeEclipse/blob/master/README-MelnormeEclipse.md#no-imodelelementijavaelement-model-hierarchy ). I think that the only significant similarity is that Melnorme's structure are also immutable, equals by value.

The first point we need to look at is, why is the hierarchy for the source-derived elements (ISourceConstruct) part of the hierarchy for project elements ( FooProject, FooSourceFolder, FooSourceFile, etc.) ? These sub-hierarchies are typically used in different ways, different contexts: ISourceConstruct's for outline for example (or code folding), and the project elements for other stuff. The only place, as far I know, they seem to connect to each is when you view ISourceConstructs under an ISourceFile in the Project Explorer tree (or in Search results). However such functionality can be trivially achieved by customizing the element provider of the underlying Tree Viewer. It doesn't justify having them both kinds of elements on the same hierarchy. In Melnorme they are separate: there is ISourceFileStructure with ISourceElement's, and then there is an INavigatorElement hierarchy for the Project Explorer elements.

With this split in place, other assumptions start to fall, namely the handle/body design. For example, in JDT if an ISourceConstruct is open, that means all the other ISourceConstructs from the same file are open/available as well. And nearly all the time you want to use an ISourceConstructs, you want it open, and you want all other elements of the same file open too (for the Outline for example). You rarely (if ever) want just a handle. As such, if ISourceConstructs is split from the other hierarchy, then it can be "open" all the time, it can contain all the model information for that source file. This means for example, methods like getChildren always return successfully, they can't throw a CoreException (another simplification that adds up in overall design).

If there is some functionality there requires a handle-only to an ISourceConstruct, it's easy to create a different class that provides that functionality (like a handle info object that is just a file path and a source range, for example).

Also, in my opinion the project model hierarchy doesn't need the handle/body design either: if it is decoupled from the ISourceConstruct elements, the size of the whole project model hierarchy is significantly smaller (one or two orders of magnitude smaller), and as such the whole thing can be kept in memory at once and recreated whole when underlying resource changes (since it no longer requires parsing source files). In other words, the project model is in the order of magnitude of (at most) the number of source files that exist in the workspace (but not on their size or contents). This should be insignificant to keep in memory.


The other thing that has me confused is, why do we need functionality to manage buffer/working-copy of the associated ISourceFile ? the #reconcile() method has been a source of confusion for me in JDT for 8+ years, -__-' (yes, that long). I mean, I understand what it does, but not why, why is it necessary. Same for the other related methods. I suspect it's unnecessarily complex API design.

In Melnorme, the ISourceFileStructure is an immutable data structure created from a source code snapshot (a String). The ISourceFileStructure does not care from where the source comes from, or whether its "reconciled" or not. Same for the Outline view that feeds on the ISourceFileStructure: it doesn't care about reconciliation or a working copy. All it cares is about an outline structure to display. These concerns are shifted to a source structure manager, that connects to IDocument's, listens to changes, creates a new ISourceFileStructure for each change, stores it in a map for each document/source that it is connected to. With this, the source code editor simply connects to the source structure manager, with its document, and listens to when a new ISourceFileStructure is created (and updates the outline view when such happens). I don't see the point of adding all that IBuffer functionality, when all it seems to be doing is replicating TextFileBufferManager functionality.

I think this is all for now, I hope this feedback is welcome. I look to forward to hearing your thoughts on this.

Also, what projects are currently using Handly? Is XText one of them?
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1725932 is a reply to message #1725812] Tue, 08 March 2016 14:51 Go to previous messageGo to next message
Vladimir Piskarev is currently offline Vladimir PiskarevFriend
Messages: 48
Registered: January 2014
Member
Hi Bruno,

And what a lucky chance that was! I'm really grateful to that "someone" who brought Handly to your attention. I have re-read your excellent comment in Doug's blog and the main points seem just as valid today as they were at the time of writing, and deeply resonate with the intention behind the Handly project. Needless to say, I'm very glad to meet you!

Thanks a lot for the great feedback and food for thought you have provided, including the suggestion to use the ide-dev list. I must admit that I didn't know about the list when the project was started, but even now, when I have been subscribed to it for some time, I was a bit hesitant about using it for introducing the project since some might consider such usage a plug. Your story gives me a good excuse for doing that, thinking of all the people who might be interested in such a project but still don't know that it even exists...

It is, indeed, unfortunate that Handly remains a best-kept secret in Eclipse despite my efforts to draw attention to the project. Besides being announced on the regular communication channels like the Eclipse's Project Proposals forum, it was presented at the first XtextCon, and I'm trying to blog about it more or less regularly, with my posts syndicated to Planet Eclipse. Recently, Wayne and I made an attempt to bring the project to the AC's attention in Eclipse. Apparently, all of it have not been successful enough.

Retrospectively, a large part of the failure is due to my initial inexperience in doing open-source in general and to being a newcomer at Eclipse in particular, so I have yet to learn how to do it right. Another part probably has to do with the project's nature itself. The current fashion is all about the Cloud and the IoT, and projects like Handly don't make a nice buzzword and remain relatively in the shadow. Perhaps NIH also has a part in that.

Of course, there are many other reasons to it, including technical ones: the current design makes it difficult to adopt Handly in projects that must maintain backward compatibility with their existing model API. It is possible with the model adaptation facility introduced in Handly 0.4, but in practice, it is complicated.

(So, no, JDT is not using Handly. If they were, there would be no reason for me to provide the performance patch as it would have been sufficient to fix it directly in Handly. The story behind the performance patch is that Handly had copied some of the JDT code initially and a bit later I discovered a scalability issue in Handly lurking in the copied code, so I had it fixed first in Handly and then contributed the fix back to JDT. The problem is that other projects like CDT that had probably copied the same code could not benefit from the fix.)

However, big changes are coming to Handly 0.5. Thanks to our current adopters (1C Enterprise Development Tools, Erlide, and Codasip Studio) who essentially gave me a green light for migration to Java 8 and for making a significant breaking change, Handly will hopefully become much less invasive with regard to the model API, so even well-established projects like JDT could use this library if they would so like. Please see this thread for more details:

http://dev.eclipse.org/mhonarc/lists/handly-dev/msg00160.html

Meanwhile, I'll take a deeper look at your project, it certainly looks interesting. I'm sorry for a delay in responding due to holidays in my country.

Thanks!
Vladimir
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1726070 is a reply to message #1725827] Wed, 09 March 2016 11:33 Go to previous messageGo to next message
Vladimir Piskarev is currently offline Vladimir PiskarevFriend
Messages: 48
Registered: January 2014
Member
Hi Bruno,

Here's some of my thoughts regarding the differences between the approaches taken by Melnorme and Handly. One caveat though: I haven't (yet) had enough time to immerse myself in the Melnorme's code base, so my comments are based chiefly on your previous feedback. Please regard them as preliminary and don't hesitate to correct me if you feel that I misunderstood something.

First of all, your project seems very innovative indeed (I would even dare say "unorthodox"). It's always good to try to push the boundaries. I applaud you for this effort.

Of course, every design is based on certain assumptions and has an imprint of its designer. To put things in perspective, I'm a big believer in uniform designs like Smalltalk and Beethoven's music. I think that what makes uniform designs great is that, although they can be quite hard to come up with and implement, they tend to be (all things being equal) more comprehensible to their consumers. This property is especially important for "kernels" such as object models and music motifs, because good kernels are made once and then used many, many times. A non-uniformity in the kernel thus multiplies greatly in upper layers.

I have yet to be convinced that there are better models for "program structure representation" than the JDT Java model (as a set of basic design patterns and principles rather than the concrete implementation) and the IDEA Psi-model. Both are great uniform models, although they make very different assumptions and trade-offs. A downside of the former is implementation complexity, a downside of the latter is scalability.

Scalability has been a very important requirement from our earliest adopter that initiated the project. We have to deal, for example, with source files containing up to 100K LOC. That's one of the reasons why we have chosen the JDT approach. Handly was designed to be uniform and scalable. Implementation complexity has never been a primary concern. Client code regularity made possible by the Handly uniform design has been of much more importance, as the amount of model's client code is vastly greater than the amount of model implementation code. This is witnessed by JDT, for example, where vast amount of code consists of programmatic manipulation of the Java model (and I'm not talking about the more or less straightforward content provider code; rather, I mean complex "semantic actions" such as refactorings, etc.)

Another reason to base Handly on the JDT Java model's design patterns and principles is that, while being many years old, these patterns and principles have aged very well and have been thoroughly tested in practice, not only in JDT but also in CDT, etc. In my 10+ year Eclipse experience, there are many subtle things in the IDE that are not always readily apparent but might come to bite you afterwards. It is very tempting to push the boundaries, but following a well-approved design essentially buys you (and your adopters) a safety net. Further, if Handly is to be adopted by any of the well-established projects, it seems important to stay close to their original design concepts.

I think that it might be worthwhile to keep in mind the following quotes for further discussion:

* "... make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." often paraphrased as "Make things as simple as possible, but *not simpler*." -- Albert Einstein. (emphasis mine)

* ALGOL 60 "was not only an improvement on its predecessors but also on nearly all its successors." -- Tony Hoare. (that is to say an old design is not necessarily bad)

OK, let me try to address your points in turn now.

Quote:
The main thing I notice is that everywhere you start with the assumption that an IDE wants a JDT-like Java model for use, however, I question that such is actually the best approach. I know JDT is a very advanced IDE, certainly the most advanced Eclipse IDE, but that doesn't mean its design decisions are the best, especially older choices, taken at a time where the Eclipse Platform was vastly different from what it is today. (and indeed, some of these JDT design decisions are quite old, maybe even before there was an "Eclipse Platform")

I don't think this account is quite accurate. The handle-based Platform resources model on which the JDT Java model is based has not changed much since at least version 2.0. The Eclipse Platform *UI* has indeed evolved, while the *Core* remained stable as far as the JDT design is concerned. So the basic design patterns and principles remain as valid today as they were many years ago.

Quote:
The first point we need to look at is, why is the hierarchy for the source-derived elements (ISourceConstruct) part of the hierarchy for project elements ( FooProject, FooSourceFolder, FooSourceFile, etc.) ? These sub-hierarchies are typically used in different ways, different contexts: ISourceConstruct's for outline for example (or code folding), and the project elements for other stuff. The only place, as far I know, they seem to connect to each is when you view ISourceConstructs under an ISourceFile in the Project Explorer tree (or in Search results). However such functionality can be trivially achieved by customizing the element provider of the underlying Tree Viewer. It doesn't justify having them both kinds of elements on the same hierarchy. In Melnorme they are separate: there is ISourceFileStructure with ISourceElement's, and then there is an INavigatorElement hierarchy for the Project Explorer elements.

To be honest, I don't see a big advantage in this separation. It introduces a non-uniformity right in the "kernel" and, though it may seem quite innocent, is fundamentally of the same nature that the separation between reference and primitive types in Java. As I commented above, I'm a big believer in uniform designs. As an example, the fact that every model element in Handly is an instance of IHandle makes it possible to introduce a uniform IHandleDelta and a uniform IElementChangeEvent/Listeners. That is to say uniformity in the kernel adds up and allows to make upper layers more regular.

Quote:
With this split in place, other assumptions start to fall, namely the handle/body design. For example, in JDT if an ISourceConstruct is open, that means all the other ISourceConstructs from the same file are open/available as well. And nearly all the time you want to use an ISourceConstructs, you want it open, and you want all other elements of the same file open too (for the Outline for example). You rarely (if ever) want just a handle. As such, if ISourceConstructs is split from the other hierarchy, then it can be "open" all the time, it can contain all the model information for that source file. This means for example, methods like getChildren always return successfully, they can't throw a CoreException (another simplification that adds up in overall design).

If there is some functionality there requires a handle-only to an ISourceConstruct, it's easy to create a different class that provides that functionality (like a handle info object that is just a file path and a source range, for example).

Also, in my opinion the project model hierarchy doesn't need the handle/body design either: if it is decoupled from the ISourceConstruct elements, the size of the whole project model hierarchy is significantly smaller (one or two orders of magnitude smaller), and as such the whole thing can be kept in memory at once and recreated whole when underlying resource changes (since it no longer requires parsing source files). In other words, the project model is in the order of magnitude of (at most) the number of source files that exist in the workspace (but not on their size or contents). This should be insignificant to keep in memory.

While I agree that memory requirements of a ISourceFileStructure in Melnorme and the non-openable element structure inside a ISourceFile in Handly are basically the same, I think that it is quite important to keep the handle/body design for the "openables" like ISourceFile. Just as a mental experiment, imagine an "Expand All" button in the Project Explorer (I know it would be a bad idea in practice), which will materialize both the whole project structure and the complete source file structure. In Handly, the memory consumption of the whole model will be capped by a bounded LRU cache for the openables. If you ditched the handle/body design, there would be no predefined upper limit for the aggregate model memory usage, which is the sum (quite large, potentially) of the whole project structure and the (opened) source file structure. It is also worth keeping in mind that "openables" may include such things as hundreds of JAR files containing thousands of classes. I think that recreating the whole project structure on a resource change might be quite expensive.

Quote:
The other thing that has me confused is, why do we need functionality to manage buffer/working-copy of the associated ISourceFile ? the #reconcile() method has been a source of confusion for me in JDT for 8+ years, -__-' (yes, that long). I mean, I understand what it does, but not why, why is it necessary. Same for the other related methods. I suspect it's unnecessarily complex API design.

In Melnorme, the ISourceFileStructure is an immutable data structure created from a source code snapshot (a String). The ISourceFileStructure does not care from where the source comes from, or whether its "reconciled" or not. Same for the Outline view that feeds on the ISourceFileStructure: it doesn't care about reconciliation or a working copy. All it cares is about an outline structure to display. These concerns are shifted to a source structure manager, that connects to IDocument's, listens to changes, creates a new ISourceFileStructure for each change, stores it in a map for each document/source that it is connected to. With this, the source code editor simply connects to the source structure manager, with its document, and listens to when a new ISourceFileStructure is created (and updates the outline view when such happens). I don't see the point of adding all that IBuffer functionality, when all it seems to be doing is replicating TextFileBufferManager functionality.

Just as the bounded memory consumption of the whole model, I think that "eventual consistency" made possible by asynchronous reconciling is very important for scalability. Frankly, 'creating a new ISourceFileStructure for each change' in a document (if you really meant it) just will not cut it when editing a reasonably large source file, let alone a file with tens of thousands of LOC. Handly provides all the facilities required for asynchronous reconciling together with fine-grained change notification, which allows to greatly minimize blocking of the UI thread while the user is typing in the editor.

Regarding IBuffer, we just needed an abstraction that could work atop both ITextFileBufferManager and IDocumentProvider (the latter is necessary for integration with XtextEditor, which uses IFileDocumentProvider rather than ITextFileDocumentProvider). Besides, IBuffer also accounts for non-shared buffers of private working copies and can provide ISnapshots which are an important element of the overall design.

The bottom line is that every design is a product of different assumptions and trade-offs imprinted by its designer, and I believe that it may actually be great for Eclipse-based IDE implementors to have a choice in Melnorme and Handly. However, we probably have to agree to disagree on some of the underlying assumptions and associated trade-offs each framework makes.

Thanks again for taking the time to write this elaborate feedback! Such constructive criticism is essential for vitality of the project and is exactly what we need.

Best regards,
Vladimir
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1726209 is a reply to message #1725932] Thu, 10 March 2016 14:38 Go to previous messageGo to next message
Bruno Medeiros is currently offline Bruno MedeirosFriend
Messages: 11
Registered: September 2014
Junior Member
Vladimir Piskarev wrote on Tue, 08 March 2016 14:51


Thanks a lot for the great feedback and food for thought you have provided, including the suggestion to use the ide-dev list. I must admit that I didn't know about the list when the project was started, but even now, when I have been subscribed to it for some time, I was a bit hesitant about using it for introducing the project since some might consider such usage a plug. Your story gives me a good excuse for doing that, thinking of all the people who might be interested in such a project but still don't know that it even exists...

It is, indeed, unfortunate that Handly remains a best-kept secret in Eclipse despite my efforts to draw attention to the project. Besides being announced on the regular communication channels like the Eclipse's Project Proposals forum, it was presented at the first XtextCon, and I'm trying to blog about it more or less regularly, with my posts syndicated to Planet Eclipse. Recently, Wayne and I made an attempt to bring the project to the AC's attention in Eclipse. Apparently, all of it have not been successful enough.


Yeah, I can see why it would be tricky to draw attention to the Handly project. Like you said, Eclipse nowadays seems to be focused a lot on cloud, IoT, and also other projects that are not IDE-specific at all. It's not easy to find a communication channel that the interested parties in IDE development (and IDE tools refactorization) are listening to.

I think the IDE-Dev mailing list is the best communication channel for this, but it also took me quite a while to find out about that (until Doug pointed it out). And there might be other developers out there who might be interested in this topic, but don't know about IDE-Dev either. I think we should mention this issue in IDE-Dev, tell about how Handly/Melnorme missed each other (and possibly others developers), and see if we could make the IDE-Dev mailing list more visible. (because it has no underlying Eclipse project, it has no newsgroups as well, so I guess that's one problem that might make it less visible)

BTW, it was the developer of Erlide who pointed me out to Handly. I guess we have him to thanks. Smile

As for your second post, my reply will have to come later... there is a lot of technical comments I'd like to make but that will take time, and I'm fairly busy in the next days...
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1726214 is a reply to message #1726070] Thu, 10 March 2016 14:46 Go to previous messageGo to next message
Bruno Medeiros is currently offline Bruno MedeirosFriend
Messages: 11
Registered: September 2014
Junior Member
My full reply to second post will comer later, but here's a quick comment:

Vladimir Piskarev wrote on Wed, 09 March 2016 11:33


Frankly, 'creating a new ISourceFileStructure for each change' in a document (if you really meant it) just will not cut it when editing a reasonably large source file, let alone a file with tens of thousands of LOC. Handly provides all the facilities required for asynchronous reconciling together with fine-grained change notification, which allows to greatly minimize blocking of the UI thread while the user is typing in the editor.



'creating a new ISourceFileStructure for each change' in a document
Yes, I meant that, but I didn't say that was done in the UI thread! Melnorme does it asynchronously as well. You're absolutely right, doing this kind of stuff in UI thread would not be acceptable.

The difference in behavior in Melnorme is that whenever a document change happens, a request for a new ISourceFileStructure is sent immediately (the previous calculation, if still running, is terminated since it's no longer necessary). Whereas in JDT (and Handly I guess), there is a delay of a few ms since last change until reconciliation begins.

BTW, if you wanna see MelnormeEclipse in action with a real IDE, I recommend RustDT, that's probably the best one of the 3 to showcase the MelnormeEclipse features, or to check out the source.
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1726218 is a reply to message #1726209] Thu, 10 March 2016 15:08 Go to previous messageGo to next message
Vladimir Piskarev is currently offline Vladimir PiskarevFriend
Messages: 48
Registered: January 2014
Member
Quote:
I think we should mention this issue in IDE-Dev, tell about how Handly/Melnorme missed each other (and possibly others developers), and see if we could make the IDE-Dev mailing list more visible.

+1. Good idea!

Quote:
BTW, it was the developer of Erlide who pointed me out to Handly. I guess we have him to thanks. Smile

Yeah. Thanks Vlad! Smile

Quote:
As for your second post, my reply will have to come later... there is a lot of technical comments I'd like to make but that will take time, and I'm fairly busy in the next days...

OK. Any input would be greatly appreciated! Thanks Bruno!
Re: Thoughts on Handly from the MelnormeEclipse dev. [message #1726226 is a reply to message #1726214] Thu, 10 March 2016 15:42 Go to previous message
Vladimir Piskarev is currently offline Vladimir PiskarevFriend
Messages: 48
Registered: January 2014
Member
Quote:
Yes, I meant that, but I didn't say that was done in the UI thread! Melnorme does it asynchronously as well. You're absolutely right, doing this kind of stuff in UI thread would not be acceptable.

The difference in behavior in Melnorme is that whenever a document change happens, a request for a new ISourceFileStructure is sent immediately (the previous calculation, if still running, is terminated since it's no longer necessary). Whereas in JDT (and Handly I guess), there is a delay of a few ms since last change until reconciliation begins.

Oh, I see, sorry for misunderstanding.

Quote:
BTW, if you wanna see MelnormeEclipse in action with a real IDE, I recommend RustDT, that's probably the best one of the 3 to showcase the MelnormeEclipse features, or to check out the source.

Sure, I'll try to find the time to take a deeper look at the source. Thanks for the hint!
Previous Topic:Announcing Handly 0.4 Release
Next Topic:Handly Success Stories
Goto Forum:
  


Current Time: Mon Sep 24 14:15:33 GMT 2018

Powered by FUDForum. Page generated in 0.01880 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top