Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Sapphire » Expensive @NoDuplicates
icon4.gif  Expensive @NoDuplicates [message #1096385] Wed, 28 August 2013 06:13 Go to next message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
Hello All

I've been using @NoDuplicates a lot in my Sapphire XML Editor, for any field that I find applicable. After while I found out that using my Editor to open some large files, consumes lot of expensive time.

For a 0.5 MB file, it costed me around 7 minutes just to launch!

When I commented all occurrences of @NoDuplicates from my code, I was surprised that the 7 minutes were reduced to only 23 seconds.

Then I decided to implement a Validation Service that simply loops over the list, and see if This item is repeated or not, and returns an Error if found twice.

In order to make it more generic, I created an element called "INamedModelElement" that extends "IModelElement", and contains 1 property called "Name".

Then:
public final class NoDuplicatesValidationService extends ValidationService
{
    public Status validate() 
    {
        final INamedModelElement item = context( INamedModelElement.class );
        final ModelElementList<INamedModelElement> items = item.nearest( ModelElementList.class );
        final IRoot doc = item.nearest( IRoot.class );
        
        if( doc != null )
        {
            String myName = item.getName().getText();
            boolean foundOnce = false;
            if(myName != null){
                for(INamedModelElement named_item: items){
                    String hisName = named_item.getName().getText();
                    if(stringsEqual(myName,hisName)){
                        if(foundOnce){
                            return Status.createErrorStatus("No Duplicates Allowed!");
                        }else{
                            foundOnce = true;
                        }
                    }
                }
            }
        }    
        return Status.createOkStatus();
    }

    private static boolean stringsEqual(String myName, String hisName) {
        if(hisName == null){
            return false;
        }
        return myName.equalsIgnoreCase(hisName);
    }
}


As you can see, for my solution to work, I have to make the element interface extends "INamedModelElement" instead of simply "IModelElement". I also HAVE TO add the "@Service(impl = NoDuplicatesValidationService.class)" at the "Name" property specifically.

The reason I did all this and not simply keep using "@NoDuplicates", is that when I did so, the 7 minutes were reduced to 26 seconds .. only 3 seconds more than the one with no checking at all, and more than 6 minutes less than "@NoDuplicates".

I'm using Sapphire 0.5, and tried migration to Sapphire 0.6 but that didn't help.

After all I have 2 questions:

  1. Why does @NoDuplicates take so much time for large files, while it should in fact only do a For Loop?
  2. For my solution, what if I need to replace @NoDuplicate for a field that is not named as "Name"? Is there any other solution other than creating Another Validation Service?


Thanks!
Re: Expensive @NoDuplicates [message #1096826 is a reply to message #1096385] Wed, 28 August 2013 19:06 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
There are two parts to @NoDuplicates implementation: (a) validation logic and (b) trigger for when to re-validate. Your implementation is just (a). The performance issue you are seeing is likely caused by unnecessary re-validation. Each property should not be validated more than once on launch.

Could you open a bug and I will investigate further?
Re: Expensive @NoDuplicates [message #1098139 is a reply to message #1096826] Fri, 30 August 2013 13:25 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
I have spent some time investigating performance of @NoDuplicates implementation and writing unit tests to measure the number of times the validation is called in different cases. I did see excessive re-validation and fixed some of the problems. Could you take a look at the latest 0.6.4 build and let me know how it works with your scenario?

https://hudson.eclipse.org/sapphire/job/0.6.x/lastSuccessfulBuild/artifact/
Re: Expensive @NoDuplicates [message #1098144 is a reply to message #1098139] Fri, 30 August 2013 13:31 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
Also, could you let me know the item count where @NoDuplicates annotation is used in your scenario?
Re: Expensive @NoDuplicates [message #1099190 is a reply to message #1098144] Sun, 01 September 2013 05:08 Go to previous messageGo to next message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
I can see that my implementation doesn't always re-validate when it needs to re-validate!

I'll try 0.6.4 as fast as I can, may be this week. That will save me a lot!

For the items count I used, it was of different categories, all using @NoDuplicates:
Type1: 301 items
Type2: 186 items
Type3: 29 items
Type4: 408 items
Re: Expensive @NoDuplicates [message #1102639 is a reply to message #1099190] Thu, 05 September 2013 20:26 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
You should plan on moving up to 0.7 development builds. I've been doing further performance and scalability work on this scenario in 0.7 that cannot happen in 0.6.x due to needed API changes. I will push my changes shortly, but the results are very encouraging. I implemented a list index that eliminates the quadratic performance degradation of @NoDuplicates with the number of entries.

Scalability...

In 0.6.4, the tests pass with 100 list entries, but fail with 1000 (stack overflow).
In 0.7 (before index), the tests pass with 1000 list entries, but take too long with 10,000 (minutes).
In 0.7 (after index), the tests pass with 100,000 list entries and will likely pass with higher counts if given enough memory.

Performance...

In 0.7 (before index), the test case with 1000 entries runs in approximately 1.35 seconds, while the test case with 10,000 entries takes many minutes.
In 0.7 (after index), the test case with 1000 entries runs in approximately 0.24 seconds, 10,000 entries in 2.2 seconds, 100,000 entries in 17.2 seconds. The progression is linear.
Re: Expensive @NoDuplicates [message #1105046 is a reply to message #1102639] Mon, 09 September 2013 05:20 Go to previous messageGo to next message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
The results you're showing is more than tempting!! but when are you going to push the "index" into Sapphire 0.7 ??
Re: Expensive @NoDuplicates [message #1106172 is a reply to message #1105046] Tue, 10 September 2013 16:00 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
The changes to Sapphire 0.7 to add list indexing are out now. Take a look and let us know how this works for your scenario.
Re: Expensive @NoDuplicates [message #1107440 is a reply to message #1106172] Thu, 12 September 2013 07:59 Go to previous messageGo to next message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
35 seconds .. I successfully migrated into Sapphire 0.7 latest build @ 10/09/2013, and replaced all occurrences of my NoDuplicatesValidationService with @NoDuplicates .. and I reached 35 seconds for the same file.

That's great for me, what do you think?
Re: Expensive @NoDuplicates [message #1107642 is a reply to message #1107440] Thu, 12 September 2013 14:26 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
That's certainly a big improvement, but obviously further performance work is needed on other aspects.
Re: Expensive @NoDuplicates [message #1109936 is a reply to message #1107642] Mon, 16 September 2013 02:22 Go to previous messageGo to next message
Greg Amerson is currently offline Greg Amerson
Messages: 116
Registered: March 2010
Senior Member
Hey Konstantin,

We've been following the 0.7.x builds and when we use @NoDuplicates with a property that in the UI is using CheckBoxListPropertyEditorRenderer$Factory as factory hint, we have been getting the following error. Any ideas?

ERROR : Name property is already disposed.
java.lang.IllegalStateException: Name property is already disposed.
	at org.eclipse.sapphire.Property.assertNotDisposed(Property.java:761)
	at org.eclipse.sapphire.Property.init(Property.java:83)
	at org.eclipse.sapphire.Value.text(Value.java:244)
	at org.eclipse.sapphire.Value.text(Value.java:239)
	at org.eclipse.sapphire.services.UniqueValueValidationService.isUniqueValue(UniqueValueValidationService.java:91)
	at org.eclipse.sapphire.services.UniqueValueValidationService.compute(UniqueValueValidationService.java:77)
	at org.eclipse.sapphire.services.UniqueValueValidationService.compute(UniqueValueValidationService.java:1)
	at org.eclipse.sapphire.services.DataService.refresh(DataService.java:56)
	at org.eclipse.sapphire.services.UniqueValueValidationService$1.handleTypedEvent(UniqueValueValidationService.java:62)
	at org.eclipse.sapphire.services.UniqueValueValidationService$1.handleTypedEvent(UniqueValueValidationService.java:1)
	at org.eclipse.sapphire.FilteredListener.handle(FilteredListener.java:35)
	at org.eclipse.sapphire.ListenerContext$BroadcastJob.run(ListenerContext.java:182)
	at org.eclipse.sapphire.ListenerContext.broadcast(ListenerContext.java:120)
	at org.eclipse.sapphire.ListenerContext.broadcast(ListenerContext.java:127)
	at org.eclipse.sapphire.Property.broadcast(Property.java:709)
	at org.eclipse.sapphire.Value.refreshContent(Value.java:145)
	at org.eclipse.sapphire.Value.refresh(Value.java:67)
	at org.eclipse.sapphire.Value.write(Value.java:414)
	at org.eclipse.sapphire.ui.renderers.swt.CheckBoxListPropertyEditorRenderer$Entry.flip(CheckBoxListPropertyEditorRenderer.java:709)
	at org.eclipse.sapphire.ui.renderers.swt.CheckBoxListPropertyEditorRenderer$9.checkStateChanged(CheckBoxListPropertyEditorRenderer.java:407)
	at org.eclipse.jface.viewers.CheckboxTableViewer$1.run(CheckboxTableViewer.java:212)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
Re: Expensive @NoDuplicates [message #1110129 is a reply to message #1109936] Mon, 16 September 2013 08:12 Go to previous messageGo to next message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
I have a weird issue as well!!

Inside Eclipse, I had the 35 seconds I just mentioned. But when I exported my editor outside eclipse, it took a very long time to open the file, and still not opening!!

I'm still not able to get an error message, or to know the reason. Once I do, I'll keep you posted.
Re: Expensive @NoDuplicates [message #1110257 is a reply to message #1110129] Mon, 16 September 2013 12:13 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
Quote:
Inside Eclipse, I had the 35 seconds I just mentioned. But when I exported my editor outside eclipse, it took a very long time to open the file, and still not opening!!


That can happen if your JVM is starved for memory. All the time is spent trying to excessively scrub garbage on every memory allocation to prevent an OOM. Check your xmx settings in eclipse.ini file.
Re: Expensive @NoDuplicates [message #1110264 is a reply to message #1110257] Mon, 16 September 2013 12:22 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 937
Registered: July 2009
Senior Member
Quote:
We've been following the 0.7.x builds and when we use @NoDuplicates with a property that in the UI is using CheckBoxListPropertyEditorRenderer$Factory as factory hint, we have been getting the following error. Any ideas?


There is a similar setup in sapphie-extension.xml editor (service contexts in service definition) and I am not able to reproduce this issue. Judging by the stack trace, you are a bit out of date, so see if newest build will resolve this issue. If not, we will need to dig into what makes your scenario different.
Re: Expensive @NoDuplicates [message #1110661 is a reply to message #1110257] Tue, 17 September 2013 02:43 Go to previous message
Hazem ELRAFFIEE is currently offline Hazem ELRAFFIEE
Messages: 59
Registered: September 2012
Member
Konstantin Komissarchik wrote on Mon, 16 September 2013 12:13

That can happen if your JVM is starved for memory. All the time is spent trying to excessively scrub garbage on every memory allocation to prevent an OOM. Check your xmx settings in eclipse.ini file.


Exactly Smile Thank you Smile
Previous Topic:Enable/disable individual FormEditorPage (or page contents) based on model properties?
Next Topic:CountConstraint does not show any error message
Goto Forum:
  


Current Time: Mon Jul 28 20:51:02 EDT 2014

Powered by FUDForum. Page generated in 0.02034 seconds