Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc) » teneo - Performance penalty for larger models in HibernateResource
| teneo - Performance penalty for larger models in HibernateResource [message #64995] |
Tue, 19 December 2006 18:14  |
Eclipse User |
|
|
|
Hi Martin,
I'm trying to load a large model into a tree based editor (using a
HibernateResource). I got the impression that the performance drop
over-proportionally with the model size.
The tree that shows up consists of the following hierarchy:
children per node: 1->580->120
total #objects per hierarchy level: 1->580->70,000
I am having problems already when the tree expands down to level 1 (the 580),
that is because the editor must determine number of children in order to decide
whether or not display the expansion icon or not.
While I was waiting for my tree to be displayed I monitored java and the running
mysql and figured, that the DB was mostly idle but java was on full load for
several minutes.
I have run the profiler and found that these two stack frames count for roughly
about 35% of the CPU:
TRACE 301131:
java.util.ArrayList.indexOf(ArrayList.java:219)
java.util.ArrayList.contains(ArrayList.java:201)
org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
TRACE 301145:
java.util.ArrayList.indexOf(ArrayList.java:220)
java.util.ArrayList.contains(ArrayList.java:201)
org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
rank self accum count trace method
1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
4 4.91% 90.20% 5004 301047 org.eclipse.swt.internal.win32.OS.WaitMessage
Both stack frames stem from the same line of code in StoreResource:
StoreResource.java:444:
if (newEObjects.contains(eObject) || loadedEObjects.contains(eObject))
I know that the checks are necessary but may be the ArrayLists can be replaced
by some hash based collection or additionally index for this existence check?
Regards
Andre
|
|
| | | |
| Re: teneo - Performance penalty for larger models in HibernateResource [message #65427 is a reply to message #65238] |
Thu, 21 December 2006 18:36  |
| Eclipse User |
|
|
|
Hi Martin,
I've tested for a few hours now without any problems. I would like to add that I
have only classes which do not override hashCode or equals. Also I don't do
unit testing, I did only a few load/modify/save cycles.
I'm still fighting against performance problems during save, which might have to
do with the save-update cascades which might cascade too deep. I have ~160000
objects in the loadedEObjects set which might be the cause. Maybe I have to
remove some of the to-one associations which as I understand can not be lazily
loaded by hibernate...
Thanks
Andre
Andre Pareis wrote:
> Hi Martin,
>
> I was unable to test this today time-wise but I will do it this week and
> get back to you as soon as possible.
>
> But let me thank you in advance for this quick implementation :)
>
> Regards
> Andre
>
>
>
> Martin Taal wrote:
>> Hi Andre,
>> I have changed this to hashset and checked it in. If you are on cvs
>> can you check it?
>>
>> gr. Martin
>>
>> Andre Pareis wrote:
>>> I can just answer myself:
>>>
>>> I have just changed these two collection to HashSet and performance
>>> improved as expected (load case):
>>>
>>> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
>>> rank self accum count trace method
>>> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
>>> 2 13.76% 63.05% 2278 301076
>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>> 3 5.65% 68.71% 936 301206
>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>
>>> 4 4.30% 73.01% 712 301222
>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>
>>> 5 3.46% 76.47% 573 300743 java.net.SocketInputStream.socketRead0
>>>
>>>
>>> The StoreResource existence check does not count any more. Load and
>>> display time is now acceptable.
>>>
>>> Looks good except that I don't know how much your code does rely on
>>> the insertion order of the elements? If not then you might consider
>>> this change.
>>>
>>> Thanks
>>> Andre
>>>
>>>
>>>
>>>
>>> Andre Pareis wrote:
>>>> Hi Martin,
>>>>
>>>> I'm trying to load a large model into a tree based editor (using a
>>>> HibernateResource). I got the impression that the performance drop
>>>> over-proportionally with the model size.
>>>>
>>>> The tree that shows up consists of the following hierarchy:
>>>>
>>>> children per node: 1->580->120
>>>> total #objects per hierarchy level: 1->580->70,000
>>>>
>>>> I am having problems already when the tree expands down to level 1
>>>> (the 580), that is because the editor must determine number of
>>>> children in order to decide whether or not display the expansion
>>>> icon or not.
>>>>
>>>> While I was waiting for my tree to be displayed I monitored java and
>>>> the running mysql and figured, that the DB was mostly idle but java
>>>> was on full load for several minutes.
>>>>
>>>> I have run the profiler and found that these two stack frames count
>>>> for roughly about 35% of the CPU:
>>>>
>>>>
>>>>
>>>> TRACE 301131:
>>>> java.util.ArrayList.indexOf(ArrayList.java:219)
>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>
>>>>
>>>> TRACE 301145:
>>>> java.util.ArrayList.indexOf(ArrayList.java:220)
>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>>>> rank self accum count trace method
>>>> 1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
>>>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>>>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>>>> 4 4.91% 90.20% 5004 301047
>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>>
>>>>
>>>> Both stack frames stem from the same line of code in StoreResource:
>>>>
>>>> StoreResource.java:444:
>>>> if (newEObjects.contains(eObject) ||
>>>> loadedEObjects.contains(eObject))
>>>>
>>>>
>>>>
>>>> I know that the checks are necessary but may be the ArrayLists can
>>>> be replaced by some hash based collection or additionally index for
>>>> this existence check?
>>>>
>>>>
>>>> Regards
>>>> Andre
>>>>
>>
>>
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #65512 is a reply to message #65427] |
Fri, 22 December 2006 15:45 |
| Eclipse User |
|
|
|
Hi Andre,
Does it load 160000 objects because of the one-to-one relations?
Another question is if hibernate actually saves all 160000 objects even if many of them did not
change. Does it do that?
gr. Martin
Andre Pareis wrote:
> Hi Martin,
>
> I've tested for a few hours now without any problems. I would like to
> add that I have only classes which do not override hashCode or equals.
> Also I don't do unit testing, I did only a few load/modify/save cycles.
>
> I'm still fighting against performance problems during save, which might
> have to do with the save-update cascades which might cascade too deep. I
> have ~160000 objects in the loadedEObjects set which might be the cause.
> Maybe I have to remove some of the to-one associations which as I
> understand can not be lazily loaded by hibernate...
>
> Thanks
> Andre
>
>
>
>
> Andre Pareis wrote:
>> Hi Martin,
>>
>> I was unable to test this today time-wise but I will do it this week
>> and get back to you as soon as possible.
>>
>> But let me thank you in advance for this quick implementation :)
>>
>> Regards
>> Andre
>>
>>
>>
>> Martin Taal wrote:
>>> Hi Andre,
>>> I have changed this to hashset and checked it in. If you are on cvs
>>> can you check it?
>>>
>>> gr. Martin
>>>
>>> Andre Pareis wrote:
>>>> I can just answer myself:
>>>>
>>>> I have just changed these two collection to HashSet and performance
>>>> improved as expected (load case):
>>>>
>>>> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
>>>> rank self accum count trace method
>>>> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
>>>> 2 13.76% 63.05% 2278 301076
>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>> 3 5.65% 68.71% 936 301206
>>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>>
>>>> 4 4.30% 73.01% 712 301222
>>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>>
>>>> 5 3.46% 76.47% 573 300743
>>>> java.net.SocketInputStream.socketRead0
>>>>
>>>>
>>>> The StoreResource existence check does not count any more. Load and
>>>> display time is now acceptable.
>>>>
>>>> Looks good except that I don't know how much your code does rely on
>>>> the insertion order of the elements? If not then you might consider
>>>> this change.
>>>>
>>>> Thanks
>>>> Andre
>>>>
>>>>
>>>>
>>>>
>>>> Andre Pareis wrote:
>>>>> Hi Martin,
>>>>>
>>>>> I'm trying to load a large model into a tree based editor (using a
>>>>> HibernateResource). I got the impression that the performance drop
>>>>> over-proportionally with the model size.
>>>>>
>>>>> The tree that shows up consists of the following hierarchy:
>>>>>
>>>>> children per node: 1->580->120
>>>>> total #objects per hierarchy level: 1->580->70,000
>>>>>
>>>>> I am having problems already when the tree expands down to level 1
>>>>> (the 580), that is because the editor must determine number of
>>>>> children in order to decide whether or not display the expansion
>>>>> icon or not.
>>>>>
>>>>> While I was waiting for my tree to be displayed I monitored java
>>>>> and the running mysql and figured, that the DB was mostly idle but
>>>>> java was on full load for several minutes.
>>>>>
>>>>> I have run the profiler and found that these two stack frames count
>>>>> for roughly about 35% of the CPU:
>>>>>
>>>>>
>>>>>
>>>>> TRACE 301131:
>>>>> java.util.ArrayList.indexOf(ArrayList.java:219)
>>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>>
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>>
>>>>>
>>>>> TRACE 301145:
>>>>> java.util.ArrayList.indexOf(ArrayList.java:220)
>>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>>
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>>>>> rank self accum count trace method
>>>>> 1 49.85% 49.85% 50789 300545
>>>>> java.net.PlainSocketImpl.socketAccept
>>>>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>>>>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>>>>> 4 4.91% 90.20% 5004 301047
>>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>>>
>>>>>
>>>>> Both stack frames stem from the same line of code in StoreResource:
>>>>>
>>>>> StoreResource.java:444:
>>>>> if (newEObjects.contains(eObject) ||
>>>>> loadedEObjects.contains(eObject))
>>>>>
>>>>>
>>>>>
>>>>> I know that the checks are necessary but may be the ArrayLists can
>>>>> be replaced by some hash based collection or additionally index for
>>>>> this existence check?
>>>>>
>>>>>
>>>>> Regards
>>>>> Andre
>>>>>
>>>
>>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #597166 is a reply to message #64995] |
Tue, 19 December 2006 18:25 |
| Eclipse User |
|
|
|
I can just answer myself:
I have just changed these two collection to HashSet and performance improved as
expected (load case):
CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
rank self accum count trace method
1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
2 13.76% 63.05% 2278 301076 org.eclipse.swt.internal.win32.OS.WaitMessage
3 5.65% 68.71% 936 301206
org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
4 4.30% 73.01% 712 301222
org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
5 3.46% 76.47% 573 300743 java.net.SocketInputStream.socketRead0
The StoreResource existence check does not count any more. Load and display time
is now acceptable.
Looks good except that I don't know how much your code does rely on the
insertion order of the elements? If not then you might consider this change.
Thanks
Andre
Andre Pareis wrote:
> Hi Martin,
>
> I'm trying to load a large model into a tree based editor (using a
> HibernateResource). I got the impression that the performance drop
> over-proportionally with the model size.
>
> The tree that shows up consists of the following hierarchy:
>
> children per node: 1->580->120
> total #objects per hierarchy level: 1->580->70,000
>
> I am having problems already when the tree expands down to level 1 (the
> 580), that is because the editor must determine number of children in
> order to decide whether or not display the expansion icon or not.
>
> While I was waiting for my tree to be displayed I monitored java and the
> running mysql and figured, that the DB was mostly idle but java was on
> full load for several minutes.
>
> I have run the profiler and found that these two stack frames count for
> roughly about 35% of the CPU:
>
>
>
> TRACE 301131:
> java.util.ArrayList.indexOf(ArrayList.java:219)
> java.util.ArrayList.contains(ArrayList.java:201)
> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>
> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>
>
> TRACE 301145:
> java.util.ArrayList.indexOf(ArrayList.java:220)
> java.util.ArrayList.contains(ArrayList.java:201)
> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>
> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>
>
>
>
>
> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
> rank self accum count trace method
> 1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
> 4 4.91% 90.20% 5004 301047
> org.eclipse.swt.internal.win32.OS.WaitMessage
>
>
> Both stack frames stem from the same line of code in StoreResource:
>
> StoreResource.java:444:
> if (newEObjects.contains(eObject) || loadedEObjects.contains(eObject))
>
>
>
> I know that the checks are necessary but may be the ArrayLists can be
> replaced by some hash based collection or additionally index for this
> existence check?
>
>
> Regards
> Andre
>
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #597203 is a reply to message #65060] |
Wed, 20 December 2006 04:40 |
| Eclipse User |
|
|
|
Hi Andre,
I have changed this to hashset and checked it in. If you are on cvs can you check it?
gr. Martin
Andre Pareis wrote:
> I can just answer myself:
>
> I have just changed these two collection to HashSet and performance
> improved as expected (load case):
>
> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
> rank self accum count trace method
> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
> 2 13.76% 63.05% 2278 301076
> org.eclipse.swt.internal.win32.OS.WaitMessage
> 3 5.65% 68.71% 936 301206
> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
> 4 4.30% 73.01% 712 301222
> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
> 5 3.46% 76.47% 573 300743 java.net.SocketInputStream.socketRead0
>
>
> The StoreResource existence check does not count any more. Load and
> display time is now acceptable.
>
> Looks good except that I don't know how much your code does rely on the
> insertion order of the elements? If not then you might consider this
> change.
>
> Thanks
> Andre
>
>
>
>
> Andre Pareis wrote:
>> Hi Martin,
>>
>> I'm trying to load a large model into a tree based editor (using a
>> HibernateResource). I got the impression that the performance drop
>> over-proportionally with the model size.
>>
>> The tree that shows up consists of the following hierarchy:
>>
>> children per node: 1->580->120
>> total #objects per hierarchy level: 1->580->70,000
>>
>> I am having problems already when the tree expands down to level 1
>> (the 580), that is because the editor must determine number of
>> children in order to decide whether or not display the expansion icon
>> or not.
>>
>> While I was waiting for my tree to be displayed I monitored java and
>> the running mysql and figured, that the DB was mostly idle but java
>> was on full load for several minutes.
>>
>> I have run the profiler and found that these two stack frames count
>> for roughly about 35% of the CPU:
>>
>>
>>
>> TRACE 301131:
>> java.util.ArrayList.indexOf(ArrayList.java:219)
>> java.util.ArrayList.contains(ArrayList.java:201)
>>
>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>
>>
>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>
>>
>> TRACE 301145:
>> java.util.ArrayList.indexOf(ArrayList.java:220)
>> java.util.ArrayList.contains(ArrayList.java:201)
>>
>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>
>>
>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>
>>
>>
>>
>>
>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>> rank self accum count trace method
>> 1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>> 4 4.91% 90.20% 5004 301047
>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>
>>
>> Both stack frames stem from the same line of code in StoreResource:
>>
>> StoreResource.java:444:
>> if (newEObjects.contains(eObject) ||
>> loadedEObjects.contains(eObject))
>>
>>
>>
>> I know that the checks are necessary but may be the ArrayLists can be
>> replaced by some hash based collection or additionally index for this
>> existence check?
>>
>>
>> Regards
>> Andre
>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #597253 is a reply to message #65154] |
Wed, 20 December 2006 19:39 |
| Eclipse User |
|
|
|
Hi Martin,
I was unable to test this today time-wise but I will do it this week and get
back to you as soon as possible.
But let me thank you in advance for this quick implementation :)
Regards
Andre
Martin Taal wrote:
> Hi Andre,
> I have changed this to hashset and checked it in. If you are on cvs can
> you check it?
>
> gr. Martin
>
> Andre Pareis wrote:
>> I can just answer myself:
>>
>> I have just changed these two collection to HashSet and performance
>> improved as expected (load case):
>>
>> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
>> rank self accum count trace method
>> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
>> 2 13.76% 63.05% 2278 301076
>> org.eclipse.swt.internal.win32.OS.WaitMessage
>> 3 5.65% 68.71% 936 301206
>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>
>> 4 4.30% 73.01% 712 301222
>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>
>> 5 3.46% 76.47% 573 300743 java.net.SocketInputStream.socketRead0
>>
>>
>> The StoreResource existence check does not count any more. Load and
>> display time is now acceptable.
>>
>> Looks good except that I don't know how much your code does rely on
>> the insertion order of the elements? If not then you might consider
>> this change.
>>
>> Thanks
>> Andre
>>
>>
>>
>>
>> Andre Pareis wrote:
>>> Hi Martin,
>>>
>>> I'm trying to load a large model into a tree based editor (using a
>>> HibernateResource). I got the impression that the performance drop
>>> over-proportionally with the model size.
>>>
>>> The tree that shows up consists of the following hierarchy:
>>>
>>> children per node: 1->580->120
>>> total #objects per hierarchy level: 1->580->70,000
>>>
>>> I am having problems already when the tree expands down to level 1
>>> (the 580), that is because the editor must determine number of
>>> children in order to decide whether or not display the expansion icon
>>> or not.
>>>
>>> While I was waiting for my tree to be displayed I monitored java and
>>> the running mysql and figured, that the DB was mostly idle but java
>>> was on full load for several minutes.
>>>
>>> I have run the profiler and found that these two stack frames count
>>> for roughly about 35% of the CPU:
>>>
>>>
>>>
>>> TRACE 301131:
>>> java.util.ArrayList.indexOf(ArrayList.java:219)
>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>
>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>
>>>
>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>
>>>
>>> TRACE 301145:
>>> java.util.ArrayList.indexOf(ArrayList.java:220)
>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>
>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>
>>>
>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>
>>>
>>>
>>>
>>>
>>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>>> rank self accum count trace method
>>> 1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
>>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>>> 4 4.91% 90.20% 5004 301047
>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>
>>>
>>> Both stack frames stem from the same line of code in StoreResource:
>>>
>>> StoreResource.java:444:
>>> if (newEObjects.contains(eObject) ||
>>> loadedEObjects.contains(eObject))
>>>
>>>
>>>
>>> I know that the checks are necessary but may be the ArrayLists can be
>>> replaced by some hash based collection or additionally index for this
>>> existence check?
>>>
>>>
>>> Regards
>>> Andre
>>>
>
>
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #600191 is a reply to message #65238] |
Thu, 21 December 2006 18:36 |
| Eclipse User |
|
|
|
Hi Martin,
I've tested for a few hours now without any problems. I would like to add that I
have only classes which do not override hashCode or equals. Also I don't do
unit testing, I did only a few load/modify/save cycles.
I'm still fighting against performance problems during save, which might have to
do with the save-update cascades which might cascade too deep. I have ~160000
objects in the loadedEObjects set which might be the cause. Maybe I have to
remove some of the to-one associations which as I understand can not be lazily
loaded by hibernate...
Thanks
Andre
Andre Pareis wrote:
> Hi Martin,
>
> I was unable to test this today time-wise but I will do it this week and
> get back to you as soon as possible.
>
> But let me thank you in advance for this quick implementation :)
>
> Regards
> Andre
>
>
>
> Martin Taal wrote:
>> Hi Andre,
>> I have changed this to hashset and checked it in. If you are on cvs
>> can you check it?
>>
>> gr. Martin
>>
>> Andre Pareis wrote:
>>> I can just answer myself:
>>>
>>> I have just changed these two collection to HashSet and performance
>>> improved as expected (load case):
>>>
>>> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
>>> rank self accum count trace method
>>> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
>>> 2 13.76% 63.05% 2278 301076
>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>> 3 5.65% 68.71% 936 301206
>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>
>>> 4 4.30% 73.01% 712 301222
>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>
>>> 5 3.46% 76.47% 573 300743 java.net.SocketInputStream.socketRead0
>>>
>>>
>>> The StoreResource existence check does not count any more. Load and
>>> display time is now acceptable.
>>>
>>> Looks good except that I don't know how much your code does rely on
>>> the insertion order of the elements? If not then you might consider
>>> this change.
>>>
>>> Thanks
>>> Andre
>>>
>>>
>>>
>>>
>>> Andre Pareis wrote:
>>>> Hi Martin,
>>>>
>>>> I'm trying to load a large model into a tree based editor (using a
>>>> HibernateResource). I got the impression that the performance drop
>>>> over-proportionally with the model size.
>>>>
>>>> The tree that shows up consists of the following hierarchy:
>>>>
>>>> children per node: 1->580->120
>>>> total #objects per hierarchy level: 1->580->70,000
>>>>
>>>> I am having problems already when the tree expands down to level 1
>>>> (the 580), that is because the editor must determine number of
>>>> children in order to decide whether or not display the expansion
>>>> icon or not.
>>>>
>>>> While I was waiting for my tree to be displayed I monitored java and
>>>> the running mysql and figured, that the DB was mostly idle but java
>>>> was on full load for several minutes.
>>>>
>>>> I have run the profiler and found that these two stack frames count
>>>> for roughly about 35% of the CPU:
>>>>
>>>>
>>>>
>>>> TRACE 301131:
>>>> java.util.ArrayList.indexOf(ArrayList.java:219)
>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>
>>>>
>>>> TRACE 301145:
>>>> java.util.ArrayList.indexOf(ArrayList.java:220)
>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>
>>>>
>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>>>> rank self accum count trace method
>>>> 1 49.85% 49.85% 50789 300545 java.net.PlainSocketImpl.socketAccept
>>>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>>>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>>>> 4 4.91% 90.20% 5004 301047
>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>>
>>>>
>>>> Both stack frames stem from the same line of code in StoreResource:
>>>>
>>>> StoreResource.java:444:
>>>> if (newEObjects.contains(eObject) ||
>>>> loadedEObjects.contains(eObject))
>>>>
>>>>
>>>>
>>>> I know that the checks are necessary but may be the ArrayLists can
>>>> be replaced by some hash based collection or additionally index for
>>>> this existence check?
>>>>
>>>>
>>>> Regards
>>>> Andre
>>>>
>>
>>
|
|
|
| Re: teneo - Performance penalty for larger models in HibernateResource [message #600222 is a reply to message #65427] |
Fri, 22 December 2006 15:45 |
| Eclipse User |
|
|
|
Hi Andre,
Does it load 160000 objects because of the one-to-one relations?
Another question is if hibernate actually saves all 160000 objects even if many of them did not
change. Does it do that?
gr. Martin
Andre Pareis wrote:
> Hi Martin,
>
> I've tested for a few hours now without any problems. I would like to
> add that I have only classes which do not override hashCode or equals.
> Also I don't do unit testing, I did only a few load/modify/save cycles.
>
> I'm still fighting against performance problems during save, which might
> have to do with the save-update cascades which might cascade too deep. I
> have ~160000 objects in the loadedEObjects set which might be the cause.
> Maybe I have to remove some of the to-one associations which as I
> understand can not be lazily loaded by hibernate...
>
> Thanks
> Andre
>
>
>
>
> Andre Pareis wrote:
>> Hi Martin,
>>
>> I was unable to test this today time-wise but I will do it this week
>> and get back to you as soon as possible.
>>
>> But let me thank you in advance for this quick implementation :)
>>
>> Regards
>> Andre
>>
>>
>>
>> Martin Taal wrote:
>>> Hi Andre,
>>> I have changed this to hashset and checked it in. If you are on cvs
>>> can you check it?
>>>
>>> gr. Martin
>>>
>>> Andre Pareis wrote:
>>>> I can just answer myself:
>>>>
>>>> I have just changed these two collection to HashSet and performance
>>>> improved as expected (load case):
>>>>
>>>> CPU SAMPLES BEGIN (total = 16553) Wed Dec 20 00:19:58 2006
>>>> rank self accum count trace method
>>>> 1 49.29% 49.29% 8159 300547 java.net.PlainSocketImpl.socketAccept
>>>> 2 13.76% 63.05% 2278 301076
>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>> 3 5.65% 68.71% 936 301206
>>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>>
>>>> 4 4.30% 73.01% 712 301222
>>>> org.hibernate.engine.StatefulPersistenceContext.afterTransac tionCompletion
>>>>
>>>> 5 3.46% 76.47% 573 300743
>>>> java.net.SocketInputStream.socketRead0
>>>>
>>>>
>>>> The StoreResource existence check does not count any more. Load and
>>>> display time is now acceptable.
>>>>
>>>> Looks good except that I don't know how much your code does rely on
>>>> the insertion order of the elements? If not then you might consider
>>>> this change.
>>>>
>>>> Thanks
>>>> Andre
>>>>
>>>>
>>>>
>>>>
>>>> Andre Pareis wrote:
>>>>> Hi Martin,
>>>>>
>>>>> I'm trying to load a large model into a tree based editor (using a
>>>>> HibernateResource). I got the impression that the performance drop
>>>>> over-proportionally with the model size.
>>>>>
>>>>> The tree that shows up consists of the following hierarchy:
>>>>>
>>>>> children per node: 1->580->120
>>>>> total #objects per hierarchy level: 1->580->70,000
>>>>>
>>>>> I am having problems already when the tree expands down to level 1
>>>>> (the 580), that is because the editor must determine number of
>>>>> children in order to decide whether or not display the expansion
>>>>> icon or not.
>>>>>
>>>>> While I was waiting for my tree to be displayed I monitored java
>>>>> and the running mysql and figured, that the DB was mostly idle but
>>>>> java was on full load for several minutes.
>>>>>
>>>>> I have run the profiler and found that these two stack frames count
>>>>> for roughly about 35% of the CPU:
>>>>>
>>>>>
>>>>>
>>>>> TRACE 301131:
>>>>> java.util.ArrayList.indexOf(ArrayList.java:219)
>>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>>
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>>
>>>>>
>>>>> TRACE 301145:
>>>>> java.util.ArrayList.indexOf(ArrayList.java:220)
>>>>> java.util.ArrayList.contains(ArrayList.java:201)
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attachedHelper( StoreResource.java:444)
>>>>>
>>>>>
>>>>> org.eclipse.emf.teneo.resource.StoreResource.attached(StoreR esource.java:421)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> CPU SAMPLES BEGIN (total = 101885) Tue Dec 19 23:43:27 2006
>>>>> rank self accum count trace method
>>>>> 1 49.85% 49.85% 50789 300545
>>>>> java.net.PlainSocketImpl.socketAccept
>>>>> 2 19.36% 69.21% 19728 301131 java.util.ArrayList.indexOf
>>>>> 3 16.07% 85.29% 16377 301145 java.util.ArrayList.indexOf
>>>>> 4 4.91% 90.20% 5004 301047
>>>>> org.eclipse.swt.internal.win32.OS.WaitMessage
>>>>>
>>>>>
>>>>> Both stack frames stem from the same line of code in StoreResource:
>>>>>
>>>>> StoreResource.java:444:
>>>>> if (newEObjects.contains(eObject) ||
>>>>> loadedEObjects.contains(eObject))
>>>>>
>>>>>
>>>>>
>>>>> I know that the checks are necessary but may be the ArrayLists can
>>>>> be replaced by some hash based collection or additionally index for
>>>>> this existence check?
>>>>>
>>>>>
>>>>> Regards
>>>>> Andre
>>>>>
>>>
>>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
Goto Forum:
Current Time: Thu Oct 30 22:27:57 EDT 2025
Powered by FUDForum. Page generated in 0.04604 seconds |
Home