Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » NatTable » How fix "OutOfMemoryError" when Nattable contain huge data
How fix "OutOfMemoryError" when Nattable contain huge data [message #1369528] Tue, 20 May 2014 16:30 Go to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
Hi,
I use nattable for display, it also contains the "Filter" function, sometimes the table has more than ten million rows data. i use a "baseEventList ", every time i add the data to eventlist.

error messages:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2245)
at java.util.Arrays.copyOf(Arrays.java:2219)
at java.util.ArrayList.grow(ArrayList.java:213)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:187)
at java.util.ArrayList.add(ArrayList.java:411)
at ca.odell.glazedlists.BasicEventList.add(BasicEventList.java:138)
at ca.odell.glazedlists.impl.ThreadSafeList.add(ThreadSafeList.java:177)
at cn.kisters.kiscript.rcp.views.ResultTableModel.fillNext(ResultTableModel.java:197)


codes:
baseEventList = GlazedLists.threadSafeList(GlazedLists.eventList(rowData));
ObservableElementList<T> observableElementList = new ObservableElementList<T>(baseEventList, connector);
FilterList<T> filterList = new FilterList<T>(observableElementList);
SortedList<T> sortedList = new SortedList<T>(filterList, null);

KiBodyLayerStack<T> bodyLayer =
new KiBodyLayerStack<T>(sortedList, rowIdAccessor, propertyNames, configRegistry, columnGroupModel,
columnPropertyAccessor);

ListDataProvider<T> bodyDataProvider = bodyLayer.getBodyDataProvider();
propertyChangeListener = bodyLayer.getGlazedListEventsLayer();

// blinking
// registerBlinkingConfigCells(configRegistry);
selectionLayer = bodyLayer.getSelectionLayer();

moveCommandHandler = new KiMoveCellSelectCommandHandler(selectionLayer);
// Column header
KiColumnHeaderLayerStack<T> columnHeaderLayer =
new KiColumnHeaderLayerStack<T>(sortedList, filterList, propertyNames, propertyToLabelMap, bodyLayer,
bodyLayer.getSelectionLayer(), columnGroupModel, configRegistry, columnPropertyAccessor);

they don't want separate multi-page, is there a good way for that? don't need to modify so much codes.

is it possible save the table's data in local file? now i need to a eventList, when put the data to "list",it already shows a error, so maybe we can't use glazedlist, and implement all of interfaces by ourselves, however i think it is a huge works.
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1369537 is a reply to message #1369528] Tue, 20 May 2014 16:31 Go to previous messageGo to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
Hi dirk,
can you give me some advices about that?
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1369879 is a reply to message #1369537] Tue, 20 May 2014 19:42 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 1407
Registered: July 2012
Senior Member
Hi,

sorry but NatTable in its current architecture does not come with memory management features. It is a "simple" control to visualize data. As there are currently no scroll events that would allow to add/remove elements by default, any solution would be complicated.

You could try to implement scroll listeners that lazy loads the data, and in your case more important removes the data if it is not visible.

But maybe an easier solution is to set the Xmx and Xms JVM settings to be bigger.

http://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1371016 is a reply to message #1369879] Wed, 21 May 2014 07:18 Go to previous messageGo to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
-------------------
You could try to implement scroll listeners that lazy loads the data, and in your case more important removes the data if it is not visible.
-----------------------------
Hi dirk,
Thank you for your reply. it can't fix this problem by set memory size, because user can display any huge data in the table.
At first,i got a wrong information about nattable,they said nattable already implemented lazy loads the data. if i want to implement the lazy loads data,whether i can't use all of glazedlist implement class.(org.eclipse.nebula.widgets.nattable.extension.glazedlists) or others way? can you give me some details information?

if the nattable has the "filter" function, whether we need to put all of the data to a list? if that is true,it still has the memory problem.(because the memory has a size).

how control "more important removes the data if it is not visible"? and don't effect nattable all of functions.
if i am wrong,please forgive me. thank you very much.
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1371051 is a reply to message #1371016] Wed, 21 May 2014 07:35 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 1407
Registered: July 2012
Senior Member
Quote:
they said nattable already implemented lazy loads the data


Then "they" are wrong. You could implement that within a custom IDataProvider, but it is not a feature NatTable comes with.

Quote:
if i want to implement the lazy loads data,whether i can't use all of glazedlist implement class


GlazedLists is "just" a java.util.List implementation. A quite fast implementation when it comes to filtering and sorting as it using fast algorithms on a list view instead of operating on the underlying collection itself. And it comes with the tree feature. Lazy loading is about adding elements to the list, so I don't see a reason why you shouldn't be able to use GlazedLists.

Quote:
if the nattable has the "filter" function, whether we need to put all of the data to a list? if that is true,it still has the memory problem.(because the memory has a size).


In general you are right. And it is part of your lazy loading IDataProvider. You could try to implement a filter that operates directly on a database instead of filtering in memory for example.

Quote:
how control "more important removes the data if it is not visible"? and don't effect nattable all of functions.


If you are not implementing sorting, filtering, etc. with implementations that operate against a database, you are not able to solve it.

Again:
- NatTable is a framework to visualize data in a table/grid/tree representation.
- It is able to handle large data sets very fast because of its virtual nature and the usage of GlazedLists in some cases.
- All provided functions operate in-memory.
- Its API allows to create mechanisms that operate on other datasources like e.g. a database, but that requires you to do something. It is not part of NatTable core!

We could think of an extension that also supports directly attaching a database, but that doesn't exist at the moment.
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1374717 is a reply to message #1371051] Thu, 22 May 2014 19:25 Go to previous messageGo to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
Hi Dirk,
thank you very much. i will try it first.

-------------------------------------------------
Lazy loading is about adding elements to the list, so I don't see a reason why you shouldn't be able to use GlazedLists.
-----------------------------------------------------------
I amn't sure about that, but i remember i did some tests about Nattable's example beore, i guess if we still push data to glazedlist,it will show "OutOfMemoryError", because it don't save the data in some temp file.
now i test Nattable example "_900_Everything_but_the_kitchen_sink.java", i just modify "DATA_SIZE" to 5Million
private static final int DATASET_SIZE = 5200000;
and run this example, when load about 2 millon datas, it will show this error.
so i guess


java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.eclipse.nebula.widgets.nattable.layer.CompositeLayer.getCellByPosition(CompositeLayer.java:495)
at org.eclipse.nebula.widgets.nattable.layer.AbstractLayerTransform.getCellByPosition(AbstractLayerTransform.java:291)
at org.eclipse.nebula.widgets.nattable.layer.AbstractLayer.getBoundsByPosition(AbstractLayer.java:310)
at org.eclipse.nebula.widgets.nattable.layer.CompositeLayer.getBoundsByPosition(CompositeLayer.java:520)
at org.eclipse.nebula.widgets.nattable.layer.cell.AbstractLayerCell.getBounds(AbstractLayerCell.java:68)
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1374720 is a reply to message #1374717] Thu, 22 May 2014 19:30 Go to previous messageGo to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
I will try to implement it like this´╝î replace the glazedlist by ourserves list(implements EventList), i will save data to a temp file when the list is so big(eg bigger than 1 million). just a idea, i will try it later.
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1385918 is a reply to message #1371051] Thu, 12 June 2014 09:24 Go to previous messageGo to next message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
Hi Dirk,
can you tell me how implement the filter without the glazedlist? i found you said "use IFilterStrategy" in another case, can you tell me some detail informations for that?
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1385931 is a reply to message #1385918] Thu, 12 June 2014 10:15 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 1407
Registered: July 2012
Senior Member
The FilterRow is part of NatTable Core. It needs a IFilterStrategy to work. We suggest to use GlazedLists and provide a DefaultGlazedListsFilterStrategy which operates on a GlazedLists FilterList on applying a filter.

You need to implement a custom IFilterStrategy if you want to implement filtering without GlazedLists. What kind of filter algorithm you are using and which way you want to use is up to you. There are several ways to filter data in memory. You could even implement a custom data provider that filters data (saying that it doesn't return rows that should be filtered). But I don't think that is better looking at performance for huge data sets.
Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1386058 is a reply to message #1385931] Fri, 13 June 2014 09:22 Go to previous message
neal zhang is currently offline neal zhangFriend
Messages: 35
Registered: July 2012
Member
Hi Dirk,
thank you for your reply, at first i used the glazedlist, when testing data is huge,eg. 10 million rows, it will show a memory error in GC(About glazedlist, it has a list which save the data,user can show any huge data in our product. so i think use the glazedlist will show memory error in some situations,right? maybe i just understand part of functions for nattable,if i misunderstand, sorry).
however i implemented a custom data provider, when show 50 million rows data,it still works, i will implment a custom IFilterStrategy, thank you very much.
Previous Topic:Filtering grouped columns
Next Topic:Listeners for scrolling
Goto Forum:
  


Current Time: Tue Nov 25 23:36:35 GMT 2014

Powered by FUDForum. Page generated in 0.02434 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software