Home » Eclipse Projects » NatTable » How fix "OutOfMemoryError" when Nattable contain huge data
| How fix "OutOfMemoryError" when Nattable contain huge data [message #1369528] |
Tue, 20 May 2014 16:30  |
 neal zhang
Messages: 45
Registered: July 2012 |
Member |
|
|
Hi,
I use nattable for display, it also contains the "Filter" function, sometimes the table has more than ten million rows data. i use a "baseEventList ", every time i add the data to eventlist.
error messages:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2245)
at java.util.Arrays.copyOf(Arrays.java:2219)
at java.util.ArrayList.grow(ArrayList.java:213)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:187)
at java.util.ArrayList.add(ArrayList.java:411)
at ca.odell.glazedlists.BasicEventList.add(BasicEventList.java:138)
at ca.odell.glazedlists.impl.ThreadSafeList.add(ThreadSafeList.java:177)
at cn.kisters.kiscript.rcp.views.ResultTableModel.fillNext(ResultTableModel.java:197)
codes:
baseEventList = GlazedLists.threadSafeList(GlazedLists.eventList(rowData));
ObservableElementList<T> observableElementList = new ObservableElementList<T>(baseEventList, connector);
FilterList<T> filterList = new FilterList<T>(observableElementList);
SortedList<T> sortedList = new SortedList<T>(filterList, null);
KiBodyLayerStack<T> bodyLayer =
new KiBodyLayerStack<T>(sortedList, rowIdAccessor, propertyNames, configRegistry, columnGroupModel,
columnPropertyAccessor);
ListDataProvider<T> bodyDataProvider = bodyLayer.getBodyDataProvider();
propertyChangeListener = bodyLayer.getGlazedListEventsLayer();
// blinking
// registerBlinkingConfigCells(configRegistry);
selectionLayer = bodyLayer.getSelectionLayer();
moveCommandHandler = new KiMoveCellSelectCommandHandler(selectionLayer);
// Column header
KiColumnHeaderLayerStack<T> columnHeaderLayer =
new KiColumnHeaderLayerStack<T>(sortedList, filterList, propertyNames, propertyToLabelMap, bodyLayer,
bodyLayer.getSelectionLayer(), columnGroupModel, configRegistry, columnPropertyAccessor);
they don't want separate multi-page, is there a good way for that? don't need to modify so much codes.
is it possible save the table's data in local file? now i need to a eventList, when put the data to "list",it already shows a error, so maybe we can't use glazedlist, and implement all of interfaces by ourselves, however i think it is a huge works.
|
|
| | |
| Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1371016 is a reply to message #1369879] |
Wed, 21 May 2014 07:18  |
neal zhang
Messages: 45
Registered: July 2012 |
Member |
|
|
-------------------
You could try to implement scroll listeners that lazy loads the data, and in your case more important removes the data if it is not visible.
-----------------------------
Hi dirk,
Thank you for your reply. it can't fix this problem by set memory size, because user can display any huge data in the table.
At first,i got a wrong information about nattable,they said nattable already implemented lazy loads the data. if i want to implement the lazy loads data,whether i can't use all of glazedlist implement class.(org.eclipse.nebula.widgets.nattable.extension.glazedlists) or others way? can you give me some details information?
if the nattable has the "filter" function, whether we need to put all of the data to a list? if that is true,it still has the memory problem.(because the memory has a size).
how control "more important removes the data if it is not visible"? and don't effect nattable all of functions.
if i am wrong,please forgive me. thank you very much.
|
|
|
| Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1371051 is a reply to message #1371016] |
Wed, 21 May 2014 07:35 |
Dirk Fauth
Messages: 2902
Registered: July 2012 |
Senior Member |
|
|
Quote:they said nattable already implemented lazy loads the data
Then "they" are wrong. You could implement that within a custom IDataProvider, but it is not a feature NatTable comes with.
Quote:if i want to implement the lazy loads data,whether i can't use all of glazedlist implement class
GlazedLists is "just" a java.util.List implementation. A quite fast implementation when it comes to filtering and sorting as it using fast algorithms on a list view instead of operating on the underlying collection itself. And it comes with the tree feature. Lazy loading is about adding elements to the list, so I don't see a reason why you shouldn't be able to use GlazedLists.
Quote:if the nattable has the "filter" function, whether we need to put all of the data to a list? if that is true,it still has the memory problem.(because the memory has a size).
In general you are right. And it is part of your lazy loading IDataProvider. You could try to implement a filter that operates directly on a database instead of filtering in memory for example.
Quote:how control "more important removes the data if it is not visible"? and don't effect nattable all of functions.
If you are not implementing sorting, filtering, etc. with implementations that operate against a database, you are not able to solve it.
Again:
- NatTable is a framework to visualize data in a table/grid/tree representation.
- It is able to handle large data sets very fast because of its virtual nature and the usage of GlazedLists in some cases.
- All provided functions operate in-memory.
- Its API allows to create mechanisms that operate on other datasources like e.g. a database, but that requires you to do something. It is not part of NatTable core!
We could think of an extension that also supports directly attaching a database, but that doesn't exist at the moment.
|
|
|
| Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1374717 is a reply to message #1371051] |
Thu, 22 May 2014 19:25 |
neal zhang
Messages: 45
Registered: July 2012 |
Member |
|
|
Hi Dirk,
thank you very much. i will try it first.
-------------------------------------------------
Lazy loading is about adding elements to the list, so I don't see a reason why you shouldn't be able to use GlazedLists.
-----------------------------------------------------------
I amn't sure about that, but i remember i did some tests about Nattable's example beore, i guess if we still push data to glazedlist,it will show "OutOfMemoryError", because it don't save the data in some temp file.
now i test Nattable example "_900_Everything_but_the_kitchen_sink.java", i just modify "DATA_SIZE" to 5Million
private static final int DATASET_SIZE = 5200000;
and run this example, when load about 2 millon datas, it will show this error.
so i guess
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.eclipse.nebula.widgets.nattable.layer.CompositeLayer.getCellByPosition(CompositeLayer.java:495)
at org.eclipse.nebula.widgets.nattable.layer.AbstractLayerTransform.getCellByPosition(AbstractLayerTransform.java:291)
at org.eclipse.nebula.widgets.nattable.layer.AbstractLayer.getBoundsByPosition(AbstractLayer.java:310)
at org.eclipse.nebula.widgets.nattable.layer.CompositeLayer.getBoundsByPosition(CompositeLayer.java:520)
at org.eclipse.nebula.widgets.nattable.layer.cell.AbstractLayerCell.getBounds(AbstractLayerCell.java:68)
|
|
| | | |
| Re: How fix "OutOfMemoryError" when Nattable contain huge data [message #1386058 is a reply to message #1385931] |
Fri, 13 June 2014 09:22 |
neal zhang
Messages: 45
Registered: July 2012 |
Member |
|
|
Hi Dirk,
thank you for your reply, at first i used the glazedlist, when testing data is huge,eg. 10 million rows, it will show a memory error in GC(About glazedlist, it has a list which save the data,user can show any huge data in our product. so i think use the glazedlist will show memory error in some situations,right? maybe i just understand part of functions for nattable,if i misunderstand, sorry).
however i implemented a custom data provider, when show 50 million rows data,it still works, i will implment a custom IFilterStrategy, thank you very much.
|
|
|
Goto Forum:
Current Time: Thu Apr 25 13:35:42 GMT 2024
Powered by FUDForum. Page generated in 0.03160 seconds |
] 
Search
Help
Register
Login
Home
