Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » NatTable » Handling very large data with Nattable
Handling very large data with Nattable [message #1275332] Sat, 22 March 2014 23:03 Go to next message
Mohamed SADI is currently offline Mohamed SADIFriend
Messages: 38
Registered: December 2013
Member
Hello, i have a list of 30 000 entries (for example invoices), i've created a nattable with multiple layers (Group by, sort by, filtering). But when i try to load the 30 0000 entries, it's take more than 40 second. Can any one explain me the necessary step to load all this amount of entries faster ? or explain me the _900_Everything_but_the_kitchen_sink example to do the same think. Or is there a way to use pagination ?

[Updated on: Sat, 22 March 2014 23:14]

Report message to a moderator

Re: Handling very large data with Nattable [message #1275521 is a reply to message #1275332] Sun, 23 March 2014 06:47 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 2495
Registered: July 2012
Senior Member
There is no pagination in NatTable.

The kitchen sink example loads more data AFAIK.

Are you sure that it's NatTable that takes the time and not the loading of data into the memory?

Are you applying some initial group by?
Re: Handling very large data with Nattable [message #1275522 is a reply to message #1275332] Sun, 23 March 2014 06:49 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 2495
Registered: July 2012
Senior Member
There is no pagination in NatTable.

The kitchen sink example loads more data AFAIK.

Are you doing some initial group by on startup?
Re: Handling very large data with Nattable [message #1275590 is a reply to message #1275522] Sun, 23 March 2014 09:21 Go to previous messageGo to next message
Mohamed SADI is currently offline Mohamed SADIFriend
Messages: 38
Registered: December 2013
Member
I have two problems, loading data to the memory (hibernate) take 3 seconds and defining all the layers before instantiate the nattable take 4-5 seconds (where the body layer stack takes 2 seconds). For the nattable creation, i have an abstract class that contains all the common thinks to create a nattable, for each Business Object list (like invoices), i extend the abstract class where i define the bodylayer stack and columns mapping ....

I have observed that the first loaded Business object list with nattable take 4-5 seconds (where the body layer stack takes 2 seconds) for defining layers but when i loaded other businnes object list it takes <= 1 second with the same amount of data entries.

Here is the code where i define the body layer stack :

public BodyLayerStack(List<T> values,
				IColumnPropertyAccessor<T> columnPropertyAccessor,
				ConfigRegistry configRegistry) {
			// wrapping of the list to show into GlazedLists
			// see http://publicobject.com/glazedlists/ for further information
			long startTime= System.currentTimeMillis();
			this.eventList = GlazedLists.eventList(values);
			TransformedList<T, T> rowObjectsGlazedList = GlazedLists
					.threadSafeList(eventList);

			// use the SortedList constructor with 'null' for the Comparator
			// because the Comparator
			// will be set by configuration
			this.sortedList = new SortedList<T>(rowObjectsGlazedList, null);
			// wrap the SortedList with the FilterList
			this.filterList = new FilterList<T>(sortedList);

			// Use the GroupByDataLayer instead of the default DataLayer
			this.bodyDataLayer = new GroupByDataLayer<T>(getGroupByModel(),
					filterList, columnPropertyAccessor, configRegistry);
			// get the IDataProvider that was created by the GroupByDataLayer
			this.bodyDataProvider = bodyDataLayer.getDataProvider();

			// layer for event handling of GlazedLists and PropertyChanges
			GlazedListsEventLayer<T> glazedListsEventLayer = new GlazedListsEventLayer<T>(
					bodyDataLayer, filterList);

			ColumnReorderLayer columnReorderLayer = new ColumnReorderLayer(
					glazedListsEventLayer);
			ColumnHideShowLayer columnHideShowLayer = new ColumnHideShowLayer(
					columnReorderLayer);
			this.selectionLayer = new SelectionLayer(columnHideShowLayer);

			// add a tree layer to visualise the grouping
			TreeLayer treeLayer = new TreeLayer(selectionLayer,
					bodyDataLayer.getTreeRowModel());

			ViewportLayer viewportLayer = new ViewportLayer(treeLayer);

			setUnderlyingLayer(viewportLayer);
			long stop= System.currentTimeMillis();
			
			System.out.println("Body layer stack take : "+(stop-startTime)/1000+" seconds");
		}




Re: Handling very large data with Nattable [message #1276178 is a reply to message #1275590] Mon, 24 March 2014 07:37 Go to previous messageGo to next message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 2495
Registered: July 2012
Senior Member
There is the _706_SortableGroupByWithFilterExample example in the repository. If I set the number of items to load to 30.000, the creation takes less than 1 second.

IMHO the issue is somewhere in your code. Maybe your column accessor or somewhat else.

NatTable was created to deal with large amount of data. That's the main intention of the widget. So I don't believe that it is NatTable code that is so slow! Only the group by feature is not the fastest for huge data sets, which is related to the feature itself, as creating the dynamic tree structure needs to iterate over the whole list.

Please check your code first, the creation of the layer stack looks like some of the examples. The only unknown thing is your IColumnPropertyAccessor.
Re: Handling very large data with Nattable [message #1276417 is a reply to message #1276178] Mon, 24 March 2014 14:42 Go to previous messageGo to next message
Mohamed SADI is currently offline Mohamed SADIFriend
Messages: 38
Registered: December 2013
Member
Hello, thank you very much for your help, i've rerunned the _SortableGroupByWithFilterExample example, this method createExampleControl takes 5581 ms. I have a Mac Book Pro Core 2 duo 2.53 GHZ with 8 Go of ram and ths os is Mac os x 10.6.8.
Re: Handling very large data with Nattable [message #1276465 is a reply to message #1276417] Mon, 24 March 2014 15:46 Go to previous message
Dirk Fauth is currently offline Dirk FauthFriend
Messages: 2495
Registered: July 2012
Senior Member
On my machine createExampleControl takes about 1700ms. I have an Intel i7 2.7 GHz with 8GB RAM, OS is Windows 7 64bit.

I'm not sure what's the issue in your environment, but I don't think it's NatTable related. Otherwise the main purpose of NatTable would be broken and all of the NatTable users would stop using it, as it doesn't do what it is intended to do.
Previous Topic:Modifications to tree rendering
Next Topic:How can summaryrow change immediately after filter
Goto Forum:
  


Current Time: Fri Sep 22 22:38:13 GMT 2017

Powered by FUDForum. Page generated in 0.02019 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software