Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Scout » Custom Table Sorting(Sort not only the visible rows in the table, but fire a sorted query based on the column's built-in sort properties (multisort))
Custom Table Sorting [message #1713202] Mon, 02 November 2015 10:29 Go to next message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
Hi,

As some of you might know, I'm writing a Scout application to visualize data coming from Elasticsearch. I implemented a table paging mechanism which only shows a defined amount of rows on one page. When changing the displayed page, a new filtered query is generated that fetches the data with the proportionate offset.
Now I want to (multi-)sort my results by adding the sort configurations to the Elasticsearch query instead of only sorting the displayed results on the page. Building the query itself is a no-brainer with Elasticsearch...
My problem is that I would like to use the Scout sorting features in order to set up the sorted query and perform the appropriate header decorations. It also cool that the table configurations are persistent through the sessions. I would resent implementing this all by myself again.
That's why I adapted the default OrganizeColumnsForm to fit my needs (basically, I only left out the filter section), set the table's isSortEnabled to 'true' and overwrote the sort() method of the table where iterate over the sort columns and build a new list with the sort configurations for Elasticsearch. The result is an incomprehensible mess: Sometimes it works as desired, but most of the time it doesn't. The column header highlighting doesn't work properly and the organization form doesn't show the correct sort configuration that is actually stored in the column set's sort columns list. When I comment out the queryHelper.setSorting part, the scout default sorting works as It should.
Basically I thought the column highlighting etc. would work independently from the the actual sort algorithm (or sort query in my case) and it would be enough to overwrite the sort() method, but this is obviously wrong. On my implementation you will notice that I call the super implementation at first. This was just a makeshift to have the column header decorations correctly set, but might be a reason for the confusion.

So how would you implement something like this in Scout? Maybe it would be easier, to disable sorting for the table and implement the sorting completely on my own. If this is the recommended approach, I would appreciate some hints how to get the correct header events and how to decorate the column header cells to visualize the sort configurations.

My current sort method:
          @Override
          public void sort() {
            super.sort();
            setTableChanging(true);
            if (isSortEnabled()) {
              List<IColumn<?>> sortColumns = getColumnSet().getSortColumns();
              Map<String, Boolean> sorting = new LinkedHashMap<>();
              for (IColumn<?> sortCol : sortColumns) {
                ITypedColumn typedSortcol = (ITypedColumn) sortCol;
                sorting.put(typedSortcol.getFieldName(), sortCol.isSortAscending());
              }
              try {
              // queryHelper is an Observable object which fetches the new data 
              // and triggers an update notification to it's Observers 
              // when the sorting is set.
              queryHelper.setSorting(sorting);
              }
              catch (Exception e) {
                logger.error("Unable to set sorting");
              }
              setTableChanging(false);
            }
          }

[Updated on: Wed, 25 November 2015 08:10]

Report message to a moderator

Re: Custom Table Sorting [message #1713218 is a reply to message #1713202] Mon, 02 November 2015 13:19 Go to previous messageGo to next message
Jeremie Bresson is currently offline Jeremie BressonFriend
Messages: 1252
Registered: October 2011
Senior Member
This seems to be a very interesting topic, but it is really hard to follow without being able to experiment and to see code.

Could you post some sketches (or annotated screenshots) corresponding to what you have described. I am not sure I have understood all your use cases (your expectations with several scout pages and ElasticSearch facets).

I never used ElasticSearch, can you recommend something like a 10 minutes "setup & hello world" example?
Is it possible for you to share more scout code?

I am really interested with this topic and I will try to help you as much as I can.
Re: Custom Table Sorting [message #1713329 is a reply to message #1713218] Tue, 03 November 2015 11:37 Go to previous messageGo to next message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
Well, Elasticsearch is a huge topic itself, but it shouldn't be that important in conjunction with my sort question: You can assume that the data coming from Elasticsearch is sorted the way I want it to be...
Nevertheless, I try to give a quick overview about what Elasticsearch actually is and how to get started.
As the name reveals, Elasticsearch is basically a powerful search engine, which allows you to set up clustered structures and load balancing very easily.
Elasticsearch is fed with any type of JSON documents via JSON over HTTP or the native Java API.
One of the strengths is the flexible type mapping. You don't need to have a schema predfined as you know from SQL databases. Elasticsearch is often described as schema-free. In fact this isn't completely true as the schema is either derived heuristically (dynamically) for the given document, or explicitly defined by the developer.
Here you find the official glossary of terms that helps you to understand the basic structure and Elasticsearch/Lucene-specific vocabulary a bit better.
To start an own instance of Elasticsearch you simply need to download a version, unpack it and start the executable in /bin/elasticsearch of the unpacked directory. If everything works fine, Elasticsearch should be listening on HTTP-Port 9200 already. You can test this with your browser. Browsing to "http://localhost:9200" should now show something like this:
{
  "status" : 200,
  "name" : "Shellshock",
  "cluster_name" : "el_hap",
  "version" : {
    "number" : "1.6.0",
    "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
    "build_timestamp" : "2015-06-09T13:36:34Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

The configurations can be made in the file "config/elasticsearch.yml". It could be useful to change the clustername of your instance at first, since instances in the same local network running the same clustername try to join the cluster and share data and queries per default. This could also be disabled by setting the property "discovery.zen.ping.multicast.enabled" to false.

Each indexed document is accesible through a RESTful URI following the schema:
http://hostname:port/indexname/doctype/docid

You can easily hand a document to elasticsearch via the HTTP-Api by settling a curl command like:
curl -X POST http://localhost:9200/addresses/wumba '{
    "name": "John Doe",
    "street": "Elm Street",
    "number" : "1",
    "city" : "Pokanasi",
    "postal" : 56368,
    "land" : "Rumbazumbi"
}'

and Elasticsearch search should answer with a success message.

What I want to achieve with my application is just a flexible tool to visualize the indexed documents like tools like Navicat etc. do with SQL Databases, whereas data modification is not needed at the moment. You should be able to set strucured queries and sort the results hierarchically. An old ugly plugin for Elasticsearch that does something similar is Elasticsearch-Head.
You can follow the installation instructions and test it, but I'm not sure right now, whether it is still compatible with the latest Elasticsearch release. I'm still using V1.6 in production.
Integration of Elasticsearch into a Scout application is also very easy as the Java-API is admittedly poorly documented but still very easy to use.
The backend of my application basically consists of one service processing the Elasticsearch queries.
I'm first building a tree structure of the form cluster name -> index name -> mapping/type name by adding them as Scout pages to the outline (see the tree on the left). After that, each of the Mapping/Type pages instantiates an own QueryForm when it is activated. The QueryForm shows a table which is generated based on the mapping I get from Elasticsearch. I therefor created an AbstractDynamicTable super class.
You can see the QueryForm on the right side in the first screenshot. Each QueryForm is connected to an QueryHelper instance via Observer/Observable semantics, which separates the UI from the business logic.
I now want to reconstruct the Scout multi-sort features (which only sort the rows that have already been loaded into the table) in a way that a new Elasticsearch-Query is generated when a column is sorted.
So I want to make Elasticsearch do my work:
Elasticsearch enables the user to set a size and an offset to only retrieve an excerpt of the whole result set of the query. So you can iterate through the result set by changing the offset send to Elasticsearch whereas the actual query stays the same. That's what I've done for the paging.
One of the problems of overwriting the sort()-method is that sort is called from several places in Scout and I don't have the complete control of it. This leads to weird behaviour especially with the presentation of the configured sorting.

Here an impression of my program, maybe this helps to understand, what I want.
[1]
index.php/fa/23812/0/
[2]
index.php/fa/23813/0/

As I'm writing this application for a company I need to be a bit careful about what to publish in the Internet. Maybe I will post a more detailed explanation some time.

[Updated on: Tue, 10 November 2015 08:23]

Report message to a moderator

Re: Custom Table Sorting [message #1713578 is a reply to message #1713329] Thu, 05 November 2015 09:40 Go to previous messageGo to next message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
My main problem at the moment is that the sort() method is called multiple times after a UI sort event. This confuses my Observer's update logic and at the end Scout's presentation of the sorting configurations (header highlighting etc.). Whereas I found that the highlighting is actually independent from the sort() method. I think, if a click on a column or a click on the Ascending/Descending button in the organization form would only result in one call of the sort() method I would be able to solve the problem to fit my needs.

The desired behaviour is:
1. click on column/button
2. the corresponding column is added to the scout-internal sort configurations (you can get them with getTable().getColumnSpecs().getSortColumns()); highlighting is done based on this configuration
3. the sort method is called (only once); the sort method sets the new sorting in the queryHelper (Observable)
4. the queryHelper sends a new query with the configured sorting to Elasticsearch
5. the queryHelper notifies its Observers of the changes (the table among others)
6. the table updates its data according to the new data in the queryHelper
Re: Custom Table Sorting [message #1714054 is a reply to message #1713578] Mon, 09 November 2015 15:37 Go to previous messageGo to next message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
Wow, sorting really drives me crazy. Shocked
Not calling the super implementation of sort() (see code below) causes the sort method to be called three times per click on the column header or the button in the organize columns menu. You can verify this by testing it with a simple application with only one form and a table. I did that too to make sure it's not just my application doing weird things.
On the other hand, if you call 'super.sort()' it is only executed once. So one of the statements in this implementation prevents multiple execution. Unfortunately, most fields and methods used in the default implementation are private and poorly documented, so that it's hard to find out what is actually happening or how to change behaviour by overwriting.
Again, I can't believe it is not possible to use Scout's provided sorting mechanism and replace just the way it actually sorted.
If you replace sort() with an empty implementation the UI presentation of the sorting still works consistently (certainly without any changes of the row order).
You can also view the order of the multisort columns in the organize columns form.
To repeat myself, my biggest problem at this point is that sort() is called multiple times and I can't find any solution to avoid this. But I'm afraid this is only the tip of the iceberg Confused

Ok, while writing this post I played around a little bit more and found an awful workaround to avoid multiple execution:
@Override
public void sort() {
setSortEnabled(false);
super.sort();
setSortEnabled(true);
}

Apparently it's enough to get 'm_sortValid' to be set to true Razz

But as supposed there are still some problems when I try to reload the new data into the table. As described in the previous posts I want to execute a new query instead of changig the order of the existing data set. Reloading the table has various side effects whose impacts I'm not yet able to completely comprehend.
For each query the table's column set is created dynamically based on the mapping of the corresponding elasticsearch index.
This reload mechanism disturbs the correct setting of the sort columns

The default sort() implementation in AbstractTable:
  @Override
  public void sort() {
    try {
      if (isSortEnabled()) {
        // Consider any active sort-column, not only explicit ones.
        // This is to support reverse (implicit) sorting of columns, meaning that multiple column sort is done
        // without CTRL-key held. In contrast to explicit multiple column sort, the first clicked column
        // is the least significant sort column.
        List<IColumn<?>> sortCols = getColumnSet().getSortColumns();
        if (sortCols.size() > 0) {
          // first make sure decorations and lookups are up-to-date
          processDecorationBuffer();
          List<ITableRow> a = new ArrayList<ITableRow>(getRows());
          Collections.sort(a, new TableRowComparator(sortCols));
          sortInternal(a);
        }
      }
    }
    finally {
      m_sortValid = true;
    }
  }

[Updated on: Tue, 10 November 2015 08:25]

Report message to a moderator

Re: Custom Table Sorting [message #1715561 is a reply to message #1714054] Tue, 24 November 2015 11:36 Go to previous messageGo to next message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
Hey guys,
custom sorting like described is still a problem in my application.
So the above posts still show the current state Confused It would be great, if someone could provide some hints, what's going wrong.

But I have another question: Where the heck does Scout store the table's sort configurations between the sessions and how can I disable that?
It is very confusing to send sorted queries to elasticsearch after program start whereas you haven't configured any sorting yet.

Best regards,
Janniki

[Updated on: Tue, 24 November 2015 14:40]

Report message to a moderator

Re: Custom Table Sorting [message #1715586 is a reply to message #1715561] Tue, 24 November 2015 13:40 Go to previous messageGo to next message
Marco Dörfliger is currently offline Marco DörfligerFriend
Messages: 46
Registered: January 2015
Member
I'm planning to trial using elasticsearch as the back-end for something in the upcoming weeks. I have to say though, I don't plan altering the features of scout's front end (sorting, column order, or pagination). Is it possible to achieve what you want by letting scout's default implementation handle that part?
Re: Custom Table Sorting [message #1715590 is a reply to message #1715586] Tue, 24 November 2015 14:30 Go to previous message
Jannik Malken is currently offline Jannik MalkenFriend
Messages: 44
Registered: October 2015
Member
This exactly what I am trying to achieve. If you are interested in this topic, please take the time to read my previous posts.
Even though this thread has drifted off towards Elasticsearch, it is more a problem with Scout.

As Scout is naturally quite static, you will have problems if you don't know which kind of data you will be dealing with at runtime. You will have to do a lot of persuation Very Happy I could give you some tips though.
On the other hand some Scout features work scarily easy and don't cause any problems.

In general, it is not a big problem to use Elasticsearch with Scout. You can define how the tables load their data by overwriting their parent field's 'execReloadTableData()' for example.

The paging you will have to implement on your own by the use of some additional form fields. The Elasticsearch part for that case is - as most of the time - very easy. You only need to pass the correct resultSize and offset and you will get the desired result, no matter what query you sent. There are just some client-side calculations needed to determine the correct interval and the appropriate page number.
Changing the column order in the UI by drag&drop or the ColumnOrganization form will work without problems. This comes with Scout per default and only needs to be enabled.

Sorting is, as far as I know (and this is the topic of this thread), limited to sort the data set that is already loaded into the table. For an application that loads data spread over various pages, this is make no real sense.
If you want to use the internal sort mechanism, but change the actual behavior at a sort event, you will face the same difficulties we're (or at least I am *g*) actually discussing in this thread.

[Updated on: Tue, 24 November 2015 14:39]

Report message to a moderator

Previous Topic:Scout and Apache Commons Collection Vulnerability
Next Topic:Neon M3 no Scout perspective
Goto Forum:
  


Current Time: Thu Mar 28 15:23:29 GMT 2024

Powered by FUDForum. Page generated in 0.02929 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top