Re: [geomesa-users] Temporal filter bug? Temporal filtering seems to be

[Jim Hughes <jnh5y@xxxxxxxx>]

Hi Beau,

Good question about access to the GeoMesa Jira.  Somehow I thought it was public, but I was mistaken.  I think Chris or Anthony may be able to help with the Jira administration.

So far, no one has asked for a 'delete-all' GeoMesa tables function.  Honestly, we haven't really considered that anyone might want to delete their finely indexed spatio-temporal data;)

With some seriousness, we consider GeoMesa to be akin to PostGIS.  I don't think there's a single command which will drop any and all PostGIS indexes.  That said, I'm excited about the command line tools which are coming soon.  Some of the initial features will let one list features cataloged and delete a given feature type.  I imagine we'll be able to chain the calls together to blow away (most*) all your GeoMesa tables.

Great suggestions.  Are there other things we can be providing in the 1.0 release?

Thanks,

Jim

* We are using one table for as a 'catalog' table which stores metadata information.  Without this table, we wouldn't know what features are in what tables.  At the minute, I hesitate to think about deleting that table.  Once all other GeoMesa tables have been deleted, it should remain and be empty. 

On 08/01/2014 09:33 AM, Beau Lalonde wrote:

Jim,

 

Thanks for the reply and the update about the temporal filtering bug.  Also thanks for the accumulo shell hints. 

 

Can you tell me how I can get an account on the JIRA page so that I can view the GEOMESA JIRA issues?

 

For other people’s reference, I have started to try out the deleteSchema() method within AccumuloDataStore.  Here is some Java example code:

    import geomesa.core.data.AccumuloDataStore;

    …

    DataStore dataStore = DataStoreFinder.getDataStore(params);

    ((AccumuloDataStore) dataStore).deleteSchema(featureName, numThreads);

 

Is there a convenient method to remove all GeoMesa-created accumulo tables, rather than having to specify each feature name one-by-one?  I’m just curious.

 

Thanks,

Beau

 

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Thursday, July 31, 2014 5:35 PM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Temporal filter bug? Temporal filtering seems to be truncated to the second.

 

Beau,

I've had a little bit of time to figure out why temporal filtering is being rounded to the second.  When an ECQL filter is serialized with ECQL.toCQL, the date is recorded without the millis.  I'll follow-up with GeoTools, and I've opened https://geomesa.atlassian.net/browse/GEOMESA-243.

In terms of splitting out the tables, we may have swung a little wide in terms of creating 3 tables for each simple feature type.  If we can sort out the configuration overhead, perhaps it'd be reasonable to consider an approach where a feature type can have its own tables or being a bigger table. 

If you are looking to delete old GeoMesa tables, I'd point you to deleteSchema(String featureName, Int numThreads) on AccumuloDataStores.  Since writing Java program to drop 3 tables would be a little tedious, we're developing a command line tool to do some of the basic maintenance one might want to do.  Hopefully we'll have something sorted tomorrow or early next week.

In terms of regex and/or more powerful Accumulo shell use, I'd point to 'accumulo -u user -p pass -f /path/toScript' where you can put shell commands in that file.  The output of the commands is piped to stdout, so you can write the first script to call 'tables'.  Once you have the list of tables, you can use your favorite chain of tools/editors to cook up a second script to delete tables or do whatever repetitive task.

As usual, great questions; thanks!

Jim

On 07/31/2014 12:31 PM, Beau Lalonde wrote:

Jim,

 

Thanks for the response.  My workaround for the temporal truncating bug right now is to add a second to the time that I use along with the BEFORE portion of my query.  For my purposes, this workaround has no ill effects.

 

I am definitely setting the "SF_PROPERTY_START_TIME" in userData to the “myTime” attribute (see example in my original post).  I am doing it in a way analogous to the following which is how I’ve been doing it for a while now:

    SimpleFeatureType featureType = null;

    … // instantiate featureType

    featureType.getUserData().put(Constants.SF_PROPERTY_START_TIME, “myTime”);

 

As far as the “explainQuery” function/feature in Java.  I would be content setting the log level for the geomesa.core.index.IndexQueryPlanner class to TRACE in order to see the explain query output.  I would probably only do this during development/debugging.

 

I do see that you have added ‘secondary’ indexes for non-spatio-temporal attirbutes.  I notice that the latest version of GeoMesa generates multiple accumulo tables for each feature type, e.g.

    TableName

    …

    TableName_FeatureName_attr_idx

    TableName_FeatureName_records

    TableName_FeatureName_st_idx

 

I am not an accumulo expert, so I yield to your expertise.  But I thought I would ask if it is necessary to create separate accumulo tables for the “_attr_idx”, “_records”, and “_st_idx” tables instead of simply creating new feature types within the original accumulo table?  I ask because I find that I now have tons of GeoMesa-created accumulo tables which seem to clutter things up.  I have been sticking a lot of different feature types in the same GeoMesa/accumulo table and I used to only have one GeoMesa-created accumulo table.  Now I have 34 GeoMesa-created accumulo tables.  Again, if indexing/querying is faster using multiple tables, then I probably need to get over this “clutter”.  What are your thoughts?

 

Is there a quick command to delete accumulo tables using a regular _expression_ via the accumulo shell?

 

Thanks,

 

Beau

 

 

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Wednesday, July 30, 2014 5:11 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Temporal filter bug? Temporal filtering seems to be truncated to the second.

 

Hi Beau,

Great question.  Let me start with two general notes:

First, our current goal is to support ECQL as implemented by GeoTools.  By that I mean that if you have a collection of features, add them to GeoMesa and then apply a filter to the collection and to GeoMesa, you should see the same results.  In a quick test, I was able to use your ECQL and cook up a SimpleFeature with a Date between the end points of the filter.  The filter did accept the feature, so, yes, I'm calling this a bug.*

Second, as a more general note, we have added an "explainQuery" function/feature recently so that users (and developers) can take a peek under the hood and see how their query is being planned and executed.  Some notes about how to use it from the Scala console are in the commit message: https://github.com/locationtech/geomesa/commit/775634b831255f0f787415decd1e705dbbbd50af  The query explanation is also available when the log level for the class geomesa.core.index.IndexQueryPlanner is set to TRACE.

At the moment, I'm working on how we plan queries from the ECQL filter.  We have also added 'secondary' indexes for non-spatio-temporal attributes.  In the future, we are considering allowing additional spatio- and/or temporal- indexes (with different resolutions).  As that work continues, the query explainer will be able to describe why we choose to treat a query in one way or another. 

As a practical note about the query explainer, I haven't had a chance to think through using it from Java.  If you try it out and there are inter-op issues, let us know.

* Back to your original question:  Hunter is working up some unit tests.  From a cursory look, I'd ask if you are setting the "SF_PROPERTY_START_TIME" in userData on the SimpleFeatureTypes.  While encoding/ingesting data, if there was miscommunication about the name of the time field to index, GeoMesa might not be encoding the Date in the index entries.  Without the userData set, on the query side of things, we may misunderstand the query and do something silly.

Thanks again,

Jim

On 07/30/2014 03:40 PM, Beau Lalonde wrote:

Hi,

 

We just installed the latest GeoMesa (actually it was from yesterday).  I am happy to report that I have noticed that several bugs have been fixed since the mid-June version; however, I have noticed a new bug related to temporal querying.

 

I am indexing the temporal component of my data using a Date object, and I am querying using CQL analogous to the following:

        ((myTime BEFORE 2014-07-30T19:29:07.917Z) AND (myTime AFTER 2014-07-30T19:29:07.519Z))

 

It appears to me that the temporal bounds of my above CQL query are truncated to the second before the query is performed.  I have verified that the actual Date data in GeoMesa is not truncated or modified.

 

For example, the following scenario fails to return any query results (even though logically it should):

      Indexed time: 2014-07-30T19:29:07.520Z

      CQL query string: ((myTime BEFORE 2014-07-30T19:29:07.917Z) AND (myTime AFTER 2014-07-30T19:29:07.519Z))

      My guess at the effective query string based upon observation: ((myTime BEFORE 2014-07-30T19:29:07) AND (myTime AFTER 2014-07-30T19:29:07))

 

But the following scenario does return results as expected:

      Indexed time: 2014-07-30T19:34:40.746Z

      CQL query string: ((myTime BEFORE 2014-07-30T19:34:41.143Z) AND (myTime AFTER 2014-07-30T19:34:40.745Z))

      My guess at the effective query string based upon observation: ((myTime BEFORE 2014-07-30T19:34:41) AND (myTime AFTER 2014-07-30T19:34:40))

 

 

I know that most people are probably not querying on a per-millisecond basis, but my code does.  Is this a GeoMesa bug?  Or do I need to modify my code to reflect that querying can only be performed at a resolution of one second?

 

Thanks in advance,

 

Beau

_______________________________________________

geomesa-users mailing list

geomesa-users@xxxxxxxxxxxxxxxx

To change your delivery options, retrieve your password, or unsubscribe from this list, visit

http://www.locationtech.org/mailman/listinfo/geomesa-users

 

_______________________________________________

geomesa-users mailing list

geomesa-users@xxxxxxxxxxxxxxxx

To change your delivery options, retrieve your password, or unsubscribe from this list, visit

http://www.locationtech.org/mailman/listinfo/geomesa-users

 

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users