Re: [geomesa-users] Possible problems and issues re updating from 1.2.1

[Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>]

Hi Ben,

1. In 1.3 we've dropped support for the old geohash index used in 1.2.1. Queries will continue to work, but if you query something that would have hit the geohash index (e.g. spatial query without a time component) then it will be extremely slow, especially with 40B records. If all your queries are hitting the Z3 index (spatial + time), record index (feature ID), or attribute index then you will be ok.

1a. Indexing strategies have improved, but the index format is set when you call createSchema. Thus, you likely won't see much query improvement, although you will get any bug fixes. The indices associated with different geomesa versions is outlined here:

http://www.geomesa.org/documentation/user/accumulo/data_management.html#upgrading-existing-indices

An explanation of the different indices we currently support is here:

http://www.geomesa.org/documentation/user/datastores/index_structure.html

To get the best performance, we provide a process to migrate data to a newer index format through our command line tools, described here:

http://www.geomesa.org/documentation/user/accumulo/commandline_tools.html#add-index-command

Given the size of your data, this may or may not be feasible - if you do want to migrate indices it may be better to create a new schema and manually backfill a few months' worth of data at a time.

2. Yes, CQL queries should yield the same results

3. As we use GeoTools as our main abstraction, the general API has not changed. If you use more geomesa specific features, then certain things may have changed. If you can describe your current process I can provide more details. We have upgraded to GeoTools 15 and GeoServer 2.9, so you would probably want to upgrade to those versions as well.

4. Date processing has not changed - the data binding is java.util.Date, but all our date operations use joda time classes. This was true in 1.2.1 as well.

Thanks,

Emilio

On 02/12/2017 12:49 PM, Benjamin Weaver wrote:

Hi all,

We would like to identify the problems and issues we are likely to face when we upgrade from Geomesa 1.2.1 to 1.3.0. I ask our biggest questions below, but would greatly appreciate information about any other problems you know of.

1. Will our existing data, ingested via geomesa 1.2.1, work correctly in 1.3.0? By 'work' I mean can we query this data properly, and will this older data set be fully co-functional and compatible with data ingested via 1.3.0?

Some context: Our largest and most important table defines a single feature (~40 billion records and growing), and uses an additional attribute index (full) one 1 of our own fields, a 'uuid'.

1a. Do the indexing strategies of the two geomesa versions differ?  (I notice that 1.2.1 creates both [featurename]_z3 and an st_idx indexes). Are featureIDs created in the same way?

2. Will CQL queries yield the same results in 1.3.0 (assuming the same data and query string)?

3. What are the major differences in API that will bear on our ingest and query of data in 1.3.0?

4. 1.2.1 uses java.util.Date. Does 1.3.0 use joda DateTime classes? If so, what bearing does this difference have on date handling in the later geomesa version?

[any and all comments and perspectives are welcome. Our apologies in advance if necessary--we could not just now reach the geomesa archives to consult possible previous answers to my questions] 

Thank you in advance,

Ben Weaver

This email (and any attachments) may contain confidential information and is intended solely for the recipient(s) to whom the email is addressed. If you received this email in error, please inform us immediately and delete the email and all attachments without further using, copying or disclosing the information. This email and any attachments are believed to be, but cannot be guaranteed to be, secure or virus-free. Satellite Applications Catapult Limited is registered in England & Wales. Company Number: 7964746. Registered office: Electron Building, Fermi Avenue, Harwell Oxford, Didcot, Oxfordshire OX11 0QR.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users