For sure I see
that the key changes on the date (it is the date that I am
using for the simple feature type to be part of that z3
index) but when I issue the WFS call (and provide date to
try and trigger z3) it still just returns one record and not
the duplicates from what I can tell.
I was using the
java api to add features so I was getting a FeatureStore off
of the DataStore and so I am not sure that is what you mean
about using an updating feature writer. Is there a way to
do that when accessing ingest via the java api? Or is there
another method to do what you are suggesting? So where I
was left was try to identify the modification records and
then really remove and then add them again vs records that
are new. So how does one use an updating feature writer.
Thanks,
Diane
Hi Diane,
A 'key' in accumulo consists of a row, a column family, a
column qualifer, a timestamp and a visibiility marking. In
general, timestamps are 'hidden' by the versioning iterator,
which always applied by default. Thus, inserting a key with
the same row/cf/cq/vis but a newer timestamp will overwrite an
older timestamp. Technically the older value is still there,
but it will be hidden and eventually compacted out by
accumulo.
The key used to write in geomesa depends on the particular
index, but it will always include the feature ID (FID). The Z3
index also includes the default date and geometry.
In general, to be safe, we recommend always using an updating
feature writer if you are modifying an existing feature. That
can be slow though, so sometimes you can get away with
appending, and relying on the versioning iterator to clean up
the duplicates. But if any of the indexed values (date,
geometry, indexed attributes) change, you will end up with a
different key in some of the indices, and thus duplicate data.
I'm not sure why you would be seeing no change with the date
modification - maybe it's not the date being indexed? I wrote
a quick script to check the z3 row keys generated for a few
different inputs. You can see the key is changing each time:
original
Sat Dec 31 19:00:00 EST 2011, POINT (45 45)
%00;%08;%8f;5`%90;H$%12;%09; fid0
lon +1
Sat Dec 31 19:00:00 EST 2011, POINT (46 45)
%00;%08;%8f;5`%90;Xl6%09;!fid0
lat +1
Sat Dec 31 19:00:00 EST 2011, POINT (45 46)
%00;%08;%8f;5`%91;L%a6;R%09;2fid0
dtg +1h
Sat Dec 31 20:00:00 EST 2011, POINT (45 45)
%00;%08;%8f;5`%92;@$%90;%09;$fid0
dtg +1d
Sun Jan 01 19:00:00 EST 2012, POINT (45 45)
%00;%08;%8f;<D%02;%01;%00;%80;@ fid0
On 02/10/2017 01:01 PM, Diane Griffith
wrote:
We have data that will see updates to its
data. These updates can be to any of the fields including
the time field as well as the latitude and longitude fields
used to create the default Point geometry.
What we have observed is if the datetime
changed, even over a month difference, we still just got one
record back via WFS regardless of how many were still in the
z3 table and the record got updated the z2 and the records
tables in accumulo.
Again if we updated the latitude value,
even a slight change to it, and added the feature again
(instead of modify the feature or remove then add), then we
got 2 copies of the record that have the same id field that
we hint to use for FID.
I found an old post that sort of talked
to this around how a Versioning Iterator is configured to
return 1 record for scan time and both minor and major
compactions and that had to so specifically for the Accumulo
side of the question. Then the response went on to talk to
the GeoMesa side of it and asked do the 2 copies of a
SimpleFeature have the same Feature ID? If yes then
GeoMesa will write the same Accumulo keys for the data. If
not then different keys will be written. So what does that
mean, Feature ID…is that the field hinted to use for FID or
is that the 3 pieces that make up the index key?
So why does time field differences not
impact duplicates coming back from WFS requests (using z3
index) but changes in latitude and longitude do? So is the
recommendation if updates to the latitude/longitude/point
data will happen then to identify the record is changed and
either modify or remove/add?
We are seeing this in GeoMesa 1.2.7.2
against Accumulo 1.6.2.
Thanks,
Diane Griffith
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
