Re: [geomesa-users] Possible bug with removing features, cannot add a fe

[Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>]

Hi Beau,

I think that this issue has been fixed with some of our recent refactoring. The following pseudo-code worked for me against a local accumulo mini-cluster. Could you try your test again with the latest branch? (note, you might need to pull a PR that hasn't been merged yet that fixes a bug in deleting: https://github.com/locationtech/geomesa/pull/141)

Thanks,

Emilio


  void deleteThenAdd() {
      DataStore dataStore = DataStoreFinder.getDataStore(params);
      SimpleFeatureStore featureStore = (SimpleFeatureStore) dataStore.getFeatureSource(sftName);

      String filterString = "...";
      Filter filter = CQL.toFilter(filterString);

      Query query = new Query(sftName, filter);

      // query returns a single feature
      SimpleFeature feature = this.query(featureStore, query);

      logger.info("removing feature");
      featureStore.removeFeatures(filter);

      // this query returns 0 features
      this.query(featureStore, query);

      logger.info("re-adding feature");

      DefaultFeatureCollection featureCollection = new DefaultFeatureCollection(sftName, feature.getFeatureType);
      feature.getUserData.put(Hints.USE_PROVIDED_FID, true);
      featureCollection.add(feature);

      featureStore.addFeatures(featureCollection);

      // returns the single feature again
      this.query(featureStore, query);
  }

  SimpleFeature query(SimpleFeatureStore featureStore, Query query) {
    int count = 0
    SimpleFeature first = null

    // execute the query
    SimpleFeatureIterator features = featureStore.getFeatures(query).features();
    while (features.hasNext()) {
      count = count + 1;
      if (first == null) {
        first = features.next();
      } else {
        features.next();
      }
    }
    logger.info("queried " + count + " features");
    return first
  }

On 07/21/2014 01:14 PM, Beau Lalonde wrote:

Anthony,

Thanks for the reply.

In my test and also in our real system we have witnessed this problem writing to accumulo from a single client (not using a map-reduce job).  I should mention that accumulo is not on the same machine as the client that we are using to write. 

In our real system I am confident that there is elapsed system time between the remove and add calls - in our junit test (partial code in original post), there might not be much elapsed time because of the simplicity of the test.

Thanks for your help,

Beau

Sent from my mobile device.

On Jul 21, 2014 8:13 AM, Anthony Fox <anthonyfox@xxxxxxxx> wrote:

Beau,

Yes, you definitely can add/remove/add without problems in Accumulo.  The only time you cannot do it without strange behaviour is if you set the timestamp exactly the same for each 'add'.  GeoMesa does not explicitly set the timestamp, so it should get the current system time for each mutation.  Can you tell me a bit more about how you are writing to GeoMesa?  Is it from the same client (with consistent system time) each time or is it from within a map-reduce job?  If the latter, do you have ntpd setup to synchronize system time?  I'll put together a test to see if we can reproduce your issue.

Thanks,
Anthony

On Fri, Jul 18, 2014 at 8:35 PM, Beau Lalonde <blalonde@xxxxxxxx> wrote:

Chris,

Thanks for the suggestion.  I'll try it out when I can.  I will test the add/remove/add in accumulo before I test in Geomesa.

I am not an accumulo expert, but it seems to me that an add (after a delete) should remove the cutting short of a scan that is caused by a delete.  If this is how accumulo works, it seems like an accumulo design flaw.  Intuitively it seems to me that you should be able to add/remove/add without any problems (and without having to wait for a major compaction) - analogous to adding a file to a folder, removing it, and then adding another one with the same name.  Unless, of course, the accumulo delete and add happens so fast that the accumulo timestamp for both are the same - which, if memory serves correctly, results in undefined behavior.

It may be a week or so before I can do more testing on this.

Thanks,
Beau

Sent from my mobile device.

On Jul 18, 2014 4:37 PM, Chris Eichelberger <cne1x@xxxxxxxx> wrote:

Beau,

There may be a reasonable explanation for what you're seeing in your
test.

Removing a row from Accumulo -- and I apologize for lecturing --
actually inserts a DELETE record in the table that cuts short subsequent
scans for matching keys.  Records are not actually removed (with the
corresponding DELETE markers) until a major compaction occurs.  Because
the new record contains the same indexed data (ID and geometry), it will
receive the same key as the entry it replaces, which means that it will
not be visible in the table until after major compaction.

To test this out, try scheduling a major compaction on your test table
(after the insertions and deletions run), and scan it to see if your new
id1 record is present.  My suspicion is that it will be.

Please let me know if this doesn't address your issue.

Thanks, as always!

Sincerely,
  -- Chris


On Fri, 2014-07-18 at 17:46 +0000, Beau Lalonde wrote:
> Hi,
>
> 
>
> I am using a mid-June version of GeoMesa.  Perhaps this issue has been
> fixed since mid-June, but if it hasn’t been I thought I should report
> it.
>
> 
>
> I am having trouble with the featureStore.removeFeatures(filter)
> capability in GeoMesa.  My problem is that I cannot seem to be able to
> add a feature once I have removed a feature with the same feature ID.
> When I invoke “featureStore.addFeatures(features)”, I do not get an
> exception; however, the feature is not added to GeoMesa.  I would like
> to be able to add/remove/add features with the same ID.
>
> 
>
> Here is some partial code that illustrates my problem.  Please note
> the line that I have commented with “this assertion fails”.
>
> 
>
>                 @Test
>
>                 public void testRemoveFeatureByID_AddRemoveAdd()
>
>                 {
>
>                                 // add three features, then remove one
> and verify its removal
>
>                                
>
>                                 String id1 = "ID1";
>
>                                 assertTrue(addFeature(id1, 45.1d,
> 123.4, 15L));
>
>                                 String id2 = "ID2";
>
>                                 assertTrue(addFeature(id2, 45.0d,
> 123.3, 16L));
>
>                                 String id3 = "ID3";
>
>                                 assertTrue(addFeature(id3, 45.2d,
> 123.5, 17L));
>
>                                
>
>
> this.geoMesaHelper.removeFeatureByID(GeoMesaHelperTest.FEATURE_NAME,
> id1);
>
>                                
>
>                                 Set<String> ids =
> this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, ""));
>
>                                
>
>                                 assertNotNull(ids);
>
>                                 assertEquals(2, ids.size());
>
>                                 assertTrue(ids.contains(id2));
>
>                                 assertTrue(ids.contains(id3));
>
>                                
>
>                                 // add back feature with id1 with
> different SKU
>
>                                 assertTrue(addFeature(id1, 45.1d,
> 123.4, 25L));
>
>                                
>
>                                 // add a uniquely new feature that
> hasn't been added before
>
>                                 String id4 = "ID4";
>
>                                 assertTrue(addFeature(id4, 45.25d,
> 123.55, 18L));
>
>                                
>
>                                 ids =
> this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, ""));
>
>                                                                
>
>                                 assertNotNull(ids);
>
>                                 assertTrue(ids.contains(id4));
>
>                                 assertTrue(ids.contains(id2));
>
>                                 assertTrue(ids.contains(id3));
>
>                                 assertTrue(ids.contains(id1)); // this
> assertion fails
>
>                                 assertEquals(4, ids.size());
>
>                 }
>
> 
>
>                 private boolean addFeature(String uniqueID, double
> latitude, double longitude, long sku)
>
>                 {
>
>                                 String featureSchema =
> "NAME:String,SKU:java.lang.Long,COST:Double,SELL_BY:Date," +
>
>                                                                 "*" +
> GeoMesaHelperTest.GEOMESA_GEOMETRY  + ":Geometry:srid=4326," +
>
>
> GeoMesaHelperTest.GEOMESA_TIME + ":Date";
>
> 
>
>                                 // Create feature
>
>                                 SimpleFeatureType featureType =
> this.geoMesaHelper.createFeatureType(dataStore,
> GeoMesaHelperTest.FEATURE_NAME, featureSchema,
> GeoMesaIndexer.GEOMESA_TIME_ATTRIBUTE_NAME);
>
>                                 SimpleFeatureBuilder
> simpleFeatureBuilder = new SimpleFeatureBuilder(featureType);
>
>                                 Object[] noValues = {};
>
>                                 SimpleFeature feature =
> simpleFeatureBuilder.buildFeature(uniqueID, noValues);
>
>
> feature.getUserData().put(Hints.USE_PROVIDED_FID,
> java.lang.Boolean.TRUE);
>
>                                
>
>                                 // create a geometric rectangle
> centered at the input latitude/longitude
>
>                                 double DLAT = 2.0d;
>
>                                 double DLON = 2.0d;
>
>                                 double minlat = latitude-DLAT/2.0d;
>
>                                 double minlon = longitude-DLON/2.0d;
>
>                                 double maxlat = latitude+DLAT/2.0d;
>
>                                 double maxlon = longitude+DLON/2.0d;
>
>                                 int srid = 4326;
>
>                                 Geometry geometry =
> GeoIndexerUtilities.createRectangleFromLatLon(minlat, minlon, maxlat,
> maxlon, srid);
>
>                                 feature.setAttribute(GEOMESA_GEOMETRY,
> geometry);
>
>                                
>
>                                 feature.setAttribute("NAME",
> GeoMesaHelperTest.NAME_ATTRIBUTE);
>
>                                 feature.setAttribute("SKU", sku);
>
>                                 feature.setAttribute("COST", 1.23);
>
>                                 feature.setAttribute("SELL_BY", new
> Date());
>
>                                 feature.setAttribute(GEOMESA_TIME, new
> Date());
>
> 
>
>                                 // Add feature to GeoMesa
>
>                                 FeatureStore<SimpleFeatureType,
> SimpleFeature> featureStore =
> this.geoMesaHelper.createFeatureStore(dataStore,
> GeoMesaHelperTest.FEATURE_NAME);
>
>                                 DefaultFeatureCollection features =
> new DefaultFeatureCollection();
>
>                                 features.add(feature);
>
>                                 try
>
>                                 {
>
>
> featureStore.addFeatures(features);
>
>                                 }
>
>                                 catch (IOException e)
>
>                                 {
>
>                                                 return false;
>
>                                 }
>
> 
>
>                                 return true;
>
>                 }
>
> 
>
> 
>
> Is there something I am doing wrong?  Is there a workaround?
>
> 
>
> Thanks,
>
> 
>
> Beau
>
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users