Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] IndexOutOfBoundsException from Geomesa while Creating Shards.
Hello,

This appears to be a fun bug, where we run into trouble due to the fact that Math.abs of Int.MinValue is still negative: https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#abs-int-

I opened a ticket here: https://geomesa.atlassian.net/browse/GEOMESA-2763

Due to the implementation of hashing, if you set the z-shards config to a power of 2 you won't hit the error (the default value is 4):

scala> (1 to 64).filterNot(i => Int.MinValue % i < 0)
res1: scala.collection.immutable.IndexedSeq[Int] = Vector(1, 2, 4, 8, 16, 32, 64)

Thanks,

Emilio

On 12/27/19 3:37 PM, Amit Srivastava wrote:
Hi,

Geomesa is returning below exception while we are ingesting data. Can anyone let us know why we are seeing java.lang.IndexOutOfBoundsException: -8? Below is our table schema:

[hadoop@ip-10-0-16-129 ~]$ geomesa-hbase describe-schema -c atlas -f OSMNodes
INFO  Describing attributes of feature 'OSMNodes'
geometry           | Point     (Spatio-temporally indexed)
ingestionTimestamp | Timestamp (Spatio-temporally indexed)
nextTimestamp      | Timestamp
serializerVersion  | String    
featurePayload     | String    

User data:
  geomesa.index.dtg | ingestionTimestamp
  geomesa.indices   | z3:6:3:geometry:ingestionTimestamp,id:4:3:
  geomesa.z.splits  | 60


Exception:

27 Dec 2019 20:10:53,742 [32m[INFO][m (pool-23424-thread-1) com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy: Some features were not written:

java.lang.IllegalArgumentException: Some features were not written:

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore.addFeatures(GeoMesaFeatureStore.scala:59) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put_aroundBody20(GeoToolsStorageClientStrategy.java:200)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy$AjcClosure21.run(GeoToolsStorageClientStrategy.java:1)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver.jar:1.8.1]

at com.test.metrics.declarative.aspectj.JoinpointInvocationHandle.proceed(JoinpointInvocationHandle.java:60)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.AbstractMethodMetricInterceptor.handleInvocation(AbstractMethodMetricInterceptor.java:283)~[DeclarativeCoralMetrics-2.1.jar:?]

at com.test.metrics.declarative.aspectj.MetricMethodAspect$ConfiguredMethodAspect.invoke(MetricMethodAspect.java:101)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.aspectj.MetricMethodAspect.captureMethodMetrics(MetricMethodAspect.java:52)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put_aroundBody22(GeoToolsStorageClientStrategy.java:192)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy$AjcClosure23.run(GeoToolsStorageClientStrategy.java:1)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)~[aspectjweaver.jar:1.8.1]

at com.test.metrics.declarative.aspectj.JoinpointInvocationHandle.proceed(JoinpointInvocationHandle.java:60)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.AbstractServiceMetricsInterceptor.handleInvocation(AbstractServiceMetricsInterceptor.java:59)~[DeclarativeCoralMetrics-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.aspectj.ServiceMetricsMethodAspect$ConfiguredMethodAspect.invoke(ServiceMetricsMethodAspect.java:59)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.aspectj.ServiceMetricsMethodAspect.invoke(ServiceMetricsMethodAspect.java:37)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put(GeoToolsStorageClientStrategy.java:192)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.DelegateStorageClientStrategy.put(DelegateStorageClientStrategy.java:81)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.helper.GeoMesaSimpleFeatureUpdater.putFeature(GeoMesaSimpleFeatureUpdater.java:121)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.helper.SimpleFeatureHandler.lambda$getCallableForSingleFeaturePut$0(SimpleFeatureHandler.java:190)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Suppressed: java.lang.IllegalArgumentException: Error indexing feature '1371494157#3638946185:POINT (88.3176015 22.5988557)|2019-12-23 01:00:00.0|9999-12-30 23:59:59.0|geojson|

{ "geometry": { "type": "Point", "coordinates": [ 88.3176015, 22.5988557 ] }, "id": "3638946185", "properties": { "tags": {}, "changesetId": 32467734, "version": 1, "uid": 2900596, "user": "harisha", "featureSource": "OSM", "sourceTimestamp": "2015-07-07 09:51:31", "ingestionTimestamp": "2019-12-23 01:00:00" } }

'

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$class.writeFeature(GeoMesaFeatureWriter.scala:55) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$TableFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:141) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$GeoMesaAppendFeatureWriter$class.write(GeoMesaFeatureWriter.scala:227) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$$anon$3.write(GeoMesaFeatureWriter.scala:108) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore$$anonfun$addFeatures$2.apply(GeoMesaFeatureStore.scala:47) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore$$anonfun$addFeatures$2.apply(GeoMesaFeatureStore.scala:42) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.utils.io.WithClose$$anonfun$apply$2$$anonfun$apply$3.apply(SafeClose.scala:65) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.utils.io.WithClose$.apply(SafeClose.scala:63) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.utils.io.WithClose$$anonfun$apply$2.apply(SafeClose.scala:65) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.utils.io.WithClose$.apply(SafeClose.scala:63) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.utils.io.WithClose$.apply(SafeClose.scala:65) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore.addFeatures(GeoMesaFeatureStore.scala:42)~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put_aroundBody20(GeoToolsStorageClientStrategy.java:200)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy$AjcClosure21.run(GeoToolsStorageClientStrategy.java:1)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)~[aspectjweaver.jar:1.8.1]

at com.test.metrics.declarative.aspectj.JoinpointInvocationHandle.proceed(JoinpointInvocationHandle.java:60)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.AbstractMethodMetricInterceptor.handleInvocation(AbstractMethodMetricInterceptor.java:283)~[DeclarativeCoralMetrics-2.1.jar:?]

at com.test.metrics.declarative.aspectj.MetricMethodAspect$ConfiguredMethodAspect.invoke(MetricMethodAspect.java:101)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.aspectj.MetricMethodAspect.captureMethodMetrics(MetricMethodAspect.java:52)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put_aroundBody22(GeoToolsStorageClientStrategy.java:192)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy$AjcClosure23.run(GeoToolsStorageClientStrategy.java:1)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)~[aspectjweaver.jar:1.8.1]

at com.test.metrics.declarative.aspectj.JoinpointInvocationHandle.proceed(JoinpointInvocationHandle.java:60)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.AbstractServiceMetricsInterceptor.handleInvocation(AbstractServiceMetricsInterceptor.java:59)~[DeclarativeCoralMetrics-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.aspectj.ServiceMetricsMethodAspect$ConfiguredMethodAspect.invoke(ServiceMetricsMethodAspect.java:59)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.metrics.declarative.servicemetrics.aspectj.ServiceMetricsMethodAspect.invoke(ServiceMetricsMethodAspect.java:37)~[DeclarativeCoralMetricsAspectJ-2.1.jar:?]

at com.test.gaia.atlas.geotools.accessor.GeoToolsStorageClientStrategy.put(GeoToolsStorageClientStrategy.java:192)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.accessor.DelegateStorageClientStrategy.put(DelegateStorageClientStrategy.java:81)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.helper.GeoMesaSimpleFeatureUpdater.putFeature(GeoMesaSimpleFeatureUpdater.java:121)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at com.test.gaia.atlas.geotools.helper.SimpleFeatureHandler.lambda$getCallableForSingleFeaturePut$0(SimpleFeatureHandler.java:190)~[GaiaAtlasGeoMesaClient-1.0.jar:?]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Caused by: java.lang.IndexOutOfBoundsException: -8

at scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) ~[scala-library.jar:?]

at scala.collection.immutable.Vector.apply(Vector.scala:122) ~[scala-library.jar:?]

at org.locationtech.geomesa.index.api.ShardStrategy$ShardStrategyImpl.apply(ShardStrategy.scala:72) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.index.z3.Z3IndexKeySpace.toIndexKey(Z3IndexKeySpace.scala:78) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.api.WriteConverter$WriteConverterImpl.convert(WriteConverter.scala:32) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.api.IndexAdapter$IndexWriter.write(IndexAdapter.scala:101)~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$class.writeFeature(GeoMesaFeatureWriter.scala:52) ~[geomesa-hbase-spark-runtime_2.11-2.3.0.jar:?]

... 33 more

 


--

Regards,

Amit Kumar Srivastava
+1 206 518 8339 / Mobile 


Back to the top