Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] ArrayIndexOutOfBoundException in AttributeWriter
Title: Default Disclaimer Daimler AG
Good to hear. Another option to consider is using a shared object pool, such as Apache commons pool.

Thanks,

Emilio

On 4/30/19 10:01 AM, siegmar.zuefle@xxxxxxxxxxx wrote:

Hi Emilio,

 

your first hyptothesis of a multiple thread issue was correct. We used feature writers in context with Guava Cache which is multi-threaded. After Guava Cache removal the exception was not seen anymore. Currently we do not use partition timeout for retaining immediate writing but we would think of using it instead of the Guava Cache.

 

Thanks

Siegmar

 

Von: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> Im Auftrag von Emilio Lahr-Vivaz
Gesendet: Montag, 29. April 2019 15:09
An: geomesa-users@xxxxxxxxxxxxxxxx
Betreff: Re: [geomesa-users] ArrayIndexOutOfBoundException in AttributeWriter

 

Hello,

No one has reported that issue before - is it possible that you are using a feature writer in multiple threads? A feature writer is single threaded, so if you share writers across requests you may get errors. The FSDS will cache, then periodically flush and re-create writers to ensure that there are not too many or too large files being written, so that is likely what is causing the error to reset periodically. See https://www.geomesa.org/documentation/user/filesystem/configuration.html#geomesa-fs-writer-partition-timeout for details on the timeout.

Thanks,

Emilio

On 4/29/19 8:51 AM, siegmar.zuefle@xxxxxxxxxxx wrote:

Dear Geomesa support,

 

we currently run Geomesa in a Filesystem (FS) environment on a S3 bucket store using

 

·         Geomesa 2.2.0

·         Geomesa FS 2.11 data store (parquet storage)

 

On request, geo-referenced data is written to the data store. The corresponding GeoMesaFeatureWriter is instantiated using getFeatureWriteAppend.

 

After a while – especially in case when the number of requests increase - it happens that we receive an ArrayIndexOutOfBoundsException, in detail

 

Apr 24, 2019 7:10:03 AM org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2] with root cause

java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2

     at org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.addLong(MessageColumnIO.java:443)

     at org.locationtech.geomesa.parquet.AttributeWriter$LongWriter.write(AttributeWriter.scala:113)

     at org.locationtech.geomesa.parquet.AttributeWriter$AbstractAttributeWriter.apply(AttributeWriter.scala:67)

     at org.locationtech.geomesa.parquet.jobs.SimpleFeatureWriteSupport.writeFields(SimpleFeatureWriteSupport.scala:58)

     at org.locationtech.geomesa.parquet.jobs.SimpleFeatureWriteSupport.write(SimpleFeatureWriteSupport.scala:43)

     at org.locationtech.geomesa.parquet.jobs.SimpleFeatureWriteSupport.write(SimpleFeatureWriteSupport.scala:21)

     at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)

     at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:292)

     at org.locationtech.geomesa.parquet.ParquetFileSystemStorage$ParquetFileSystemWriter.write(ParquetFileSystemStorage.scala:86)

     at org.locationtech.geomesa.parquet.ParquetFileSystemStorage$$anon$1.org$locationtech$geomesa$fs$storage$common$MetadataObservingFileSystemWriter$$super$write(ParquetFileSystemStorage.scala:39)

     at org.locationtech.geomesa.fs.storage.common.MetadataObservingFileSystemWriter$class.write(MetadataObservingFileSystemWriter.scala:24)

     at org.locationtech.geomesa.parquet.ParquetFileSystemStorage$$anon$1.write(ParquetFileSystemStorage.scala:39)

     at org.locationtech.geomesa.fs.FileSystemFeatureStore$$anon$1.write(FileSystemFeatureStore.scala:75)

     at org.geotools.data.store.EventContentFeatureWriter.write(EventContentFeatureWriter.java:122)

     at org.geotools.data.InProcessLockingManager$1.write(InProcessLockingManager.java:304)

     at com.athena.vcc.geomesaservice.geomesa.datastore.GeoMesaFeatureWriter.write(GeoMesaFeatureWriter.java:81)

     at com.athena.vcc.geomesaservice.geomesa.datastore.GeoMesaFileSystemDataStore.insert(GeoMesaFileSystemDataStore.java:130)

     at com.athena.vcc.geomesaservice.geomesa.GeoMesaFacade.saveNetworkPerformanceEvents(GeoMesaFacade.java:72)

     at com.athena.vcc.geomesaservice.rest.StorageController.saveNetworkPerformanceEvents(StorageController.java:80)

     at jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)

     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

     at java.base/java.lang.reflect.Method.invoke(Method.java:566)

     at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)

     at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)

     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)

     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)

     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)

     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)

     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)

     at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)

     at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)

     at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)

     at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117)

     at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)

     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

     at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

     at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

     at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)

     at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

     at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

     at java.base/java.lang.Thread.run(Thread.java:834)

 

From this point every call to the storage wirte fails according the exception above. However from time to time the behaviour recovers itself until the next period of above exception.

 

Did you already see this issue before or do you have any ideas how isolate the issue? Any suggestions are welcome.

 

Thank you and best regards

Siegmar Züfle

Daimler AG
RD/AFC

Onboard Crowd Functions for Automated Driving
HPC: 059-G006

Gebäude 738, Zimmer 3.132

Hanns-Klemm-Str. 45
D-71034 Böblingen

email: siegmar.zuefle@xxxxxxxxxxx

 


If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.


 



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 

Default Disclaimer Daimler AG
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.

 

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top