[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
|
Jim,
I believe we have evidence that GeoServer AND the Accumulo TabletServer may be experiencing out of memory conditions. I'm experimenting with the memory-related configuration parameters, but since I can't duplicate the error without running our web-app, it's a bit difficult to diagnose.
I think we're just running GeoServer via Jetty.
Thanks,
Bob
-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Friday, December 19, 2014 2:24 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
Hi Bob,
Is that error happening on GeoServer or Accumulo? I imagine the GeoServer JVM is likely reporting its OOM.
How much memory are giving GeoServer? Are you running it via Tomcat, Jetty, or JBoss?
Thanks,
Jim
On 12/19/2014 01:45 PM, Barnhart, Bob M. wrote:
> Chris,
>
> We're not [yet] talking "big data" here. We've seen failures trying to retrieve 55 FMV metadata records constituting ~68K.
>
> I don't think we're running a web-server on this server--just GeoServer (etc.).
>
> This morning one of our engineers used WizTools.org RESTClient 3.1 to hit our server every second and got 1 failure out of 50 requests:
>
> <?xml version="1.0" ?>
> <ServiceExceptionReport
> version="1.2.0"
> xmlns="http://www.opengis.net/ogc";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd";>
> <ServiceException>
> java.lang.OutOfMemoryError: unable to create new native thread
> unable to create new native thread
> </ServiceException></ServiceExceptionReport>
>
> This seems to confirm our earlier suspicions that somehow, the JVM is [intermittently] running out of memory on our 16GB ("ram") VM.
>
> Bob
>
> -----Original Message-----
> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
> Eichelberger
> Sent: Friday, December 19, 2014 8:55 AM
> To: Geomesa User discussions
> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
> accessing table via GeoServer using GeoMesa Plugin
>
> Bob,
>
> As this is a query for all locations and all time, how many records are being exchanged? Is your web server also running on this VM?
> GeoServer's WFS service and Accumulo+GeoMesa should be using iterators, but I wonder if some other down-stream process on the VM is trying to hold all of the response features in memory at one time, forcing something else to swap. We *have* seen issues where the Accumulo server starts swapping, Zookeeper starts timing out, and then everything collapses.
>
> Just a thought.
>
> Sincerely,
> -- Chris
>
>
> On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
>> Thanks for responding Chris.
>>
>> The query is identical to the ones we've used, except for the SERVER IP address.
>>
>> I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates.
>>
>> This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".
>>
>> -----Original Message-----
>> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
>> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
>> Eichelberger
>> Sent: Friday, December 19, 2014 8:18 AM
>> To: geomesa-users@xxxxxxxxxxxxxxxx
>> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
>> accessing table via GeoServer using GeoMesa Plugin
>>
>> Bob,
>>
>> I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.
>>
>> Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?
>>
>> Best of luck!
>>
>> Sincerely,
>> -- Chris
>>
>>
>> On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
>>> Our web-application uses GoogleEarth technology and has the ability
>>> to request from Accumulo (via GeoServer using the GeoMesa Accumulo
>>> plugin) geospatial feature-data that has been ingested using GeoMesa.
>>> Once a user has zoomed into a particular area of the map, the
>>> web-application uses AJAX to issue an HTTP request (such as the
>>> following) in order to bring back meta-data (in this case for
>>> “Full-Motion-Videos”) for any “features” that lie within the visible
>>> map area:
>>>
>>>
>>>
>>> http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
>>> eq
>>> uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
>>>
>>>
>>>
>>> Our web-application (asynchronously) submits such URL requests using
>>> AJAX. The problem is that the request USUALLY (though not always)
>>> fails and “hangs” with the side-effect that the Accumulo
>>> Table-Server process will have terminated (possibly due to a
>>> memory-compaction error). At this point it is not possible to access
>>> Accumulo via GeoServer/GeoMesa-Plugin until the
>>> Zookeeper/Hadoop/Accumulo/GeoServer
>>> stack is restarted.
>>>
>>>
>>>
>>> However, if I manually send this request to GeoServer using the
>>> following wget(1) command, everything works fine:
>>>
>>>
>>>
>>> wget -t 1 -O /tmp/gs-fmv.json
>>> http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
>>> eq
>>> uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
>>>
>>>
>>>
>>> I seem to be able to submit such requests at any frequency, even
>>> submitting multiple concurrent requests, and experience no such
>>> problems with the Accumulo Tablet Servers dying.
>>>
>>>
>>>
>>> At this point, we’re stymied as to what might be happening and hoped
>>> that perhaps one of you had seen some similar behavior or might have
>>> a theory about what is happening, or even suggestions about how to
>>> debug this.
>>>
>>>
>>>
>>> FYI, we have been running the following software versions in our
>>> “data-stack” since our work with GeoMesa/Accumulo technology began
>>> April/May 2014:
>>>
>>>
>>>
>>> accumulo-1.5.1
>>>
>>> geomesa 1.0.0
>>>
>>> geoserver-2.5
>>>
>>> /webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoserv
>>> er
>>> -plugin.jar
>>>
>>> /webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar
>>>
>>> hadoop-2.4.0
>>>
>>> zookeeper-3.4.6
>>>
>>>
>>>
>>> We would appreciate any help or insight you might be able to provide.
>>>
>>>
>>>
>>> Bob Barnhart
>>>
>>> Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489
>>> (Mobile)
>>> | Robert.M.Barnhart@xxxxxxxxxx
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or
>>> unsubscribe from this list, visit
>>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>>
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users
