[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
|
Chris,
We're not [yet] talking "big data" here. We've seen failures trying to retrieve 55 FMV metadata records constituting ~68K.
I don't think we're running a web-server on this server--just GeoServer (etc.).
This morning one of our engineers used WizTools.org RESTClient 3.1 to hit our server every second and got 1 failure out of 50 requests:
<?xml version="1.0" ?>
<ServiceExceptionReport
version="1.2.0"
xmlns="http://www.opengis.net/ogc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd">
<ServiceException>
java.lang.OutOfMemoryError: unable to create new native thread
unable to create new native thread
</ServiceException></ServiceExceptionReport>
This seems to confirm our earlier suspicions that somehow, the JVM is [intermittently] running out of memory on our 16GB ("ram") VM.
Bob
-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris Eichelberger
Sent: Friday, December 19, 2014 8:55 AM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
Bob,
As this is a query for all locations and all time, how many records are being exchanged? Is your web server also running on this VM?
GeoServer's WFS service and Accumulo+GeoMesa should be using iterators, but I wonder if some other down-stream process on the VM is trying to hold all of the response features in memory at one time, forcing something else to swap. We *have* seen issues where the Accumulo server starts swapping, Zookeeper starts timing out, and then everything collapses.
Just a thought.
Sincerely,
-- Chris
On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
> Thanks for responding Chris.
>
> The query is identical to the ones we've used, except for the SERVER IP address.
>
> I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates.
>
> This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".
>
> -----Original Message-----
> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
> Eichelberger
> Sent: Friday, December 19, 2014 8:18 AM
> To: geomesa-users@xxxxxxxxxxxxxxxx
> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
> accessing table via GeoServer using GeoMesa Plugin
>
> Bob,
>
> I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.
>
> Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?
>
> Best of luck!
>
> Sincerely,
> -- Chris
>
>
> On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
> > Our web-application uses GoogleEarth technology and has the ability
> > to request from Accumulo (via GeoServer using the GeoMesa Accumulo
> > plugin) geospatial feature-data that has been ingested using GeoMesa.
> > Once a user has zoomed into a particular area of the map, the
> > web-application uses AJAX to issue an HTTP request (such as the
> > following) in order to bring back meta-data (in this case for
> > “Full-Motion-Videos”) for any “features” that lie within the visible
> > map area:
> >
> >
> >
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
> > eq
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> >
> >
> >
> > Our web-application (asynchronously) submits such URL requests using
> > AJAX. The problem is that the request USUALLY (though not always)
> > fails and “hangs” with the side-effect that the Accumulo
> > Table-Server process will have terminated (possibly due to a
> > memory-compaction error). At this point it is not possible to access
> > Accumulo via GeoServer/GeoMesa-Plugin until the
> > Zookeeper/Hadoop/Accumulo/GeoServer
> > stack is restarted.
> >
> >
> >
> > However, if I manually send this request to GeoServer using the
> > following wget(1) command, everything works fine:
> >
> >
> >
> > wget -t 1 -O /tmp/gs-fmv.json
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
> > eq
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> >
> >
> >
> > I seem to be able to submit such requests at any frequency, even
> > submitting multiple concurrent requests, and experience no such
> > problems with the Accumulo Tablet Servers dying.
> >
> >
> >
> > At this point, we’re stymied as to what might be happening and hoped
> > that perhaps one of you had seen some similar behavior or might have
> > a theory about what is happening, or even suggestions about how to
> > debug this.
> >
> >
> >
> > FYI, we have been running the following software versions in our
> > “data-stack” since our work with GeoMesa/Accumulo technology began
> > April/May 2014:
> >
> >
> >
> > accumulo-1.5.1
> >
> > geomesa 1.0.0
> >
> > geoserver-2.5
> >
> > /webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoserv
> > er
> > -plugin.jar
> >
> > /webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar
> >
> > hadoop-2.4.0
> >
> > zookeeper-3.4.6
> >
> >
> >
> > We would appreciate any help or insight you might be able to provide.
> >
> >
> >
> > Bob Barnhart
> >
> > Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489
> > (Mobile)
> > | Robert.M.Barnhart@xxxxxxxxxx
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > geomesa-users mailing list
> > geomesa-users@xxxxxxxxxxxxxxxx
> > To change your delivery options, retrieve your password, or
> > unsubscribe from this list, visit
> > http://www.locationtech.org/mailman/listinfo/geomesa-users
>
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users