Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin

Hi Bob,

Apologies for the delay in responding. I wanted to ask if you had figured out any more about the memory problems you had reported.

We typically have been running GeoServer with Tomcat. It may be worth trying out Tomcat or JBoss.

Thanks,

Jim

On 12/22/2014 05:57 PM, Barnhart, Bob M. wrote:
Jim,

I believe we have evidence that GeoServer AND the Accumulo TabletServer may be experiencing out of memory conditions. I'm experimenting with the memory-related configuration parameters, but since I can't duplicate the error without running our web-app, it's a bit difficult to diagnose.

I think we're just running GeoServer via Jetty.

Thanks,
Bob

-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Friday, December 19, 2014 2:24 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin

Hi Bob,

Is that error happening on GeoServer or Accumulo?  I imagine the GeoServer JVM is likely reporting its OOM.

How much memory are giving GeoServer?  Are you running it via Tomcat, Jetty, or JBoss?

Thanks,

Jim

On 12/19/2014 01:45 PM, Barnhart, Bob M. wrote:
Chris,

We're not [yet] talking "big data" here. We've seen failures trying to retrieve 55 FMV metadata records constituting ~68K.

I don't think we're running a web-server on this server--just GeoServer (etc.).

This morning one of our engineers used WizTools.org RESTClient 3.1 to hit our server every second and got 1 failure out of 50 requests:

<?xml version="1.0" ?>
<ServiceExceptionReport
     version="1.2.0"
     xmlns="http://www.opengis.net/ogc";
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
     xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd";>
     <ServiceException>
        java.lang.OutOfMemoryError: unable to create new native thread
unable to create new native thread
</ServiceException></ServiceExceptionReport>

This seems to confirm our earlier suspicions that somehow, the JVM is [intermittently] running out of memory on our 16GB ("ram") VM.

Bob

-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
[mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
Eichelberger
Sent: Friday, December 19, 2014 8:55 AM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
accessing table via GeoServer using GeoMesa Plugin

Bob,

As this is a query for all locations and all time, how many records are being exchanged?  Is your web server also running on this VM?
GeoServer's WFS service and Accumulo+GeoMesa should be using iterators, but I wonder if some other down-stream process on the VM is trying to hold all of the response features in memory at one time, forcing something else to swap.  We *have* seen issues where the Accumulo server starts swapping, Zookeeper starts timing out, and then everything collapses.

Just a thought.

Sincerely,
    -- Chris


On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
Thanks for responding Chris.

The query is identical to the ones we've used, except for the SERVER IP address.

I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates.

This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".

-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
[mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
Eichelberger
Sent: Friday, December 19, 2014 8:18 AM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
accessing table via GeoServer using GeoMesa Plugin

Bob,

I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.

Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?

Best of luck!

Sincerely,
    -- Chris


On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
Our web-application uses GoogleEarth technology and has the ability
to request from Accumulo (via GeoServer using the GeoMesa Accumulo
plugin) geospatial feature-data that has been ingested using GeoMesa.
Once a user has zoomed into a particular area of the map, the
web-application uses AJAX to issue an HTTP request (such as the
following) in order to bring back meta-data (in this case for
“Full-Motion-Videos”) for any “features” that lie within the visible
map area:

http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
eq
uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json

Our web-application (asynchronously) submits such URL requests using
AJAX. The problem is that the request USUALLY (though not always)
fails and “hangs” with the side-effect that the Accumulo
Table-Server process will have terminated (possibly due to a
memory-compaction error). At this point it is not possible to access
Accumulo via GeoServer/GeoMesa-Plugin until the
Zookeeper/Hadoop/Accumulo/GeoServer
stack is restarted.

However, if I manually send this request to GeoServer using the
following wget(1) command, everything works fine:

wget -t 1 -O /tmp/gs-fmv.json
http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
eq
uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json

I seem to be able to submit such requests at any frequency, even
submitting multiple concurrent requests, and experience no such
problems with the Accumulo Tablet Servers dying.

At this point, we’re stymied as to what might be happening and hoped
that perhaps one of you had seen some similar behavior or might have
a theory about what is happening, or even suggestions about how to
debug this.

FYI, we have been running the following software versions in our
“data-stack” since our work with GeoMesa/Accumulo technology began
April/May 2014:

accumulo-1.5.1

geomesa 1.0.0

geoserver-2.5

/webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoserv
er
-plugin.jar

/webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar

hadoop-2.4.0

zookeeper-3.4.6

We would appreciate any help or insight you might be able to provide.

Bob Barnhart

Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489
(Mobile)
| Robert.M.Barnhart@xxxxxxxxxx


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



Back to the top