Hi,
Have you verified with the 'listscans' command in an Accumulo shell
that your queries are using the Attribute Index Iterator?
When you said that you set the number of threads to 50, do you mean
that you are using 50 threads in your query code? In general, I'd
love to see your test project; it'd let us reproduce the issues you
are seeing and verify that any fix or advice is helpful.
Since our iterator stack is non-trivial, I wonder if the high
concurrency is causing transient classes to be tenured mistakenly.
Are you running more than one tablet servers on your 3 nodes? It is
not well documented, and it may not help alleviate an underlying
issue with GeoMesa, but it may be of general interest.
Thanks for your great questions,
Jim
On 03/09/2015 03:50 AM, 焦小超 wrote:
Hi,
I wrote and run a multithread clients to query features
(Attribute Index Only Iterator) from geomesa
according to geomesa-quickstart, and set thread nums=50,
request send interval=1s. After running a while, i found the
following problems:
1. some clients got the result features after a long
duration, maybe 6 0r 7s or more, and the latency will be
longer and longer;
After studying the paper (http://ieee-hpec.org/2013/index_htm_files/28-2868615.pdf),
i guess, not sure, the root of this problem maybe thrift,
communication stack of accumulo. because i did not find any
abnormal data or activity in the scan metrics of accumulo
and tablet server's jvm profile.
2. I opened the jconsole view of tservers, found the
usage of the jvm old gen exceeds the threshold i set to
trigger the CMS GC, and after gc, nothing has changed!!!
the worst is the usage of old gen will be 100%,
OutOfMemory!
I wrote another test case in accumulo client api to do
the same query, no spatial and temporal process, and never
found the above problems!
I don't know how to solve above problems, anybody
help me? Thanks a lot!
PS: environment
and settings:
accumulo cluster: 4 node, 1 master, 3 tablet
servers. every node: 126G memory, 40 cores x Intel Xeon
E5-2670 v2@ 2.50GHZ, 769G hard disk, 1000Mbps network card.
accumulo-env.sh: ACCUMULO_MASTER_OPTS="${POLICY}
-Xmx3806m -Xms3806m -Xmn1g -XX:MaxPermSize=512m
-XX:SurvivorRatio=8 "
ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx3806m -Xms3806m -Xmn1g
-XX:MaxPermSize=512m -XX:SurvivorRatio=8 "
ACCUMULO_GENERAL_OPTS="-XX:MaxTenuringThreshold=10
-XX:GCTimeRatio=19 -XX:+DisableExplicitGC
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:CMSInitiatingOccupancyFraction=70
-Djava.net.preferIPv4Stack=true"
accumulo-site.xml: <property>
<name>tserver.memory.maps.max</name>
<value>1G</value>
</property>
<property>
<name>tserver.memory.maps.native.enabled</name>
<value>true</value>
</property>
<property>
<name>tserver.cache.data.size</name>
<value>256M</value>
</property>
<property>
<name>tserver.cache.index.size</name>
<value>256M</value>
</property>
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
|
