Hi,
I wrote and run a multithread clients to query features (Attribute Index Only Iterator) from geomesa according to geomesa-quickstart, and set thread nums=50, request send interval=1s. After running a while, i found the following problems:
1. some clients got the result features after a long duration, maybe 6 0r 7s or more, and the latency will be longer and longer;
After studying the paper (http://ieee-hpec.org/2013/index_htm_files/28-2868615.pdf), i guess, not sure, the root of this problem maybe thrift, communication stack of accumulo. because i did not find any abnormal data or activity in the scan metrics of accumulo and tablet server's jvm profile.
2. I opened the jconsole view of tservers, found the usage of the jvm old gen exceeds the threshold i set to trigger the CMS GC, and after gc, nothing has changed!!! the worst is the usage of old gen will be 100%, OutOfMemory!
I wrote another test case in accumulo client api to do the same query, no spatial and temporal process, and never found the above problems!
I don't know how to solve above problems, anybody help me? Thanks a lot!
PS: environment and settings:
accumulo cluster: 4 node, 1 master, 3 tablet servers. every node: 126G memory, 40 cores x Intel Xeon E5-2670 v2@ 2.50GHZ, 769G hard disk, 1000Mbps network card.
accumulo-env.sh: ACCUMULO_MASTER_OPTS="${POLICY} -Xmx3806m -Xms3806m -Xmn1g -XX:MaxPermSize=512m -XX:SurvivorRatio=8 "
ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx3806m -Xms3806m -Xmn1g -XX:MaxPermSize=512m -XX:SurvivorRatio=8 "
ACCUMULO_GENERAL_OPTS="-XX:MaxTenuringThreshold=10 -XX:GCTimeRatio=19 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true"
accumulo-site.xml: <property>
<name>tserver.memory.maps.max</name>
<value>1G</value>
</property>
<property>
<name>tserver.memory.maps.native.enabled</name>
<value>true</value>
</property>
<property>
<name>tserver.cache.data.size</name>
<value>256M</value>
</property>
<property>
<name>tserver.cache.index.size</name>
<value>256M</value>
</property>
