Hi Jyotsna,
just to mention:
Though californium is designed to work efficient on a small number of datagram sockets, it’s not really recommended to try to start larger number of clients
using then a large number of datagram sockets! If the maximum number of processed messages is reached, increasing the number of clients will lower that number of messages.
On “my machine” 3000 clients is already too much, 2000 clients processes more messages.
Mit freundlichen Grüßen / Best regards
Achim Kraus
(INST/ECS4)
Bosch Software Innovations GmbH | Stuttgarter Straße 130 | 71332 Waiblingen |
GERMANY |
www.bosch-si.com
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Michael Hahn
From: leshan-dev-bounces@xxxxxxxxxxx [mailto:leshan-dev-bounces@xxxxxxxxxxx]
On Behalf Of Simon Bernard
Sent: Mittwoch, 21. März 2018 17:52
To: leshan developer discussions <leshan-dev@xxxxxxxxxxx>; Bhonde, Jyotsna <Jyotsna.Bhonde@xxxxxxxxx>
Subject: Re: [leshan-dev] Running in to OutOfMemoryError: unable to create new native thread while scale testing
The number of thread which could be allocated by the JVM is limited by severals OS settings.
I suppose you are running on a linux machine as you are talking about "ulimit".
On my laptop I was limited around 10,800 threads by default.
After tweaking my debian configuration I was able to run ~120,000 threads.
A default Leshan client uses 10 threads :
- 1 UDP sender (from element-connector)
- 1 UDP receiver (from element-connector)
- 2 Deduplicator (from californium)
- 1 Coap Server (from californium)
- 1 DTLS sender (from scandium)
- 1 DTLS receiver (from scandium)
- 1 DTLS retransmit (from scandium)
- 1 DTLS connectionHandler (from scandium)
- 1 registration engine (from leshan)
You can reduce it by removing 3(UDP+1deduplicator) or 5(DTLS+1deduplicator) threads if your are using only UDP or only DTLS.
To do that LeshanClientBuilder.disableUnsecuredEndpoint() or disableSecuredEndpoint()...
But this is still 5 or 7 threads by clients for a simulator which would run thousands of clients that's means thousands of threads ... there is no way this was a good design... We should fix this so I open an issue about that :
https://github.com/eclipse/leshan/issues/491
I suppose we should change the code a bit to be able to use only one thread pool executor for all the clients.
Currently this is partially done, you can set a common executor for DTLSConnectionHandler and CoAP server so you will save 2 more thread by clients.
With Leshan 1.0.0-M5, the tricks looks like this :
final ScheduledExecutorService executor = Executors.newScheduledThreadPool(1); // the executor to use for all clients
builder.setEndpointFactory(new EndpointFactory() {
@Override
public CoapEndpoint createUnsecuredEndpoint(InetSocketAddress address, NetworkConfig coapConfig,
ObservationStore store) {
return new CoapEndpoint(address, coapConfig);
}
@Override
public CoapEndpoint createSecuredEndpoint(DtlsConnectorConfig dtlsConfig, NetworkConfig coapConfig,
ObservationStore store) {
DTLSConnector dtlsConnector = new DTLSConnector(dtlsConfig);
dtlsConnector.setExecutor(new StripedExecutorService(executor));
return new CoapEndpoint(dtlsConnector, coapConfig, null, null);
}
});
client = builder.build();
client.getCoapServer().setExecutor(executor);
Waiting we are better you can try to play with linux limits configuration but maybe you will face Azure limitation.
Personnaly, I was first limited by UserTasksMax=33% in /etc/systemd/logind.conf and /proc/sys/kernel/pid_max set to 32768. (~10900)
Some resources which could help you :
https://stackoverflow.com/questions/344203/maximum-number-of-threads-per-process-in-linux/26190804#26190804
https://web.archive.org/web/20111209081734/http://research.cs.wisc.edu/condor/condorg/linux_scalability.html#Process%20Identifiers
http://geekswing.com/geek/quickie-tutorial-ulimit-soft-limits-hard-limits-soft-stack-hard-stack/
I was using this kind of code to check my modification :
public class MaxThreadCounter {
private static int count = 0;
public static void main(String[] argv) {
System.out.println("Counting ...");
while (true) {
count++;
try {
new Thread(new Runnable() {
public void run() {
while (true) {
try {
Thread.sleep(1000);
} catch (Throwable e) {
e.printStackTrace();
}
}
}
}).start();
} catch (Throwable e) {
System.err.println("Number of thread :" + count);
e.printStackTrace();
try {
Thread.sleep(5000);
} catch (InterruptedException e1) {
}
System.exit(0);
}
}
}
}
(This is a bit painful, some modification need reboot, others does not persist after a reboot ...If I well understand this is also not a so good practice to allow so much thread...)
If you are interested by the issue 491, do not hesitate to contribute code, I can even guide you a little.
HTH
Simon
Le 21/03/2018 à 01:48, Bhonde, Jyotsna a écrit :
Hi
We are scale testing. Our simulator is in Azure and what we have noticed is that we get following exception when we have about 3500 leshan simulators.
EXCEPTION:OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:717)
org.eclipse.californium.elements.UDPConnector.start(UDPConnector.java:169)
org.eclipse.californium.core.network.CoapEndpoint.start(CoapEndpoint.java:418)
org.eclipse.californium.core.CoapServer.start(CoapServer.java:215)
org.eclipse.leshan.client.californium.LeshanClient.start(LeshanClient.java:173)
....
....
Has anybody come across this problem and any suggestions on how to go about resolving this problem?
Available memory to the container is 25gb ram and it has used up 10gb.
Settings at the docker engine level are are follows:
/ # ulimit -a
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) 0
-m: resident set size (kb) unlimited
-l: locked memory (kb) 82000
-p: processes unlimited
-n: file descriptors 1048576
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
I was looking in to tweaking these properties: NETWORK_STAGE_RECEIVER_THREAD_COUNT and NETWORK_STAGE_SENDER_THREAD_COUNT in californium.properties. But I don't think that will help much since it seems the resource issue is at the system
level.
Thanks
J.
NOTICE OF CONFIDENTIALITY:
This message may contain information that is considered confidential and which may be prohibited from disclosure under applicable law or by contractual agreement. The information is intended solely for the use of the individual or entity named above. If you
are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. If you have received this email transmission in error, please notify
the sender by replying to this email and then delete it from your system.
_______________________________________________
leshan-dev mailing list
leshan-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/leshan-dev
