Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Paho » Paho java client - cannot create more than 5000 connections(Getting out of memory with Paho java client after creating 5000 connections)
Paho java client - cannot create more than 5000 connections [message #1723882] Thu, 18 February 2016 17:11 Go to next message
Dee Sog is currently offline Dee SogFriend
Messages: 2
Registered: February 2016
Junior Member
Hi,
I am trying to create multiple connections to mqtt broker (mosquitto) using Paho's java client. But After ~5000 connections, I get error:

"java.lang.OutOfMemoryError: unable to create new native thread".

This is a big VM with 16 CPU/ 16g ram.
- I have already increased the ulimit parameters on this ubuntu box.
- tried creating multiple connections within one java thread
- one connect per java thread.

None of the above has helped. Based on some of the other forums, people have been able to make close to 100k connections at a time from one box. What am I missing ?

Here is my code:
for (int i = 0; i < connectionCountPerThread ; i++) {
MqttClient newClient = new MqttClient("tcp://" + url, credsList.get(i).getClientId());
sampleClient.add(newClient);
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
connOpts.setPassword(credsList.get(i).getPassword().toCharArray());
connOpts.setUserName(credsList.get(i).getUsername());
newClient.connect(connOpts);
newClient.setTimeToWait(1000000);

}


MqttMessage message1;
int qos = 0;
MockObservations mo = new MockObservations();

for (int i = 0 ; i < msgCount ; i++) {
for (int conn = 0; conn < connectionCountPerThread ; conn++) {

message1 = new MqttMessage(mo.getSensorReading(credsList.get(conn).getPayloadType()).getBytes());
message1.setQos(qos);
sampleClient.get(conn).publish(credsList.get(conn).getTopicId(), message1);

try {Thread.sleep(5000); } catch (Exception e) {}

}


727b8ad00873:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128311
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 3000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 200000
virtual memory (kbytes, -v) unlimited
Re: Paho java client - cannot create more than 5000 connections [message #1724228 is a reply to message #1723882] Mon, 22 February 2016 11:45 Go to previous messageGo to next message
James Sutton is currently offline James SuttonFriend
Messages: 71
Registered: July 2015
Member
Hi,

The Paho Java library wasn't optimized for that many connections. It starts up 4 or 5 threads per connection, so you're quite likely to run out of memory or hit a thread limit just because of that.
What are you trying to achieve by making 5000+ connections from one application? Could your application be redesigned to share a single connection instead?
Re: Paho java client - cannot create more than 5000 connections [message #1724270 is a reply to message #1724228] Mon, 22 February 2016 16:48 Go to previous messageGo to next message
Dee Sog is currently offline Dee SogFriend
Messages: 2
Registered: February 2016
Junior Member
Thanks for the response.
Good to know that Java paho client may not be able to scale up to my requirements. Is there another client - python /C that would allow me to do so ?

The purpose here is to simulate a IoT scenario where million devices are connected and sending messages. In this case, each device is one paho client connection. So, in the ideal case, I would like to have million connections. In order to do so, I should at least scale upto 50k connections per VM/box.

Thanks.
Re: Paho java client - cannot create more than 5000 connections [message #1724845 is a reply to message #1724270] Fri, 26 February 2016 14:09 Go to previous message
James Sutton is currently offline James SuttonFriend
Messages: 71
Registered: July 2015
Member
You might find that the C client is more lightweight and so you could fit more onto one box. An alternative would be to set up a number of either Mosquitto or RSMB brokers in bridge mode with bridges connected to bridges. You should be able to get enough connected that would create enough connections for your needs. IIRC, RSMB only takes up around 200K of memory per instance and should be able to create a couple of hundred connections!
Previous Topic:Detecting timeout between device and server
Next Topic:Java client with websocket : Invalid Frame : Opcode 15 error
Goto Forum:
  


Current Time: Sat Nov 17 21:49:04 GMT 2018

Powered by FUDForum. Page generated in 0.01935 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top