Eclipse Community Forums: Paho » Paho java client - cannot create more than 5000 connections

Home » Eclipse Projects » Paho » Paho java client - cannot create more than 5000 connections(Getting out of memory with Paho java client after creating 5000 connections)

Show: Today's Messages :: Show Polls :: Message Navigator

Paho java client - cannot create more than 5000 connections [message #1723882]

Thu, 18 February 2016 17:11

Dee Sog

Messages: 2
Registered: February 2016

Junior Member

Hi,
I am trying to create multiple connections to mqtt broker (mosquitto) using Paho's java client. But After ~5000 connections, I get error:

"java.lang.OutOfMemoryError: unable to create new native thread".

This is a big VM with 16 CPU/ 16g ram.
- I have already increased the ulimit parameters on this ubuntu box.
- tried creating multiple connections within one java thread
- one connect per java thread.

None of the above has helped. Based on some of the other forums, people have been able to make close to 100k connections at a time from one box. What am I missing ?

Here is my code:
for (int i = 0; i < connectionCountPerThread ; i++) {
MqttClient newClient = new MqttClient("tcp://" + url, credsList.get(i).getClientId());
sampleClient.add(newClient);
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
connOpts.setPassword(credsList.get(i).getPassword().toCharArray());
connOpts.setUserName(credsList.get(i).getUsername());
newClient.connect(connOpts);
newClient.setTimeToWait(1000000);

}

MqttMessage message1;
int qos = 0;
MockObservations mo = new MockObservations();

for (int i = 0 ; i < msgCount ; i++) {
for (int conn = 0; conn < connectionCountPerThread ; conn++) {

message1 = new MqttMessage(mo.getSensorReading(credsList.get(conn).getPayloadType()).getBytes());
message1.setQos(qos);
sampleClient.get(conn).publish(credsList.get(conn).getTopicId(), message1);

try {Thread.sleep(5000); } catch (Exception e) {}

}

727b8ad00873:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128311
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 3000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 200000
virtual memory (kbytes, -v) unlimited

Report message to a moderator

Re: Paho java client - cannot create more than 5000 connections [message #1724228 is a reply to message #1723882]

Mon, 22 February 2016 11:45

James Sutton

Messages: 71
Registered: July 2015

Member

Hi,

The Paho Java library wasn't optimized for that many connections. It starts up 4 or 5 threads per connection, so you're quite likely to run out of memory or hit a thread limit just because of that.
What are you trying to achieve by making 5000+ connections from one application? Could your application be redesigned to share a single connection instead?

Report message to a moderator

Re: Paho java client - cannot create more than 5000 connections [message #1724270 is a reply to message #1724228]

Mon, 22 February 2016 16:48

Dee Sog

Messages: 2
Registered: February 2016

Junior Member

Thanks for the response.
Good to know that Java paho client may not be able to scale up to my requirements. Is there another client - python /C that would allow me to do so ?

The purpose here is to simulate a IoT scenario where million devices are connected and sending messages. In this case, each device is one paho client connection. So, in the ideal case, I would like to have million connections. In order to do so, I should at least scale upto 50k connections per VM/box.

Thanks.

Report message to a moderator

Re: Paho java client - cannot create more than 5000 connections [message #1724845 is a reply to message #1724270]

Fri, 26 February 2016 14:09

James Sutton

Messages: 71
Registered: July 2015

Member

You might find that the C client is more lightweight and so you could fit more onto one box. An alternative would be to set up a number of either Mosquitto or RSMB brokers in bridge mode with bridges connected to bridges. You should be able to get enough connected that would create enough connections for your needs. IIRC, RSMB only takes up around 200K of memory per instance and should be able to create a couple of hundred connections!

Report message to a moderator

Previous Topic:	Detecting timeout between device and server
Next Topic:	Java client with websocket : Invalid Frame : Opcode 15 error

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 05:14:58 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter