| Re: [mosquitto-dev] A decentralize Mosquitto cluster design. |
@Tatsuzo Osawa, thanks for the information about share subscription, I would like to make the adaption for cluster if it is implemented on current Mosquitto. For the 60K clients limitation, maybe you can try to allocate more ip address to your NIC and bind the "clients'" socket with them.
@Oegma2, I have tested your scenario in 3 brokers mosquitto cluster by Tsung, include 100 clients connected with will msg, and another client subscribed with topic filter "#", and found that only 25 will msgs received by the subscriber, so I have checked the mosquitto.log and found that the subscription was happend AFTER 75 clients' disconnection, so it only receive 25 will msgs for remained disconnection. Could you check with this point in your env.?
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Saturday, January 20, 2018 7:11:55 AM
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.Hi Jianhui, Izak
Subscribing "#" is an interesting scenario, because a lot of IoT use cases consist one subscriber handles many publishers. But, under this cluster design, I think the idea of shared subscription in MQTTv5 becomes essential. I'd like to implement the shared subscription not only on MQTTv5 but also on current MQTT version as non-standard features of mosquitto in the near future.
Regards,Tatsuzo
2018-01-20 0:19 GMT+09:00 jianhui zhan <hui6075@xxxxxxxxxxx>:
Hi Smit,
Thank you very very much for the testing, I have not tested with "#" which will bring a substantial messages forwarding inside the cluster. There's a traffic flow control for each client, and in the cluster implementation, the bandwidth of the pipeline between different brokers is the same as each broker with it's client, no special optimization. Is there any notice with "dropped" in the mosquitto log file if you enabled mosquitto.conf with log_type notice?
And could you do the same test for mosquitto bridge mode? Because bridge also has a such kind of traffic flow control.
Anyway, I will try to simulate a high latency network and try to reproduce the scenario!
BRs,
Jianhui
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Friday, January 19, 2018 23:11
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.I've repeated this test now 3x using old setup and cluster - think I will need to do some more testing first! So try the above but this is not yet 100% confined the results valid
Will continue to see if I can get to a better answer for everybody - for now, just take note of my testing setup - and ignore the results - possible that my sub client or even network playing a role here.
On Fri, Jan 19, 2018 at 4:39 PM, Oegma2 <oegma2@xxxxxxxxx> wrote:
Hi everybody
I've done a very basic test - used my baseline setup as starting point to qualify that the system is working - so taking out any network, setup ext before switch over to the new mosquitto clustered option. Very basic test - 2 servers running mosquitto behind LB and 1000 client connection to the LB, splitting the load 50%/50% with one subscriber listening for all messages using "#"
The client all connect but the subscriber fails to receive the published online message from the clients. So each client starts up will inform mosquitto it will be using LWT & once connected, publish an online message
Only received about 280 connected message out of the 1000....small test that showing some problem somewhere. Using a more real world scenario where network latency will play a role and the clients & subscriber for message maybe on a different network
So the setup is simple:* 2xServers running mosquitto on private network on AWS using t2.micro EC2 instance* Using latest AWS LB (TCP option) forwardnig to those to EC2 instances* Then running my 1000 client from my own network in office to sim clients out in the field - so not running them in AWS* To make thing more handy - running the subscriber client on different laptop in the office
The above scenario pass my normal setup using 2 subscribers to mosquitto and 1000 clients - I get all my connected messages and can also push message to all clients with no issue - so not the EC2 or LB or network in our around AWS/OFficeSwitching then to mosquitto cluster and do the same test, I can only get about 1/4 of the message & pushing message fails to get them...i have to send a lot of pub message to trigger the client to respond. . . but I never get the result on the subscriber side
Hope this helps and provide some idea where problem may be - - - but setup should be clients & workers/sub not in the same network as the broker and see if you can replicate this issue(Or maybe my testing is flawed :))...but this is a real world scenario ..the clients and sub can be on different networks with latency but should not stop the system from working
Warm regardsIzak S Smit (Oegma)
On Wed, Jan 17, 2018 at 4:08 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:
______________________________Hi tatsuzo,
I have made a simple test for the cluster, use krylovsk/mqtt-benchmark as a benchmark tool. But with number of brokers increase to 3, the mean throughput get the limitation. I think the reason is that the client cannot offer enough stress to cluster. See appendix for detail.
It is not easy to test a cluster, especially for MQTT service, Tsung and Jmeter support client clustered, but they are not friendly for MQTT..
Anyway, I'll do more tests, and hope someone can also give a test for mosquitto cluster .. thanks!
BRs,
Jianhui
Appendix:
No. brokers 1 2 3 4 9 n10k c100 17548.83 25509.4 28878.05 28610.25 28947.78 n1k c1k 9928.573 16533.82 24105.12 24034.03 24328.37
Unit: Total Bandwidth (msg/sec), QoS=2, MsgSize=1000bytes.
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Wednesday, January 17, 2018 20:20
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.Hi Oegma,cc Jianhui
> Will it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....? Normal Mosquitto> is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the> traffic/load ...
I think applying the cluster on multicore environment might be a cool idea.Because, as you described, normal mosquitto cannot use multicore efficiently.
Regards,Tatsuzo
_________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev