Re: [mosquitto-dev] A decentralize Mosquitto cluster design.

[jianhui zhan <hui6075@xxxxxxxxxxx>]

All the publishes forwarding inside the cluster was taken as QoS=0 for efficiency because I assume that the cluster is deploying under a low latency private network. And what is the QoS of publishes and subscription? The message also possible lost on the way between broker<==>subscriber with QoS=0.

client<==line1==>broker<==line2==>broker<==line3==>

In general, the messages under your scenario (with # and high latency and if sub qos=0) would possible lost both on line2 or line3.

---

From: mosquitto-dev-bounces@xxxxxxxxxxx <mosquitto-dev-bounces@xxxxxxxxxxx> on behalf of jianhui zhan <hui6075@xxxxxxxxxxx>
Sent: Friday, January 19, 2018 11:19:14 PM
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.

 

Hi Smit,

Thank you very very much for the testing, I have not tested with "#" which will bring a substantial messages forwarding inside the cluster. There's a traffic flow control for each client, and in the cluster implementation, the bandwidth of the pipeline between different brokers is the same as each broker with it's client, no special optimization. Is there any notice with "dropped" in the mosquitto log file if you enabled mosquitto.conf with log_type notice? 

And could you do the same test for mosquitto bridge mode? Because bridge also has a such kind of traffic flow control.

Anyway, I will try to simulate a high latency network and try to reproduce the scenario!

BRs,

Jianhui

---

From: mosquitto-dev-bounces@xxxxxxxxxxx <mosquitto-dev-bounces@xxxxxxxxxxx> on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Friday, January 19, 2018 23:11
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.

 

I've repeated this test now 3x using old setup and cluster - think I will need to do some more testing first! So try the above but this is not yet 100% confined the results valid 

Will continue to see if I can get to a better answer for everybody - for now, just take note of my testing setup - and ignore the results - possible that my sub client or even network playing a role here.

On Fri, Jan 19, 2018 at 4:39 PM, Oegma2 <oegma2@xxxxxxxxx> wrote:

Hi everybody

I've done a very basic test - used my baseline setup as starting point to qualify that the system is working - so taking out any network, setup ext before switch over to the new mosquitto clustered option. Very basic test - 2 servers running mosquitto behind LB and 1000 client connection to the LB, splitting the load 50%/50% with one subscriber listening for all messages using "#"

The client all connect but the subscriber fails to receive the published online message from the clients. So each client starts up will inform mosquitto it will be using LWT & once connected, publish an online message

Only received about 280 connected message out of the 1000....small test that showing some problem somewhere. Using a more real world scenario where network latency will play a role and the clients & subscriber for message maybe on a different network

So the setup is simple:

  * 2xServers running mosquitto on private network on AWS using t2.micro EC2 instance

  * Using latest AWS LB (TCP option) forwardnig to those to EC2 instances

  * Then running my 1000 client from my own network in office to sim clients out in the field - so not running them in AWS 

  * To make thing more handy - running the subscriber client on different laptop in the office

The above scenario pass my normal setup using 2 subscribers to mosquitto and 1000 clients - I get all my connected messages and can also push message to all clients with no issue - so not the EC2 or LB or network in our around AWS/OFfice

Switching then to mosquitto cluster and do the same test, I can only get about 1/4 of the message & pushing message fails to get them...i have to send a lot of pub message to trigger the client to respond. . . but I never get the result on the subscriber side

Hope this helps and provide some idea where problem may be - - - but setup should be clients & workers/sub not in the same network as the broker and see if you can replicate this issue

(Or maybe my testing is flawed :))...but this is a real world scenario ..the clients and sub can be on different networks with latency but should not stop the system from working

Warm regards

Izak S Smit (Oegma)

On Wed, Jan 17, 2018 at 4:08 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:

Hi tatsuzo,

I have made a simple test for the cluster, use krylovsk/mqtt-benchmark as a benchmark tool. But with number of brokers increase to 3, the mean throughput get the limitation. I think the reason is that the client cannot offer enough stress to cluster. See appendix for detail.

It is not easy to test a cluster, especially for MQTT service, Tsung and Jmeter support client clustered, but they are not friendly for MQTT..

Anyway, I'll do more tests, and hope someone can also give a test for mosquitto cluster .. thanks!

BRs,

Jianhui

Appendix:

No. brokers		1		2		3		4		9
n10k c100		17548.83		25509.4		28878.05		28610.25		28947.78
n1k c1k		9928.573		16533.82		24105.12		24034.03		24328.37

Unit: Total Bandwidth (msg/sec), QoS=2, MsgSize=1000bytes.

---

From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Wednesday, January 17, 2018 20:20
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.

 

Hi Oegma,

cc Jianhui

> Will it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....? Normal Mosquitto

> is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the 

> traffic/load ...

I think applying the cluster on multicore environment might be a cool idea.

Because, as you described, normal mosquitto cannot use multicore efficiently.

Regards,

Tatsuzo

_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev