Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mosquitto-dev] A decentralize Mosquitto cluster design.

thanks, that will be helpful! In my platform, storage was happened in the http server before push messages to the mosquitto brokers, so I have pool experience about HDD I/O, but I wonder that in a 16 CPUs single box, HDD, I/O controller maybe also a bottleneck if you use built-in storage, further more, power outage/ OS panic..this is the single point failure.😉



From: mosquitto-dev-bounces@xxxxxxxxxxx <mosquitto-dev-bounces@xxxxxxxxxxx> on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Tuesday, January 16, 2018 20:19
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.
 
Hi Jian

Thanks for the feedback on the question above - agree with you on both points :).. so will rather test the system using smaller servers behind LB...never seen LVS LB so will do some R&D on that (thx)


BTW, I have tested with RabbitMQ, I found that it doesn't support MQTT retain message and QoS=2, is that I should install the newest version?
Oegma: If you publish a MQTT message as QoS1 it will "store" the message on "disc" if you create an exchange to route the MQTT "topic" message to a queue and that queue is durable. If you publish Qos0 message, the message will only be stored in RMQ memory. As for Qos2, I don't think RMQ supports that - ever since I've tested RMQ it was only Qos0 = memory and Qos1 = disc ...haven't looked at the latest RMQ, <but> in the same view, AWS IoT broker is doing the same thing - support Qos0/QoS1 but not Qos2

Another area I found with RMQ was that if you persist message to disc, those message ends up on a single HDD - and can become a hot node if you push say all you message to a single queue...facing typical firehose problem...will have to look at sharing or other load balancing option when really pushing high volumes..but it totally possible...just keep in mind ...queue = single HDD (that time I checked)


On Tue, Jan 16, 2018 at 1:21 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:

Cool, hope it can cover your test.


Question
Will it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....? 
Jianhui: It will work, but I don't know whether the memory bus would be a bottleneck for so many instances. It is hard to give a desired performance, hardware architecture(SMP, NUMA, MPI) are uncertain factors, theoretically speaking, for each SMP machine/NUMA node, 4 instance would be the limitation..

Normal Mosquitto is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the traffic/load ....<or> is the idea to use small 2 cpu servers in clustered network mode behind LB like nginx or aws ELB?
Jianhui: I think single process would be better than multi-thread, Nginx/Redis gave the answer. For software LB, LVS is much better then nginx and ELB in my opinion. And if you let me choose, I will choose 16 CPU single box, taskset each broker to one core under memory bus limitation. Hardware BUS is much faster then the cable.

BTW, I have tested with RabbitMQ, I found that it doesn't support MQTT retain message and QoS=2, is that I should install the newest version?


From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Tuesday, January 16, 2018 17:32

To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.
 
Hi Jian

I've pulled the clustered mosquitto branch code, compiled with those flags and got it running now with 2 instances - did a quick test connecting to one broker on node1 and then another client connect to node2 and sub on both client to topic "#" - when I pub a test msg on client1, client2 recv it, so the cluster is working - the 2 brokers also ack that they see each other - fairly easy to set this up :)

I am using one server with diff ports to run this cluster ...node1 is 127.0.0.1:8000 and node2 is 127.0.0.1:9000 ...seem 2 be working just fine on same server with different ports ^_^.

Question
Will it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....? Normal Mosquitto is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the traffic/load ....<or> is the idea to use small 2 cpu servers in clustered network mode behind LB like nginx or aws ELB?

But in short - got it running and can assist in some testing scenarios - exiting times 2 be honest, - if this really works under load, means we can scale the awesome mosquitto broker to bigger loads ^_^ - "big up for C & mosquitto i say" ^_^



On Mon, Jan 15, 2018 at 6:57 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:

Hi Oegma,


That will be wonderful. And please be attention that:

  •  the cluster is based on mosquitto develop branch and there's a known crash issue #505 under some scenario(such as a huge mount of clients disconnect at the same time) that may disturb the testing.
  • SSL/TLS between brokers and persistent session with QoS>0 has not been verified, I will test it asap.
  • If your brokers distribute in different data center with a high latency, please increase cluster_retain_delay in mosquitto.conf to ensure retain message would be receive as expected.
Please feel free to let me know for any confusion😉



From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Monday, January 15, 2018 23:42

To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.
 
Hi Jian

I will be able to deploy it to my own env - compiling with those flags should not be an issue :)...LB is also not an issue - can set the env up with or without a LB 

Never tried Tsung, but looks like solid testing system 

On Mon, Jan 15, 2018 at 5:36 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:

Hi Oegma,

It's my honor that you are willing to make the test.

Currently I have no open resources/servers for a public testing, so could you deploy the cluster on your environment? build mosquitto with WITH_CLUSTER=yes, WITH_BRIDGE=no, fill all the nodes' ip address and port in mosquitto.conf, then take them as the back end server of the LB. If you use Tsung, LB is not needed. And it is recommend to use plain TCP between brokers..


I'll also make some testing with RabbitMQ.
I believe that even if Erlang is suitable for message queue, it must slower then C😉
And maybe a centralized system(as you mentioned, to make some clever routing service) maybe more efficient but NOT reliable due to single point failure.

From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Monday, January 15, 2018 21:42
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.
 
Hi - how can I get involved with testing this setup? Been looking at diff options to scale mosquitto myself in the past, and even bridge options was on the table, creating some clever routing broker/cluster - mapping between RabbitMQ and mosquitto..the list goes on - even custom ZMQ code was once an option to look at and I even learned erlang to write a middle layer clustered system...

So would love to assist with the cluster design testing/setup

What do I need to start

@Roger: Is this cluster mosquitto something you be looking at going forward to scale mosquitto out? Or do you have other plans on making this powerful broker scale beyond a single machine?



On Mon, Jan 15, 2018 at 2:46 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:

Hi Roger, Tatsuzo, Tifaifai, anyone who is interested with in mosquitto cluster,

I've tested the mosquitto cluster by appropriate amount of simultaneous users, connect/subscribe/publish TPS, which gave a proper pressure(carry as much TPS as possible w/o causing latency) to the system.
It is meaningless to give a plain benchmark w/o any comparison, so I've also tested mosquitto bridge under a same scenario which is quite similar with our smart home service platform architecture and scenario(10 brokers, 20k subscribers, 1k publishes from 10 publishers which use persistent TCP from a http server): 
9 brokers on 3 OpenStack VMs(4cores 8G RAM), after 30k persistant subscribers setup, send 10k publishes per second(actually only 2.5kps publishes due to client's bottleneck) from publishers(one publish per client use non-persistant TCP), with payload length = 744 bytes.

With QoS=0, both cluster and bridge(topic # both 0) can work as normal, with QoS=1(topic # both 1), the CPU usage for each broker stabilise at at 65%-75% in cluster, but 100% for the bridge broker during the publish phase, and meanwhile 30% messages lost due to bridge broker's overload(see appendix). More detail test reports which include connect/request response time, network throughput, server monitoring are available under https://github.com/hui6075/mosquitto/tree/develop/benchmark .

I believe that the situation will be worse for bridge under QoS=2, but will not deteriorate for cluster since publish messages forward with it's origin QoS and process with QoS=0 in the cluster. The mosquitto cluster equalized all brokers' load, and it is different with bridge, it is an entire MQTT logic broker for external clients such as duplicate client id elimination, persistent session inheritable after client's reconnection, and the most important is that it's a autonomy system which provide continuous service under single point of failure which bridge doesn't have, so I sincerely hope that you can make any comments, code review, make performance testing under your scenario, etc., to make mosquitto cluster be better.

Thanks!

BRs,
Jianhui

PS. an oProfile report has attached in appendix, which shows that a more efficient timer management should be involved to save the CPU cycles which bring from expiration polling.

Appendix:

bridge cpu usage snapshot(18225 is the bridge):
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18225 mosquitt  20   0 45232 5576 1776 R 100.0  0.1   1:59.91 mosquitto(bridge)
18224 mosquitt  20   0 44704 5052 1760 R 91.0  0.1   1:28.08 mosquitto
18223 mosquitt  20   0 44708 5076 1760 R 82.7  0.1   1:30.28 mosquitto
4869 mosquitt  20   0 44708 5008 1764 R 79.4  0.1   1:41.62 mosquitto
4875 mosquitt  20   0 44720 5004 1764 R 78.0  0.1   1:38.25 mosquitto
4876 mosquitt  20   0 44724 5008 1764 R 75.7  0.1   1:38.51 mosquitto
2900 mosquitt  20   0 24480 4892 1572 R 71.4  0.1   1:25.55 mosquitto
2898 mosquitt  20   0 24480 4872 1572 S 68.1  0.1   1:26.22 mosquitto
2899 mosquitt  20   0 24488 4860 1572 R 66.1  0.1   1:25.79 mosquitto

cluster cpu usage snapshot:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19715 mosquitt  20   0 47632 7968 1796 R 73.4  0.1   2:17.25 mosquitto
19716 mosquitt  20   0 47660 7996 1796 R 72.4  0.1   2:20.55 mosquitto
19717 mosquitt  20   0 47512 7868 1796 R 70.7  0.1   2:16.84 mosquitto
6574 mosquitt  20   0 47796 8148 1800 R 64.4  0.1   2:14.92 mosquitto
6573 mosquitt  20   0 47928 8180 1800 S 63.1  0.1   2:17.60 mosquitto
6572 mosquitt  20   0 47808 8100 1800 R 62.8  0.1   2:15.44 mosquitto
3580 mosquitt  20   0 27364 7728 1604 R 62.8  0.1   1:48.19 mosquitto
3581 mosquitt  20   0 27552 7936 1604 R 62.1  0.1   1:48.50 mosquitto
3582 mosquitt  20   0 27824 8260 1604 S 60.8  0.1   1:47.63 mosquitto

Oprofile report:
CPU: Intel Haswell microarchitecture, speed 3500 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 6000
samples  %        linenr info                 image name               symbol name
5786584  52.8127  loop.c:101                  mosquitto                mosquitto_main_loop
2749353  25.0926  subs.c:388                  mosquitto                sub__search
546670    4.9893  database.c:856              mosquitto                db__message_write
267198    2.4386  subs.c:692                  mosquitto                retain__search.isra.2
...

               :int mosquitto_main_loop(struct mosquitto_db *db, mosq_sock_t *listensock, int listensock_count, int listener_max)
               :{
 /* mosquitto_main_loop total: 5786584 52.8127 */
 26163  0.2388 : HASH_ITER(hh_sock, db->contexts_by_sock, context, ctxt_tmp){
3211672 29.3121 : if(time_count > 0){
                            ...
540500  4.9330 : context->pollfd_index = -1;
                            ...
439382  4.0101 : if(context->events & EPOLLOUT) {
                            ...
691792  6.3138 : if(context->current_out_packet || context->state == mosq_cs_connect_pending || context->ws_want_write){



From: jianhui zhan
Sent: Friday, December 29, 2017 9:34
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.
 

yes, 2000 PUB/SUBs testing is more of a functional testing then a stress testing, I will do some more testing to verify the performance.



From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Friday, December 29, 2017 9:12
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.
 
Hi Jianhui,

Thank you for the further information, but I'm not sure the cluster can expand the performance.
The amount of '2000 PUB/SUBs' seems too small, it can be handled by using only one broker.
Could you simples the scenarios, and  show the performance change in according with the number of brokers?

Regards,
Tatsuzo


2017-12-29 2:32 GMT+09:00 jianhui zhan <hui6075@xxxxxxxxxxx>:

Hi Osawa,


I have continuously write and debug the non-centralized cluster, and test it with a light load, about 2000 SUB/PUB pair, fortunately it work as expected, without errors. Attached is the test report. More stress is needed in future testing.


These days I've also thought of some other business MQTT cluster, almost Erlang or Java based, which contains too much nodes/services such as broker, event dispatcher, subscription in-memory cache, persistence database, distributed coordinator, etc., with high latency(400ms~500ms), and depend on the availability of each key service.

In some ways, I think Mosquitto should take it's role as Redis in NoSQLs, ZeroMQ in Message Queues, make the very very low latency as it's most significant advantage..

From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org> on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Saturday, December 9, 2017 11:54
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.
 
Hi Jianhui,

Your e-mail software made the subjects include multi-byte characters.
Therefore, the subjects cannot be read on the archive:
https://dev.eclipse.org/mhonarc/lists/mosquitto-dev/ .
Mailing List Archives - mosquitto-dev
dev.eclipse.org
Eclipse is probably best known as a Java IDE, but it is more: it is an IDE framework, a tools framework, an open source project, a community, an eco-system, and a ...



Could you correct the subjects from the next post?

Regards,
Tatsuzo
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev

_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev


_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev


_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev


_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev


_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev

Back to the top