| Re: [mosquitto-dev] A decentralize Mosquitto cluster design. |
Cool, hope it can cover your test.
Jianhui: I think single process would be better than multi-thread, Nginx/Redis gave the answer. For software LB, LVS is much better then nginx and ELB in my opinion. And if you let me choose, I will choose 16 CPU single box, taskset each broker to one core under memory bus limitation. Hardware BUS is much faster then the cable.QuestionWill it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....?Jianhui: It will work, but I don't know whether the memory bus would be a bottleneck for so many instances. It is hard to give a desired performance, hardware architecture(SMP, NUMA, MPI) are uncertain factors, theoretically speaking, for each SMP machine/NUMA node, 4 instance would be the limitation..
Normal Mosquitto is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the traffic/load ....<or> is the idea to use small 2 cpu servers in clustered network mode behind LB like nginx or aws ELB?
BTW, I have tested with RabbitMQ, I found that it doesn't support MQTT retain message and QoS=2, is that I should install the newest version?
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Tuesday, January 16, 2018 17:32
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.Hi Jian
I've pulled the clustered mosquitto branch code, compiled with those flags and got it running now with 2 instances - did a quick test connecting to one broker on node1 and then another client connect to node2 and sub on both client to topic "#" - when I pub a test msg on client1, client2 recv it, so the cluster is working - the 2 brokers also ack that they see each other - fairly easy to set this up :)
I am using one server with diff ports to run this cluster ...node1 is 127.0.0.1:8000 and node2 is 127.0.0.1:9000 ...seem 2 be working just fine on same server with different ports ^_^.
QuestionWill it even make sense to try this - get server with 16 CPU's and run 8 brokers on single box....? Normal Mosquitto is only using a single CPU at time - with this option you can already use one server with more CPU's to handle the traffic/load ....<or> is the idea to use small 2 cpu servers in clustered network mode behind LB like nginx or aws ELB?
But in short - got it running and can assist in some testing scenarios - exiting times 2 be honest, - if this really works under load, means we can scale the awesome mosquitto broker to bigger loads ^_^ - "big up for C & mosquitto i say" ^_^
On Mon, Jan 15, 2018 at 6:57 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:
Hi Oegma,
That will be wonderful. And please be attention that:
Please feel free to let me know for any confusion😉
- the cluster is based on mosquitto develop branch and there's a known crash issue #505 under some scenario(such as a huge mount of clients disconnect at the same time) that may disturb the testing.
- SSL/TLS between brokers and persistent session with QoS>0 has not been verified, I will test it asap.
- If your brokers distribute in different data center with a high latency, please increase cluster_retain_delay in mosquitto.conf to ensure retain message would be receive as expected.
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Monday, January 15, 2018 23:42
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.Hi Jian
I will be able to deploy it to my own env - compiling with those flags should not be an issue :)...LB is also not an issue - can set the env up with or without a LB
Never tried Tsung, but looks like solid testing system
On Mon, Jan 15, 2018 at 5:36 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:
Hi Oegma,
It's my honor that you are willing to make the test.
Currently I have no open resources/servers for a public testing, so could you deploy the cluster on your environment? build mosquitto with WITH_CLUSTER=yes, WITH_BRIDGE=
no, fill all the nodes' ip address and port in mosquitto.conf, then take them as the back end server of the LB. If you use Tsung, LB is not needed. And it is recommend to use plain TCP between brokers.. I'll also make some testing with RabbitMQ.
I believe that even if Erlang is suitable for message queue, it must slower then C😉And maybe a centralized system(as you mentioned, to make some clever routing service) maybe more efficient but NOT reliable due to single point failure.
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Oegma2 <oegma2@xxxxxxxxx>
Sent: Monday, January 15, 2018 21:42
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A decentralize Mosquitto cluster design.Hi - how can I get involved with testing this setup? Been looking at diff options to scale mosquitto myself in the past, and even bridge options was on the table, creating some clever routing broker/cluster - mapping between RabbitMQ and mosquitto..the list goes on - even custom ZMQ code was once an option to look at and I even learned erlang to write a middle layer clustered system...
So would love to assist with the cluster design testing/setup
What do I need to start
@Roger: Is this cluster mosquitto something you be looking at going forward to scale mosquitto out? Or do you have other plans on making this powerful broker scale beyond a single machine?
On Mon, Jan 15, 2018 at 2:46 PM, jianhui zhan <hui6075@xxxxxxxxxxx> wrote:
Hi Roger, Tatsuzo, Tifaifai, anyone who is interested with in mosquitto cluster,
I've tested the mosquitto cluster by appropriate amount of simultaneous users, connect/subscribe/publish TPS, which gave a proper pressure(carry as much TPS as possible w/o causing latency) to the system.It is meaningless to give a plain benchmark w/o any comparison, so I've also tested mosquitto bridge under a same scenario which is quite similar with our smart home service platform architecture and scenario(10 brokers, 20k subscribers, 1k publishes from 10 publishers which use persistent TCP from a http server):9 brokers on 3 OpenStack VMs(4cores 8G RAM), after 30k persistant subscribers setup, send 10k publishes per second(actually only 2.5kps publishes due to client's bottleneck) from publishers(one publish per client use non-persistant TCP), with payload length = 744 bytes.
With QoS=0, both cluster and bridge(topic # both 0) can work as normal, with QoS=1(topic # both 1), the CPU usage for each broker stabilise at at 65%-75% in cluster, but 100% for the bridge broker during the publish phase, and meanwhile 30% messages lost due to bridge broker's overload(see appendix). More detail test reports which include connect/request response time, network throughput, server monitoring are available under https://github.com/hui6075/mosquitto/tree/develop/benchmark .
I believe that the situation will be worse for bridge under QoS=2, but will not deteriorate for cluster since publish messages forward with it's origin QoS and process with QoS=0 in the cluster. The mosquitto cluster equalized all brokers' load, and it is different with bridge, it is an entire MQTT logic broker for external clients such as duplicate client id elimination, persistent session inheritable after client's reconnection, and the most important is that it's a autonomy system which provide continuous service under single point of failure which bridge doesn't have, so I sincerely hope that you can make any comments, code review, make performance testing under your scenario, etc., to make mosquitto cluster be better.
Thanks!
BRs,Jianhui
PS. an oProfile report has attached in appendix, which shows that a more efficient timer management should be involved to save the CPU cycles which bring from expiration polling.
Appendix:
bridge cpu usage snapshot(18225 is the bridge):PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND18225 mosquitt 20 0 45232 5576 1776 R 100.0 0.1 1:59.91 mosquitto(bridge)18224 mosquitt 20 0 44704 5052 1760 R 91.0 0.1 1:28.08 mosquitto18223 mosquitt 20 0 44708 5076 1760 R 82.7 0.1 1:30.28 mosquitto4869 mosquitt 20 0 44708 5008 1764 R 79.4 0.1 1:41.62 mosquitto4875 mosquitt 20 0 44720 5004 1764 R 78.0 0.1 1:38.25 mosquitto4876 mosquitt 20 0 44724 5008 1764 R 75.7 0.1 1:38.51 mosquitto2900 mosquitt 20 0 24480 4892 1572 R 71.4 0.1 1:25.55 mosquitto2898 mosquitt 20 0 24480 4872 1572 S 68.1 0.1 1:26.22 mosquitto2899 mosquitt 20 0 24488 4860 1572 R 66.1 0.1 1:25.79 mosquitto
cluster cpu usage snapshot:PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND19715 mosquitt 20 0 47632 7968 1796 R 73.4 0.1 2:17.25 mosquitto19716 mosquitt 20 0 47660 7996 1796 R 72.4 0.1 2:20.55 mosquitto19717 mosquitt 20 0 47512 7868 1796 R 70.7 0.1 2:16.84 mosquitto6574 mosquitt 20 0 47796 8148 1800 R 64.4 0.1 2:14.92 mosquitto6573 mosquitt 20 0 47928 8180 1800 S 63.1 0.1 2:17.60 mosquitto6572 mosquitt 20 0 47808 8100 1800 R 62.8 0.1 2:15.44 mosquitto3580 mosquitt 20 0 27364 7728 1604 R 62.8 0.1 1:48.19 mosquitto3581 mosquitt 20 0 27552 7936 1604 R 62.1 0.1 1:48.50 mosquitto3582 mosquitt 20 0 27824 8260 1604 S 60.8 0.1 1:47.63 mosquitto
Oprofile report:CPU: Intel Haswell microarchitecture, speed 3500 MHz (estimated)Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 6000samples % linenr info image name symbol name5786584 52.8127 loop.c:101 mosquitto mosquitto_main_loop2749353 25.0926 subs.c:388 mosquitto sub__search546670 4.9893 database.c:856 mosquitto db__message_write267198 2.4386 subs.c:692 mosquitto retain__search.isra.2...
:int mosquitto_main_loop(struct mosquitto_db *db, mosq_sock_t *listensock, int listensock_count, int listener_max):{/* mosquitto_main_loop total: 5786584 52.8127 */26163 0.2388 : HASH_ITER(hh_sock, db->contexts_by_sock, context, ctxt_tmp){3211672 29.3121 : if(time_count > 0){...540500 4.9330 : context->pollfd_index = -1;...439382 4.0101 : if(context->events & EPOLLOUT) {...691792 6.3138 : if(context->current_out_packet || context->state == mosq_cs_connect_pending || context->ws_want_write){
From: jianhui zhan
Sent: Friday, December 29, 2017 9:34
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.yes, 2000 PUB/SUBs testing is more of a functional testing then a stress testing, I will do some more testing to verify the performance.
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Friday, December 29, 2017 9:12
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.Hi Jianhui,
Thank you for the further information, but I'm not sure the cluster can expand the performance.
The amount of '2000 PUB/SUBs' seems too small, it can be handled by using only one broker.
Could you simples the scenarios, and show the performance change in according with the number of brokers?
Regards,Tatsuzo
2017-12-29 2:32 GMT+09:00 jianhui zhan <hui6075@xxxxxxxxxxx>:
Hi Osawa,
I have continuously write and debug the non-centralized cluster, and test it with a light load, about 2000 SUB/PUB pair, fortunately it work as expected, without errors. Attached is the test report. More stress is needed in future testing.
These days I've also thought of some other business MQTT cluster, almost Erlang or Java based, which contains too much nodes/services such as broker, event dispatcher, subscription in-memory cache, persistence database, distributed coordinator, etc., with high latency(400ms~500ms), and depend on the availability of each key service.
In some ways, I think Mosquitto should take it's role as Redis in NoSQLs, ZeroMQ in Message Queues, make the very very low latency as it's most significant advantage..
From: mosquitto-dev-bounces@eclipse.org <mosquitto-dev-bounces@eclipse.org > on behalf of Tatsuzo Osawa <tatsuzo.osawa@xxxxxxxxx>
Sent: Saturday, December 9, 2017 11:54
To: General development discussions for the mosquitto project
Subject: Re: [mosquitto-dev] A non-centralize Mosquitto cluster design.Hi Jianhui,
Your e-mail software made the subjects include multi-byte characters.
Therefore, the subjects cannot be read on the archive:
https://dev.eclipse.org/mhonarc/lists/mosquitto-dev/ .
Could you correct the subjects from the next post?
Regards,
Tatsuzo
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev