| Re: [mosquitto-dev] Advice needed (long, sorry) |
I'll continue with you privately, although of course you are welcome to share any results or experience.You will need to be comfortable building mosquitto 1.4. All the existing features are supported normally. It is straightforward to install ZMQ. Cassandra is easy to install on a single instance for dev/testing.A companion exec pgm (python) launches and manages the microservices (python and C) to support the swarm, enabling it to efficiently share retained storage and PUBLISHs using asynchronous message pipelines.The architecture is experimental. For production we will eliminate all SPFs and add auto-discovery, auto-scaling, etc.Federated swarms of swarms can be run by employing RabbitMQ - working on that feature now - goes across regions like our current nyt⨍aбrik.My team and I will package up a version today.mlOn Tue, Mar 10, 2015 at 3:54 AM, Manuel Domínguez Dorado <manolodd@xxxxxxxxx> wrote:Hi Michael,It sounds really interesting. Of course I would want to try!! This is exactly what I need. I'm now convinced of using AWS from the beginning but AWS does not offers "Mosquitto high scalability" therefore, we still have the same problem of making Mosquitto more scalable.As I said at the beginning of this thread, we are considering Cassandra, but not to provide a shared persistence storage (great idea!). I'm not sure where to put ZeroMQ and I don't control pyhton very much. Certainly, I'm waiting for hearing more from you :-)Best regards.2015-03-09 22:35 GMT+01:00 Laing, Michael <michael.laing@xxxxxxxxxxx>:Hi Manuel,If you are willing to work with experimental code based on mosquitto 1.4, I may be able to help you.I have a solution that allows you to run multiple brokers and interconnect them in a high performance manner.The brokers can be on the same machine (different ports), or on multiple machines in a LAN - you may need to load balance w HAProxy or equivalent.The brokers share retained storage using Cassandra - but I may be able to cut out Cassandra if that is an issue and use the in-memory store.ZeroMQ is required to connect the python microservices to the broker.It would give you much bigger numbers of connections - almost linear... I am planning to package and open source the code in the next few weeks, but could provide you with a preview.MichaelOn Mon, Mar 9, 2015 at 3:45 PM, Manuel Domínguez Dorado <manolodd@xxxxxxxxx> wrote:_______________________________________________With this configuration we achieved "only" 31K concurrent connections. We decided to change the server certificate, that was using 4096 bits, and create a new one using 1024 that should consume less resources. After this change,This time we used a set of 5 VM each one of them bombing the Mosquitto broker, creating idle connections (no susbcriptions, no publishings).Hi all,I'm still on my challenge to achieve 100K concurrent connections. I'm learning lots of things :-)this time we achieved up to 40K connections. it was still possible to create new connections to the broker, but connections already established started to be disconnected because no keepalive was received. We changed keepalive value to a higher value and this time we obtained 52K concurrent connections. After this number, the vistualization server we were using to bomb the broker become exhausted, reason why the number of concurrent connections is not greater.After this experiment, Roger, we have some suggestions:- Is there any way of making OpenSSL lib to use more than a single processor even being Mosquitto single threaded? It could be interesting as in our case the broker's server was almost unused except for the processor being used by Mosquitto (something similar to use "openssl speed -multi 4")- Keepalive mesages are not being sent properly when this number of establishes connections is maintained. I suppose this is for the same reason that the processor used by Mosquitto is exhausted and cannot process this amount of keepalive messages on time).In general, It would be desirable to implement muti-threading in Mosquitto, because it is a bit extrange to have a 8 core server almost idle and a Mosquitto running on it completely overloaded. Today the trend in new processors is not to have a greater frequency but a greater number of cores, so it seems logic to make use of them. Is there a plan to develop a multi-threaded Mosquitto? The possibility of doing TLS offloading over the "free" processors would be great, too.Ok, I'll continue posting here our advances.Kindest regards.2015-03-04 8:55 GMT+01:00 Manuel Domínguez Dorado <manolodd@xxxxxxxxx>:Hi,yes we started doing client tests using Java. It has no effect on the broker side, but on the client side, the test die at ~2K concurrent connections due tojava.lang.OutOfMemoryError (1/10 of connections achieved from nodejs). But it is a matter of memory resources. In real conditions neither the client nor the backend will support this great number of MQTT connections. Only the broker and the haproxy will support them.We will repeat the test tuning the memory used by nodejs and using several VM clones to start connecting to the broker.Ciao--2015-03-04 7:47 GMT+01:00 Stefano Costa <stefano.costa@xxxxxxxxxxx>:Il 03/03/2015 23:18, Manuel Domínguez Dorado ha scritto:
Hi,
- Desktop PC (4 cores, 8 GB RAM) runing a single nodejs test that
establishes TLS connections to the Mosquitto broker at 10 new
connections/sec.
Again, did you try with anything different from nodejs at client side? Python etc? (should not make it any different anyway)
--
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev--
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev