Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[mosquitto-dev] Advice needed (long, sorry)

Hi guys,

I would like to request some advice from you. I've been reading a lot about MQTT and all technologies around it in the last year.

We (three people) are trying to create an small company based on a product we're developing. It is a mobile app (20%) and a backend (80%) that will be conected using MQTT.

We have some constraints: budget and time (always the same) so we have decided to put it in the market progressively:

1st) Deploying our infrastructure on premise to test the platform with a very reduced number of users (~200), mainly friends and 2nd grade friends :-)

2nd) After that, we have a rack with three medium-size proliant servers we want to use. We expect to use them to host our infrastructure until:

a) The number of users reach a number that makes the bussines sustainable from an economic point of view (~100K). No matter if it is profitable at this point; we only expect at least that it is not a cost.

or b) Our infrastructure cannot scale more and we have to migrate it to AWS althoug the number of users does not garantees the business sustainability (this would be bad).

3rd) Once reached the number of users to make the platform profitable, we will migrate it to AWS and became a "serious" platform.

All of us have a job so this is a part-time "hobby". This is an attempt to have our company in about 2 years.

During 1st) and 2nd) all costs will be supported by our familiar budget as the platform will be free for all users. Once in 3rd) we expect to start charging a fee; this is the moment when we will be really a company.

For that reasons, we want to "assure" that our infrastructure can support an increasing number of users until 100K before we start charging a fee.

Our base is composed by:

- Android app + MQTT paho.
- Mosquitto 1.3.5 (1.4 next week) + mosquitto-auth-plug

- Haproxy load balancer between clients and Mosquitto.
- Haproxy load balancer between Mosquitto and the backend.
- A Java backend highly threaded.
- A MySQL cluster in master-master configuration.
- HirakiCP as connection pool to the MySQL cluster.
- Haproxy between the backend and the MySQL cluster.

- All running on Ubuntu Server 14.04.1 hosts that have been tuned to improve performance and scalability (the number of connections allowed, the parameters for TCP, number of sockets...).

- The messages we are sending through Mosquitto are API messages (client-backend, backend-client and client-client). We need TLS+QoS2+Persistence.

We are designing API messages very compact and taking advantages of topics to avoid transferring unneeded data (less bandwith, less ram, less processing). Also, we are designing the application to minimize the number of messages that need to be exchanged and the sending rate, Applications I/O to disk is being progressively monimized, too.

Really, everything is redundant; everything except the broker. We have tried some bridge configuration but they don't scale so it is not an option, but a temporary patch. After all, we still need a solution to make Mosquitto scale. Notice that 100K users does not mean
that all of them will be connected concurrently.

We don't know:

- How fast is the application being adopted by users (the faster, the better, but we need to be prepared to migrate to AWS or to scale fast).
- The number of concurrent connection we could expect having 100K users.
- Traffic patterns.

After deep analyses we are thinking on changing MySQl and use Cassandra instead, althoug we are not very experienced with Cassandra. But it seems to be not very complicated.

But Mosquitto...

We have read some experienced os people using Mosquitto sustaining 100K concurrent connection and more. But in our test we have reached ~25K concurrent connections at maximum, which is not enough (and operating not very well, to be realistic).

Could you give us some advice on how to improve these numbers at least to be able to reach our 100K users? We don't know, but maybe 20K concurrent connections are enough to support 100K users that are not connected at the same time.

Also, any general advice? for example we have read that zmq can help to scale Mosquitto, but we do not know if it should be used in the backend, to connect some mosquittos... or where.

We are used to build backends, but is the fist time using MQTT as the center of the platform and we feel nervous because cannot make the broker to grow. And we expect to have ~1M-2M users (no connections) in three years.

Any consideration, advice, criticism that can help us are wellcome.

Thank you. Kindest regards.

PS: once in 3rd) we should have funds to use HiveMQ, MessageSight applicance or whatever. We would like to continue using mosquitto if posible although is seem difficult.

Manuel Domínguez Dorado
ingeniero ARROBA ManoloDominguez PUNTO com

Back to the top