Re: [mosquitto-dev] bridges for dynamically assigned hosts.

[Roger Light <roger@xxxxxxxxxx>]

Hi Duncan,

Your solutions 1 and 2 would work, but as you say they aren't ideal.

Solution 3 would be great. Reloading the bridge config on a SIGHUP
would be the most straightforward approach for sure. The reason this
isn't already done is the panic attacks I had trying to consider all
of the insane things users might try and do, and having to cope with
them :) In principle it's not that bad though. If you'd like to take a
look at it, I'd be very happy to receive a patch. I would do something
like load the bridge config into an array, then compare to the
existing bridges, disconnecting or creating bridges as appropriate.
Any bridges that already exist (based on the  connection name) would
ideally be kept connected and just the topics/other changed - but
you'd need to see what parameters had changed to determine whether or
not that was possible.

Runtime control like $SYS - yes, I've considered it. Again, I'd be
happy to have that functionality but defining the interface requires
some thought.

Cheers,

Roger



On Wed, Aug 19, 2015 at 12:45 AM, Duncan Law <mrdunk@xxxxxxxxx> wrote:
> TLDR: my thoughts about dynamic clustering of brokers.
>
> hi, first post!
> i'm enjoying playing with Mosquitto but
> i'm looking for a little advice on the bast way to achieve the following:
>
> my current hobby project has many small embedded nodes acting as MQTT
> clients.
> a subset of these nodes are also brokers running on OpenWRT.
> brokers advertise them selves via mDNS.
> clients pick a broker at random from those currently online.
> the brokers will all bridge to each other so clients connected to different
> brokers can communicate.
>
> so far, so good.
>
> where things become less obvious:
> once this is all working i don't want to have to configure anything.
> i would like the broker nodes to dynamically build the bridges with their
> neighbors.
>
> currently under Mosquitto it's not possible to change the broker information
> of a running instance so
> when new broker nodes come online i have no way to update the existing
> brokers bridge records.
>
>
> possible solution 1:
> re-generate mosquitto.conf periodically with the updated broker list and
> restart the mosquitto binary.
> this is less drastic than it sounds because all clients already re-try other
> brokers in the event a broker drops off the network.
> but it's an ugly solution. we can do better.
>
>
>
> possible solution 2:
> a binary running on *all* the broker nodes that acts as the bridge
> functionality.
> something like this:
>  - builds a list of all brokers on the network from mDNS information.
>  - subscribes to "#/" on the broker and forwards all topics to all other
> brokers (having modified the topic with a "/broker/" identifier).
>  - re-writes any incoming topics with the "/broker/" identifier in the topic
> (these will have been sent by other bridge programs) removing the "/broker/"
> identifier. it will need to keep a temporary record of these topics so it
> knows not to re-send when they are re-published.
>
> this should work well but re-implementing the bridge functionality seems
> pointless when the Mosquitto broker already has most of what we need...
> also, it's not possible to ask Mosquitto what topics are currently
> subscribed on a broker so i'll have to send *everything* *everywhere* unless
> we explicitly tell it which topics... (the current bridge implementation
> doesn't do any better though.)
>
>
>
> possible solution 3:
> modify the Mosquito codebase to allow dynamic updates of Bridges.
> can any one tell me if this is on the roadmap for Mosquitto?
> it would be useful for anyone trying to dynamically scale a cluster of
> Mosquitto servers without re-starting jobs.
>
> presuming nobody has done any work on this already,
> what would people's thoughts on an interface be?
> re-writing mosquitto.conf and performing a SIGHUP would be closest to the
> current model...
>
> can anyone with knowledge of the codebase suggest whether there is anything
> fundamental to the current bridging code that means it can't be modified
> while Mosquitto is running?
> obviously we'll need to disable a bridge whle working on it. probably
> easiest to remove it completely, modify, then bring it up again from
> scratch...
>
>
>
> other tangents:
> has anybody considered the possibility of modifying running config using an
> interface similar to "$SYS/#" ?
> security would need to be handled carefully... but mosquitto.conf shows we
> can limit certain topics to certain users.
> no reason something like this would have to be part of the main binary; a
> separate binary could subscribe to the control topic and build a new
> configuration file according to the results.
>
>
> anyway, thoughts, suggestions, etc welcome.
> dunk.
>
> _______________________________________________
> mosquitto-dev mailing list
> mosquitto-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/mosquitto-dev