Re: [cyclonedds-dev] Intra-process transport mechanism

[Erik Boasson <eb@xxxxxxxxxxx>]

Hi Alan,

Cyclone has a “local delivery” mechanism that is used to forward data to all local readers (except when Iceoryx is involved), as long as they are in the same process/domain and regardless of the DDS DomainParticipant. The implementation is a simple loop over an array of pointers to reader history caches, and updating each history cache in turn.

When Iceoryx is involved, then the above is bypassed because it would interfere with Iceoryx (otherwise everything would get delivered twice).

The network stack (so UDP, normally) gets involved when it is not in the same process (and not via Iceoryx).

In Cyclone, this *always* happens, you can’t configure it not to do it this way. So I presume the “INTRA” transport mechanism refers to Connext and/or FastDDS, because I believe they used to perform communication between DomainParticipants in the same process via the network stack, and only later added an alternative transport for within the process. I don’t know all that much about them, but if I am correct, then in their approach data for which there are only readers in the same process will still traverse (most) of the RTPS stack, which is not the case for Cyclone.

> Also, if a thread is sleeping in dds_waitset_wait, how is it woken by
> the thread sending a message? It looks like it uses
> pthread_cond_timedwait()? What about when UDP is being used?

The reader history cache update causes the “data available” status to be raised and the read/query conditions to be re-evaluated. If waitsets are waiting for any of these, they get triggered by “pthread_cond_broadcast” (on POSIX platforms). Because the trigger is the reader history cache update, it doesn’t matter whether the data came from a local writer or the network.

Since you mention the performance_test tool, it might be interesting to know that while I personally prefer waitsets and read conditions, the fastest way to get data out of Cyclone is to use a listener for the “data available” event. The listeners in Cyclone get called by the thread updating the reader history cache, so it saves a context switch and avoids possible contention. The downside is that you have to be careful what you do in a listener, just like in POSIX you have to be careful what you do inside a signal handler.

As an aside, most of the community interactions happen on GitHub and some on Discord. You’re welcome to use the mailing list, but for questions where you think a wider audience might find it interesting, especially GitHub is probably the better choice.

Best regards,
Erik

> On 6 Jun 2023, at 15:49, Alan Beadle <ab.beadle@xxxxxxxxx> wrote:
> 
> Hi,
> 
> I understand that CycloneDDS would normally use UDP as the transport
> mechanism between processes. However, I was wondering what mechanism
> is used for intra-process communication.
> 
> I am using the performance_test tool [1] from Apex AI and would like
> to understand the results in better detail. Their docs only state that
> CycloneDDS uses an "INTRA" transport mechanism, which does not refer
> to anything in the CycloneDDS docs. Apparently this is meant to
> indicate that Cyclone will bypass the networking stack for
> intra-process. Can anyone provide more detail?
> 
> Also, if a thread is sleeping in dds_waitset_wait, how is it woken by
> the thread sending a message? It looks like it uses
> pthread_cond_timedwait()? What about when UDP is being used?
> 
> Thanks in advance,
> -Alan
> 
> [1] https://gitlab.com/ApexAI/performance_test/-/tree/master/performance_test#eclipse-cyclone-dds
> _______________________________________________
> cyclonedds-dev mailing list
> cyclonedds-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cyclonedds-dev