Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » 4DIAC - Framework for Distributed Industrial Automation and Control » mqtt(Content of mqtt message is not correct)
mqtt [message #1754123] Tue, 14 February 2017 15:59 Go to next message
Raimund Bruening is currently offline Raimund BrueningFriend
Messages: 4
Registered: June 2016
Junior Member
I'm testing mqtt in forte running under Linux.
The subscriber does not receive the correct string. External paho or mosquitto subscriber are also receiving this string.
Please have a look a my attached 4diac screen shot.

index.php/fa/28441/0/
Re: mqtt [message #1754127 is a reply to message #1754123] Tue, 14 February 2017 16:05 Go to previous messageGo to next message
Jose Maria Jesus Cabral Lassalle is currently offline Jose Maria Jesus Cabral LassalleFriend
Messages: 66
Registered: February 2016
Member
Try adding "raw[]." before the mqtt ID like:

raw[].mqtt[tcp://localhost....

See:

https://www.eclipse.org/forums/index.php/t/1074332/

Also, you don't need to connect an event to the RSP of the subscriber. When a packet arrives, the IND will trigger an event.
Re: mqtt [message #1754289 is a reply to message #1754127] Thu, 16 February 2017 08:30 Go to previous messageGo to next message
Raimund Bruening is currently offline Raimund BrueningFriend
Messages: 4
Registered: June 2016
Junior Member
Thanks,

it works.
Re: mqtt [message #1754400 is a reply to message #1754289] Fri, 17 February 2017 09:39 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Great.

Just for clarification. The main reason for this additional parameter is that mqtt does not define a payload format. With the additional param you specify in 4diac an so called communication layer to be used on top of MQTT. This communication layer is now in charge of taking the data from IEC 61499 and generate an according payload format or take the payload and transform it for the FB's outputs. the raw-layer is just accepting strings and sends the raw string content. If you need other payload formats you can also write your own layer. Please have a look on the communication architecture documentation which can be found in the FORTE development section of 4diac's documentation.

Cheers,
Alois
Re: mqtt [message #1756319 is a reply to message #1754400] Wed, 15 March 2017 11:06 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
I will ask here to keep the forum tidy, but if you think this should be a separate post, split it or let me know so I can do it myself.

I was trying to communicate over MQTT (for that, I compiled Paho MQTT C library and added the following command line arguments to cmake in the setup_posix.sh: -DFORTE_MODULE_SysFs=ON -DFORTE_COM_PAHOMQTT=ON -DFORTE_COM_PAHOMQTT_DIR="../../../paho.mqtt.c") and added a publisher and subscriber FBs as shown in the images:

Publisher
Subscriber
EDIT: The http:// should not be there and the port should be 1883. These are just typos when restoring to the first version after the tests I carried out and they are not the source of the problem.

I have also tries swapping raw[] for fbdk[] as suggested in MQTTComLayer.h (line 21):

MQTTComLayer.h (line 21)

And the only difference is that I'm getting a INVALID_ID message instead of TERMINATED in both cases.

MQTTComLayer.cpp only has 3 posible spots for TERMINATED, messages as shown in the next image: the first one is overwritten before returning, and the second and third ones are for MQTT clients and servers which are not implemented. INVALID_ID is just present once and can also be seen in the image.

MQTTComLayer.cpp (lines 73-104)

My guess is that the error comes from a lower layer, so I will inspect raw and fbdk layers now but if you guys have any idea about what could be the cause please let me know.

UPDATE: I only found TERMINATED in rawdatacomlayer.cpp as seen in the following image.

rawdatacomlayer.cpp (lines 48-64)

Correct me if I'm mistaken, but I think raw only works for publishers with at least one data input, so this means that fbdk should be my option. This also tells me that the ID I showed in the first 2 images must have some kind of error, are domain names allowed or do I need to give the IP address?

UPDATE 2: I've tried to use the IP address and it stays yielding the same.

[Updated on: Wed, 15 March 2017 11:46]

Report message to a moderator

Re: mqtt [message #1756364 is a reply to message #1756319] Wed, 15 March 2017 20:52 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Thank for the elaborate description and analysis. You are right the raw layer is currently only working with a single in or output and this in or output has to be of type STRING. The problem for PUB and sUb 0 is that the raw layer wouldn't know what to generate as payload for the MQTT messsage. With FBDK a 0x5 will be send for pure event PUBs. So this could be better on your case.

Regarding the invalid ID I see two issues. First of all I think you mixed up ClientID and Topic name. The ClientID should be the second mqtt parameter and currently it needs to be the same for all MQTT pubs and subs within one device. We know this is a little bit inconvenient and we should improve this. The second thing can really be that the domain names is also an issue. However this depends on the paho lib and if your device supports name resolving. For example on the Bosch Rexroth PLC name resolving was not working and therefore I had to replace the the domain names with ips.

I hope this helps.
Re: mqtt [message #1756381 is a reply to message #1756364] Thu, 16 March 2017 08:29 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
Alois Zoitl wrote on Wed, 15 March 2017 20:52
Thank for the elaborate description and analysis. You are right the raw layer is currently only working with a single in or output and this in or output has to be of type STRING. The problem for PUB and sUb 0 is that the raw layer wouldn't know what to generate as payload for the MQTT messsage. With FBDK a 0x5 will be send for pure event PUBs. So this could be better on your case.


FBDK then. Is it possible to use a single MQTT layer? Just including as ID:
mqtt[tcp://ip-address:port, client-id, topic]


Alois Zoitl wrote on Wed, 15 March 2017 20:52
Regarding the invalid ID I see two issues. First of all I think you mixed up ClientID and Topic name. The ClientID should be the second mqtt parameter and currently it needs to be the same for all MQTT pubs and subs within one device. We know this is a little bit inconvenient and we should improve this.


The PUBLISH_0 and SUBSCRIBE_0 FBs are in different HW devices, each with its own client id (4diac-pub-test and 4diac-sub-test respectively) and they are both using the same topic (4diac-events). The idea is that one device triggers an event on the other device, as a first test for MQTT communication with 4DIAC.

Alois Zoitl wrote on Wed, 15 March 2017 20:52
The second thing can really be that the domain names is also an issue. However this depends on the paho lib and if your device supports name resolving. For example on the Bosch Rexroth PLC name resolving was not working and therefore I had to replace the the domain names with ips.


Name resolving is working on my HW. I tried to ping the domain name and the correct ip address is shown. Have also tried to input this ip address manually but it didn't help.

I think that the problem may be inside the MQTTHandler::registerLayer() method. Is there anyhow that I can check that MQTT paho was properly linked to FORTE?
Re: mqtt [message #1756384 is a reply to message #1756381] Thu, 16 March 2017 09:38 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
I made the following code change in MQTTComLayer.cpp as previously both eWrongClientID and eConnectionFailed where being turned into eInitInvalidId to try to debug my case and compiled FORTE again with the same cmake configuration:
MQTTComLayer.cpp (lines 73-109)

This should not have fixed the problem, I think I must have had something not properly compiled previous times. The change may be interesting to keep for future versions though. The setup is properly working without the use of FBDK.

[Updated on: Thu, 16 March 2017 09:44]

Report message to a moderator

Re: mqtt [message #1756435 is a reply to message #1756381] Thu, 16 March 2017 20:09 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Adrian Orive wrote on Thu, 16 March 2017 08:29

FBDK then. Is it possible to use a single MQTT layer? Just including as ID:
mqtt[tcp://ip-address:port, client-id, topic]


No. The problem is that MQTT does not define a payload format and therfore you always need in 4diac a layer on top of MQTT layer.

Adrian Orive wrote on Thu, 16 March 2017 08:29

The PUBLISH_0 and SUBSCRIBE_0 FBs are in different HW devices, each with its own client id (4diac-pub-test and 4diac-sub-test respectively) and they are both using the same topic (4diac-events). The idea is that one device triggers an event on the other device, as a first test for MQTT communication with 4DIAC.


Not sure if i got your idea. This will only work if the pub and sub are mapped on two different devices. In your screenshot both where mapped onto the same device and then you need to use the same clientid for both FBs otherwise the second FB will not work.

Adrian Orive wrote on Thu, 16 March 2017 08:29

I think that the problem may be inside the MQTTHandler::registerLayer() method. Is there anyhow that I can check that MQTT paho was properly linked to FORTE?


I have a hard time to believe this as the MQTT is used by many users which din't had such problems.
Re: mqtt [message #1756458 is a reply to message #1756435] Fri, 17 March 2017 08:41 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
Alois Zoitl wrote on Thu, 16 March 2017 20:09
Adrian Orive wrote on Thu, 16 March 2017 08:29

FBDK then. Is it possible to use a single MQTT layer? Just including as ID:
mqtt[tcp://ip-address:port, client-id, topic]


No. The problem is that MQTT does not define a payload format and therfore you always need in 4diac a layer on top of MQTT layer.


I understand that. I meant for PUBLISH_0 and SUBSCRIBE_0 FBs, as they do not need any payload. I tried it and worked just fine, so I can confirm that PUBLISH_0 and SUBSCRIBE_0 MQTT FBs do not require an additional communication layer.

Alois Zoitl wrote on Thu, 16 March 2017 20:09
Adrian Orive wrote on Thu, 16 March 2017 08:29

The PUBLISH_0 and SUBSCRIBE_0 FBs are in different HW devices, each with its own client id (4diac-pub-test and 4diac-sub-test respectively) and they are both using the same topic (4diac-events). The idea is that one device triggers an event on the other device, as a first test for MQTT communication with 4DIAC.


Not sure if i got your idea. This will only work if the pub and sub are mapped on two different devices. In your screenshot both where mapped onto the same device and then you need to use the same clientid for both FBs otherwise the second FB will not work.


Those FBs are in two different HW devices, sorry if I didn't make that clear from the start. They are both with a white background as they are not defined in the app view, but inside each HW resource. I already show the part of the code that raises an INVALID ID error if either the host or the client id of every mqtt FB in a HW device are different.

Alois Zoitl wrote on Thu, 16 March 2017 20:09
Adrian Orive wrote on Thu, 16 March 2017 08:29

I think that the problem may be inside the MQTTHandler::registerLayer() method. Is there anyhow that I can check that MQTT paho was properly linked to FORTE?


I have a hard time to believe this as the MQTT is used by many users which din't had such problems.


Sorry, I didn't mean that there was a bug there, I was just saying that I suspect that the error was being raised from that function. I have it working properly both for empty payloads with just a single communication layer (mqtt[...]) and for strings as payloads with two communication layers (raw[].mqtt[...]).

[Updated on: Fri, 17 March 2017 09:34]

Report message to a moderator

Re: mqtt [message #1756501 is a reply to message #1756458] Fri, 17 March 2017 22:13 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Adrian Orive wrote on Fri, 17 March 2017 08:41

I understand that. I meant for PUBLISH_0 and SUBSCRIBE_0 FBs, as they do not need any payload. I tried it and worked just fine, so I can confirm that PUBLISH_0 and SUBSCRIBE_0 MQTT FBs do not require an additional communication layer.


I don't know that MQTT allows also empty messages. Interesting and good to know. But it is a bit of a hack and can be fragile. Maybe it would be better to expand the raw layer also to PUB/SUB 0.

Adrian Orive wrote on Fri, 17 March 2017 08:41

Those FBs are in two different HW devices, sorry if I didn't make that clear from the start. They are both with a white background as they are not defined in the app view, but inside each HW resource. I already show the part of the code that raises an INVALID ID error if either the host or the client id of every mqtt FB in a HW device are different.

Sorry I mixed up the screenshots. Yes with your explanation it your intention and setup is clearer.


Adrian Orive wrote on Fri, 17 March 2017 08:41

Sorry, I didn't mean that there was a bug there, I was just saying that I suspect that the error was being raised from that function. I have it working properly both for empty payloads with just a single communication layer (mqtt[...]) and for strings as payloads with two communication layers (raw[].mqtt[...]).

That's good to here. But still I'm a bit curios why it didn't work in the first place.

[Updated on: Fri, 17 March 2017 22:15]

Report message to a moderator

Re: mqtt [message #1757806 is a reply to message #1756501] Mon, 20 March 2017 07:34 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
Alois Zoitl wrote on Fri, 17 March 2017 22:13
Adrian Orive wrote on Fri, 17 March 2017 08:41

I understand that. I meant for PUBLISH_0 and SUBSCRIBE_0 FBs, as they do not need any payload. I tried it and worked just fine, so I can confirm that PUBLISH_0 and SUBSCRIBE_0 MQTT FBs do not require an additional communication layer.


I don't know that MQTT allows also empty messages. Interesting and good to know. But it is a bit of a hack and can be fragile. Maybe it would be better to expand the raw layer also to PUB/SUB 0.


Empty payloads are allowed in MQTT's specification, so I would not consider it to be fragile. Using a single layer sounds fine to me, but if you want to have 2, fbdk[].mqtt[...] also works. From a network efficiency point of view this 2 layer messages are sending non-needed bytes that in a constrained application may be relevant.

Alois Zoitl wrote on Fri, 17 March 2017 22:13
That's good to here. But still I'm a bit curios why it didn't work in the first place.


I was using a WSTRING data output for one of the SUBSCRIBE_1 FBs using the raw data layer which requires STRING outputs.

So my conclussions are:

  1. A single layer setup (mqtt[...]) can only be used for PUBLISH_0 and SUBSCRIBE_0 FBs as MQTT does not enforce any payload.
  2. The raw data layer can be used in a two layer setup (raw[].mqtt[...]) only for PUBLISH_1 and SUBSCRIBE_1 FBs as this layer enforces a single input/output. Only STRING inputs or outputs are allowed, not even WSTRING ones will work.
  3. The fbdk layer can be used as two layer alternative (fbdk[].mqtt[...]) for PUBLISH_0 and SUBSCRIBE_0 FBs, including a non-meaningful payload. This increases both the message size and the processing time so I would not recommend it as it is not needed.
  4. Using the fbdk layer for higher ammount of inputs or outputs has not been tested.
Re: mqtt [message #1757864 is a reply to message #1757806] Mon, 20 March 2017 21:15 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Adrian Orive wrote on Mon, 20 March 2017 07:34
Alois Zoitl wrote on Fri, 17 March 2017 22:13
Adrian Orive wrote on Fri, 17 March 2017 08:41

I understand that. I meant for PUBLISH_0 and SUBSCRIBE_0 FBs, as they do not need any payload. I tried it and worked just fine, so I can confirm that PUBLISH_0 and SUBSCRIBE_0 MQTT FBs do not require an additional communication layer.


I don't know that MQTT allows also empty messages. Interesting and good to know. But it is a bit of a hack and can be fragile. Maybe it would be better to expand the raw layer also to PUB/SUB 0.


Empty payloads are allowed in MQTT's specification, so I would not consider it to be fragile. Using a single layer sounds fine to me, but if you want to have 2, fbdk[].mqtt[...] also works. From a network efficiency point of view this 2 layer messages are sending non-needed bytes that in a constrained application may be relevant.


With fragile I meant that it is fragile from 4diacs' mqtt layer implementation. Which some further investigation may be needed.

Compared to the overhead of a TCP message sending one or two additional bytes seams to be negligible. One could for example send the event count that ocured so far. This would allow to detect if messages where lost on the way.

Re: mqtt [message #1757907 is a reply to message #1757864] Tue, 21 March 2017 15:53 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
Alois Zoitl wrote on Mon, 20 March 2017 21:15
Adrian Orive wrote on Mon, 20 March 2017 07:34
Alois Zoitl wrote on Fri, 17 March 2017 22:13
I don't know that MQTT allows also empty messages. Interesting and good to know. But it is a bit of a hack and can be fragile. Maybe it would be better to expand the raw layer also to PUB/SUB 0.


Empty payloads are allowed in MQTT's specification, so I would not consider it to be fragile. Using a single layer sounds fine to me, but if you want to have 2, fbdk[].mqtt[...] also works. From a network efficiency point of view this 2 layer messages are sending non-needed bytes that in a constrained application may be relevant.


With fragile I meant that it is fragile from 4diacs' mqtt layer implementation. Which some further investigation may be needed.


If the MQTT layer is the top most one, the sendData() function will be called with no arguments a 0 in as the second argument if PUBLISH_0 is being used so an empty payload will be sent. I don't think it needs further checking but it could be added.

Alois Zoitl wrote on Mon, 20 March 2017 21:15
Compared to the overhead of a TCP message sending one or two additional bytes seams to be negligible. One could for example send the event count that ocured so far. This would allow to detect if messages where lost on the way.


TCP already offers you losless guarranty. However the event number may be useful for other reasons. The two bytes may be negligible compared with TCP overhead (20 bytes, 10%) in most situations but for ultra-optimized applications even these two bytes may count. The communication layers allow both options to be implemented and be the user the one who selects which to use.

[Updated on: Wed, 22 March 2017 07:40]

Report message to a moderator

Re: mqtt [message #1757972 is a reply to message #1757907] Wed, 22 March 2017 12:51 Go to previous messageGo to next message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Adrian Orive wrote on Tue, 21 March 2017 15:53

If the MQTT layer is the top most one, the sendData() function will be called with no arguments a 0 in as the second argument if PUBLISH_0 is being used so an empty payload will be sent. I don't think it needs further checking but it could be added.

The only reason I have some doubts is the pointer which is pointing to SD_1 which is not valid in case of size 0. Maybe there is a general issue here. Have to think this through a bit.

Adrian Orive wrote on Tue, 21 March 2017 15:53

TCP already offers you losless guarranty. However the event number may be useful for other reasons. The two bytes may be negligible compared with TCP overhead (20 bytes, 10%) in most situations but for ultra-optimized applications even these two bytes may count. The communication layers allow both options to be implemented and be the user the one who selects which to use.

This depends to whome the TCP connection is. In MQTT only to the broker. I noticed but can not confirm that if the subscriber is gone or connects later it does not get any information how many messages have been. Sometimes dont even get the latest value but only updates. But I've to say I'm not very into MQTT.
Re: mqtt [message #1757998 is a reply to message #1757972] Wed, 22 March 2017 16:30 Go to previous messageGo to next message
Adrian Orive is currently offline Adrian OriveFriend
Messages: 38
Registered: March 2017
Member
Alois Zoitl wrote on Wed, 22 March 2017 12:51
Adrian Orive wrote on Tue, 21 March 2017 15:53

TCP already offers you losless guarranty. However the event number may be useful for other reasons. The two bytes may be negligible compared with TCP overhead (20 bytes, 10%) in most situations but for ultra-optimized applications even these two bytes may count. The communication layers allow both options to be implemented and be the user the one who selects which to use.

This depends to whome the TCP connection is. In MQTT only to the broker. I noticed but can not confirm that if the subscriber is gone or connects later it does not get any information how many messages have been. Sometimes dont even get the latest value but only updates. But I've to say I'm not very into MQTT.


I agree. Just as a side note: MQTT offers a way of getting messages previous to the connection called retained messages, but they can only store one message per topic. This is probably not implemented in the MQTT layer for 4DIAC as other MQTT features but should be easy to implement them.

[Updated on: Wed, 22 March 2017 16:32]

Report message to a moderator

Re: mqtt [message #1758061 is a reply to message #1757998] Thu, 23 March 2017 08:30 Go to previous message
Alois Zoitl is currently offline Alois ZoitlFriend
Messages: 454
Registered: January 2014
Senior Member
Sounds interesting. If you like to do experiments and enhancments we are happy to hear about it. Also improvements to the MQTT layer are warmly welcome. In the docs there is a section on how you can rather easily contribute to FORTE using git and gerrit.
Previous Topic:4DIAC: Copy-Paste FBs
Next Topic:4DIAC Demo Model
Goto Forum:
  


Current Time: Sun Feb 18 03:26:44 GMT 2018

Powered by FUDForum. Page generated in 0.02960 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software