| Forte stops running on Raspberry Pi after a certain time [message #1772713] |
Thu, 14 September 2017 04:49  |
Eclipse User |
|
|
|
Hello again.
I have my RPi set up so that forte is started automatically in rc.local.
However, after some time (usually a few hours, up to a bit less than a day), the process just stops. It does not appear that the RPi has rebooted or anything, because there is no forte process running when this happens.
I have tested the exact same application (boot files) on a Win32 version of FORTE on my Windows PC and it runs without interruption (I tested it for 4 days).
So it appears to be a problem with either the Posix build or the RPi. Has anyone encountered this behaviour before? Maybe I am missing an important setting.
Otherwise, I would be happy for any hints on how to debug this (i.e. does forte have a logger that can be enabled)?
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Forte stops running on Raspberry Pi after a certain time [message #1772799 is a reply to message #1772791] |
Fri, 15 September 2017 08:17  |
| Eclipse User |
|
|
|
Good that you found more clues.
First thing, from the little I read about the fsanitizer, it's not available for windwos or arm compilers, at least in most cases.
Second, the build error was weird, but good that you solved it.
Third, I wasn't sure about the maximum response data, but cleaning cppcheck is always good.
Fourth. Let's go to the important stuff. Let me see if I understood correctly. You have a CLIENT FB using yout http layer, and some other FBs that when you get and error (QO = 0), you re-initialize the CLIENT, and then try a REQ? Is that right?
If yes: what do you mean with re-initialize? De-initiliaze and initialize again, or just initialize again? Is always better to de-initialize first.
It looks like when you are re-initializing, it doesn't finish it and get stuck (this might explaing why you can't send a CNF event manually). When sending the REQ after reinitilizing, do you wait for INITO of CLIENT FB? This will give you a clue if it gets stuck there.
|
|
|
| Re: Forte stops running on Raspberry Pi after a certain time [message #1772817 is a reply to message #1772799] |
Fri, 15 September 2017 11:45 |
| Eclipse User |
|
|
|
Thanks for going through this with me :)
Here's the function block network.
I think it's easiest to understand if I explain how the http layer works.
Using forte's available CIPComLayer class to receive the data is too slow, because usually, the REST server will close the connection before the data is received.
So I created a slightly modified copy of the CIPComLayer (CHttpIPComLayer) that performs all operations - opening the connection, sending data, getting the response and closing the connection from within the sendData() method.
This way it is fast enough to get the response before the peer closes the connection.
An incoming INIT+ event doesn't actually open a connection, it parses the ID and
writes the connection params to a cache, because the socket handler's methods consume the data.
Since closeConnection() is already called at the end of every sendData() call, an INIT- event effectively does nothing.
The com layer is designed in such a way that there are 4 possible outputs:
INITO+
INITO- is only possible if the ID is invalid or QI is false
CNF+ (if an HTTP OK response is received and the body contains valid data)
CNF- (if an unexpected response is received, the timeout is exceeded or the response is invalid)
I only just added the checks for invalid responses, so maybe that'll fix it.
To explain the most important parts of the function block network:
If a CNF- event is output, it loops back to the INIT+ event and then to the E_DELAY FB, which then goes back to the REQ event. This loop goes on until a CNF+ event occurs.
Attachment: Capture.PNG
(Size: 39.08KB, Downloaded 1113 times)
|
|
|
|
| Re: Forte stops running on Raspberry Pi after a certain time [message #1772871 is a reply to message #1772822] |
Mon, 18 September 2017 03:44 |
| Eclipse User |
|
|
|
Thanks for the detailed info! I went through your http layer code. If forte freezes is normally because of an inifinite loop. I think you might have a problem with this section of code:
while ((cg_unIPLayerRecvBufferSize - m_unBufFillSize) <= 0) {
#ifdef WIN32
Sleep(0);
#else
sleep(0);
#endif
}
Once the code enters this loop, I don't see how it can go out. Maybe I missed something, but you could put a DEVLOG to see when this section of code is entered. If your buffer is fulled, it won't leave this loop.
Here:
openConnection();
if (0 >= CIPComSocketHandler::sendDataOnTCP(m_nSocketID, request, pa_unSize)) {
m_eInterruptResp = e_ProcessDataSendFailed;
}
you could check first the return value of openConnection() before trying to send, and also break the while if it fails, to avoid waiting the timeout and skip the handledConnectedDataRecv().
Regarding the CSV_WRITER I'm not sure why it freezes. I see only a possible problem that you could have if one of the inputs of the FB is longer than 100 characters, but in any case it shouldn't freeze, just write a shorter string.
And as always, I recommend to put a lot of DEVLOG information, specially when the FB could fail (and its corresponding success branch), since this code is not compiled when selecting a NODEVLOG configuration
[Updated on: Mon, 18 September 2017 03:46] by Moderator |
|
|
| Re: Forte stops running on Raspberry Pi after a certain time [message #1772884 is a reply to message #1772871] |
Mon, 18 September 2017 07:29 |
| Eclipse User |
|
|
|
Thanks for the hint. That may have been just what I needed.
The while loop in handleConnectedDataRecv() is actually residue from forte's built in ipcomlayer (on which the httpipcomlayer is based). Technically, it shouldn't be needed, because the HTTP response should never have a character width greater than 55.
But I was able to catch the class with the following corrupted buffer in debug mode:
HTTP/1.1 200 OK
Content-Type: text/plain
1858.656
e="http[192.168.10.193:7000/rest/devices/battery/M03]" forced="false"></Data></Port><Port name="QI"><Data value="TRUE" forced="false"></Data></Port><Port name="QO"><Data value="FALSE" forced="false"></Data></Port><Port name="STATUS"><Data value="" forced="false"></Data></Port><Port name="RD_1"><Data value="0" forced="false"></Data></Port><Port name="INIT"><Data value="0" time="0"></Data></Port><Port name="REQ"><Data value=
It appears to be mixed with monitoring info being sent to 4diac-IDE.
I changed the while loop to the following:
if ((cg_unIPLayerRecvBufferSize - m_unBufFillSize) <= 0) {
// If buffer is full, clear and return
memset(&m_acRecvBuffer[0], 0, sizeof(m_acRecvBuffer));
m_unBufFillSize = 0;
m_eInterruptResp = e_ProcessDataRecvFaild;
DEVLOG_INFO("HTTP recv buffer full\n");
return;
}
So if it enters, it clears the buffer and notifies a failure.
I also noticed some more potential issues.
The REST server neither includes a content-length in its response headers, nor does it send the data in encoded chunks. I was relying on the detection of "\r\n\r\n" to end the loop in which the response is received. I didn't take into account that a response to a GET request has an additional body.
As a result, maybe the '\0' terminator wasn't always received (although I would have assumed that the socket handler's receiveDataFromTCP() method terminates the data it receives).
I changed it so that it always waits for the peer to close the connection (so far I haven't noticed much of a performance impact). That seems to be the only way to ensure that all of the data is received.
If it works well, I will later add some code to detect if the header has a content-length field or if the data is sent in encoded chunks.
By the way, the code I am currently using is on the development branch of the project. I will merge it over to master if there are no new issues.
I have DEVLOGs in all critical places and will see if I can start forte it in a terminal that saves the output to a text file.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Forte stops running on Raspberry Pi after a certain time [message #1772985 is a reply to message #1772959] |
Tue, 19 September 2017 13:54 |
| Eclipse User |
|
|
|
No problem :)
The first iteration of the HTTP com layer actually worked in that way, using forte's standard IP com layer that the FBDK layer uses as its bottom layer.
But it turned out to have a lot of issues.
HTTP servers usually close the connection immediately after sending the response. In the HTTP standard, this is intended behaviour, while in forte, it is treated as unwanted.
As a result, the process would be interrupted (often before the response was handled), causing the function block to issue an INIT- event almost immediately after opening the connection and receiving data.
Attempts to work around the processInterrupt quickly got messy, so I decided to create my own IP layer that is based on the FORTE one.
But if the current version works and I end up adding it to FORTE after my thesis, I could create a common parent class for the two IP layers with shared code in its protected methods.
Update:
I played around with it and got it working with forte's select loop (see development branch). But I think either changing the standard CIPComLayer's private methods to protected and subclassing it or creating a common parent class would be necessary to truly reduce code duplication.
I will leave the changes in the development branch for now, since the master branch seems to be working so far.
[Updated on: Wed, 20 September 2017 02:09] by Moderator |
|
|
|
|
|
Home