My initial suspicion was the same as Bora’s. These are separate devices, though. What seems to be the gist of the problem is that there is a race condition on the Modbus device, where it is still responding to a read when it receives the
write command. It then abandons the read partway through, and that errors out. The solution is therefore probably to time the write command such that it doesn’t interfere with the response from the slow device. I suggested a couple workarounds in the thread
below. I suspect that Joe’s new solution should work as well.
@Thompson, Joe, did I understand correctly that it seem to be working now, since
your last changes?
For context to the rest of the list-members, we had some discussion already on this out of band. I had started to reply directly to Joe, thought better of removing it from the mailing list, and re-added the (wrong) list. Sorry for the confusion.
From: Raker, David M
Sent: Thursday, February 22, 2024 2:07 PM
To: Thompson, Joe <jthompson@xxxxxxxx>
Cc: ^DL PNNL Volttron <volttron@xxxxxxxx>; Bonicillo, Mark A <mark.bonicillo@xxxxxxxx>
Subject: RE: [EXTERNAL] RE: Custom Controller Agent Causes Modbus Read Error
So, you are likely correct that the problem will be the write on the inverter interfering with the ongoing read. This is a problem, generally, for a lot of Modbus devices. A couple of things you may be able to do in this case (you should
only need to do one):
- Call gevent.sleep(0.5) after you receive the subscription to the BMS and before you make the write to the inverter. That should allow sufficient time for the read on the inverter to
complete before you write to it.
- Configure the drivers to use separate “group” numbers and a “group_offset_interval” (these are options in the devices/… configuration file). This will separate the polling of the
two devices by whatever the group_offset_interval is. If you put the inverter in the lower-numbered group, and separate them by 0.5 seconds, you should get the results closer in time than they are now with the write going out well after the read on the inverter
has completed.
- Make sure that you are not skipping registers in your poll of the Inverter. This one is an oddity of Modbus. Let’s say you want registers 1, 5, and 12. If you request all 12 registers,
it will require one network request. If you request only the three you want, it will make 3 network requests to complete the full query. This is because the protocol uses only the address a start register and the number of registers to read. Requesting all
will take up more space in the historian, but will make much more efficient use of the network.
Any of these three options are likely to get you around the problem, but let me know how it goes.
Dave
Hi David,
Here are some answers to your questions:
- Do these devices share a serial bus (including behind a modbus gateway)?
- Do these devices share an IP address? (e.g., again, a gateway)?
- No, these are 2 separate devices with 2 separate IP addresses
- Which method is the agent using to write the point? I assume this is probably using the set_point method of either the driver or the actuator?
result
=
self.vip.rpc.call(
'platform.actuator',
'set_multiple_points',
self.core.identity,
topic_values).get(
timeout=4)
- Does the write (and/or the read) fail every time, or is this intermittent?
- The write to the inverter always succeeds, the read from the inverter always fails
To your point about concurrency issues, I think I see what is going on.. I have a timing / order of operations issue.
When my controller is not running: every 5 seconds the platform.driver tries to read from my 2 devices, “devices/STAC/EnervenueBMS” and “devices/STAC/DynapowerInverter”.
- It takes ~0.08 seconds to read all data from “devices/STAC/EnervenueBMS” and ~0.53 seconds to read all data from “devices/STAC/DynapowerInverter”.
2024-02-22
12:45:30,127 (listeneragent-3.3
27215) __main__
INFO: Peer: pubsub, Sender:
platform.driver:, Bus: , Topic: devices/STAC/EnervenueBMS/all, Headers: {'Date':
'2024-02-22T19:45:30.086216+00:00',
'TimeStamp':
'2024-02-22T19:45:30.086216+00:00',
'SynchronizedTimeStamp':
'2024-02-22T19:45:30.000000+00:00',
'min_compatible_version':
'3.0',
'max_compatible_version':
''}, Message:
[{'EnerStation Capacity':
480.0,
'EnerStation Current':
46.3,
'EnerStation Max Charge Current':
240.0,
..........
2024-02-22
12:45:30,545 (listeneragent-3.3
27215) __main__
INFO: Peer: pubsub, Sender:
platform.driver:, Bus: , Topic: devices/STAC/DynapowerInverter/all, Headers: {'Date':
'2024-02-22T19:45:30.528178+00:00',
'TimeStamp':
'2024-02-22T19:45:30.528178+00:00',
'SynchronizedTimeStamp':
'2024-02-22T19:45:30.000000+00:00',
'min_compatible_version':
'3.0',
'max_compatible_version':
''}, Message:
[{'AC Energy':
117637120000,
'AC Power Factor':
1.0,
'Automatic Transfer':
1,
..........
My control loop is triggered by a subscription to “devices/STAC/EnervenueBMS”. So,
- the platform.driver first published data from “devices/STAC/EnervenueBMS” after ~0.08 seconds
- This triggers my agent to write to “devices/STAC/DynapowerInverter” after only ~0.13 seconds
- the platform.driver is still working through getting data from “devices/STAC/DynapowerInverter” and my write to the same device seems to cut that reading off.
2024-02-22
13:54:25,091 (enervenue_ctrlagent-0.1
29470) __main__
INFO: schedule_request [['STAC/DynapowerInverter',
'2024-02-22T20:54:25.090972+00:00',
'2024-02-22T21:54:25.090972+00:00']]
2024-02-22
13:54:25,107 (enervenue_ctrlagent-0.1
29470) __main__
INFO: Modbus points to write: [('STAC/DynapowerInverter/Output
Power Command', -20)]
2024-02-22
13:54:25,111 (listeneragent-3.3
29609) __main__
INFO: Peer: pubsub, Sender:
platform.driver:, Bus: , Topic: devices/STAC/EnervenueBMS/all, Headers: {'Date':
'2024-02-22T20:54:25.082792+00:00',
'TimeStamp':
'2024-02-22T20:54:25.082792+00:00',
'SynchronizedTimeStamp':
'2024-02-22T20:54:25.000000+00:00',
'min_compatible_version':
'3.0',
'max_compatible_version':
''}, Message:
[{'EnerStation Capacity':
480.0,
'EnerStation Current':
26.5,
'EnerStation Max Charge Current':
240.0,
............
2024-02-22
13:54:25,126 (enervenue_ctrlagent-0.1
29470) __main__
INFO: Modbus write response: {'STAC/DynapowerInverter/Output
Power Command':
"ConcurrentObjectUseError('This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0xb39995f0>>')"}
2024-02-22
13:54:25,126 (enervenue_ctrlagent-0.1
29470) __main__
INFO: Sent charge command of
-44.0kW
2024-02-22
13:54:25,139 (platform_driveragent-4.0
25845)
platform_driver.driver
ERROR: Failed to scrape STAC/DynapowerInverter:
Traceback (most recent call last):
File "/home/pi/.volttron/agents/b89cc86f-c623-44c7-bf70-74511e776b02/platform_driveragent-4.0/platform_driver/driver.py",
line 244, in periodic_read
............
I guess what I need is a way to subscribe to “devices/STAC/EnervenueBMS” and “devices/STAC/DynapowerInverter” In a way that only prompts my control loop one time in each 5 second interval
after the reads from both “devices/STAC/EnervenueBMS” and “devices/STAC/DynapowerInverter”.
I could set my agent to subscribe to “devices/STAC/DynapowerInverter”, but then I will not have the data I need from “devices/STAC/EnervenueBMS” in the callback.
Any ideas on how to handle this?
Thanks!
Hi Joe, It should not be a problem to be both reading and writing. That said, there are a couple of things which could cause concurrency issues, and from the error
being returned, this looks like it could be a race condition in the Modbus TK
Hi Joe,
It should not be a problem to be both reading and writing. That said, there are a couple of things which could cause concurrency issues, and from the error being returned, this looks like it could be a race
condition in the Modbus TK library. A couple questions to try to narrow this down:
1. Do these devices share a serial bus (including behind a modbus gateway)?
2. Do these devices share an IP address? (e.g., again, a gateway)?
3. Which method is the agent using to write the point? I assume this is probably using the set_point method of either the driver or the actuator?
4 Does the write (and/or the read) fail every time, or is this intermittent?
Thanks,
Dave
Check twice before you click! This email originated from outside PNNL.
Hello Volttron Team,
Background:
I am using Volttron 8.1.3 on a Raspberry Pi to coordinate the operation / testing of 2 devices over Modbus:
- An energy storage system with a BMS that Volttron’s platform.driver communicates with over modbus
- An inverter that also communicates with Volttron’s platform.driver via modbus
Using the Listener Agent I can see that the platform.driver is reading all of my modbus points perfectly every 5 seconds (see the attached volttron_Without_Controller.log).
I have a simple custom agent, enervenue_ctrlagent (see the “agent.txt” file, changed from “agent.py” for attaching), that is subscribed to the "devices/STAC/EnervenueBMS" topic and runs a simple read from the battery, decide what to do,
and write to the inverter loop every 5 seconds when new "devices/STAC/EnervenueBMS" data comes in. The writing is just to a single “Output Power Command” register on the inverter.
The Problem:
When enervenue_ctrlagent is installed and started, modbus reads of the inverter begin to fail. The controller’s attempts to write power setpoints are successful, but I start to receive this modbus_tk error (see the attached volttron_With_Controller.log):
Expected Behavior:
I was expecting that my controller agent should be able to run in tandem with the platform agent with no conflicts between reading and writing, but something funny is happening that I don’t understand.
Any and all help is greatly appreciated!
Joe Thompson
Engineer / Scientist
Electric Power Research Institute
Energy Storage and Distributed Generation
(912) 663-3407
*** This email message is for the sole use of the intended recipient(s) and may contain information that is confidential, privileged or exempt from disclosure under applicable law. Unless otherwise expressed in this message by the sender
or except as may be allowed by separate written agreement between EPRI and recipient or recipient’s employer, any review, use, distribution or disclosure by others of this message is prohibited and this message is not intended to be an electronic signature,
instrument or anything that may form a legally binding agreement with EPRI. If you are not the intended recipient, please contact the sender by reply email and permanently delete all copies of this message. Please be advised that the message and its contents
may be disclosed, accessed and reviewed by the sender's email system administrator and/or provider. ***