We recently had a failure in which a PLC “PROD” (ControlLogix L72S with L7SP) was producing tags without any faults while the consumer PLC “CONS” (ControlLogix L72S with L72SP) kept timing out trying to connect. Both PLCs are at 20.1x firmware. PROD had produced tags for both safety and non-safety configured with a maximum of 1 consumer and safety networks in both directions matched where necessary. Both PLCs actually shared tags both directions so each had produced safety and non-safety tags and consumed safety and non-safety tags from the other one. The tags were user-defined AOIs with a connection status object, DINT[0..31] array, and REAL[0..99] array. These tags and CPUs have been in use for 10 years so not a new application.

Initial indications of the failures were that both bits (DINTs) and REALs were being lost before reaching the CONS processor. The values were being set properly at the PROD cpu but not reaching CONS. Loaded both CPU programs on Logix 5000 and found that on the PROD controller organizer there were no faults shown in the tree and communication status showed RUN, no FAULTS constantly. CONS controller, however, showed exclamation icon for PRODs L72S cpu. Physically looking at both CPUs in their respective showed no problems with EN2T cards or CPUs.

Tried resetting EN2T on CONS cpu and finally replaced with new EN2T, still same results. Verified IP addresses were correct, verified configuration, reloaded program into CONS cpu thinking maybe an issue with memory corruption had occurred. No changes.

Researched via Internet and found a couple of references to similar issues, basically networking possibly a problem. Double-checked all settings again. One thing of note is that the producing CPU doesn’t seem to care if it actually has a connection to any other device. From what I can tell it will allow the maximum number of consumers but will not complain if there are none at all. I had expected a fault on the PROD plc tree similar to the CONS plc tree but this explained why the PROD plc didn’t show issues.

Using Studio 5000 Task Monitor, checked the EN2T module on CONS plc and noted that both the safety and standard messages were faulting; the organizer tree did not show safety messages faulting, though. Sure enough, comparison of data showed this failure as well (no data changes on CONS for a change on PROD).

Finally went to double-check cables. When doing so, found one of the two Stratix 8000 switches in the CONS control cabinet had 5 connections but all were staying on solid green, no flashing lights at all. Cycled power to switch and everything started working properly after switch finished booting. This problem was very strange because all connections to the failed switch still showed network traffic and no indication of failure. Thought I’d post this in case anyone searches for similar issues as there wasn’t a lot of information for similar configurations on this fault (16#203 or 16#0203 ?).

Cheers