Almost exactly the same here.
I have only 35 devices, all Z-Wave plus, added with mixed levels of security, mostly mains-powered.
According to the "best practice" I took my sweet time while including them.
Experiencing only some lags before, things were getting really severe after adding my TRVs, which are all FLiRS.
Simple motion trigger lights on-automations didn't work anymore or with a delay of a minute or more.
While in this state, also voice commands are being ignored.
At the latest when one of my TRVs is not responding anymore, it's time for a safe power-off, need this almost once a day.
Did some testing using the Hub Watchdog-App with the following results:

As you can see, the response time for my tested device is up to 45s.
In this case an Aeotec SmartSwitch7 included with S2 security, within direct range to HE.
The 4 "faster" results above are Oomi bulbs without any security.
Another finding I'd like to share is that i find always the same mains-powered S2-enabled devices stated as "not responding" or more rarely as "failed".
What they have in common is the fact, that each of them is routed directly to HE.
Intereresting, 4 of them are Aeotec SmartSwitch 7, and the only one I had been not able to catch with such issues is the one routing through another device.
Anything in common?