C8 RM Delays, Custom Driver

After migrating to C8 (BTW, it was very successful) I noticed many of my RM Rules started to
respond slower. The behavior is very random, so logs are not really helpful and since random
rules exhibit slowness randomly I am not sure which logs to turn on. Obviously turning logs
on for every rule is not a god idea.
@ogiewon
However one custom driver for the Hubduino exhibits very huge delay pretty much consistently.
I am using Hubduino project as a AutoSlide Controller. This project has 3 relays and 2 contact
sensors. On C7 there was not any single problem with this integration. Problem started after
migrating to C8. The communication is LAN/WiFi (C8 hub is hard wired, Hubduino is WiFi).
Here is two sequential actions in the rure:
image
And here is a correspondent log for these two actions:

In this case actions are spaced by about 8 sec but I saw 10+ sec delays.
And here is a log for the related Hubduino On/Off switch:

The ON command was sent at 10:49:21.9 but device responded at 10:49:29.1

C7 did not exhibit any delays neither with all my ZWave/Zigbee devices nor
with custom drivers (there are few more).
I wonder why this is started right after migration to C8?
And what is more important - how to fix this very annoying problem?
I understand, there is new ZWave and Zigbee chips in the C8. This could
somewhat explain inconsistent behavior for the ZWave and Zigbee devices.
But why clear local LAN integration became very unstable?

This is the first report of any type of issue using HubDuino on a C8 hub. I migrated my HubDuino instances from a C7 to a C8 many weeks ago. I have not experienced any sort of performance related issues.

Any chance something else may have changed in you home networking? Just a hypothesis... :thinking:

Are you using the exact same Ethernet cable and network port for your C8 that you were using with your C7? If not, maybe try a new patch cable and a new port on your network switch.

If you use the HubDuino child devices to turn the switch devices on and off, do you see any delays? It is always a good idea to try and split the problem in half to determine if there is an issue on the Device side OR if the issue is with the APP that is trying to use the device.

1 Like

Says who? It might make it more confusing for you to find the entries you're looking for, or impossible to find if you're looking at "Past Logs" and are already past the cutoff, but other than that, there is minimal effect.

One thing you could try is making a backup and doing a restore of that backup. This will clean your database and would eliminate corruption as a possible cause of this problem. (Yes, I know this is unlikely if you just moved to the C-8 and haven't had any unexpected shutdowns, etc., but it's a safe process to try and might help.)

Other than that, the usual steps of looking at App Stats, Device Stats, anything concerning in Logs themselves, etc., would be good things to try.

1 Like

I did not touch major network components (Router, Switch, etc.) for about half a year.
Nothing was changed in a wired network area.
However few WiFi devices were added and few removed.

Just tested On/Off commands for one relay from the Device page.
(Hubduino is configured for auto off after 1 sec).


First click produced lust only "Read timed out) warning from parent device.
and switch did not turned on. {I forgot to turn on login for parent at this time.)
Few sequential followed clicks produced the expected On/Off events and
switch physically followed the commands.
However after 5 min delay first click again was significantly delayed
but "Read timed out" warning was not present.
It looks like after long inactivity (not sure how long it is) first command is
not going through. This is very well correspond to the RM rule behavior.
I.e. first attempt to trigger rule fails but all followed sequential triggers are
working as expected. It looks like missing first command after long inactivity
timeout is very consistent.

Me saying this.
If I turn login for everything

I will be completely lost in this forest.

Yes, I am doing this pretty much regularly after migrating to the C8.
C7 was running flawlessly for months (until I was doing something not quite right.

Yes, I had some issues with few RM rules generating elevated or even severe hub load.
I had to redo them and now hub is happy as far as load goes.
And again, surprisingly C7 was very happy with original rules but C8 was not.
I am not saying the original rules were perfect.

I already mentioned this but let me repeat my message.
After migrating to C8 many RM rules (I have more than 200) started exhibit random
but very noticeable delays. Turning logs on did not produce anything reasonable.
Fist - because I did not turn logs on for every rule.
And second - by Murphy Law as soon as log turned on this specific rule runs perfectly fine.

This is a fairly easy process that is described, among other places, here: How to Troubleshoot Apps or Devices | Hubitat Documentation

1 Like

Well, I checked the suggested doc but frankly I did not find anything new or unknown.
My point is/was: If I turn on logs for EVERY app (200+) I will be lost in a forest of these logs.
And as I said, Murphy Law works very well: i.e. as soon as log for the app in question
is turned on this app is not failing anymore.
If problem is relatively easy reproduceable I have no problem to debug it.
(Please keep in mind , I am EE with huge experience how to design and debug things.)

ST_Anything has a built-in automatic refresh that typically runs every 5 minutes. Depending on how many โ€œdevicesโ€ are defined in your sketch, this refresh can take a few seconds and result in degraded performance for a few seconds.

You may have simply happened to notice this after your move to the C8.

If you want to adjust the timing of, or disable, this automatic refresh, you can modify CONSTANTS.H and the rebuild and upload your sketch.

Then it sounds like you should already know how to do the soft reset I mentioned, which is the specific section of the document that I linked to and the topic for the material you quoted. Have you tried it?

Well, this is a whole point!
After migrating to the C8 I noticed abnormal (but pronounced very randomly which makes it
next to impossible to catch and debug) behavior for many (if not all) of my RM rules (200+).
My observation tells me something was changed in the Platform leading to all this
suddenly popped up delays, etc. WAF is quickly declining.

Yes, to...o many times in a past few weeks.

Then what was the reply to (seemingly) my suggestion to do a soft reset, where you implied that it was confusing, referring to? Is there something else that needs clarification?

1 Like

Now I am confused what I said wrong or unclear?
As far as Soft reset goes, I know what it is, what it does and how to use it.
Unfortunately this Soft Reset becomes almost a daily routine.

I'm not sure there is any point in continuing this part of the discussion, but what I'm saying should be clear if you re-read your reply above and my reply to that: I suggested a soft reset, your replied to that (quoting that specific part of my post) implying that you didn't know how (this is the only way I can interpret the "lost in the forest" metaphor), and then a later reply made it clear that not only do you know how but you have done so.

So, not much more to discuss with that particular suggestion. :slight_smile: (Unless it actually helps for a while, then maybe something else is going on...)

3 Likes

Oh, "lost in a forest" was related to many log entries to go through if logs are turned on for many rules. I am sorry this was not clear (my old style former USSR thinking?).

1 Like

Just to add more pizazz to this discussion, C-8 has nothing to do with this. It possesses no new demons that cause rules or devices to run slowly. But, having just migrated, it does make a convenient target as an explanation, just as new releases are the target for things "suddenly" not working (even though a particular release has nothing to do with the reported problem).

Use filters and search. I have logging on for just about everything. Without logs, there is no hope of finding what is going on.

4 Likes

Well, you specifically called out your HubDuino devices as being problematic, and I gave you a hypothesis with a possible explanation for what you have observed. Since you mentioned every 5 mins, that helped me to focus on what I know to be a 300s default setting in the ST_Anything Arduino library. Have you tried my suggestion for modifying CONSTANTS.H to see if the rebuilt sketch still exhibits any sort of delay every 5 minutes?

If you have other RM rules having issues, that are not using HubDuino devices, then I don't know what to tell you... I have personally not had any such issues with my C3, C4, C5, C7, or C8 hubs over the years.

1 Like

Please don't get me wrong. I like Hubduino because I can create just anything I want.
I mentioned Hubduino only because this extra long delay between two sequential actions
is somewhat reproducible in one of my RM rule. All other cases are very random.
5 min I mentioned was just a random number I picked up while I was doing some testing with
Hubduino 3 relays + 2 sensors device. Missing of very first ON command after relatively long
inactivity period happens to be very consistent. How long inactivity timeout should be for failure
to manifest itself I have no idea but it is definitely a lot longer than 5 min.
I did not know about built-in 5 min refresh period and was about to add this in a dedicated
RM rule. But now I am very confused why this failure is happening . This is pretty much consistent
behavior even though the timeout must be very long. This specific RM rule and Hubduino device is
used to control Balcony Door Autoslide. During the day there are few times when we need to
open/close balcony door and everything works as expected. But at the morning time my wife tries
to open balcony door, pushes a button and nothing happens on a first try. Following button push
does the job. This never failed on C7.

My observation for RM delays is not new and yes (sorry @bravenel) clearly started to pop up
after migrating to C8. I cannot go back to C7 because there is no simple and easy way to do
this. But my C7 was running flawlessly for months (please don't ask me why I migrated to C8).
From the other side C8 started to exhibit different problems right away despite the migration
went very smoothly. I didn't/don't have any problems (I think) with ZWave and/or Zigbee
meshes and devices. The only problem what cough my and my wife's eyes was a lot of random
delays.

Yes, I am using filters and search but unfortunately forest is very dense.

Well, I am sorry to say this say this but how many times something working very well
became broken with a new release?
Don't worry, this is normal for all development stages. And that is why testing and
debugging is very important.