Hub not responding

blanghor · June 7, 2021, 5:24pm

Hello,

A few weeks ago, I had an issue where my C7 was locking up every few hours. I was finally able to restore a previous evening's backups and everything seemed to work after that, with no additional lockups.

A couple of weeks later, the hub started locking up again and has been doing so ever since every couple of days. I'm on the latest firmware. I'm going to be blunt - these issues are not acceptable and are getting REALLY old. I stuck with the C7 through the growing pains but my patience has run out. I'm going to open a support ticket but wanted to put this here as well. The last time this happened, the logs showed NOTHING.

aaiyar · June 7, 2021, 5:30pm

Tagging @gopher.ny

gopher.ny · June 7, 2021, 5:46pm

Do you need history for the power reporting devices?
If not, I think running these two endpoints may help. At the minimum, it's the easiest thing to change that will also likely have the most impact.

http://your.hubs.ip.here/hub/advanced/deviceStateHistorySize/11
http://your.hubs.ip.here/hub/advanced/event/limit/11

blanghor · June 7, 2021, 6:29pm

Thanks, I ran those two endpoints. Those disabled history for the power reporting devices?

gopher.ny · June 7, 2021, 6:32pm

They reduce history size for all devices, making data easier to process.
An hourly cleanup job will do the actual history trimming in an hour or less. Alternatively, this endpoint can be run to force immediate cleanup: http://your.hubs.ip.here/hub/cleanupDatabase
I'll check on your hub in an hour. The database should shrink.

blanghor · June 7, 2021, 6:43pm

Thanks, I ran that endpoint to force the db cleanup.

blanghor · June 9, 2021, 12:08pm

Hub was completely locked up again this morning. No devices - zwave, zigbee, etc - would respond. According to the logs, the hub stopped logging at 12:00 AM this morning and these were the final log entries:

dev:1472021-06-09 12:00:32.323 am debugSensorMultilevelReport(precision:3, scale:0, sensorType:4, sensorValue:[0, 0, 0, 0], size:4, scaledSensorValue:0.000)

dev:1472021-06-09 12:00:32.321 am debugparse:zw device: 12, command: 3105, payload: 04 64 00 00 00 00 , isMulticast: false

dev:1472021-06-09 12:00:32.244 am debugMeterReport(meterType: 1, precision: 3, scale: 0, size: 4, meterValue: [0, 0, 47, 109], rateType: 1, deltaTime: 10,

What next?

thebearmay · June 9, 2021, 12:28pm

What device corresponds to dev:147?

blanghor · June 9, 2021, 12:50pm

It's a zwave plug. However, I am not sure if that device was also the last device logged on all the other lockups.

gopher.ny · June 9, 2021, 1:11pm

What do the runtime stats show at http://your.hubs.ip.address/runtimeStats? App/driver tabs should have busiest devices/drivers at the top.

I see a bunch errors like this one in the back end log:

app19Recur.refreshDevices has been firing for NNN seconds

with NNN varying, but time is at least several minutes. App 19 is Sensibo Integration. I'm not sure if it's a cause or a symptom, but it may be worth trying to disable it in the meantime.

Couple of general observations. One, a soft reset tends to help busy systems in general, Please try creating a download a local backup, doing soft reset, and restoring from the fresh backup. Another, a daily (typically early morning) scheduled reboot may be in order. There are quite a few custom apps, and we have no insight (other than runtime stats) into what they do. A reboot will not address a root cause, but it may keep things from freezing up.

blanghor · June 9, 2021, 1:30pm

I actually do have a daily reboot scheduled but it's at 2 or 3 AM and it seems the hub locks before that. Here is the screenshot of the apps/drives tab for the runtime stats:

It would really suck if I had to disable my Sensibo application . Any chance we could get native Sensibo functionality?

gopher.ny · June 9, 2021, 1:39pm

Does Sensibo have a polling interval setting by any chance? If yes, setting it higher may help.
If C3 is still around (I can't ping it from the support tool), try offloading the app on it. The AC device can be shared in hub mesh if there are C7 specific automations that use it.

aaiyar · June 9, 2021, 1:40pm

Three of your apps are ones that have caused issues on a hub that I maintain - Sensibo, Echo Speaks and Lifx Master.

Make sure you’re running the latest versions of Echo Speaks and LiFX Master. And that none of your LiFX devices are powered off.

blanghor · June 9, 2021, 2:45pm

I do have a C-3 which I could possibly move Sensibo over to. I'll have to look into updating the other two.

rlithgow1 · June 9, 2021, 7:12pm

Are you running any zooz stuff? Any ghosts? (this may have been addressed in your last ticket so me asking may be redundant)

blanghor · June 9, 2021, 7:41pm

I have the Zooz dual outlet (ZEN25?). I don't think I have any other Zooz devices currently connected.

aaiyar · June 9, 2021, 7:44pm

FYI ...

rlithgow1 · June 9, 2021, 7:53pm

What @aaiyar said

blanghor · June 9, 2021, 11:06pm

Yes, that was months ago and I took all of those steps.

It locked up again - interestingly, the hub web interface is accessible but no devices are working and logging stopped about 4 hours ago - the last entries are shown below. Device 28 is Lutron Telnet, 127 is a Zwave plug, 128 is a Zwave plug, 147 is a Zwave plug, and 146 is a Zwave plug. Hmmmmmm......

dev:282021-06-09 03:32:48.593 pm inforcvd: DEVICE,13,3,3

dev:1272021-06-09 03:00:21.529 pm debugMeterReport(meterType: 1, precision: 3, scale: 0, size: 4, meterValue: [0, 0, 65, 239], rateType: 1, deltaTime: 10, previousMeterValue: [0, 0, 65, 239])

dev:1272021-06-09 03:00:21.526 pm debugparse:zw device: 0D, command: 3202, payload: 21 64 00 00 41 EF 00 0A 00 00 41 EF , isMulticast: false

dev:1282021-06-09 03:00:19.703 pm debugSensorMultilevelReport(precision:3, scale:0, sensorType:4, sensorValue:[0, 8, 44, 22], size:4, scaledSensorValue:535.574)

dev:1282021-06-09 03:00:19.700 pm debugparse:zw device: 0E, command: 3105, payload: 04 64 00 08 2C 16 , isMulticast: false

dev:1282021-06-09 03:00:19.444 pm debugMeterReport(meterType: 1, precision: 3, scale: 0, size: 4, meterValue: [0, 1, 68, 14], rateType: 1, deltaTime: 10, previousMeterValue: [0, 1, 68, 14])

dev:1282021-06-09 03:00:19.440 pm debugparse:zw device: 0E, command: 3202, payload: 21 64 00 01 44 0E 00 0A 00 01 44 0E , isMulticast: false

dev:1472021-06-09 03:00:17.297 pm debugSensorMultilevelReport(precision:3, scale:0, sensorType:4, sensorValue:[0, 0, 0, 0], size:4, scaledSensorValue:0.000)

dev:1472021-06-09 03:00:17.294 pm debugparse:zw device: 12, command: 3105, payload: 04 64 00 00 00 00 , isMulticast: false

dev:1472021-06-09 03:00:17.203 pm debugMeterReport(meterType: 1, precision: 3, scale: 0, size: 4, meterValue: [0, 0, 47, 109], rateType: 1, deltaTime: 10, previousMeterValue: [0, 0, 47, 109])

dev:1472021-06-09 03:00:17.199 pm debugparse:zw device: 12, command: 3202, payload: 21 64 00 00 2F 6D 00 0A 00 00 2F 6D , isMulticast: false

dev:1462021-06-09 03:00:14.503 pm debugSensorMultilevelReport(precision:3, scale:0, sensorType:4, sensorValue:[0, 0, 0, 0], size:4, scaledSensorValue:0.000)

The Zwave plugs are the older Aeotec DSC06106 models and have worked great all this time. Maybe I should start moving them over to the cheap Centralite Zigbee ones? I tried accessing Runtime stats but it times out - going to need to reboot.

blanghor · June 9, 2021, 11:58pm

It looks like the default refresh interval is 1 minute for Sensibo. I just brought my C-3 back online, updated it, and just added the Sensibo integration. From what I remember, the Sensibo integration always acted kind of weird on the C-3. Anyway, I have a webcore piston which controls Sensibo on my C-7 based on the temp of a Konke Temp and Humidity sensor in my office. It has been super rock solid and stable.

I upgraded Echo Speaks (also removed it as an active app, since I am not currently using it) and Lifx. Let's see what happens and if the lockups continue, Sensibo is the next step.