I'm having trouble with instability on my C8 hub. Various devices work sometimes and fail to work other times. These are usually wall switches (GE & Zooz) but can occasionally be other devices like battery-powered remotes or motion sensors. Most of the time I know something has been triggered, I can observe the trigger in the log, but do not see the action occur. Then, upon attempting to activate the switch from the device page get no response. Later, that same device will work just fine. This issue started small and has gotten progressively worse over several weeks in terms of frequency and devices affected.
A culprit that has been called out in other threads has been ghost nodes. I wonder if this is my issue. I have 15 nodes that potentially look like ghosts. I think this because the Device column is empty.
Some posts have indicated that ghosts usually have a blank Route column but mine all have routes. Here's what they look like:
I do wonder if these qualify as ghost nodes. I don't see how to relate them to a physical device, but I don't have any actual devices missing so I'm guessing they are ghosts/duplicates/old-partials/whatever. Undesirable and potentially interfering.
You'll note some Pending and Not Responding entries in the Status column. I've been experimenting with the Refresh/Repair/Remove options based on what I've read in other posts. No luck getting rid of them, however.
Some things I've tried:
1. Shutdown, unplug, wait, boot up
2. Multiple Refresh button attempts
3. Multiple Remove button attempts (when I can get Remove to show up)
4. Individual node Repair
5. Complete Z-Wave Repair
6. Verifying region for proper frequency
7. Relocating Hub a few times
8. Waiting a day or two in between attempts to see if anything heals on its own
I have not tried cutting power to the device that the ghost is a ghost of. This is mainly because I don't know which devices the ghosts are partially tied to. If there is a way to determine that, I would give that a try as other posters have had at least mixed success with it.
I can grab whatever other info would be helpful. Any guidance would be appreciated. I really want to like my Hubitat but sometimes it makes it hard on me...
When I refresh, I sometimes get a 'responded with busy message' WARN in the log. Not always, though. Maybe one time out of three. The other times nothing logs.
When I attempt to remove, I often (even usually) get an INFO in the log that says 'Failed node ## remove status: 0 SDK failure node is no longer in the failed node list". This sounds like it removed it, but it doesn't actually go away. The ghost node remains after the removal attempt.
I have played with multiple attempts at Refresh and mostly get the Remove button. There are two that show a 'Not Responding' status and no Remove button - even after a few tries. But the rest get the Remove button and a 'Pending' status. When I hit Remove, on the other hand, it doesn't do anything (except show a removal on the log that doesn't really happen). I have not been shutting down and restarting in between attempts (as the How To suggests), however. That'll be time-consuming. I'll throw a few hours at it tomorrow.
The How-To warns about Ghost Nodes with 'too many' neighbors.
Some of my Ghost Nodes have only 1 neighbor, some have more. The one with the most has 32. I'm guessing that's not the issue, though, since removal fails regardless. I'm also not sure what I can do about it - I can't remove neighbors, can I?
Do you think I am on the right track? Are these really Ghost Nodes even though they have a Route indicated? Could they be responsible for my instability issues? The problem could be something else completely, for all I know. I just thought these looked suspicious.
No, that is a failure error, it is saying the node is not failing anymore which is obviously false. Some sort of bug in the SiLabs software/firmware.
If that on neighbor is not that hub that could be why it fails. Usually if the only neighbor is the hub you can remove it. Same bug as noted above.
Yes, there is no device attached to them, that is the key indicator.
Its hard to know for sure what he issue is with all those in there. Sometimes a lot of dead/ghost nodes will cause issues, usually a small number of them does not cause a problem.
No luck with the standard tools in Settings. I've done the Refresh+Remove thing four or five times with shutdowns in between on 3 of my ghosts. But it always reverts back to the original state (Status = check or Pending and Device is blank). I will try more and in different combinations, I guess. Maybe I'm just not hitting it right.
I also installed the Z-Wave Mesh Details app as suggested by your link. It is handy for identifying neighbors which the link said might prevent removal. Most of my ghosts have a handful of neighbors and some have a lot. I did have one with only one neighbor (besides the hub). This neighbor I was able to remove via the instructions. But I had no luck removing the ghost node via the remove button. Same reaction as before. Even if I did get that to work, I'd still have the problem where many of my ghost nodes have other ghost nodes as neighbors. And I can only remove neighbors for which I can access the physical device.
When attempting to remove a ghost node, I seem to get that log WARN that indicates that the 'node is no longer in the failed node list'. This rings true as the status is either a checkmark or Pending (both of which show as 'OK' in the mesh tool. It's like it doesn't want to remove it because it is acceptably connected. Is there something I can do to get it out of 'OK' status and maybe then the remove command will be accepted?
Other than repeatedly trying these same things in different sequences or with longer time between steps, I'm not sure what else I can do. Any ideas?
Like I said it is a bug in the software from the zwave vendor that causes that error even when the node is dead. There is nothing that can be done about it.
That’s a bummer. Can you point me to any available info on this bug? Maybe I can devise a work-around. Separately, this sounds like something I should elevate to the developers. Do you know how best to do that?
I think the question is how many Zwave devices do you really have. If they are largely one brand maybe 2 then the exclude process shouldn't be to bad. If you have a bunch of devices with various brands and exclude process will vary allot then it may be less of a headache to get the zwave stick and remove the ghost one at a time. Need to figure out.
If all of your devices are zwave 700 or newer i would look into thr SmartStat Manager app to help with the adding back in.
Either way take your time and check the zwave settings table periodically to ensure ghosts are not occurring.
I did a zwave reset and rebuild not long ago. It wasn't to bad with good planning because almost all of my devices used the same 2 exclude procedures. It did take a little bit of time, but it was pretty easy. Just remember to start near the hub first and work your way out so you can build out your mesh. I also limited my adds with SmartStart to 4 devices at a time to get maximum reliability of the include process.
The removal of the ghosts with the stick isn't to bad once you get it setup,
I'm not against the Z-Wave stick idea, though I've never used one. I read the post on it but I'm not 100% clear on how it would work. Is the idea to use the stick to remove the ghost node? Or is it to more easily remove all the neighbors which would free the ghost node to be removed?
If the first one (directly remove ghost node), a problem I have is that I don't know which physical devices are associated with the ghosts. I'm guessing that I would need physical access to the device to use the stick, but I don't know what device it is. And it might even be a device that I don't have anymore. Do I have that right?
If the second one (remove neighbors), many of my ghost nodes have other ghost nodes as neighbors. So I fall back into the problem of the previous paragraph. Furthermore, even on the one ghost node where I was able to remove all neighbors, I still couldn't remove the ghost node.
Does any of this mean that the stick option won't work for me or is it still viable? I feel like I'd rather do that than a full reset, so if it's viable then I'll buy a stick and try it. But it kinda seems like it won't work because I don't know what my ghosts were when they were still alive.
I was thinking a reset might be in my future. Perhaps if I could get a clean slate then I could be more diligent about creating ghost nodes.
I have about a dozen different devices, many with different exclusion operations. That's not a deal-breaker, though. I don't mind spending the time if the results are good. What I don't want to have to do is re-write all my rules and re-build all my apps. Can I do the reset operation without having to do that? Maybe through swapping everything to virtual devices?
What might that procedure look like?
1. Swap everything to virtual devices.
2. Exclude everything manually.
3. Reset the radio (not sure how I do this)
4. Include everything manually.
5. Swap everything back from the virtual device.
6. Delete the virtual device.
Would that work? Or by 'reset the radio', do you mean a full wipe of the hub which would lose my rules? Perhaps necessitating a backup and restore?
When i say reset i would mean go into the diagnostic page and click on the option to reset the Zwave Radio.
Swapping could be a option. So if you decide to go that route you will need to do something with those left over devices. I believe the directions provided in the past were something like this.
Reset the radio
(Very Important) Go into each zwave device and add a _old or _# at the end for both the DNI and the Name.
Then go include the devices again.
Step 4 is where I am a little foggy. I think I read somewhere you can just assign the new DNI for the new working device to the old device and then force remove the new one and it would be good. I think i would prefer to just do the app swap between devices if it is a option. In my case i ended up having to do that part manually mainly because I had another problems that I created while troubleshooting.
Rremove the old device once the apps are swapped to the new one.
The one thing i would suggest is to look at the SmartStart Manager if your stuff is Zwave 700 or newer. It will make it allot easier to add devices at any time in the future once everything is loaded in there. It takes a little bit to get everything in Smartstart and saved, but once it is there it is super easy. I reloaded almost all of my 56 Zwave devices with that tool from @jtp10181 in about a hour 4 at a time. Getting everything in it can be a little bit of a pain though depending on how you have handled the QR Codes.
You attach the stick to hubitat as a secondary controller. You can look at the hex numbers on the hubitat table to determine which ones need gone and match those numbers in the silabs software (free)
Thanks, @rlithgow1. There’s a lot of maybes and finger-crossing in there but it seems worth a shot. I’m not clear on how the stick can work on ghost nodes when the hub can’t but I’ll give it the ol’ college try.
I suppose I’ll buy an Aeotec stick. I have both 500 series and 700 series. Would their Stick-7 handle both, do you think? I’m not sure it matters if the physical device doesn’t exist.
I’ll save the radio reset plan @mavrrick58 described for if the stick doesn’t work.
Because the silabs software bypasses certain things. This can't be built into hubitat as they would lose their z-wave certification. The overall issue is actually a SiLabs issue and not a hubiat one. Simply follow those instructions and you will be fine.
Ok. Got the Z-Stick. Loaded the Silicon Labs software. Removed the ghost nodes. The whole process went easier than expected. My Z-Wave Details now is ghost-free. I'll know in about a week if that is the cause of my instability issues. Thanks for your help!
