What do you do with a ghost node?

I get ghost "routes" - I currently have 2 devices reporting as routing through devices that do not exist no longer exist, either in HE or using a secondary controller. The route shows a speed of 9.6Kbits. Repairs, reboot do nothing... it's really annoying because it seems to persist for a while and my devices are flaky - slow, sometimes fail to respond.

@dman2306 several weeks ago I had a GE switch fail and I couldn’t get it to remove. I shut down the hub and went out to dinner and came back booted up hub and entire mesh was bricked. Device would not remove via HE after reboot. Several devices lost their routes similar to what you are experiencing too. I paired my Aeon Zstick and PC Controller and finally got the ghost node removed and the mesh immediately got back to normal. I bring this all up to say that a single ghost can cause all the problems you are experiencing and unfortunately sometimes a secondary stick like the Zstick may be required to remove it but things should quickly get back to normal. I equate the Zstick as another tool I have in my tool chest to deal with issues along with its added benefit to exclude devices and upgrade firmware too.

2 Likes

Yeah I believe you on the ghost node causing havoc, I'm more skeptical that 2.2.8 will fix the ghost node. It also seems weird to me that certain devices are completely unroutable too. But I'll definitely admit ignorance of the zwave protocol and why it might be related.

As a note, 2 of the devices eventually started routing but I have 1 device with no route still. It is my furthest device from the hub, however, it is only 5 feet away from another repeating device that works perfectly so I'm surprised it can't seem to route it.

Interestingly, I just noticed Flagpole light (which does work) is routing through Lamppost light (which is the one with no route)... That doesn't seem to make sense to me?

3 Likes

Are you Vladimir, the optimistic intellectual, or Estragon, the pessimistic misanthrope?

2 Likes

Just saw the release notes for 2.2.7.128...

" * C-7: Periodic Z-Wave node neighbor updated to assist the SDK in making better routing decisions."

2 Likes

That change happened when 2.2.7 was released (I think 2.2.7.119 was the first official release).

The specific release notes for 2.2.7.128 are here:

oops. I must have read the wrong thing notes. My bad. And bummer.

Unfortunately the release notes shown on the Hubitat itself for each update are not the incremental notes, but the main notes for the major release version ...

3 Likes

And that has tripped me up before. Sigh. Thanks for reminding me.

1 Like

An interesting thing I noticed that really makes me skeptical that a ghost is the cause is that even devices directly routed to the hub fail to respond:

Can anyone explain to me how, from a zwave perspective, a ghost node would cause this? That's not a rhetorical question, I really don't understand how this would cause issues if things aren't working when they're directly routed to the hub? Maybe I don't fully understand how the mesh routing works!

There are also some devices reporting an insanely high number of route changes. Though the device that isn't routable (lamppost light), only has 3 reroutes. Seems really weird that devices that are working fine and are only about 5ft from the hub (granted, 5ft with a cement wall in between), have been rerouted 51 times,

But a device that doesn't work at all, only 3 :man_shrugging:

Seems like a ghost could tie up the network as the hub attempts to communicate with it and things get delayed. Network Busy?

Not to mention other devices trying to route through it that have to fall back on other slower ways of getting back to the hub.

1 Like

Ok that seems reasonable. Would that show in the zwave logs? I don't see that device showing up in the logs at all.

Ghosts are invisible to most of the ui so may not show up in the logs properly but that's just a guess..

1 Like

I haven't scrolled back to check but wanted to ask, did you try the z-wave stick to remove the ghost?

1 Like

No. The problem is I don't own one. @672southmain was nice enough to ship me a spare one he had. However, there is this horrible organization called the USPS between him and me. They claim it will be delivered tomorrow, however, since it's still a little over 1000 miles away, I'm skeptical. I'll report back whether it works or not though.

6 Likes

That was super nice of @672southmain.

7 Likes

I have six or so extra Aeotec gen 5 z-sticks if anyone needs one. $25 each. I have removed ghosts with HE and with the z-sticks. They do not all remove the same way. It is still hit and miss. The one thing I do use the stick for is the update neighbor button. If you have a z-wave non plus device you can join it close to the hub and the move it 15 feet at a time and update neighbors each time you move it with the z-stick. By the time you get it to it's final destination it will be functioning.

Sigh, well thanks to @672southmain that ghost node is gone... however the zstick became a ghost node itself. It never paired successfully on the HE side, it was discovered, but it never created a device. It still let me mark the bad node as Is Failed so I did that. That all worked and the node was removed pretty instantly. But then I noticed my shiny new ghost node. Fortunately, after yanking zstick out, I was able to get that one to die via normal means.

So at this point I have no remaining ghost nodes and all nodes report routes. Now the big question, will this help my mesh? I guess time will tell? Any ideas how long it should take for the mesh to clean itself up?

1 Like

I would try a power-down, unplug, power-up cycle. See what comes back following that. (It may still need to rebuild the routes… Not sure…)

1 Like

Glad you were able to get it done. I didn’t have that issue. As for how long, as I have reported before, the routes churn continuously on my C-7 with 2.2.7.xxx. Has happened ever since 2.2.7 was introduced. Now that there are no ghosts, everything is working well, I have 6 Ring Extender Gen 2 and 2 Aeotec Range Extender 7 devices in our 1800 sf home, and a rather small # of Z-Wave devices. Because everything works reliably now and no errors, I just avoid looking at the route changes. :man_shrugging:

1 Like