12 Ghost Nodes. Nuke and Reboot?

I migrated my c5 to a c7 a few months ago. Been plagued with ghost nodes. Have 12 of them now. Adding any new devices has become much more challenging than it ever was. Since the upgrade I've not had a single instance of putting the hub into discovery mode, putting the device into pairing mode, and having the device found on the first try. I've usually had to exclude the brand new devices several times, factory reset them, and just try over and over again. This seems to have led to an even larger number of ghost nodes.

I've followed Hubitat's suggestions for getting rid of the ghosts, but that seldom has worked. I think I've been able to clear out 3 or 4 of them, but the others just won't die. Some of them I've clicked remove on 20 and 30 times over the course of a week. Still freaking there.

First question is whether its just naive to think you can pair a zwave device successfully the first try. Does it ever work out that easily for people?

Second question is that given the number of ghost nodes I have, is it time to just reset this hub to defaults and start completely over?

EEK! That's not good.

Sometimes it does work good like that but there are other times you get the ghost issue. I think this has got better lately, but I still occasionally get a ghost. I had a bunch of ghosts while migrating from a C5 to C7, but the key is to not go on adding stuff until you get rid of the ghost.

Some things that help are to work from the hub outward, adding repeating devices (either dedicated repeaters or line powered switches/outlets) first and go back and add things like battery powered sensors and locks in a second round.

And in some cases, I had to bring the hub closer to a device, or bring a device to the hub to get it to pair without ghosting. That is against any advice you see here or elsewhere, but there it is. Sometimes you just don't have a good signal to pair in place for whatever reason.

Clicking on something by itself probably won't help if the device that caused the ghost is still powered up. Try flipping the breaker for this device, or pull batteries as appropriate before attempting removal. Some have even run the hub (and router) from a battery pack, and flipped the main breaker to cut power to everything if they didn't know what was causing the ghost.
I have been somewhat successful removing ghosts, but you have to click on Refresh, then when Remove comes up, click it and wait. If nothing happens, reboot and try again. You have to be persistent, and not try just once a week or whatever.

Now you shouldn't in theory have to do this, but it is much easier to use a USB Z-stick like the Zooz ZST10 (there are lots of similar devices) and Zwave PC Controller (or its new version Simplicity Studio) to remove these stubborn ghosts. There is a really good writeup here with links to everything and very clear instructions. This works virtually 100% of the time, and in my opinion is faster and quicker. But costs about $20 for the USB stick.
Dropbox - Hubitat UZB Stick How-To.pdf - Simplify your life

I guess that depends upon how many Zwave you have, and how many rules or other things you will break by doing this. If you are just starting, and have maybe 10 devices, and a couple rules, I say "go for it" But it sounds like you are much deeper into it than that. So I would try removing the ghosts, and see if you can get it to clean up.

2 Likes

i think i have around 80 devices, so nuking and starting from scratch isnโ€™t something iโ€™d look forward to.

the idea of cutting power to everything sounds interesting. What if I took my hub to work, hooked it up there, and then tried removing the ghosts there?

Iโ€™ll have to check out Simplicity Studio. Zwave stick is cheap, so not worried about that.

1 Like

You could try, but depending upon your work, you might not be able to access the hub. At my work, they actively block connections like this. But generally speaking, yep that could work.

1 Like

You'll be glad you did. Even once you get rid of these 12 ghosts there will be others in your future unless you never buy another zwave device. I have also found it is sometimes a lot easier to add new devices to the hub using the stick instead of the hub itself.

3 Likes

Also sometimes easier/faster to update firmware with a Z-Wave stick as well..

4 Likes

I think that anyone who has paired zwave devices to a C7 for the last few months has has ghost devices. I know that I certainly have.
I know that you said that you have been able to get rid of some of them, via the "remove" button. That's certainly the easiest way. Try it a couple of times, just in case.
I've also used the PC Controller program with a Zooz stick. I wish I didn't have to, but that's reality.
It's my personal opinion, that the longer the time that the device is exchanging data with the hub, the less likely you are to get a ghost. Therefore, I have been getting the device ready into pairing mode before I get the Hub into pairing mode. Those 60 seconds are very precious seconds for the Hub and the device to exchange data - make the most of them. (I think that's why it may be advantageous to get the Hub as close as possible to the device.)

1 Like

Resetting before exclusion would lead to ghost creation. Successful exclusion prior to reset would not. Before going forward, there are a few questions that need to be answered. The first, is the method in which you've "migrated". Did you use Hub Protect or you've manually rebuilt your mesh? Second, and probably the most important, is what are the specific make and model of the devices that you had hard time including?

Based on other users' experience and my own, you can certainly build a healthy mesh only by using the hub itself without any additional 3rd party software or hardware, and that includes the removal of any ghosts left behind by failed inclusions. It is possible that too many failed inclusions may have corrupted your Z-Wave database, that may explain the difficulty of including new devices.

For removal of ghosts, I would try the following steps: go to Z-Wave Details page, then click "Refresh" button for the node you'd like to remove. After the page reloads, then click "Remove". Note that if the node doesn't show as failed after the page reloads, you may need to refresh the browser page manually (not using the refresh button). For some stubborn nodes, this process may need to be repeated several times.

In your case, I would also recommend updating the Z-Wave firmware. Please PM me so I can give you more details on how to perform the update.

1 Like

Not I :slight_smile: And many others.

This practice is truly a double edged sword. If you include the device close to the hub, then move the hub in its permanent location, the device may not be able to reach it, it goes unresponsive, exclusion cannot be performed, force removal happens, then a stubborn ghost that cannot be easily removed is born. Of course, this doesn't apply to some non-Z-Wave Plus devices and secure devices (garage doors, locks) those may refuse to include via repeaters and a direct connection to the hub is required.

1 Like

Why are we even having ghost issues? This seems to have become really bad with the C7. Are these things not tested and vetted before release, if not why and if so, why these problems?

3 Likes

I reached the same conclusion, but then I realized the folks that come here for assistance are the ones having problems. Few people come to post "Hey no issues with ZWave, everything working fine," yet that is probably the majority of users.

1 Like

Well I haven't really had any ghost device issues in the last few months just some very persistent and annoying "ghost routes" - devices showing routing through non-existent devices. Things eventually cleared up though, not sure if it was multiple reboots or firmware updates etc...

Z-Wave exclusion is a handshake between the hub's Z-Wave radio and the device. The radio sends a message to the device and the device agrees to "detach" from the controller. If one or the other doesn't get the memo, a failed node is created. On the device side, it means that association with a controller is still active (this is the reason why an exclusion must be done before the device can be included on a different network). On the controller side, the device remains active until the radio realizes that it is no longer responding to wake up calls. Only then, the radio marks it ready for removal and the so-called "ghost" can be successfully removed from the network. When the radio marks the device as dead, is outside of Hubitat's control.

2 Likes

It seems like since HE does less hand-holding that this sometimes can occur. I am not saying this is a bad thing - rather have the :zap: power :zap: . To be sure you can get ghosts on other platforms as well but are usually limited in what you can do and sometimes their existence remains hidden..

1 Like

I just added a Aoetec Multisensor to my C7 this week and it created a ghost node. I just kept at refreshing and removing It took me about 8 or 10 tries before it finally removed it.

I was determined I wasn't going to have to pull out my Z-stick and try to remember what I did to use it. I had the same issue a few revisions back when I migrated to the C7. I had 2 or 3 nodes I couldn't get rid of. It wasn't that bad but I have since changed computers and I am going to have to reinstall everything and set it up again.

2 Likes

If you have failure in removing ghosts, you can get a z-wave stick to pair as a secondary controller and use that for removal. That said, anytime you pair something and it fails. Stop what you're doing and remove it from the z-wave table.

1 Like

During my migration of about 80 Zwave devices, I got about 10 ghost nodes. Trying to remove these ghost nodes via the hub is futile in my opinion. Occasionally it works when you click Remove. Occasionally it works when you click remove 50 times. And most of the time it doesn't work at all. A zstick seems to work 99% of the time. While I understand the radio in use in the HE doesn't give us the ability to mark things as "failed" it's super disappointing that SILabs (the same company that makes the radio inside HE) offers the zstick with all of the functionality missing. A zstick is so necessary, I almost feel like if HE can't get SILabs to add this functionality to the radio, a ZStick should just be bundled with the hub. I can't imagine supporting my hub without one now that I have it. It's really the only surefire way to remove ghosts and it takes under a minute.

I can't say why it seems so common. But I agree, ghost nodes aren't rare and after I experienced issues I had lots of people reach out to me and say they had almost identical problems to me, but for a particular reason, they decided not to post about it. So I think the issue is more widespread than they say, or in fact, more widespread than they're even aware when people choose not to report it!

Eliminating the ghost nodes is crucial though, it killed my mesh. Things work A LOT better when you've eliminated the ghost nodes. I'll also say that it's important any time you add a device to check for ghost nodes because I've had times where everything appeared to successfully pair but really there was a ghost node created so now I always check.

6 Likes

I've actually said the exact same thing before. :slight_smile:

7 Likes

In your recent experience, have you found this to be true, or am I just imagining it?

1 Like

Knock on wood, I have never had a ghost node. I have always paired devices in place and always run device exclusion on new device prior to inclusion. Almost without fail, Hubitat finds something to exclude. I have no idea if excluding first is why I don't get ghost nodes.

2 Likes