EEK! That's not good.
Sometimes it does work good like that but there are other times you get the ghost issue. I think this has got better lately, but I still occasionally get a ghost. I had a bunch of ghosts while migrating from a C5 to C7, but the key is to not go on adding stuff until you get rid of the ghost.
Some things that help are to work from the hub outward, adding repeating devices (either dedicated repeaters or line powered switches/outlets) first and go back and add things like battery powered sensors and locks in a second round.
And in some cases, I had to bring the hub closer to a device, or bring a device to the hub to get it to pair without ghosting. That is against any advice you see here or elsewhere, but there it is. Sometimes you just don't have a good signal to pair in place for whatever reason.
Clicking on something by itself probably won't help if the device that caused the ghost is still powered up. Try flipping the breaker for this device, or pull batteries as appropriate before attempting removal. Some have even run the hub (and router) from a battery pack, and flipped the main breaker to cut power to everything if they didn't know what was causing the ghost.
I have been somewhat successful removing ghosts, but you have to click on Refresh, then when Remove comes up, click it and wait. If nothing happens, reboot and try again. You have to be persistent, and not try just once a week or whatever.
Now you shouldn't in theory have to do this, but it is much easier to use a USB Z-stick like the Zooz ZST10 (there are lots of similar devices) and Zwave PC Controller (or its new version Simplicity Studio) to remove these stubborn ghosts. There is a really good writeup here with links to everything and very clear instructions. This works virtually 100% of the time, and in my opinion is faster and quicker. But costs about $20 for the USB stick.
Dropbox - Hubitat UZB Stick How-To.pdf - Simplify your life
I guess that depends upon how many Zwave you have, and how many rules or other things you will break by doing this. If you are just starting, and have maybe 10 devices, and a couple rules, I say "go for it" But it sounds like you are much deeper into it than that. So I would try removing the ghosts, and see if you can get it to clean up.