Sometimes if you do a power down and remove power for 10+ seconds then restart it will help it work. You can also watch the logs for the error message if the remove fails. There is an entire post dedicated on this subject: How To: Remove Z-Wave Ghosts using a UZB Stick
Sometimes those "ghost" nodes can cause issues with the Zwave mesh, sometimes they are harmless. Since yours is showing 3 neighbors still it may be causing some routing problems.
