You can certainly remove Hubitat from Assistant and Google Home from Hubitat. Especially since you say that you have nothing configured. When you add Hubitat back to Google Assistant it will add it back to Hubitat again.
Understand the direction is Hubitat > Google Assistant (Google Home). Google Home devices do not show up in Hubitat. Also keep in mind that not all devices on your Hubitat hub are compatible with Google Home. If they don't show up in the Google Assistant app, then you can select them to add from the Google Home Hubitat app, but they will not show up in Google Home or Google Assistant.
Make sure you're using your Hubitat Administrator email (the one that you used to register your hub) when you activate Hubitat in Google Assistant. It's the email address you see at the top left when you go to portal.hubitat.com