There's a setting in Home to tell your devices the type. Is this the integration you're using?
You can also create a routine in Google Assistant to trigger one of the virtual switches. Just remove the harmony switch from that room. It's better to generally not include these special switches in rooms because you have that issue. I have a routine on Alexa (same thing roughly) and I say "turn the TV off". That trigger my switch to turn off the AV. Only other routines it's included in are a "Goodnight", one called "I'm leaving", and one called "I'm going upstairs"
Here's an example of how to change the device type in Google Home
