This is more of a Google issue than a Hubitat issue. If you configured this group into Room Lighting and used an Activator Device, it would be possible to send multiple On commands to get the group back to the desired state. You would just need the options "Activate if already partially activated" and "Turn off even if partially off". With these settings, from Hubitat, you could turn on and off the Activator Device and it will always send either an On or Off command regardless of which lights are on/off.
As for Google, that is going to be a trickier problem to solve. This came up in Fire an action when switch is turned on or off while already on or off - #7 by bertabcd1234 and seems to work for Alexa. You might want to try the virtual driver in this thread to see if it works for your virtual Downstairs switch.
