You would have to use something like Cast-Web-API which exposes each GH device as a "castable" device. You can't broadcast to it but you can speak text to it to get an announcement. It uses the cast capability of the GH though, so any music you have playing will stop and not restart after the announcement. This applies only to announcements to single devices. Group broadcast is different.
The alternative in assistant-relay is to create other google accounts and link those only to some speakers. Then when you issue the broadcast command under that user, the broadcast would only play on some speakers. But you would have to have another google account and set up another google console project and get another oauth file. Very cumbersome.