Rule Machine isn't the only way, but it's probably the easiest. Hear me out. If you make a new rule with a "Button Device" trigger (not "Button"), you'll basically get the Button Controller UI you already know, but you'll have access to RM features like conditionals, which you are correct that Button Controller does not have.
But even Button Controller has some things you can do per mode without writing conditionals: set color temperature per mode, set (dimmer) level per mode, and probably a couple more. RM has these too, though RM will also let you do whatever you want per mode with conditionals instead too. If BC doesn't cut it for you, I'd try RM next.
Thanks for offer, I probably go for red for the grey switch and have to remember what the buttons do on black.
Thought it would increase WAF but didn't think it through!