With that said, please provide logs so the issue you describe can be debugged. I have two instances in my house where input from a Pico button controller is used to control a zigbee in-box dimmer. The response is pretty much instantaneous.
We should determine where in the chain of events the slowdown is occurring. The logs should display the times between each step of the process.
How is the delay if you turn on the bulb from the device page?
When you turn on the switch, do you see the change on the device page immediately?
On another topic, I recently installed several Wiz bulbs using the Wiz integration app, and they really work well. Is there any particular reason you connected them via Matter. I had a poor experience with the one Matter device I experimented with.