That shouldn't be happening. But, I think there is a bug in the way override is working (or not, as the case may be).
The way it's supposed to work is that if the lights have come on by motion to some level, and you adjust the level with they physical dimmer or Alexa, then it is supposed to stop changing the level or turning off until it is turned off by some means. I just discovered that it's missing the Alexa element of that. This will be fixed in an upcoming release.