After a bit more study I was able to determine that their is an issue with the driver. What is happening is that sometimes the state.LastButtonNumber is getting stuck at 1 even after the timeout. When that happens it then ignores the next button press (considers it the dup) and then the actual dup button press gets treated like the real thing.
What I suspect is happening as I've run into this before is that it is caused by caching of state variables. When you write a state variable and then almost immediately read it back there is no guarantee that you will actually get the most recent value. In apps you can guarantee it by using Atomic state but not in drivers.
However I do think this link has an approach will resolve the problem. Basically, the driver waits until the end of the debounce period and then only submits a single event, no matter how many events are received.
The only disadvantage is that the button press is delayed until the debounce period ends so it reduces the responsiveness of the system, but that is a small price for stability when the response is sub second.
Hope that helps.