If your rule is otherwise working as expected, adding a simple conditional (Conditional Actions > Simple Conditional Action in the menu) to check for the time and run an "Exit Rule" if it's not a match would be an easy way to do this. Another option is wrapping everything you have around a full-blown IF THEN ... END-IF with that same condition, but that takes a bit more clicking.
Here's all the first option would entail:
