I've looked at a few examples, still can't figure out how to make the notice repeat for 10 minutes and then stop when either the door is closed or motion changes to active.
You should be able to put that if block inside a repeat block and have it repeat 2 times since your notification is delayed by 50 minutes and you want it to notify for 10, although I've not tried it with notifications so it may queue up two notifications for 5 minutes out as is and you may need to delay actions (cancelable) as the first step of the repeat block and turn off the delay function for that notification line.
I often refer people to the thread below when they ask how to do repeated notifications in Rule Machine. There are several examples, and you should be able to find one that closely matches what you're looking for (the one with multiple conditions, I'd say) and adjust it from there:
With that said, why not use the built-in Notification app? I think it will do what you want without a somewhat complex rule. The only downside is you cannot select more than one type sensor (motion vs tilt), so you would need two notifiers. But that actually wouldn't be a bad thing, you would get different notifications for door and motion so you could distinguish the two.
Also, the app Event Engine has an option for this specific use-case. If x happens, remind me repeatedly until y happens, then stop reminding. I use it for a trash reminder...every Monday night at 7PM it starts reminding me to take out the trash every 20 minutes. The reminders stop when a contact sensor on an outside gate opens, indicating that I've taken out the trash.