In use by bug

It is true that removing a device from a rule will often not clear the In-Use setting for the device. It turns out that doing this would actually be quite difficult. You can see the selected devices in the App Status Settings for the rule. If this bothers you the fix is to remove that rule and recreate it without deleting any devices. To avoid it in the first place you would have to diligently deselect any device you don’t want. If you delete an action odds are good that any device in that action is still “in use”.