There was a fix for this (stopped rules running on their own again) in 2.4.1.152, but I think it might only apply to new 2.4.1 upgrades (from 2.4.0 or earlier) to this build (or later).
Assuming you went for one of the earlier 2.4.1 releases first, that was probably what caused this for you, so you'll need some workaround like re-stopping them or restoring your old database backup.
