In the past few months I have been trying to do every automation in the simpler tool - i.e. Motion Sensors in Motion sensing.
The reasons I did this was:
Potential Performance Issues
Better Organization of my Apps page
But as I tweak and optimize I realize that the best way to do automations that are a bit more complex is in the Rules Machine using IF Statements.
It's the best way because it has all the functions (for example, I cannot find Run Custom actions in Motion) and I will have all the logic for each Room/Light group in one place.
I think that in most programming languages, doing things in nested IF Statements is the most inefficient way and I assume it's the same with Hubitat.
Am I overthinking this, or if I create lets say 20 Rules with 4 If Statements in each, Hubitat might start having performance issues?
In general, if all your apps are doing "reasonable" things, this seems unlikely to be a problem no matter how you approach the automation. (By "reasonable," I mean avoiding things that could cause problems no matter how you do them: sending a bunch of commands at the same time, especially on large networks or slow networks; repeating commands in quick succession; etc.).
You are correct that Motion Lighting doesn't have custom commands (it's geared towards lighting, so those are mostly the kind of "actions" you're going to find there). Rule Machine is probably the best option if you need those, though in some cases there are ways to make it work somehow, so consider asking about your specific use case if you aren't sure.
That being said, I still generally prefer to use the simplest built-in app that meets my needs. In some cases, that still means Rule Machine (and I confess that I actually have a lot of custom-written apps I use instead of any built-in option, which is something you might want to look into doing if you're interested in "programming" since you used that word--it should be noted that writing Rule Machine rules is not really writing code, though the UI does display your action selections in a sort of pseudo-code).
In any case, there's no need to guess about performance. If you're on a reasonably new platform version (2.2.5-ish or later), you have runtime statistics available (under "Logs" on the current platform version, a separate page if you're not). This can help narrow down overly-active apps or drivers.
