Another Wait for Expression RM failure

📐 Rule Machine® All things Rule Machine
66

Because I am dealing with a "black box" with a lot of hidden don'ts.
Maybe I have to learn Groovy in order to have better control and understanding of what is going on.

So far I did not use a Duration at all. let me see how this will fit.

For years I was using IF-THEN-ELSE approach for creating home automation rules on different platforms and never had any single system-related problems. Whatever was a self-inflicting problem is a very different story.
In theory RM has all power for creating rules with different complexity. But ... RM has a lot of hidden "don't do this". This hidden 'features' basically makes it very difficult to deal with. Using a multiple rules instead of one single immediately creates a huge maintainability related problem.
Let me see what I can do with Room Lights application. But lighting related RM rules is only about 25-30% of all my automations
But you are absolutely correct - I have to learn Groovy. I guess, this is the only way to keep everything under control and understanding what is going on.

67

For this specific use case you might want to have a look at a Zone Motion Controller set up for motion aggregation.

68

Usually not necessary

69

If you're going to use Room Lighting I would agree. But as a surrogate for a presence device, which behaviour can then be used to drive other rules (breaking the logic up into more manageable parts and perhaps avoid race conditions), I find it useful.

70

And this was ideal programming approach because you knew up to very little detail what is going on. These days all high level programming completely decoupled from HW. Yes, this way the SW could be designed much faster but you have to rely on too many unknowns.

And what normal and logical thinking should be?
You have set of valid statements, etc. but you cannot use them because it could be hidden and unknown implications? Very nice. Say, you have Dollars and Euros in your valet. But a hidden rule is - if you use Dollars you will lose all your Euros and vice versa (maybe not a very clear example but I hope the idea is clear).

Bottom line:
I am very experienced EE dealing with very complex designs involving a lot of custom logic and embedded processors. My point is - I know very well how to design very stable and robust logic of different complexity. But all statements I am using in my designs are doing exactly what they must/should do without any ifs and/or other hidden restrictions.
My expectation for the RM naturally is the same but unfortunately this is not a case.
Now please think about whose problem this is - end user or a platform presented to the end user as a "black box" with a lot of hidden restrictions?

It looks like with the current RM implementation these two approaches are not playing nice concert. Each one has a pros and cons.

71

There aren't really unknown implications, but some are subtle and learning is required. The hub is an event driven, multi-CPU, multi-threaded computing engine.

Apps have state variables (certainly Rules have a lot of state variables). Those state variables are stored in the database, so they persist across instantiations of the app. Every time an app exits from execution the system writes the state to the database, and every time app execution begins its state is loaded from the database. This writing and loading of state takes some amount of time -- not a lot and not zero. It can't be predicted exactly how much time it takes, but evidently in some cases longer than 4 msec.

There is only a single copy of state for all instances of the same app. So, having multiple instances executing at the same time can be problematic. The most typical problem comes from delays or repeats embedded in an IF-THEN-ELSE of a rule. Because IF-THEN-ELSE can be nested, the current point in an IF-THEN-ELSE is recorded in state. So with a delay embedded, the rule will exit for the delay, and if another instance comes along into the IF-THEN-ELSE, there is intrinsic ambiguity as to which instance is running, and the state can be put into a wrong condition by the second instance.

These multiple instance problems can be avoided for the most part in RM by using Required Expression for PB true, setting it false as the first action and setting it to true as the last action.

Rules with multiple devices as triggers pose the problem that they can be triggered multiple times, possibly resulting in multiple simultaneous instances of the rule (although not necessarily, depending on timing of the events). In the case where the rule is simple, this won't cause any problem. For example turning on a device multiple times is generally harmless. But some thought about what can happen is required -- just as it would be with assembly language or any programming language.

Some apps are "Single Threaded", a setting that causes the underlying system to only allow a single instance to be running at any given instant in time (but not preventing multiple instances per se). Apps like Room Lights are Single Threaded. Rule Machine is not. The main virtue of Single Threaded execution is that we can be fairly certain there won't be simultaneous attempts to read or write state for the app. Even with Single Threaded execution, Room Lights still has to contend with multiple device events and how that might affect the state and actions the app should take.

None of the above is really 'black box'. This is all pretty normal stuff for software in a multi-CPU multi-threaded environment. Dealing with the implications of such an environment is similar whenever coding in assembly language, Groovy or Rule Machine.

5 Likes
72

Zone Motion Controller has no beneficial impact on other apps relative to the topics under discussion in this thread. An app like Room Lights has the exact same logic to deal with multiple motion sensors, and a Rule can have the same if properly implemented. ZMC actually introduces more overall complexity by creating a virtual device. It's intended purpose has to do with dealing with false events in an outdoor setting, not with reducing complexity for multiple motion devices.

2 Likes
73

First of all - very BIG Thank you for the very valuable details.
Yes, I clearly understand what is Event Driven concept. From the other side I never had a clear vision how modern SW is dealing with multi-core CPUs. In a past I did design multi-CPU systems (not a multi-core CPUs but rathe multiple physical CPUs). Deferent CPUs had a well defined dedicated tasks. For instance, one CPU handles all IOs and another one does everything else. Communication between CPUs is a sheared memory used as a mail box with semaphores and Interrupts telling each CPU something is ready in the mail box. I am totally lost how this is done these days but frankly I really don't care.

Returning to the RM.
The Database for storing all current states is similar to the HW Registers. On each Event the states from the Database is used in RM (or whatever application) as soon as Event to with rule is subscribed is occurred. Rule does something related to this Event, updates new states in DB (yes, the update takes some time and yes, this time cannot be predicted but DB update can be protected by Semaphores) and goes to sleep allowing CPU(s) do something else.

Now I am thinking as EE. The Event occurred and rule is started. While progressing suddenly anther Event occurred. So, how to handle this sequential Events (obviously could be more than two)? The rule should be just restarted with the DB reset to known state. Depend on Even Type the DB reset could be partial for Regular Event (resetting all rule related states but keeping the Device states intact) or complete reset if Event Type is Emergency (as of now all Events are Regular, there is no such thing like Emergency Event).
I am trying hard to understand what is a benefit for the creating multiple running rule instances versus just restarting already running instance with DB appropriately reset. I can see only a potential troubles like you already pointed out but cannot see any benefits.

74

OK, and Thank You for the idea. I update this rule in question (and few more) with the Wait for Expression and use Duration statement. So far so good. let me see how it will run over time.

Here is an updated rule:

image
image1580×1040 109 KB

75

I donno where this explanatory stuff should best reside,
but it should be in a place readily encountered vs stumbled upon.

2 Likes
76

I've seen (and occasionally written) rules like:
Trigger: Switch 1 turns on or Switch 2 turns on
Actions:
set local_variable_device to %device%
if local_variable_device = 'Switch 1' then...
else
if local_variable_device = 'Switch 2' then...
end if

This rule branches out into different actions depending on what device triggered it. There might be no conflict in having multiple instances running at once. While it might be cleaner to have separate rules for each device, but this sort of thing is currently a valid way of using RM and I've done it on occasion.

77

As long as there are no delays or repeats in those IF-THEN-ELSE blocks it should be fine.

78

This kind of info is extremely helpful!

Thanks!

1 Like
79

This isn't how the underlying system works. This statement would imply that there is more than one copy of state for a given rule (the current state and some original state). But there is only one state.

80

The beneficial impact is separation of concerns, which helps with cognitive load. Not to mention events that can be leveraged in other rules. Separate logging of the virtual device. No chance of a race condition (I assume?). What am I getting wrong ?

I do understand the same can be achieved within a rule, however I don't see the point of writing a rule as @vitaliy_kh did when a built-in construct will work. Agree that, if RL fits the use case, there would be no point in layering in ZMC.

The rule shown sets a virtual presence device... I rest my case.

81

Actually, that's a comment about what @vitaliy_kh is doing in that particular rule, and not about ZMC. ZMC would introduce yet another virtual device.

Look, you're free to do things in whatever way makes sense for you. I'm simply trying to provide complete information. ZMC is not some magic panacea for multiple motion devices, it's just way to do something.

This is not even relevant -- the race conditions don't come from multiple motion devices, but rather from multiple instances of a rule and timing issues associated therewith. There is not some special benefit here from ZMC.

As I said above. the intended use for ZMC has to do with filtering out extraneous motion events in an outdoor setting. But if you find it useful for other things. have at it. Just don't try to layer on more values than it actually has, or propose it as a solution for problems that have nothing to do with it.

82

All what is needed - is a single current state. Furthermore, even current state is need only for the still running instance. When rule finished state is not needed anymore.

What you just said is exactly my point. Multiple running instances are not useful but instead they are creating many problems. The solution seems to be very obvious - simply do not create multiple running instances. As you mentioned before Room Lighting app is a single threaded. So, there is no platform-related limitation.
Why not to create single threaded RM 6.X?

1 Like
83

Well, yes. Hence my use of "For this specific use case (...)" in the post where I made the suggestion.

I'm sorry Bruce, but I think you misunderstood. I suggested using ZMC instead of a rule, therefore avoiding, not solving, the issue.

Right - I totally get it and my suggestion is clearly not relevant to that. And by the way, thank you for the time you spent explaining the details in this thread, I and I am sure many others have learned a lot.

That being said, a personal opinion : while I had already pretty much made up my mind on avoiding re-triggering / multiple instances for my rules, what I've learned in this thread tends to solidify that choice. Breaking things up logically into different rules, using the pb pattern, using virtual devices to represent state outside of a rule instead of having it buried inside of it, those are tricks I use and find useful to KISS.

1 Like
84

All these trick are not a solution for the "running multiple instances" problem but just a band aids. Using multiple smaller rules and many virtual devices or variables drastically reduces maintainability. The problem must be solved globally by preventing of creation these multiple running instances instances instead of creating them and than applying all these band aids.

PS. When I joined my company back in 2007 they had many issues related to EMI (Electromagnetic Interference). The EMI test never passed on a first try. The solution was to put left and right many different shielding. I had a hard time to convince engineers and managers to apply just a best design practice in order not to create EMI in a first place. Very first design which followed this approach passed EMI test on a first try with very significant positive margins. And near all followed designs basically did not have EMI related problems.

Point is: Do not create a problem in a first place.

85

I'll say this as gently as possible. There is no issue in rule machine except in rule design and the usage of the Private Boolean within it. Your original rule before adding a PB worked fine without because every time a motion sensor triggers, it wipes out both waits and starts over again. In this scenario, your rule is only running once because it resets itself after each trigger.

If you want to use the Private Boolean, then you need to follow the best practice design with that variable. Almost everyone who uses a PB has it set false as the first action and then true as the last action. This is not what your original rule was doing and causes the issues that was seen. The new rule should work better as it is now following best practice.

1 Like