I found this thread, according to the staff responding, this app's intended function is as I thought, controlling one or more switches with a switch combined with a per mode dimming function. Here is another thread that supports my interpretation.
Update: I found the internal documentation regarding this app, it seems to have been tucked away inside the documentation for the Motion Lighting App, maybe because its bundled as part of the Motion Lighting Built-in app? Anyway, the documentation only confuses me since while it does say that:
"If you want the option to turn these lights on outside of mode changes, select Options for lights on. Choose buttons and/or switches you want to use to turn on the lights manually."
But in the same breath, it also says:
"Mode Lighting apps work similar to Motion Lighting apps, but allow you to set lighting that is triggered by mode changes, rather than motion sensors."
Which does not seem to be working for me either, and I swear I have read multiple threads saying the opposite is true.