It's a known issue in the Synolgoy NAS world. I had the same happen when I had my Pi connected to the Synolgoy though NUT. Let me dig the link out for you on how to fix it:
https://www.synoforum.com/threads/another-dsm7-regression-ups.6586/post-34007&ref=RME8R
You need to remove the ref portion to have a link others can follow:
https://www.synoforum.com/threads/another-dsm7-regression-ups.6586/page-2#post-34007
The situation described in that thread is as I mentioned above. NUT has already declared a low battery situation and that a shutdown is underway. In this condition, it will not allow the attach of new clients (thus the user/pass error).
I performed a more controlled set of experiments this morning, tl;dr at this point my UPS ecosystem seems to be working correctly.
I'm running a CyberPower CP1000 (purchased new and installed in Jan '21) and my Synology DS218+ NAS is still on DSM 6.2.4x. I have configured the upsmon driver to use the Synology NUT driver's "hardcoded" authentication with user:monuser and pw:secret
Today, without changing any other parameters in my NAS or hub's setups, I enabled DEBUG on the upsmon driver and tried interrupting the mains power for increasingly long periods to see what log entries I received on the HE hub and the NAS. I also tried the same power interruptions with and without Enable Hub Shutdown turned on. I was careful to create longer power interruptions that were guaranteed to span the upsmon polling period of 20 sec.
All the logs and behaviors were entirely nominal as I would have expected given the setup conditions, unlike the anomalous behavior resulting from the spontaneous power glitch the previous day. The NAS showed the correct log entries and the HE hub logs showed that the UPS operating parameters were being fetched correctly at the polling rate.
Furthermore, the HE hub logs correctly showed no switchover to battery power when the power interruption did not span a polling cycle, even when registered in the NAS logs. Interestingly, when the power interruption did span a polling cycle with Enable Hub Shutdown turned on, the hub did not perform an immediate shutdown; I infer from this that there must be some small additional delay required while on battery power before a properly initiated hub shutdown occurs.
Additional data from yesterday's live power interruption: my NAS logs did not show any evidence of a switchover to UPS battery power at that time like they did today. I also didn't get any of the notifications (mobile and email) that the NAS should have issued. From this difference in the behavior of yesterday vs today's test I am inclined to think that the NAS's NUT subsystem wasn't operating correctly and that simply rebooting and re-entering the NAS's UPS parameters were responsible for the fix. However that still doesn't explain to me why the HE hub went into immediate shutdown yesterday... ĀÆ\_(ć)_/ĀÆ
Rather than relying on the Enable Hub Shutdown feature of upsmon I think instead that I'll install an RM rule that gives some more leeway, notification and options to shut down the hub when it's switched over to battery. Has anyone else already cooked up something I could reuse or modify?
I used this with a different driver before and switched it over to this one. Maybe more complex than it needs to be but it does what I wanted it to do. It calls the shutdown command exposed with the Hub Info Driver v3.
Assuming you still have an internet connection it gives status updates as the battery depletes before shutting down. If power is restored it will notify on that and then give you a final update when the battery is fully charged again.
Thanks @jtp10181 ! What exactly is your trigger event for the actions and are there any preconditions?
I updated mine above with the trigger info.
Couple of things...
Regarding the client not noticing short outages, NUT is a poll model with no push capability. What this means is that the upsmon client will have no knowledge of state transitions that happen between polls. In other words, if mains are lost and restored in-between polls the clients will never know. If knowing this is important, you need to use a smaller polling interval. NUT's default, and the default I use in the upsmon driver, is 5 seconds.
Regarding the immediate hub shutdown, yes, this does make sense. There are two statuses that will trigger a shutdown in the upsmon driver, and there is no delay for either of them. The first, is a combination of 'OB' (On Battery) and 'LB' (Low Battery). This is the normal status generated by the driver when the UPS is running on battery and the battery is running low. The other is 'FSD' (Forced Shut Down). This status is used by the server to force all subordinate systems to shut down pending restart. FSD is a one-way latch, and is not reset by the return of mains. I believe that the NUT server on your Synology was in this state, waiting for power to be cut.
Here is the relevant passage from the NUT documentation:
By design, since we require power-cycling the load and don't
want some systems to be powered off while others remain running
if the "wall power" returns at the wrong moment as usual, the "FSD"
flag can not be removed from the data server unless its daemon is
restarted. If we do take the first step in critical mode, then we
intend to go all the way -- shut down all the servers gracefully,
and power down the UPS.
So when you rebooted your hub, and it contacted Synology's NUT server, it was told to immediately shut down ('FSD') because the NUT server was expecting power to be cut. No way to change this without restarting the NUT server on the Synology.
Best guess is that you encountered a bug in Synology's configuration of the NUT shutdown command, or an incompatibility between your UPS and Synology. Either way, I would suggest that you explore this, either in the Synology forums or with Synology support, because it means that your NAS will not shut down correctly.
Yes based on my research the erroneous issuance of latched "FSD OL" status from the server seems to be a known issue with the Synology DSM version 6.x and still appearing in version 7, perhaps dependent on the specific behavior of some UPS models including those like mine from Cyberpower.
So last night I opened a ticket with Synology support based on comments from some Syno forum posts that indicated there was a possible patch available to handle the issue (that for some reason did not make it into regular DSM updates) - IDK if the "patch" is a code update or perhaps some parameter changes applied to uspmon.conf. Waiting to hear back from them.
It's not clear to me why I never saw the "FSD OL" status returned from the server and displayed in the hub logs during my first encounter with this issue, nor during my power-interruption testing. I can only think it has something to do with the duration and timing of the power outage interacting with the various polling intervals employed by the server, the UPS and the hub.
Status is not in the log, it's in events. "FSD OL" will show up as status "Online Forced Shutdown". In the log, you will see "upsd requesting client shutdown".
I added the status to the shutdown log entry to make it simpler to identify in the future.
Following this current discussion with much interest as I too am using this NUT client app combined with a Synology NAS 220+ hosted NUT server which is connected to a CyberPower CP1500 UPS via USB. I have not had a power outage since installing however my son runs the same Synology NAS and setup. Only difference is he has a Eaton UPS and he HAS experienced the immediate forced shutdown action following a power loss. He was also forced to turn off the NAS to get the Hubitat hub to stay powered up. So it looks like it is a Synology issue rather than a CyberPower related issue. We both run the latest V7 firmware on our NAS.
On my Hubitat NUT client device the Current States shows āstatus: Onlineā. Would this status display as āFSD OLā if the Synology NAS NUT server was āstuckā in this state? If so, could this be used to generate a warning notification that the Synology needs a reboot?
'FSD OL' is a NUT stat word list (NUT line protocol). What you would see on the hub is a status of "Online Forced Shut Down" as noted above.
FWIW, I'm not sure you have to restart the whole Synology. You can probably just disable/re-enable the UPS in the Control Panel, or perhaps even just a disable/re-enable the network UPS Server option will do it.
I only became aware of this anomalous behavior based on the hub's log output with Debug enabled in the uspmon driver - every polling period there was another sequence of log messages like this:
On a somewhat tangent subject: this excerpt from the log was generated with the uspmon driver's Enable Hub Shutdown set to off - would that be the cause of the error entry when FSD OL status was received from the server?
Yep, upsd is instructing immediate client shutdown. FSD will be locked on. Nothing will fix that other than restarting upsd (and probably the ups driver) on the server. If the Synology isn't shutting down, and requesting the UPS to cut power, that's a bug. Either in NUT (unlikely) or in Synology's configuration of NUT. Note that this behavior would affect any remote clients (even remote Synology servers).
Just out of curiosity, would you mind posting a screenshot of Control Panel -> Hardware & Power -> UPS on the Synology?
As to the error in the log, yes, if the hub receives a status from upsd indicating a shutdown condition, and shutdown is disabled on the hub, an error is logged. It's just the class of the log message, with no other effect.
Sigh. I just looked at Synology's NUT config. It seems that they've had another attack of the clevers. They are using upssched to route everything to a complex custom script of their own, and in doing so they have completely disabled NUT's shutdown processing. I don't understand why everyone likes to think that they are smarter than the NUT developers.
As it sits, I would not trust the UPS service on Synology DSM 7.
As you can see I'm not attempting to use the UPS shutdown feature. Conserving battery power during a power outage by shutting down the UPS entirely seemed like an unnecessary frill at this stage of my system development.
Yes that's one of the reasons I'm still on 6.2.4x (as mentioned in my OP.) DSM 7 users have seen Synology go back and forth on supporting the UPS shutdown feature with attendant changes and complications to their installations.
Assuming it actually works, you probably want to use the shut down option. This is the only way that NUT is going to recover after the UPS reports a low battery. Your config doesn't seem to offer the "Until low battery option", which is sometimes preferable as not all UPS system cut the load unless there is a low battery situation.
In general, would not count on proper NUT function for Synology without explicit testing.
That is why the synoforum community recommended the script changes to the configuration that Synolgoy provided. It will bring the server back out of FSD state. The "Experts" at Synolgoy never acknowledge the issue and the only responses I ever got from their support was working as designed and that they don't support the networking of UPS's to other devices except for other Synolgoy NAS's.
Yeah that's basically what I got from Synology Support on this issue. And that they don't support users modifying the as-shipped system via the CLI. Luckily NUT and Hubitat are script-driven to a large degree so are amenable to tinkering, though it does require self-documentation to accomodate DSM updates and I'll have to revisit it again if I "upgrade" to DSM 7. I have the system pretty much working as I want for now while avoiding FSD lockup.
Download the Hubitat app
