I asked where you are seeing that message, if it was in HK or in HE, you did not really give much details. I know how Homebridge works, I use it myself. Screen shots would be useful because myself and others probably have no idea what "unpair 4 controllers" message you are talking about.
The issue with it mysteriously working after a soft reset has happened before, @lcw731 I think ran into this as well.
Is this what you are seeing but instead of 2 yours says 4?
@gopher.ny What does this number mean exactly?