[2.4.0.151] File Manager API failure (both built-in and custom apps) [C7]

[hubitrep]

Continuing the thread below.

File Manager size limits?

@gopher.ny I wrote a custom app that logs certain device events to a .csv file; the event handler does a straightforward append, using downloadHubFile() followed by uploadHubFile() with the new content. I added the singleThreaded: true flag to the app definition, but still seeing these errors from time to time (twice today) : [image] Any insight as to what could be the cause? I verified that there's only one app writing to this particular file. In the meantime I'll see if I can catch the e…

How can I get attention to this issue @support_team ? It is affecting both built-in and custom apps on two separate C7s of mine.

Just yesterday there were multiple events where my custom app caught an exception calling uploadHubFile() and had to retry to succeed (code below):

Catch-and-retry code for uploadHubFile()

def safeUploadHubFile(fileName, bytes) {
    int maxRetries = 3
    int retryCount = 0
    boolean success = false

    while (retryCount < maxRetries && !success) {
        try {
            uploadHubFile(fileName, bytes)
            success = true
        } catch (Exception e) {
            retryCount++
            if (retryCount < maxRetries) {
                log.warn "Retrying upload of $fileName... Attempt ${retryCount}"
                pauseExecution(500)
            } else {
                log.error "Failed to upload $fileName after ${maxRetries} attempts - possible data loss"
                throw e
            }
        }
    }
}

Rule Machine apps also fail to write to a file once in a while, like so (there’s obviously lots of space left on the hub):

I suppose these hubs are writing to more files than most but there’s nothing extravagant going on here. What are the limits?

These errors are resulting in data loss… any way to raise the priority please?

[user2574]

@hubitrep

This says you are out of space. Have you checked your space remaining in settings?

[hubitrep]

@user2574

checked your space

As stated above and in the quoted thread this issue has nothing to do with available space. The exception type is a total red herring.

I’m afraid only staff can answer this one.

[user2574]

@File Manager size limits?

handler does a straightforward append

I use file manager for some logging, but I just use the append feature. Why download the file and then re-upload it when append is available?

[hubitrep]

@user2574

Why download the file and then re-upload it when append is available

Because it is not available.

You might be confusing the Rule Machine actions with the File Manager API proper. Incidentally the Rule Machine “append” action fails in the same way, as I reported. Most likely it is doing the same under the hood.

[user2574]

@hubitrep

You might be confusing the Rule Machine

I honestly thought you were talking about File Manager Device. My Bad. You may want to try that as a work-around.

[hubitrep]

@user2574

thought you were talking about File Manager Device.

I wasn’t. That is a community driver written by @thebearmay . It isn’t part of the platform.

@user2574

You may want to try that as a work-around.

Thanks for trying to help, but this is a platform bug and as such there is no possible workaround, at least not for built-in apps.

[user2574]

@hubitrep

this is a platform bug and as such there is no possible workaround

I get you want to use the File Manager API in your app, but if it is not working there are workarounds, like using @thebearmay driver with your app and take advantage of his append option.

Very easy to call his device commands from your app.

Edit: I use @thebearmay's File Manager with my custom apps. I look at it like a library, and I have one instance of File Manager as a device that I use for file management with more than one app. Just tag the device on to the app and call it as needed.

[hubitrep]

@user2574

there are workarounds

There's no workaround. Either you take my word for it, or you accept to do the homework

• Built-in apps can't be modified to use something like File Manager Driver. For example, the Notifications app logs to a file, no doubt using the File Manager API directly.
• For Rule Machine, there could be a workaround, which consists of not using the File actions and instead using a driver like the File Manager driver and calling its custom commands.
• Unfortunately, the File Manager driver is not a valid workaround even for Rule Machine, because in the end it uses the same exact File Manager API, and as such is susceptible to the same bug. Unlike the code I shared above, File Manager driver doesn't have a retry mechanism (although it could be added).

@user2574

take advantage of his append option.

There isn't really an append option. Have a look at the code, and you will realize that the append() method is implemented using a readFile() followed by a writeFile() call. You might also notice along the way that those methods don't call the File Manager API directly, instead they use an http endpoint. I believe that's because the File Manager driver was created prior to the File Manager API being made available to Groovy apps and drivers. But it doesn't make any difference, under the hood, those calls all end up running the same code.

[hubitrep]

@moderators is there a more appropriate category for this bug report?

[bobbyD]

We are looking into it. Thanks for your feedback.

[thebearmay]

Could move it to Feedback.

As to what you’re seeing only time I’ve seen it is with a write collision.

[hubitrep]

@thebearmay

write collision

If you mean two threads writing to the same file, this is not what I'm seeing, or at least it is not limited to that case. I am seeing failure to write to one file apparently when another distinct file is being written to.

[user2574]

@hubitrep

There's no workaround. Either you take my word for it, or you accept to do the homework

Ok, got it. It is really a bug in 2.0.4.151. Somehow I totally missed that small fact even though it was in the title, i am an update behind and I did not realize it. Good to know, I won't be doing that upgrade yet.

[sburke781]

Thank goodness for that....

Here I thought we had one of those awkward misunderstandings where people get all pent-up and defensive for no good reason.... Let's hope it get's fixed soon....

[hubitrep]

@sburke781

people get all pent-up

Nothing like that. I posted in "Get Help" and @user2574 was trying to... help

Reporting bugs through forums... not the best process, for all involved IMO.

[sburke781]

Was only a worry on my part... likely and seemingly misguided in this case... thankfully... I just know how things can turn...

[sburke781]

And yes, would not want my response to discourage others from responding to posts like this... am I am as thankful for anyone's contribution as anyone else.... just noting how tone can be important in communicating.... (I'm probably diluting the usefulness of my response now....)

[user2574]

@sburke781

just noting how tone can be important in communicating

I didn't read the original post thoroughly, so it is on me for jumping in and offering help when I wasn't even clear of what the root cause of the issue was at the time. I read it as a user issue, not a hub platform bug, which was wrong.

[sburke781]

All good, any hesitation on either side is enough for me.. Was just worried where the conversation may have lead... Glad id would have only ever lead in the right direction...