Data persistence between threads

I looking for a way to persist a large amount of data between threads.. State is not an good option here because of the size and temporary nature of this data.. I need to keep the data around to service 5K+ individual requests for parts of this data.

And then dispose of it when complete.. Total time should be a couple of minutes

Anyone have a good idea?.. I can do without this but it would be much more efficient if I can figure out a way..

I have been researching and so far have come up empty..

One thought I had was to keep a thread running and filling small parts of the data at a time into state for the other threads... But this won't work as state isn't saved until the thread execution is complete. atomicstate is also not an option as this is driver level..

have you tried using Data (storage)?

Is this saved before thread execution is complete?

As far as I know, atomic state is the only storage that is 'saved immediately'

Yea.. That's what I thought.. not an option here..

I am building the work-around using math and small http requests for non persistent requests.. Just wanted to be sure there wasn't a more efficient method I wasn't aware of..

Maybe you could (ab)use runinmillis to save to data?

Like the man on the hill.. you said "between threads"

multiple interpretations possible. I interpreted as "inner thread communications" :slight_smile:

2 Likes

Hmm that's an interesting possibility... I see where that could work.. but could get stomped on depending on the timing of the next request thread .. But I still wouldn't have a way to know when to re-fill the buffer.. I think this is an unobtainable goal at this point.. .

I will just go with my work around..

1 Like

Why? 5k works for atomicState.

Since this is a driver, why not use a parent app as your “storage manager”

EDIT: Another idea. How about splitting data across a few attributes?

I thought the limit on storage for states was more like 100k .. it's 5k as in 5 thousand requests for parts of the data.. The actual data is 101,702 bytes in a byte array in this lab test.. could be more in real world

I suppose if they are following ST then it should be 100k. You could create a couple child devices that do nothing but store and retrieve data. :thinking:

Good thoughts .. But I also don’t want to keep creating and deleting this huge database entry.. I would be concerned at that point with potentially causing issues with the database store

I was hoping for an in-memory solution... But I don’t think there is a way in the current environment.. There are a few groovy classes that would enable this for me.. But I checked and they don’t exist in the HE environment..

5K requests??? In what time frame... The quantity of requests could be an issue... I remember a thread a while ago, regarding I believe MQTT, where there were concerns that even 75 requests per second would have the potential to bring the HE to a grinding halt...

1 Like