I looking for a way to persist a large amount of data between threads.. State is not an good option here because of the size and temporary nature of this data.. I need to keep the data around to service 5K+ individual requests for parts of this data.
And then dispose of it when complete.. Total time should be a couple of minutes
Anyone have a good idea?.. I can do without this but it would be much more efficient if I can figure out a way..
I have been researching and so far have come up empty..
One thought I had was to keep a thread running and filling small parts of the data at a time into state for the other threads... But this won't work as state isn't saved until the thread execution is complete. atomicstate is also not an option as this is driver level..
I am building the work-around using math and small http requests for non persistent requests.. Just wanted to be sure there wasn't a more efficient method I wasn't aware of..
Hmm that's an interesting possibility... I see where that could work.. but could get stomped on depending on the timing of the next request thread .. But I still wouldn't have a way to know when to re-fill the buffer.. I think this is an unobtainable goal at this point.. .
I thought the limit on storage for states was more like 100k .. it's 5k as in 5 thousand requests for parts of the data.. The actual data is 101,702 bytes in a byte array in this lab test.. could be more in real world
Good thoughts .. But I also don’t want to keep creating and deleting this huge database entry.. I would be concerned at that point with potentially causing issues with the database store
I was hoping for an in-memory solution... But I don’t think there is a way in the current environment.. There are a few groovy classes that would enable this for me.. But I checked and they don’t exist in the HE environment..
5K requests??? In what time frame... The quantity of requests could be an issue... I remember a thread a while ago, regarding I believe MQTT, where there were concerns that even 75 requests per second would have the potential to bring the HE to a grinding halt...