I’m trying to add a variable number of keys to my extension StorageManger object ‘on-the-fly’ from data coming into my system.
I’ve successfully been able to do so by setting the ‘locked’ flag in the StorageManager constructor to False and adding items via the _addItem() method tucked away in the TDStoreTools module…
/sys/TDModules/TDStoreTools
However, it appears I have to run the _sync() method as well in order for the item to write to the StorageManager object. The doc string for _sync() states…
I realize I’m accessing ‘internal’ TD class methods - however, there doesn’t appear to be any documentation around adding items to StorageManager objects outside of the (re)initialization of the storedItems list. So, I’m wondering if this is the proper method to do such things and if running _sync() can cause any issues if run outside of initialization.
Below is the generic code I’m using to add items to the StorageManager object…
You are using features that were never fully fleshed out, as we didn’t find it effective to add storage items on the fly.
However, you are using it correctly. A couple tidbits that might be helpful:
The _setItems(storedItems, sync=True) method works just like _addItem but accepts a list like the one usually provided in init. It also does a sync automatically at end if sync is True.
There is no danger with using _sync except that it is a bit slow. Be aware that _sync has an argument _sync(deleteOld=True). If deleteOld is True, any items in storage that don’t have a corresponding entry in StorageManager will be deleted.
Like I said, this is rather uncharted territory. Feel free to post any bugs you find or features you need on this thread.
Thanks for the info - I’m fleshing out some concepts using this technique and will definitely let you know if I run across any bugs or issues. So far it’s been quite stable and effective.
I’m curious why it was found to be ineffective adding items on the fly? I’m primarily dealing with new variable data coming into the system that I need to effectively store with dependencies and read settings. I find this method to be more elegant than creating new standalone properties or similar - unless there is a different method I’m not thinking of you would find superior?
As for features - it would be awesome to see these concepts fully fleshed out officially and methods fully exposed in the StorageManager class documentation…
Also, I think it would be useful to add another method such as ‘_syncItem()’ or similar that could be called upon after ‘_addItem()’ to exclusively add the single new item to the ‘self._items()’ dictionary and ‘self.storageDict’ object. As opposed to looping through every item and re-syncing as the ‘_sync()’ method is doing - which could potentially speed up the sync process.
Okay, to figure out the best method for what you’re doing, a couple questions on what your needs are…
do you really need these items in your .toe or are they recreated on load anyway? Most times people think they need storage, they actually just need dependable objects. This is better achieved with the createProperty function (derivative.ca/wiki099/index. … DFunctions)
do you need a separate storage item for each piece of data, or will a dependable list or dict work for you? You can create a deeply dependable list or dictionary by setting dependable to “deep” in createProperty or True in storage manager.
I often use ‘createProperty’ to create dependable objects when data does not need to be saved to the .toe file - such as previously saved preset data, locked configuration data, etc.
I see your point in saying that it’s not necessary to have separate storage items in the StorageManager and could simply initialize a dictionary(s) into the storedItems list on load and edit/add/remove items directly from there. I guess I like the idea of having a configurable amount of separate storage items ‘on-the-fly’ in the StorageManager - over a massive nested dictionary(s) or adding standalone (non-stored) properties. Specifically, I like the persistent storage (across initializations) and the ability to easily examine explicitly via node storage (middle-click node / examine DAT) my storage items in the network editor - which becomes cumbersome with massive nested dictionaries, etc. Likewise, when using standalone properties these do not show up in node storage and require an eval DAT to examine manually from the network editor.
I think using separate storage items allows some of the hidden data in the extension to be more easily exposed in the network editor - which can help in debugging purposes, etc. Also, using persistent storage is often quite nice to avoid re-initialization issues when saving extensions via an external editor - which often can be a headache.
That being said, if createProperty is vastly superior in speed to writing separate storage items that benefit may be moot as you are saying. I’d be interested to see these features exposed for (optional) use/exploration.
Ah okay. The main advantage of the separate items sound like visibility. I hear that. I can spend a little time looking at StorageManager features, but yours is the first request to do this kind of stuff so I suspect it won’t appear quickly. In the meantime, you could make a nice viewer for any dependable data by using an evaluateDAT.
FYI, StorageManager just manages a dependable dictionary, so there is very little functional difference between storage and a dependable dict made by createProperty. Just one less layer of abstraction, really. One big advantage of createProperty is that it won’t bloat your filesize or take time during load. Not a huge thing if your data is relatively small though.
