Correct way to save, and load, data from file.

[jackbinysh]

Hi All,

First, this package is really great, thanks David and other developers for making it available!

I have a question about the right way to save and load data (having read #39 already).

I'm running a sim, saving data to disk, as so:

storage=FileStorage(filepath)
trackers = [storage.tracker(interval=pt)]
controller1 = Controller(ExplicitSolver(eq), t_range=T, tracker=trackers)

Having saved the simulation results to disk, I want to load them and analyse them at a later time (i.e. a fresh python instance).

I'm loading data from disk as so:

storage=FileStorage(filepath)

Having loaded storage in this way, If I try running through a generator to get (field,time) pairs:

mygenerator=storage.items()
next(mygenerator)

I get an error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/var/folders/_g/mfxtwks53bl1bwb0dp644mt40000gn/T/ipykernel_20728/1344562706.py in <module>
----> 1 next(storage.items())

/lib/python3.9/site-packages/pde/storage/base.py in items(self)
    231         """iterate over all times and stored fields, returning pairs"""
    232         for i in range(len(self)):
--> 233             yield self.times[i], self[i]  # type: ignore
    234 
    235     @fill_in_docstring

/lib/python3.9/site-packages/pde/storage/base.py in __getitem__(self, key)
    217         """return field at given index or a list of fields for a slice"""
    218         if isinstance(key, int):
--> 219             return self._get_field(key)
    220         elif isinstance(key, slice):
    221             return [self._get_field(i) for i in range(*key.indices(len(self)))]

/lib/python3.9/site-packages/pde/storage/base.py in _get_field(self, t_index)
    171             # we need to determine the field type
    172 
--> 173             if self.grid is None:
    174                 raise RuntimeError(
    175                     "Could not load grid from data. Please set the `_grid` attribute "

/lib/python3.9/site-packages/pde/storage/base.py in grid(self)
    143                 attrs_serialized = self.info["field_attributes"]
    144                 attrs = FieldBase.unserialize_attributes(attrs_serialized)
--> 145                 self._grid = attrs["grid"]
    146             else:
    147                 self._logger.warning("`grid` attribute was not stored")

KeyError: 'grid'

I guess this is related to #39 , and this grid attribute not being set. My question is: What is the right way of setting this grid key in the FileStorage instance, so I can safely yield (time, value) pairs? More generally, is this the right way of loading data having run a simulation? [I see that I can access the underlying FieldStorage.data, but this feels dangerous.]

Any help much appreciated, thanks again.

[david-zwicker]

This was indeed a bug, but should be fixed in the latest version of py-pde. Conda has not yet updated the package, but the master branch and pip should have it. After updating, you should be able to load the stored data.

[jackbinysh]

Thanks for the response! I'll try this