Consider better packaging of h5 files from exported ROIs

[emlynjdavies]

Is your feature request related to a problem? Please describe.
exported ROIs are put into a folder that becomes eventually a very long list of h5 files

Describe the solution you'd like
package this into a single file that is easier to transport between discs

Describe alternatives you've considered
maybe some tools like xarray.open_mfdataset could be helpful here?

[emlynjdavies]

is this related to #207 @nepstad ?

[nepstad]

is this related to #207 @nepstad ?

Not directly, no. #207 concerns the stats netcdf file.

[nepstad]

We could supply a "merge ROI files" option to the PyOPIA command line interface. Since the ROIs have different shapes, xarray/netcdf is not ideal, but we could simply drop them into one big hdf5 file.

[nepstad]

Something like this could be a solution, producing one merged hdf5 file with one group for each processed image, containing all the ROIs as datasets under those groups:

h5files = sorted(glob('...'))
with h5py.File('silcam_rois_combined.h5', 'w') as f:
    for h5file in h5files:
        with h5py.File(h5file, 'r') as file_in:
            g = f.create_group(os.path.basename(h5file).replace('.h5', ''))
            for name in file_in:
                if 'PN' in name:
                    g.create_dataset(name, data=file_in['/'+name][:])

[nepstad]

Another option is to look into a more cloud-friendly storage solution, such as zarr: https://zarr.dev/

[emlynjdavies]

If we switched to zarr, we could put the ROIs as a subgroup within the main STATS output - then we could have all output files in the same place.

See zarr groups here