[Bug]: Can't Export NWB

[rcpeene]

What happened?

I am appending columns to the NWB electrodes table and then exporting it as a new file, described in more detail here
hdmf-dev/hdmf-zarr#179

I receive the error below when trying to export the NWB. Is this an hdmf problem? The same code works when using hdmf-zarr to export.

Steps to Reproduce

Use pynwb and run this code where locs, xs, ys, and zs are numpy arrays

    with io_class(str(scratch_nwb_path), mode='a') as read_io:
        nwb = read_io.read()

        print('Getting new electrode columns')
        locs, xs, ys, zs = get_new_electrode_colums(nwb, ccf_map)

        nwb.electrodes.location.data[:] = np.array(locs)
        nwb.electrodes.add_column('x', 'ccf x coordinate', data=xs)
        nwb.electrodes.add_column('y', 'ccf y coordinate', data=ys)
        nwb.electrodes.add_column('z', 'ccf z coordinate', data=zs)

        print('Exporting to NWB:',result_nwb_path)
        with io_class(str(result_nwb_path), "w") as export_io:
            export_io.export(src_io=read_io, nwbfile=nwb, write_args={'link_data': False})
        print(f"Done writing {result_nwb_path}")

Traceback

Traceback (most recent call last):
  File "/code/run_capsule.py", line 126, in <module>
    if __name__ == "__main__": run()
  File "/code/run_capsule.py", line 122, in run
    export_io.export(src_io=read_io, nwbfile=nwb, write_args={'link_data': False})
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/pynwb/__init__.py", line 410, in export
    super().export(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/hdf5/h5tools.py", line 458, in export
    super().export(**ckwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/io.py", line 163, in export
    bldr = src_io.manager.build(container, source=self.__source, root=True, export=True)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 730, in build
    self.__add_groups(builder, self.__spec.groups, container, manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1139, in __add_groups
    self.__add_groups(sub_builder, spec.groups, container, build_manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1152, in __add_groups
    self.__add_containers(builder, spec, attr_value, build_manager, source, container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1210, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1172, in __add_containers
    new_builder = build_manager.build(value, source=source, spec_ext=spec, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 729, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1122, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1172, in __add_containers
    new_builder = build_manager.build(value, source=source, spec_ext=spec, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 798, in build
    builder = DatasetBuilder(
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 667, in func_call
    pargs = _check_args(args, kwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 660, in _check_args
    raise ExceptionType(msg)
TypeError: DatasetBuilder.__init__: incorrect type for 'data' (got 'AstypeWrapper', expected 'ndarray, list, tuple, Dataset, Array, StrDataset, HDMFDataset, AbstractDataChunkIterator, str, int, float, bytes, bool, DataIO, DatasetBuilder, RegionBuilder, Iterable, datetime or date')

Operating System

Windows

Python Executable

Conda

Python Version

3.9

Package Versions

pynwb==2.8.2

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[stephprince]

Hi @rcpeene, I believe this would be an hdmf issue, but I wasn't able to reproduce the error with a small test example so I’m not positive what’s causing the error.

@mavaylon1 would you be able to take a look since you have been working on export?

[mavaylon1]

@stephprince happy to help
@rcpeene I will try the test code you gave to see if I can reproduce the error.

[mavaylon1]

@rcpeene Are you exporting across backends or within the same? It looks like you are using the same io_class throughout

[rcpeene]

The same backends; I am opening in h5py and exporting to h5py in the run that is failing. When I change io_class to zarr_hdmf, then the run passes and the NWB exports.

[rcpeene]

Any more updates here? Currently holding back our pipeline

[rly]

@rcpeene, could you share the file that is causing this error? We cannot seem to reproduce the error on our end.

Secondly, are the xs, ys, and zs arrays 1-D float numpy arrays?

Can you also tell us what version of pynwb and hdmf you are running? Thank you

[rcpeene]

• The file is an asset on code ocean. I could ask for permission to download it and share it. Would emailing it be an option? It will be rather large.
• That is correct, they are 1D numpy arrays of floats.
• Using hdmf-zarr 0.8.0 and pynwb 2.8.2

[stephprince]

If you could get permission to download and share that would be great! Email is an option or a google drive link would work as well.

[rcpeene]

@stephprince, @rly. I have shared with you the folder containing the nwb and a CCF file, and a link to the capsule repo :)

[stephprince]

@rcpeene thanks for sharing the file! I was able to reproduce your error. This issue was caused by a bug in HDMF during export of a specific case of a string dataset.

I am planning to get the related fix into the next hdmf release, which should occur sometime next week.

[rcpeene]

Great! Thanks for looking into it

[rcpeene]

Any updates here? Unfortunately our pipelines on Code Ocean have limited flexibility, so they mostly rely on the latest pip version. Is there an ETA for when hdmf and pynwb will have this fix released?

[stephprince]

@rcpeene thanks for the ping and apologies for the delay from the original estimate. The bug fix for this issue coincided with the upcoming hdmf 4.0 major release that required some coordination across hdmf-zarr and pynwb.

@rly is currently preparing the hdmf release that should be out by the end of the week with this fix!