Support for MultScalaData (e.g. OME-Zarr)

[jluethi]

Hey @haesleinhuepf

Very useful collection of image processing functions in this plugin!

I've been trying to use it with OME-Zarr files (using the awesome napari-ome-zarr plugin to read them lazily into napari). This creates a MultiScaleData image object:

In [1]: viewer.layers[0].data
Out[1]: <MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>

When I try to run different functions of the napari-segment-blobs-and-things-with-membranes on this, I run into some issues:

1) Functions that don't support MultiScaleData

For example, running a gaussian blur returns an AttributeError: 'MultiScaleData' object has no attribute 'ndim'

Full stack trace

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/magicgui/widgets/_bases/value_widget.py:57, in ValueWidget._on_value_change(self=PushButton(value=False, annotation=None, name='call_button'), value=False)
     55 if value is self.null_value and not self._nullable:
     56     return
---> 57 self.changed.emit(value)
        value = False
        self.changed = <SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>
        self = PushButton(value=False, annotation=None, name='call_button')

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/psygnal/_signal.py:725, in psygnal._signal.SignalInstance.emit()

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/psygnal/_signal.py:767, in psygnal._signal.SignalInstance._run_emit_loop()

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/psygnal/_signal.py:768, in psygnal._signal.SignalInstance._run_emit_loop()

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/psygnal/_signal.py:788, in psygnal._signal.SignalInstance._run_emit_loop()

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:207, in FunctionGui.__init__.<locals>._disable_button_and_call()
    205 self._call_button.enabled = False
    206 try:
--> 207     self.__call__()
        self = <FunctionGui gaussian_blur(image: <function NewType.<locals>.new_type at 0x1528baf70> = <MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma: float = 2.0, *, viewer: napari.viewer.Viewer = Viewer(axes=Axes(visible=False, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 2159.5, 2559.5), zoom=0.154453432405846, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(0.0, -815.5067337861703, 4537.442632718694), scaled=True, size=40, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 1.0, 1.0), (-1.5, 4320.0, 1.0), (-1.5, 5120.0, 1.0)), current_step=(0, 2160, 2560), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[<Image layer 'DAPI' at 0x29fef9d30>, <Image layer 'nanog' at 0x29ff45100>, <Image layer 'Lamin B1' at 0x2a99c8220>, <Labels layer 'label_DAPI' at 0x29fef9f70>], scale_bar=ScaleBar(visible=False, colored=False, ticks=True, position=<Position.BOTTOM_RIGHT: 'bottom_right'>, font_size=10, unit=None), text_overlay=TextOverlay(visible=False, color=(0.5, 0.5, 0.5, 1.0), font_size=10, position=<TextOverlayPosition.TOP_LEFT: 'top_left'>, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x15519d970>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x15519d9d0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x15519da30>, transform=<napari.utils.transforms.transforms.Affine object at 0x15519da90>, allow_new_selection=True, selected_vertex=None)), help='enter paint or fill mode to edit labels', status='label_DAPI [   0 -816 4537]', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x28cce8430>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x28ccd38b0>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x152b61040>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x28ccd3700>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x28cd111f0>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x28cd114c0>})) -> <function NewType.<locals>.new_type at 0x1528baf70>>
    208 finally:
    209     self._call_button.enabled = True

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:318, in FunctionGui.__call__(self=<FunctionGui gaussian_blur(image: <function NewT...14c0>})) -> <function NewType.<locals>.new_type>>, update_widget=False, *args=(), **kwargs={})
    316 self._tqdm_depth = 0  # reset the tqdm stack count
    317 with _function_name_pointing_to_widget(self):
--> 318     value = self._function(*bound.args, **bound.kwargs)
        self = <FunctionGui gaussian_blur(image: <function NewType.<locals>.new_type at 0x1528baf70> = <MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma: float = 2.0, *, viewer: napari.viewer.Viewer = Viewer(axes=Axes(visible=False, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 2159.5, 2559.5), zoom=0.154453432405846, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(0.0, -815.5067337861703, 4537.442632718694), scaled=True, size=40, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 1.0, 1.0), (-1.5, 4320.0, 1.0), (-1.5, 5120.0, 1.0)), current_step=(0, 2160, 2560), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[<Image layer 'DAPI' at 0x29fef9d30>, <Image layer 'nanog' at 0x29ff45100>, <Image layer 'Lamin B1' at 0x2a99c8220>, <Labels layer 'label_DAPI' at 0x29fef9f70>], scale_bar=ScaleBar(visible=False, colored=False, ticks=True, position=<Position.BOTTOM_RIGHT: 'bottom_right'>, font_size=10, unit=None), text_overlay=TextOverlay(visible=False, color=(0.5, 0.5, 0.5, 1.0), font_size=10, position=<TextOverlayPosition.TOP_LEFT: 'top_left'>, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x15519d970>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x15519d9d0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x15519da30>, transform=<napari.utils.transforms.transforms.Affine object at 0x15519da90>, allow_new_selection=True, selected_vertex=None)), help='enter paint or fill mode to edit labels', status='label_DAPI [   0 -816 4537]', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x28cce8430>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x28ccd38b0>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x152b61040>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x28ccd3700>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x28cd111f0>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x28cd114c0>})) -> <function NewType.<locals>.new_type at 0x1528baf70>>
        bound = <BoundArguments (image=<MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma=2.0, viewer=Viewer(axes=Axes(visible=False, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 2159.5, 2559.5), zoom=0.154453432405846, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(0.0, -815.5067337861703, 4537.442632718694), scaled=True, size=40, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 1.0, 1.0), (-1.5, 4320.0, 1.0), (-1.5, 5120.0, 1.0)), current_step=(0, 2160, 2560), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[<Image layer 'DAPI' at 0x29fef9d30>, <Image layer 'nanog' at 0x29ff45100>, <Image layer 'Lamin B1' at 0x2a99c8220>, <Labels layer 'label_DAPI' at 0x29fef9f70>], scale_bar=ScaleBar(visible=False, colored=False, ticks=True, position=<Position.BOTTOM_RIGHT: 'bottom_right'>, font_size=10, unit=None), text_overlay=TextOverlay(visible=False, color=(0.5, 0.5, 0.5, 1.0), font_size=10, position=<TextOverlayPosition.TOP_LEFT: 'top_left'>, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x15519d970>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x15519d9d0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x15519da30>, transform=<napari.utils.transforms.transforms.Affine object at 0x15519da90>, allow_new_selection=True, selected_vertex=None)), help='enter paint or fill mode to edit labels', status='label_DAPI [   0 -816 4537]', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x28cce8430>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x28ccd38b0>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x152b61040>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x28ccd3700>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x28cd111f0>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x28cd114c0>}))>
        self._function = <function gaussian_blur at 0x28ff945e0>
    320 self._call_count += 1
    321 if self._result_widget is not None:

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/napari_time_slicer/__init__.py:61, in time_slicer.<locals>.worker_function(*args=[<MultiScaleData at 0x29ff03d00. 5 levels, 'uint1..., (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, 2.0], **kwargs={})
     58         _break_down_4d_to_2d_kwargs(bound.arguments, current_timepoint, viewer)
     60 # call the decorated function
---> 61 result = function(*bound.args, **bound.kwargs)
        bound = <BoundArguments (image=<MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma=2.0)>
        function = <function gaussian_blur at 0x28ff94550>
     62 return result

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/napari_segment_blobs_and_things_with_membranes/__init__.py:283, in gaussian_blur(image=<MultiScaleData at 0x29ff03d00. 5 levels, 'uint1..., (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma=2.0)
    277 @register_function(menu="Filtering / noise removal > Gaussian (scikit-image, nsbatwm)")
    278 @time_slicer
    279 def gaussian_blur(image:ImageData, sigma: float = 1) -> ImageData:
    280     """
    281     Applies a Gaussian blur to an image with a defined sigma. Useful for denoising.
    282     """
--> 283     return gaussian(image, sigma)
        image = <MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>
        sigma = 2.0

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/skimage/_shared/utils.py:348, in deprecate_multichannel_kwarg.__call__.<locals>.fixed_func(*args=(<MultiScaleData at 0x29ff03d00. 5 levels, 'uint1..., (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, 2.0), **kwargs={})
    345     kwargs['channel_axis'] = convert[kwargs.pop('multichannel')]
    347 # Call the function with the fixed arguments
--> 348 return func(*args, **kwargs)
        func = <function gaussian at 0x28f092b80>
        kwargs = {}
        args = (<MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, 2.0)

File ~/opt/miniconda3/envs/napari-assistant/lib/python3.9/site-packages/skimage/_shared/filters.py:116, in gaussian(image=<MultiScaleData at 0x29ff03d00. 5 levels, 'uint1..., (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>, sigma=2.0, output=None, mode='nearest', cval=0, multichannel=None, preserve_range=False, truncate=4.0, channel_axis=None)
     16 @utils.deprecate_multichannel_kwarg(multichannel_position=5)
     17 def gaussian(image, sigma=1, output=None, mode='nearest', cval=0,
     18              multichannel=None, preserve_range=False, truncate=4.0, *,
     19              channel_axis=None):
     20     """Multi-dimensional Gaussian filter.
     21 
     22     Parameters
   (...)
    114 
    115     """
--> 116     if image.ndim == 3 and image.shape[-1] == 3 and channel_axis is None:
        image = <MultiScaleData at 0x29ff03d00. 5 levels, 'uint16', shapes: ((1, 4320, 5120), (1, 2160, 2560), (1, 1080, 1280), (1, 540, 640), (1, 270, 320))>
        channel_axis is None = True
        channel_axis = None
    117         msg = ("Images with dimensions (M, N, 3) are interpreted as 2D+RGB "
    118                "by default. Use `multichannel=False` to interpret as "
    119                "3D image with last dimension of length 3.")
    120         warn(RuntimeWarning(msg))

AttributeError: 'MultiScaleData' object has no attribute 'ndim'

2) Functions that run, but ignore the scales

For example, applying a threshold_mean runs, but returns a tiny output image in the top left corner. It looks like it just ran on the top level, because when zooming in, the thresholding is a very coarse threshold of the full image.

Example images

Overview (with threshold result in top left corner)

Zoom in to top left corner:

Thus my broader question: Is support for MultiScaleData in scope for this plugin? May require extra work for different functions and, especially when looking at large example like whole plates, it wouldn't be feasible to always run at full resolution. So could add complexity as in "What resolution is run?" and "What region do I want to run?"
There are interesting approaches to this, e.g. in StarDist: https://github.com/stardist/stardist-napari. StarDist has the ability to run on current field of view only ("Predict on field of view (only for 2D models in 2D view)"). But even that is limited to 2D data processing only. So I'm not sure whether the complexity of handling what part of the image needs to be processed should be handled by the processing plugins (would be neat, but potentially also quite complex) or whether we should just create a plugin for loading regions of interest from an OME-Zarr into memory (as a numpy array, not a MultiScaleData pyramid). Curious about your opinion on the topic @haesleinhuepf