cube.collapsed fails with multi-dimensional string coordinates #3653

PAGWatson · 2020-02-10T12:57:46Z

Hi, I have a cube like that below, with with data from multi-ensemble member climate model runs covering different years. The 'Expt ID' coordinate contains the run ID corresponding to each ensemble member for each year. I get an error when I do cube.collapsed('year', iris.analysis.MEAN).

I did previously collapse the cube over a 'season' coordinate (since removed), where each season had three time values, so perhaps this issue only arises when an entire dimension is collapsed?

print cube
air_temperature / (K)               (time: 30; Ens member: 15; latitude: 145; longitude: 192)
     Dimension coordinates:
          time                           x              -             -               -
          Ens member                     -              x             -               -
          latitude                       -              -             x               -
          longitude                      -              -             -               x
     Auxiliary coordinates:
          season_year                    x              -             -               -
          year                           x              -             -               -
          Expt ID                        x              x             -               -

cube_mean=cube.collapsed('year',iris.analysis.MEAN)

The full error message is given below. It seems that the problem is that iris tries joining the strings in 'Expt ID' into one long string, and then finds that this does not have the same size as the 'Ens member' dimension.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-43764f0e4a7d> in <module>()
----> 1 cube_seasmean.collapsed('year',iris.analysis.MEAN)

/network/home/aopp/watson/anaconda2/envs/main/lib/python2.7/site-packages/iris/cube.pyc in collapsed(self, coords, aggregator, **kwargs)
   3253                 local_dims = [coord_dims.index(dim) for dim in
   3254                               dims_to_collapse if dim in coord_dims]
-> 3255                 collapsed_cube.replace_coord(coord.collapsed(local_dims))
   3256 
   3257         untouched_dims = sorted(untouched_dims)

/network/home/aopp/watson/anaconda2/envs/main/lib/python2.7/site-packages/iris/cube.pyc in replace_coord(self, new_coord)
   1181             self.add_dim_coord(new_coord, dims[0])
   1182         else:
-> 1183             self.add_aux_coord(new_coord, dims)
   1184 
   1185         for factory in self.aux_factories:

/network/home/aopp/watson/anaconda2/envs/main/lib/python2.7/site-packages/iris/cube.pyc in add_aux_coord(self, coord, data_dims)
    964         if self.coords(coord):  # TODO: just fail on duplicate object
    965             raise ValueError('Duplicate coordinates are not permitted.')
--> 966         self._add_unique_aux_coord(coord, data_dims)
    967 
    968     def _check_multi_dim_metadata(self, metadata, data_dims):

/network/home/aopp/watson/anaconda2/envs/main/lib/python2.7/site-packages/iris/cube.pyc in _add_unique_aux_coord(self, coord, data_dims)
    996 
    997     def _add_unique_aux_coord(self, coord, data_dims):
--> 998         data_dims = self._check_multi_dim_metadata(coord, data_dims)
    999         self._aux_coords_and_dims.append([coord, data_dims])
   1000 

/network/home/aopp/watson/anaconda2/envs/main/lib/python2.7/site-packages/iris/cube.pyc in _check_multi_dim_metadata(self, metadata, data_dims)
    988                     raise ValueError(msg.format(dim, self.shape[dim],
    989                                                 metadata.name(), i,
--> 990                                                 metadata.shape[i]))
    991         elif metadata.shape != (1,):
    992             msg = 'Missing data dimensions for multi-valued {} {!r}'

ValueError: Unequal lengths. Cube dimension 0 => 15; metadata 'Expt ID' dimension 0 => 1.

A quick way of coming up with a solution is just to explicitly make a string that is the join of the individual strings, removing the string coord and re-adding the joined string as a scalar coord (I'm not good enough with iris to know if this would be very robust, but it seems to work for my case).

for coord in cube.aux_coords:
    if coord.ndim>1 and coord.dtype.char=='S':
        new_str='|'.join(coord.points.ravel())
        new_coord=iris.coords.AuxCoord(new_str, attributes=coord.attributes, long_name=coord.long_name, standard_name=coord.standard_name, units=coord.units, var_name=coord.var_name)
        cube.remove_coord(coord)
        cube.add_aux_coord(new_coord)
        
cube_mean=cube_.collapsed('year', iris.analysis.MEAN)  #now this works

It would be nice if iris would do something like this in cube.collapsed(). Even better would be a method that only collapses 'Expt ID' here along the dimension being collapsed, so the association of 'Expt ID' values with the 'Ens member' dimension would be maintained.

The text was updated successfully, but these errors were encountered:

rcomer · 2020-02-10T14:08:17Z

Even better would be a method that only collapses 'Expt ID' here along the dimension being collapsed, so the association of 'Expt ID' values with the 'Ens member' dimension would be maintained.

The aggregated_by method has string handling that does that. So I would say it’s desirable to have consistent behaviour in collapsed. I also think it should be relatively simple to implement.

rcomer · 2020-02-10T14:33:35Z

The relevant handling for aggregated_by looks like this:

iris/lib/iris/analysis/__init__.py

Lines 2182 to 2217 in c9506e6

 if coord.points.dtype.kind in "SU": 

 if coord.bounds is None: 

 new_points = [] 

 new_bounds = None 

 # np.apply_along_axis does not work with str.join, so we 

 # need to loop through the array directly. First move axis 

 # of interest to trailing dim and flatten the others. 

 work_arr = np.moveaxis(coord.points, dim, -1) 

 shape = work_arr.shape 

 work_shape = (-1, shape[-1]) 

 new_shape = (len(self),) 

 if coord.ndim > 1: 

 new_shape += shape[:-1] 

 work_arr = work_arr.reshape(work_shape) 

 for key_slice in self._slices_by_key.values(): 

 if isinstance(key_slice, slice): 

 indices = key_slice.indices( 

 coord.points.shape[dim] 

 ) 

 key_slice = range(*indices) 

 for arr in work_arr: 

 new_points.append("|".join(arr.take(key_slice))) 

 # Reinstate flattened dimensions. Aggregated dim now leads. 

 new_points = np.array(new_points).reshape(new_shape) 

 # Move aggregated dimension back to position it started in. 

 new_points = np.moveaxis(new_points, 0, dim) 

 else: 

 msg = ( 

 "collapsing the bounded string coordinate {0!r}" 

 " is not supported".format(coord.name()) 

 ) 

 raise ValueError(msg)

github-actions · 2021-08-04T00:44:06Z

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

rcomer · 2021-08-04T07:01:03Z

I believe this bug is very fixable, it just needs someone to find the time. So I say we leave this issue open.

rcomer · 2022-10-19T14:13:04Z

I have proposed a fix for this at #4294.

github-actions · 2024-03-03T00:14:46Z

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

rcomer · 2024-03-06T13:54:53Z

I still think we should make this work but if we don't make it work we should at least raise a more decipherable error message.

rcomer added the Type: Bug label Feb 10, 2020

valeriupredoi mentioned this issue Mar 5, 2020

Implemented regridding of auxiliary coordinates ESMValGroup/ESMValCore#548

Closed

9 tasks

github-actions bot added the Stale A stale issue/pull-request label Aug 4, 2021

rcomer removed the Stale A stale issue/pull-request label Aug 4, 2021

rcomer self-assigned this Aug 19, 2021

rcomer linked a pull request Aug 24, 2021 that will close this issue

Partial collapse of multi-dimensional string coords #4294

Open

github-actions bot added the Stale A stale issue/pull-request label Mar 3, 2024

rcomer removed the Stale A stale issue/pull-request label Mar 6, 2024

rcomer linked a pull request May 15, 2024 that will close this issue

Partial collapse of multi-dim string coords: take 2 #5955

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cube.collapsed fails with multi-dimensional string coordinates #3653

cube.collapsed fails with multi-dimensional string coordinates #3653

PAGWatson commented Feb 10, 2020 •

edited

rcomer commented Feb 10, 2020

rcomer commented Feb 10, 2020

github-actions bot commented Aug 4, 2021

rcomer commented Aug 4, 2021

rcomer commented Oct 19, 2022

github-actions bot commented Mar 3, 2024

rcomer commented Mar 6, 2024

cube.collapsed fails with multi-dimensional string coordinates #3653

cube.collapsed fails with multi-dimensional string coordinates #3653

Comments

PAGWatson commented Feb 10, 2020 • edited

rcomer commented Feb 10, 2020

rcomer commented Feb 10, 2020

github-actions bot commented Aug 4, 2021

rcomer commented Aug 4, 2021

rcomer commented Oct 19, 2022

github-actions bot commented Mar 3, 2024

rcomer commented Mar 6, 2024

PAGWatson commented Feb 10, 2020 •

edited