Combining distributed files with differing variables

[ceblanton]

FRE's output.stager combines distributed model output from decomposed domains using mppnccombine and combine-ncc. Each distributed file contains all variables.

In 2019, it was discovered that regional output (ability to request a lat/lon subset of output from the diag manager, commonly used for ocean channels) sometimes resulted in distributed files with differing variables. This was seen a natural consequence of the variable not being available for output and not a bug with diag manager.

The problem is that mppnccombine (and combine-ncc) do not check for differing variables, and combine them anyway, resulting in incorrect output.

As a solution, FRE's output.stager was modified to check for consistent variables in the distributed files before combining; if variables differ, the files are left uncombined.

While this has worked adequately, if the domain settings result in a large number of distributed files, the variable consistency checking becomes pathological, resulting in output staging taking multiple times longer than the simulation itself.

Should the NC tools themselves (mppnccombine and combine-ncc) do the variable consistency checking instead? Some arguments for: probably, the compiled tools can do this check faster than the cshell output.stager; and other workflows (FRE's replacement, w-group, etc) could take advantage of the variable checking.

Arguments against:

• Tool users should ensure that correct input is used; if incorrect input is used, then incorrect output is saved.
• the model itself should ensure that distributed files have identical variables