Support concatenating groups in query

[gaow]

Currently:

DSC:
  define:
     c: a * b

then c is not available for query. This is because there will be potential ambiguity in parameter conventions in the concatenated group c that we find conceptually dangerous to attempt to treat. But I think now there are appealing case from my own user case that needs this feature. In part it has something to do with better support for Shell, eg:

preprocess: R(write(...))
   $data: data

workhorse: Shell(exec ... )
   data: $data
   $res: res

postprocess: R(read(...))
   res: $res
   $res: res

DSC:
   define:
      analysis: preprocess * workhorse * postprocess

(As I've discussed before, a full support for mixing language is a separate project but I can get limited cases such as this to work within a few days)

So now it makes sense that we'll only suppose to query on analysis not any of its 3 components.

I propose we have something like this:

DSC:
   concate:
      analysis: preprocess * workhorse * postprocess

that is, one additional keyword eg concate that, when specified, will merge results from the 3 modules for query. Then for non-unique parameters only keep the one from the first module and results from the last module, as we've once discussed. This additional keyword will at least make it conceptually very clear to users. The benenifit is that under the hood the 3 modules are still executed as 3 separate modules, so changes of code in postprocess will not trigger rerun of the potentially time consuming workhorse.

[stephens999]

i would prefer to have query conceptually separated out from actual running and definition of dsc.... I agree we may want more flexible queries than we currently support. let's discuss in person though.

…

On Thu, May 3, 2018 at 7:57 AM, gaow ***@***.***> wrote: Currently: DSC: define: c: a * b then c is not available for query. This is because there will be potential ambiguity in parameter conventions in the concatenated group c that we find conceptually dangerous to attempt to treat. But I think now there are appealing case from my own user case that needs this feature. In part it has something to do with better support for Shell, eg: preprocess: R(write(...)) $data: data workhorse: Shell(exec ... ) data: $data $res: res postprocess: R(read(...)) res: $res $res: res DSC: define: analysis: preprocess * workhorse * postprocess (As I've discussed before, a full support for mixing language is a separate project but I can get limited cases such as this to work within a few days) So now it makes sense that we'll only suppose to query on analysis not any of its 3 components. I propose we have something like this: DSC: concate: analysis: preprocess * workhorse * postprocess that is, one additional keyword eg concate that, when specified, will merge results from the 3 modules for query. Then for non-unique parameters only keep the one from the first module and results from the last module, as we've once discussed. This additional keyword will at least make it conceptually very clear to users. The benenifit is that under the hood the 3 modules are still executed as 3 separate modules, so changes of code in postprocess will not trigger rerun of the potentially time consuming workhorse. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#134>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABt4xWVApbHHXBCBZ_lqIm_F33dzzk3Vks5tuv6ogaJpZM4TxCBL> .