create: cannot submit Snakemake workflows using wildcards as parameters

[tiborsimko]

Current behaviour

Consider the following workflow example:

• reana.yaml content:

inputs:
  files:
    - inputs.yaml
    - Snakefile
  parameters:
    input: inputs.yaml
workflow:
  type: snakemake
  file: Snakefile

• inputs.yaml content:

samples:
  WW: /WW_Tblahblah
  DY: /DYJetsToLL_blahbalh

• Snakefile content:

rule all:
    input:
        expand("output/dataset_{sample}.txt", sample=config["samples"].keys())

rule dataset:
    output:
        "output/dataset_{sample}.txt"
    container:
        "docker://docker.io/reanahub/reana-env-root6:6.18.04"
    params:
        dataset = lambda wc: config["samples"].get(wc.get("sample"), "/UNKNOWNN")
    resources:
        kubernetes_memory_limit="256Mi"
    shell:
        "mkdir -p $(dirname {output}) && echo {params.dataset} > {output}"

This Snakemake example is specific in that the analysis would like to use wildcards in rules's parameters, which is usually done by means of lambda functions working on the wildcard object.

The local execution works well:

$ snakemake -c1 --configfile inputs.yaml
$ head output/dataset_*
==> output/dataset_DY.txt <==
/DYJetsToLL_blahbalh

==> output/dataset_WW.txt <==
/WW_Tblahblah

The submission of the same workflow to REANA does not pass:

$ reana-client create -w test
==> ERROR: Cannot create workflow test:
Object of type function is not JSON serializable

Expected behaviour

It should be possible to create workflows that run well locally.

Notes

This problem may be best addressed as part of the "thin client" sprint when the client would send only files and the workflow creation will be fully done on the server side.

If it is possible to find a workaround in the client and server combination for the forthcoming 0.9.4 release, that would be even better.

[tiborsimko]

One possible workaround until we solve the problem is to "duplicate" the rules so that parameters don't have to be wildcards. (This could be acceptable in case the number of samples is small.) Here's the working example:

rule all:
    input:
        expand("output/dataset_{sample}.txt", sample=config["samples"].keys())

rule dataset_WW:
    output:
        "output/dataset_WW.txt"
    container:
        "docker://docker.io/reanahub/reana-env-root6:6.18.04"
    params:
        dataset = config["samples"].get("WW", "/UNKNOWNN")
    resources:
        kubernetes_memory_limit="256Mi"
    shell:
        "mkdir -p $(dirname {output}) && echo {params.dataset} > {output}"

rule dataset_DY:
    output:
        "output/dataset_DY.txt"
    container:
        "docker://docker.io/reanahub/reana-env-root6:6.18.04"
    params:
        dataset = config["samples"].get("DY", "/UNKNOWNN")
    resources:
        kubernetes_memory_limit="256Mi"
    shell:
        "mkdir -p $(dirname {output}) && echo {params.dataset} > {output}"