--singularity-args quotes not propagated to cluster job script

[dkoppstein]

Snakemake version

8.11.3

Also using:

• snakemake-executor-plugin-lsf=0.2.4
• snakemake-executor-plugin-jobstep=0.1.10
• snakemake-interface-common=1.17.2
• snakemake-interface-executor-plugins=9.1.1

Describe the bug
When running snakemake with e.g. --use-singularity --profile lsf_plugin --singularity-args "-B /tmp,/home/david", the quotes around --singularity-args are interpreted correctly by the first call of snakemake but do not get passed on to the cluster job script. Therefore, the submitted job looks like --use-singularity --singularity-args -B /tmp,/home/david and gives __main__.py: error: argument --apptainer-args/--singularity-args: expected one argument.

Logs

Error in rule all:
    message: LSF-job '26674252' failed, LSF status is: 'EXIT'For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: test.txt
    output: test.out
    log: /omics/odcf/analysis/OE0470_projects/aberrant-gene-activation/um_dktk/analyses/baltica_pdx/sandbox/.snakemake/lsf_logs/rule_all/unique/ba3e82e2-100c-4498-8d86-95a036c1ac96.log (check log file(s) for error details)
    shell:
        ls test.txt; echo "test" > test.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    external_jobid: 26674252

Sender: LSF System <lsfadmin@odcf-cn31u25>
Subject: Job 26674252: <Snakemake_rule_all___(ba3e82e2-100c-4498-8d86-95a036c1ac96)> in cluster <ODCF> Exited

Job <Snakemake_rule_all___(ba3e82e2-100c-4498-8d86-95a036c1ac96)> was submitted from host <odcf-vm135> by user <d988n> in cluster <ODCF> at Mon May 13 17:02:26 2024
Job was executed on host(s) <odcf-cn31u25>, in queue <long>, as user <d988n> in cluster <ODCF> at Mon May 13 17:02:28 2024
</home/d988n> was used as the home directory.
</omics/odcf/analysis/OE0470_projects/aberrant-gene-activation/um_dktk/analyses/baltica_pdx/sandbox> was used as the working directory.
Started at Mon May 13 17:02:28 2024
Terminated at Mon May 13 17:02:48 2024
Results reported at Mon May 13 17:02:48 2024

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
/omics/groups/OE0650/internal/d988n/mambaforge/envs/snakemake/bin/python3.12 -m snakemake --snakefile /omics/odcf/analysis/OE0470_projects/aberrant-gene-activation/um_dktk/analyses/baltica_pdx/sandbox/Snakefile --target
-jobs 'all:' --allowed-rules 'all' --cores 7000 --attempt 1 --force-use-threads  --resources 'mem_mb=4000' 'mem_mib=3815' 'disk_mb=1000' 'disk_mib=954' --wait-for-files '/omics/odcf/analysis/OE0470_projects/aberrant-gene-activation/um_dktk/analyses/baltica_pdx/sandbox/.snakemake/tmp.lhuac55j' 'test.txt' --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --rerun-triggers code software-env input mtime params --deployment-method conda apptainer --conda-frontend mamba --conda-base-path /omics/groups/OE0650/internal/d988n/mambaforge --apptainer-prefix /omics/groups/OE0650/internal/d988n/.apptainer/cache --apptainer-args -B /tmp,/home/d988n --max-threads 64 --shared-fs-usage software-deployment input-output storage-local-copies persistence sources source-cache --wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ --latency-wait 60 --scheduler ilp --local-storage-prefix .snakemake/storage --scheduler-solver-path /omics/groups/OE0650/internal/d988n/mambaforge/envs/sn
akemake/bin --set-resources base64//a2FsbGlzdG9faW5kZXg6bWVtX21iPW1pbihtYXgoMjAgKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== base64//a2FsbGlzdG9fcXVhbnQ6bWVtX21iPW1pbihtYXgoMS41ICogaW5wdXQuc2l6ZV9tYiwgNDAwMCksIDIwMDAwMC
k= base64//a2FsbGlzdG9fM3ByaW1lX2luZGV4Om1lbV9tYj1taW4obWF4KDIwICogaW5wdXQuc2l6ZV9tYiwgNDAwMCksIDIwMDAwMCk= base64//bG9nY291bnRfbWF0cml4Om1lbV9tYj1taW4obWF4KDggKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== base64//YndhX2
1lbTJfaW5kZXg6bWVtX21iPW1pbihtYXgoMjUgKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== base64//Y2lyY2xlX21hcF9yZWFsaWduOm1lbV9tYj1taW4obWF4KDIuNSAqIGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//YndhX21lbTptZW1fbWI9NCAqIHJ
vdW5kKCBpbnB1dC5zaXplX21iIC8gMTAwMDAgKSAqIDQwMDA= base64//YndhX2luZGV4Om1lbV9tYj1taW4obWF4KDIuNSAqIGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//bWFwX3JlYWRzOm1lbV9tYj1taW4odGhyZWFkcyAqIDQwMDAsIDIwMDAwMCk= base64//YmF
tX2luZGV4Om1lbV9tYj1taW4obWF4KGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//Z2V0X2tub3duX3ZhcmlhbnRzOm1lbV9tYj04MDAw base64//cmVjYWxpYnJhdGVfYmFzZV9xdWFsaXRpZXM6bWVtX21iPW1pbih0aHJlYWRzICogNDAwMCwgMjAwMDAwKQ== base64/
/YnVpbGRfc2FtcGxlX3JlZ2lvbnM6bWVtX21iPTEwMDAw base64//bXVsdGlxYzptZW1fbWI9bWluKG1heCgxLjIgKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== base64//ZnJlZWJheWVzOm1lbV9tYj1taW4odGhyZWFkcyAqIDUwMDAsIDIwMDAwMCk= base64//YW5ub3R
hdGVfdmFyaWFudHM6bWVtX21iPW1pbihtYXgoMS4yICogaW5wdXQuc2l6ZV9tYiwgNDAwMCksIDIwMDAwMCk= base64//YW5ub3RhdGVfY2FuZGlkYXRlX3ZhcmlhbnRzOm1lbV9tYj1taW4obWF4KDEuMiAqIGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//ZGF0YXZ6cmRf
dmFyaWFudHNfY2FsbHM6bWVtX21iPW1pbihtYXgoMiAqIGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//bWFya19kdXBsaWNhdGVzOm1lbV9tYj1taW4obWF4KDIgKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== base64//c3BsaXRfY2FsbF90YWJsZXM6bWVtX
21iPW1pbihtYXgoMiAqIGlucHV0LnNpemVfbWIsIDQwMDApLCAyMDAwMDAp base64//cmVuZGVyX2RhdGF2enJkX2NvbmZpZzptZW1fbWI9bWluKG1heCgxLjIgKiBpbnB1dC5zaXplX21iLCA0MDAwKSwgMjAwMDAwKQ== --set-threads base64//YndhX21lbT0zMg== base64//ZnJ
lZWJheWVzPTMy --default-resources base64//bWVtX21iPTQwMDA= base64//ZGlza19tYj1tYXgoMippbnB1dC5zaXplX21iLCAxMDAwKQ== base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= base64//bHNmX3F1ZXVlPWxvbmc= base64//bHNmX3Byb2plY3Q9ZGVmYXVsdA==
base64//bHNmX2V4dHJhPSItZW52IGFsbCxIVFRQX1BST1hZPWh0dHA6Ly93d3ctaW50Mi5ka2Z6LWhlaWRlbGJlcmcuZGU6MzEyOC8saHR0cF9wcm94eT1odHRwOi8vd3d3LWludDIuZGtmei1oZWlkZWxiZXJnLmRlOjMxMjgvLEhUVFBTX1BST1hZPWh0dHA6Ly93d3ctaW50Mi5ka2Z6LWh
laWRlbGJlcmcuZGU6MzEyOC8saHR0cHNfcHJveHk9aHR0cDovL3d3dy1pbnQyLmRrZnotaGVpZGVsYmVyZy5kZTozMTI4LyxGVFBfUFJPWFk9aHR0cDovL3d3dy1pbnQyLmRrZnotaGVpZGVsYmVyZy5kZTozMTI4LyxmdHBfcHJveHk9aHR0cDovL3d3dy1pbnQyLmRrZnotaGVpZGVsYmVyZy
5kZTozMTI4LyI= --executor lsf-jobstep --jobs 1 --mode remote
------------------------------------------------------------

Exited with exit code 2.

Resource usage summary:

    CPU time :                                   0.67 sec.
    Max Memory :                                 31 MB
    Average Memory :                             19.40 MB
    Total Requested Memory :                     4000.00 MB
    Delta Memory :                               3969.00 MB
    Max Swap :                                   -
    Max Processes :                              3
    Max Threads :                                4
    Run time :                                   8 sec.
    Turnaround time :                            22 sec.

The output (if any) follows:

usage: __main__.py [-h] [--dry-run] [--profile PROFILE]
                   [--workflow-profile WORKFLOW_PROFILE] [--cache [RULE ...]]
                   [--snakefile FILE] [--cores N] [--jobs N] [--local-cores N]
                   [--resources NAME=INT [NAME=INT ...]]
                   [--set-threads RULE=THREADS [RULE=THREADS ...]]
                   [--max-threads MAX_THREADS]
                   [--set-resources RULE:RESOURCE=VALUE [RULE:RESOURCE=VALUE ...]]
                   [--set-scatter NAME=SCATTERITEMS [NAME=SCATTERITEMS ...]]
                   [--set-resource-scopes RESOURCE=[global|local]
                   [RESOURCE=[global|local] ...]]
                   [--default-resources [NAME=INT ...]]
                   [--preemptible-rules [PREEMPTIBLE_RULES ...]]
                   [--preemptible-retries PREEMPTIBLE_RETRIES]
                   [--configfile FILE [FILE ...]] [--config [KEY=VALUE ...]]
                   [--envvars VARNAME [VARNAME ...]] [--directory DIR]
                   [--touch] [--keep-going]
                   [--rerun-triggers {code,input,mtime,params,software-env} [{code,input,mtime,params,software-env} ...]]
                   [--force] [--executor {lsf,lsf-jobstep,local,dryrun,touch}]
                   [--forceall] [--forcerun [TARGET ...]]
                   [--prioritize TARGET [TARGET ...]]
                   [--batch RULE=BATCH/BATCHES] [--until TARGET [TARGET ...]]
                   [--omit-from TARGET [TARGET ...]] [--rerun-incomplete]
                   [--shadow-prefix DIR] [--scheduler [{ilp,greedy}]]
                   [--wms-monitor [WMS_MONITOR]]
                   [--wms-monitor-arg [NAME=VALUE ...]]
                   [--scheduler-ilp-solver {PULP_CBC_CMD,COIN_CMD}]
                   [--scheduler-solver-path SCHEDULER_SOLVER_PATH]
                   [--conda-base-path CONDA_BASE_PATH] [--no-subworkflows]
                   [--precommand PRECOMMAND] [--groups GROUPS [GROUPS ...]]
                   [--group-components GROUP_COMPONENTS [GROUP_COMPONENTS ...]]
                   [--report [FILE]] [--report-stylesheet CSSFILE]
                   [--reporter PLUGIN] [--draft-notebook TARGET]
                   [--edit-notebook TARGET] [--notebook-listen IP:PORT]
                   [--lint [{text,json}]] [--generate-unit-tests [TESTPATH]]
                   [--containerize] [--export-cwl FILE] [--list-rules]
                   [--list-target-rules] [--dag] [--rulegraph] [--filegraph]
                   [--d3dag] [--summary] [--detailed-summary] [--archive FILE]
                   [--cleanup-metadata FILE [FILE ...]] [--cleanup-shadow]
                   [--skip-script-cleanup] [--unlock]
                   [--list-changes {input,params,code}] [--list-input-changes]
                   [--list-params-changes] [--list-untracked]
                   [--delete-all-output | --delete-temp-output]
                   [--keep-incomplete] [--drop-metadata]
                   [--deploy-sources QUERY CHECKSUM] [--version]
                   [--printshellcmds] [--debug-dag] [--nocolor]
                   [--quiet [{all,progress,rules} ...]] [--print-compilation]
                   [--verbose] [--force-use-threads] [--allow-ambiguity]
                   [--nolock] [--ignore-incomplete]
                   [--max-inventory-time SECONDS] [--latency-wait SECONDS]
                   [--wait-for-files [FILE ...]] [--wait-for-files-file FILE]
                   [--queue-input-wait-time SECONDS] [--notemp] [--all-temp]
                   [--unneeded-temp-files FILE [FILE ...]]
                   [--keep-storage-local-copies]
                   [--target-files-omit-workdir-adjustment]
                   [--allowed-rules ALLOWED_RULES [ALLOWED_RULES ...]]
                   [--target-jobs TARGET_JOBS [TARGET_JOBS ...]]
                   [--local-groupid LOCAL_GROUPID]
                   [--max-jobs-per-second MAX_JOBS_PER_SECOND]
                   [--max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND]
                   [--seconds-between-status-checks SECONDS_BETWEEN_STATUS_CHECKS]
                   [--retries RETRIES] [--attempt ATTEMPT]
                   [--wrapper-prefix WRAPPER_PREFIX]
                   [--default-storage-provider DEFAULT_STORAGE_PROVIDER]
                   [--default-storage-prefix DEFAULT_STORAGE_PREFIX]
                   [--local-storage-prefix LOCAL_STORAGE_PREFIX]
                   [--remote-job-local-storage-prefix REMOTE_JOB_LOCAL_STORAGE_PREFIX]
                   [--shared-fs-usage {input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} [{input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} ...]]
                   [--scheduler-greediness SCHEDULER_GREEDINESS] [--no-hooks]
                   [--debug] [--runtime-profile FILE]
                   [--mode {subprocess,default,remote}] [--show-failed-logs]
                   [--log-handler-script FILE]
                   [--log-service {none,slack,wms}] [--job-deploy-sources]
                   [--benchmark-extended] [--container-image IMAGE]
                   [--immediate-submit] [--jobscript SCRIPT] [--jobname NAME]
                   [--flux]
                   [--software-deployment-method {apptainer,conda,env-modules} [{apptainer,conda,env-modules} ...]]
                   [--container-cleanup-images] [--use-conda]
                   [--conda-not-block-search-path-envvars] [--list-conda-envs]
                   [--conda-prefix DIR] [--conda-cleanup-envs]
                   [--conda-cleanup-pkgs [{tarballs,cache}]]
                   [--conda-create-envs-only] [--conda-frontend {conda,mamba}]
                   [--use-apptainer] [--apptainer-prefix DIR]
                   [--apptainer-args ARGS] [--use-envmodules]
                   [--report-html-path VALUE]
                   [--report-html-stylesheet-path VALUE]
                   [targets ...]
__main__.py: error: argument --apptainer-args/--singularity-args: expected one argument

Minimal example

Snakefile:

rule all:
    input:
        "test.txt",
    output:
        "test.out",
    singularity:
        "docker://bash"
    shell:
        'ls {input}; echo "test" > {output}'

test.sh:

snakemake -j1 --use-singularity --profile lsf_plugin --singularity-args "-B /tmp,/home/$USER"

Additional context
Setup the lsf plugin with the following config.yaml:

### GENERAL SNAKEMAKE SETTINGS

# Tell snakemake which remote executor plugin to use, in our case the plugin
# for the cluster system LSF, which is documented here:
# https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/lsf.html
executor: "lsf"

cores: 7000

jobs: 300

latency-wait: 60

local-cores: 6

max-jobs-per-second: 10

max-threads: 64

seconds-between-status-checks: 20

software-deployment-method: "conda"

default-resources:
  mem_mb: 4000
  lsf_queue: "long"
  lsf_project: "default" 
  lsf_extra: '"-env all"'

[dkoppstein]

Likely a duplicate of #2808.