[bug] Subshells inside command arguments act in an unpredictable manner

[kikones34]

Describe the bug

When including subshells in command arguments, sometimes they work, and sometimes they don't. The number of variables that affect this outcome is too large to test every possible combination. I have found the following variables:

• Whether the Python toolchain is enabled or not
• Which project is the task defined in (e.g. in the monorepo root, in a Python project, in a non-python project)
• Whether the subshell argument is part of the original task's command, or it is added by a child task
  • In which file the parent task is defined (e.g. python.yml vs tasks.yml and global vs local)
  • In which project the child task is defined
• What command you put inside the subshell

Steps to reproduce

Choose any configuration of all of the variables mentioned above. For some configurations, the command will fail, and for others it will work.

Example 1 (does not work)

Python toolchain enabled, global task defined in python.yml, child task defined in a Python project, argument added by child task and not present in original task.

python.yml

  build:
    command: docker build --target prod -t $project -f Dockerfile $workspaceRoot

moon.yml

  build:
    args:
      - '--build-arg COMMIT_HASH=$(git rev-parse HEAD)'
    options:
      mergeArgs: append

Result:

unknown flag: --build-arg COMMIT_HASH
See 'docker buildx build --help'.

Example 2 (works)

Same as Example 1, but with the python toolchain disabled.

Result:

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 5.31kB done
#1 DONE 0.0s
...

Example 3 (works)

Python toolchain enabled, parent task defined in a Python project, child task defined in a Python project, argument added by child task and not present in parent task.

moon.yml

  build-base:
    command: docker build --target prod -t $project -f Dockerfile $workspaceRoot

  build:
    extends: build-base
    args:
      - '--build-arg COMMIT_HASH=$(git rev-parse HEAD)'
    options:
      mergeArgs: append

Result:

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 5.31kB done
#1 DONE 0.0s
...

Example 4 (does not work)

Same as Example 3, but there is no inheritance, the task already contains the argument.

moon.yml

build:
  command: docker build --target prod -t trio-cloud-search -f Dockerfile $workspaceRoot --build-arg COMMIT_HASH=$(git rev-parse HEAD)

Result:

ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

Expected behavior

The command works in all cases.

Additional context

Moon version: 1.30.0 (linux musl)

I'm also getting these logs when running any command whenever the Python toolchain is enabled:

libunwind: __unw_add_dynamic_fde: bad fde: FDE is really a CIE
libunwind: __unw_add_dynamic_fde: bad fde: FDE is really a CIE

[milesj]

@kikones34 Subshells are not supported in command, try script instead.

[kikones34]

@milesj I cannot append args to the command using a script, however :(
I thought subshells were supported since it did work all this time until I tried enabling the Python toolchain. You mean that it's just by chance that it sometimes works, then?

[milesj]

@kikones34 It's just by chance that it worked. command is constantly parsed from a string into an array, then formatted into a string, and parsed again. Through all of this, values are quoted and unquoted, so there's no guarantee that complex CLI arguments will be parsed correctly. The parser isn't a Bash compatible parser, just an args parser.

I thought subshells were supported since it did work all this time until I tried enabling the Python toolchain.

Python toolchain has nothing to do with this. None of the args/command/process code changed as part of the Python PR.

[kikones34]

Python toolchain has nothing to do with this. None of the args/command/process code changed as part of the Python PR.

Then it probably is just enabling any toolchain, it just so happened that I enabled the Python one. When I mentioned "no Python toolchain" in the examples I really meant "no toolchain of any kind".

Either way, if it's unsupported I'll just find another way around it.