SlurmExecutor env_vars does not override ENV variables defined in the container image

[gwarmstrong]

If I have a docker file like:

FROM ubuntu:latest

ENV MY_VARIABLE="Hello, Docker!"

CMD ["bash", "-c", "echo $MY_VARIABLE"]

and I use a SlurmExecutor setting env_vars={"MY_VARIABLE": "Foo!"}, the variable MY_VARIABLE will not be overwritten and instead the result of executing the container would still be "Hello, Docker!".

In order to fix this, the SlurmExecutor needs to add a --container-env argument to the srun call that is a comma-separated list of variables to be exported from the environment, e.g.,

srun ... --container-env=MY_VARIABLE1,MY_VARIABLE2

It's possible this override behavior is not always wanted, so it might also be reasonable to introduce a flag to indicate whether variables should be overwritten or not (possibly on a per-variable basis).

[hemildesai]

Hi, yes we will add an option to configure --container-env in the executor. Thanks for reporting!