Remote Taskfiles experiment

[pd93]

Warning

All experimental features are subject to breaking changes and/or removal at any time. We strongly recommend that you do not use these features in a production environment. They are intended for testing and feedback only.

Note

You can view the Remote Taskfiles experiment documentation on our website, including instructions on how to enable/disable it.

Context

This experiment attempts to solve the problems originally described by #770.

Currently, Task only supports running Taskfiles located on the local filesystem. This makes it difficult to reuse Taskfiles across various repositories and systems without making copies of a file. It would be useful to have a way to run a Taskfile located on a remote system or run a local Taskfile that includes one on a remote system.

Proposal

This experiment would allow users to run a Taskfiles located elsewhere, such as via HTTP or in a remote Git repository. These remote Taskfiles can either be run directly from the CLI or included by another Taskfile.

We're looking to gather feedback on the current draft, so please feel free to leave your thoughts/comments here.

Decision log / TODOs:

• Refactor nodes as an interface feat: node refactor #1316
• HTTP nodes feat: remote taskfiles (HTTP) #1152
• Decide between go-getter or a native implementation
  • No go-getter for now. We can re-evaluate later
• How to handle security (remote files that change etc)
  • See https://taskfile.dev/experiments/remote-taskfiles/#security
• Caching (--download/--offline flags or similar)
  • See https://taskfile.dev/experiments/remote-taskfiles/#caching--running-offline
• Implement --yes flag on prompts feat: unify prompts #1344
• Decide if we should use remote/local files first if both are available feat: prefer remote taskfiles over cached ones #1345
• Consider storing checksums/cached files with a more user-friendly filename feat(remote): prefix checksums/cached files with the filename #1636
• Command to clear the remote cache feat(remote): add a command to clear the cache #1639
• Check that local -> remote -> remote:local includes work correctly feat: enable root remote taskfiles #1347
• Auto update TTL for cached files Feature Request: Have an auto update ttl for downloaded taskfiles #1402
• Add remote flags as config in Taskfile Feature request: Allow taskfile schema to use options related to remote taskfiles #1403
• Decide how special variables should look for remote files
• Running root remote Taskfiles feat: enable root remote taskfiles #1347
  • e.g. task test --taskfile https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml
• Git Nodes
  • e.g. [email protected]/foo/bar?ref=main//Taskfile.yml feat(remote): support include git remote #1652

[task-bot]

This issue has been marked as an experiment proposal! 🧪 It will now enter a period of consultation during which we encourage the community to provide feedback on the proposed design. Please see the experiment workflow documentation for more information on how we release experiments.

[task-bot]

This experiment has been marked as a draft! ✨ This means that an initial implementation has been added to the latest release of Task! You can find information about this experiment and how to enable it in our experiments documentation. Please see the experiment workflow documentation for more information on how we release experiments.

[wburningham]

If one wants to use remote tasks in CI how does one bypass the prompt when running a task for the first time. Is there a flag or environment variable that can be set to “force yes” to the prompt on whether or not to include the remote task file?

BTW thanks for a great experiment!

[wburningham]

The documentation mentions the -y flag for tasks that require prompts, but the experimental documentation doesn't mention if that also applies to remote taskfile includes.

[pd93]

@wburningham Thanks for pointing this out. The --yes flag has not currently been set up to automatically accept the remote task prompts. This is definitely something we can consider in the next iteration. I think this is an acceptable change, but if anyone has any security concerns about being able to skip the remote taskfile warning prompts, please add a comment.

[oliver-helix]

For Running root remote Taskfiles, consider specifying the remote taskfile via the --taskfile arg?
task hello --taskfile https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml

[oliver-helix]

Any idea why the following doesn't work?

# taskfile.yml
version: '3'

include:
  my-remote-namespace: https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml

tasks:
  test:
    cmds:
      - echo test succesful

$ task --version
Task version: 3.30.1
$ task test
task: [test] echo test succesful
test succesful
$ export TASK_X_REMOTE_TASKFILES=1
$ task my-remote-namespace:packages
task: No tasks with description available. Try --list-all to list all tasks
task: Task "my-remote-namespace:packages" does not exist

[pd93]

@oliver-helix Good idea about the root remote taskfiles. As for why your example doesn't work. I think its just a simple typo:

  # taskfile.yml
  version: '3'

- include:
+ includes:
    my-remote-namespace: https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml

  tasks:
    test:
      cmds:
        - echo test succesful

Edit: We should probably give a better error message for this...

[oliver-helix]

Doh! My bad. Thanks for the help! Consider updating the experiment docs to fix this typo.

[pd93]

Ah whoops. Well spotted. I've updated the docs 👍

[plcarman]

Hi, just wanted to say this experiment is exciting!

Do you have any plans to add support for remote endpoints that require authentication? My use case is that I have a remote Taskfile stored in GitLab. I would think an easy-ish way would be to honor my .netrc file like git and other tools do.

edit: I caught up on #1152 and I see there has already been talk about authentication. looking forward to whatever comes! Thanks.

[wburningham]

The --yes flag has not currently been set up to automatically accept the remote task prompts. This is definitely something we can consider in the next iteration. I think this is an acceptable change, but if anyone has any security concerns about being able to skip the remote taskfile warning prompts, please add a comment.

@pd93 my company is ready to try out taskfiles because the remote taskfile experiment moved to draft. Being able to run on CI with a --yes flag is our next implementation step.

When you say "the next iteration" are you wanting (or willing to accept) a PR?

The Experiments Documentation didn't specify any general timelines for accepting feedback/comments. Are you planning on waiting a minimum amount of time before releasing changes to this experiment?

[pd93]

@wburningham really nice to hear that this feature is making the difference for you. Are there any other blockers or features you'd need before being able to use this fully? (besides the obvious Git/SSH integration).

When you say "the next iteration" are you wanting (or willing to accept) a PR?

Absolutely! Contributions are always welcome, though we encourage people to discuss them first to make sure multiple people aren't doing the same thing and/or wasting their time. In this particular case, I actually already have a branch which will add this functionality and I intend to post a PR soon. As a potential future user of that functionality, your feedback on this would be very welcome.

If there are other extensions/improvements you'd like to help contribute to, I'm always happy to have some help.

The Experiments Documentation didn't specify any general timelines for accepting feedback/comments. Are you planning on waiting a minimum amount of time before releasing changes to this experiment?

There is no minimum/fixed time period for changes. @andreynering and I both work on this in our spare time, so planning is hard and changes tend to happen in chunks when we find some bandwidth. A new draft of the experiment will be released alongside the next version of Task which tends to happen around once a month, but this is also subject to our availability and having a set of meaningful changes to release.

I'm hoping to address the remaining TODOs in this issue (not the future extensions) in the next week or so and hopefully we'll get some good feedback from that which will allow us to move forward.

---

Edit - See PR:

• feat: unify prompts #1344

[pd93]

I've also just created #1345 which changes the default behaviour of Remote Taskfiles to prefer remote files over cached ones as discussed in previous PRs.

I've added a couple of thoughts to the end of the PR description. I would really appreciate feedback on this from @andreynering @ryancurrah @caphrim007 and anyone else with an opinion on the matter 🙏

[oliver-helix]

Any roadmap (ETA) for when remote taskfiles with Git ssh for private repository acecss will be added? Thanks so much for your hard work on this awesome project!

[ThomasSanson]

Greetings,

First and foremost, I would like to extend my heartfelt congratulations and gratitude for the exemplary work you have undertaken.

I am curious to know whether you have plans to support 'remote' Taskfiles that incorporate other Taskfiles. For instance, in this link: GitLab - Devcontainer Taskfiles, if a directory is segmented into multiple Taskfiles, will remotely including the principal Taskfile also entail the inclusion of the associated Taskfiles?

I look forward to your insightful response and thank you in advance for your consideration.

[lorenzofaresin]

Is it possible to add completion capabilities on remote tasks?

[vmaerten]

Hi!
Thanks for your work, it's a really really appreciated feature!
We plan to use it, to mutualize all our project's Takefile
Amazing work 🔥

[yordis]

I wonder if adding something like a shasum check would be a good idea here

[titpetric]

I love the experiment, the only thing I'm missing is an allow-list style configuration of what urls the experiment is allowed to fetch from at this moment. This would allow us to adopt it in CI and just allowlist the github public urls for the trusted org/repo/branch.

TASKLIST_X_REMOTE_TASKFILES_ALLOW=https://raw.githubusercontent.com/<Owner>/<Repo>/main/[,csv?]

[pd93]

Thanks all for the kind words :) Catching up on some comments:

Firstly, I don't have any ETAs for this experiment and I don't plan to give any. The last month or so has been pretty quiet in terms of progress and this is simply a reflection of other life priorities. The time I can give to the project varies a lot and some weeks you will see lots of progress and other weeks you will see none. For this reason, it would be unfair to make any promises that I can't keep. Rest assured that this is always on my mind and I fully intend to see it through. I won't be responding to any further comments on timelines/ETAs.

With that said, I'm very grateful for all the excitement and support and I really appreciate all the feedback and ideas. Please keep it coming ❤️

I am curious to know whether you have plans to support 'remote' Taskfiles that incorporate other Taskfiles. For instance, in this link: GitLab - Devcontainer Taskfiles, if a directory is segmented into multiple Taskfiles, will remotely including the principal Taskfile also entail the inclusion of the associated Taskfiles?

@ThomasSanson This is something I definitely want to support at some point and is on my TODO list. However, I haven't put much thought into how the implementation will work yet. I don't think we can ship the experiment without this as it would result in some very unexpected behaviour. Stay tuned.

Is it possible to add completion capabilities on remote tasks?

@lorenzofaresin It's definitely possible, but this is probably low priority and something that can be added once the experiment is stable. This is also heavily linked to #293 and #1157. Completions need some love and attention in general, not just for remote Taskfiles.

I wonder if adding something like a shasum check would be a good idea here

@yordis Can you please open a ticket for this so that we can discuss it in its own thread? It would be useful if you can include your use case for this. I can see a few, but it's nice to hear things from other people's perspectives.

an allow-list style configuration of what urls the experiment is allowed to fetch from at this moment

@titpetric Same as above. Could you please open an issue for this and explain your use case in as much detail as possible. I will take a look there.

[BnGx]

It would also be very useful to have the ability to use remote files using the -t option not worry about having the file available locally and having a centralized task repository. For example, typing the command

task -t https://www.example.com/mytasks.yaml task1

[onedr0p]

Another feature that I didn't see brought up would be to allow the remote ref to include a git tag or ref so they can be versioned instead of always pulling the "latest".

Not sure the best way to implement this since taskfile would need to know is a git repo and not a static file being served on nginx or whatever 😄

https://github.com/go-task/task/main/Taskfile.yml?ref=v1.0.0
https://github.com/go-task/task/main/Taskfile.yml?ref=testing

This would be great to distribute taskfiles to a larger audience while not introducing breaking or unexpected changes.

Edit: This already looks to be on the roadmap, sorry for the comment!

[blackjid]

that would be awesome...

I think that is refered in the "Future Extensions" section of the main body of this issue...

[pd93]

It would also be very useful to have the ability to use remote files using the -t option

@BnGx as @blackjid said, this is what I'm referring to as "root remote Taskfiles" and there is a draft PR for it #1347. However, there are a couple of issue with it that need resolving (hence it being a draft). This is quite near the top of my list.

Another feature that I didn't see brought up would be to allow the remote ref to include a git tag or ref so they can be versioned instead of always pulling the "latest".

@onedr0p This can already be done with refs to commits/branches/tags via GitHub URIs. For example:

• https://raw.githubusercontent.com/go-task/task/c4dce8f506814d7af750792f53d4fa0c863b4470/Taskfile.yml
• https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml
• https://raw.githubusercontent.com/go-task/task/v3.32.0/Taskfile.yml

Ideally, I would like the HTTP implementation to remain as simple as possible as ?ref could conflict with a real param on a real website.

We do plan to add a Git implementation later. However, this is only really useful if your Git repo is behind some kind of auth. If you are able to access it via HTTP(S), then this is not necessary.

[blackjid]

Would the use of go-getter let you have git, plus http with reference query string, and some other source go-getter supports.
Why did you decided against using it? I'm just curious..

[fredrikaverpil]

Any roadmap (ETA) for when remote taskfiles with Git ssh for private repository acecss will be added? Thanks so much for your hard work on this awesome project!

Ugly workaround:

includes:
  private: https://git:[email protected]/user/repo/main/Taskfile.yml

Follow the discussion around this here: #1420 (reply in thread)

[luis-garza]

There is still no plan for the git ssh repository access?
It's a feature that prevent us to use this nice experiment... 😢

[vmaerten]

I understand that implement SSH can be time consuming and will be done in the long run but is there any plan to support env variable in Taskfile's URL ?
(Like proposed here : #1420 (reply in thread))

I prevent us (and maybe others) to use it because we have a private org

[jylenhofdecat]

Any news on the possibility to use variable (tipycally for token) in github URL ?

[prnk28]

What if we adopted a system similar to homebrew tap's where an org would have a taskfiles repository? I feel like we could simplify the import scheme.

Any roadmap (ETA) for when remote taskfiles with Git ssh for private repository acecss will be added? Thanks so much for your hard work on this awesome project!

Ugly workaround:

includes:
  private: https://git:[email protected]/user/repo/main/Taskfile.yml

Follow the discussion around this here: #1420 (reply in thread)

[titpetric]

The workaround could use sh and invoke gh auth token in a subshell $(gh auth token).

…

On Mon, Feb 5, 2024, 11:24 Prad Nukala ***@***.***> wrote: What if we adopted a system similar to homebrew tap's where an org would have a taskfiles repository? I feel like we could simplify the import scheme. Any roadmap (ETA) for when remote taskfiles with Git ssh for private repository acecss will be added? Thanks so much for your hard work on this awesome project! Ugly workaround: includes: private: ***@***.***/user/repo/main/Taskfile.yml Follow the discussion around this here: #1420 (reply in thread) <#1420 (reply in thread)> — Reply to this email directly, view it on GitHub <#1317 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABY7EAJIZ37NNPJGAU6NNDYSCXO7AVCNFSM6AAAAAA4CZOYE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRWGY2TQMJVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vmaerten]

It seems local -> remote -> remote:local includes does not work correctly
I created another issue to track it down :

• Include a Taskfile in a remote Taskfile does not work #1531

[pd93]

@vmaerten #1347 is still WIP, but will include a fix for resolving includes in remote files correctly.

[alexjball]

go-getter integration would be make go-task so much more useful for sharing tooling across our org. I think following the terraform cli's support for module sources would provide a validated approach to caching, auth, and featureset.

go-getter supports a double slash syntax to specify a subpath of folders/archives/repos. Supporting this would be amazing to give scripts access to supporting files. Developers just need to set up native credentials for the source, and then could pull in all the team's tooling using a remote include and vars. And any repo-specific tooling would remain in there

For caching, terraform always uses the local module path and exits with an error if it's missing. You run terraform init to download remote sources once, and whenever versions change. These sources are stored under the ignored tool folder, .task in this case. This makes it straightforward to debug since you can view the code locally.

To match the current behavior, go-task would download source by default, unless you used --download and --offline to try only the local version.

[titpetric]

+1 for https://github.com/hashicorp/go-getter as the integration, seems like a good package, given a cursory review

…

On Wed, Mar 20, 2024, 05:29 Alex Ball ***@***.***> wrote: go-getter integration would be make go-task so much more useful for sharing tooling across our org. I think following the terraform cli's support for module sources <https://developer.hashicorp.com/terraform/language/modules/sources#generic-git-repository> would provide a validated approach to caching, auth, and featureset. go-getter supports a double slash syntax to specify a subpath of folders/archives/repos. Supporting this would be amazing to give scripts access to supporting files. Developers just need to set up native credentials for the source, and then could pull in all the team's tooling using a remote include and vars. And any repo-specific tooling would remain in there For caching, terraform always uses the local module path and exits with an error if it's missing. You run terraform init to download remote sources once, and whenever versions change. These sources are stored under the ignored tool folder, .task in this case. This makes it straightforward to debug since you can view the code locally. To match the current behavior, go-task would download source by default, unless you used --download and --offline to try only the local version. — Reply to this email directly, view it on GitHub <#1317 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABY7EHWLPLEP5YZ7ARGQZ3YZEGD7AVCNFSM6AAAAAA4CZOYE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGY2DKMJUGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vmaerten]

I've started working on adding support of Git in remote taskfile.

If you want to try it out, you need to clone my repo https://github.com/vmaerten/task, switch to remote-git branch, and run go run ./cmd/task -d <your_dir>

We will support SSH url, SCP url and HTTPS url.
You can find the docs here : https://github.com/vmaerten/task/blob/4f85e85167b375100e3da3803973ff203045935e/website/docs/experiments/remote_taskfiles.mdx?plain=1#L65

I'll add more examples in the next couple of days.

[blackjid]

That is awesome @vmaerten thank for the work...

Why did you choose to set the filename and the git reference in the same querystring in the url? Why not just having a different querystring for the filename? https://github.com/foo/bar.git?ref=main&filename=directory/Taskfile.yml

Or if you don't want the extra query string, a more readable form I think could be to add the path to the file just after the .git https://github.com/foo/bar.git/directory/Taskfile.yml?ref=main maybe with a double slash // like terraform and others does.. https://github.com/foo/bar.git//directory/Taskfile.yml?ref=main

Sorry if this is not the place or time to give feedback. Maybe discord?

[vmaerten]

Hello @blackjid
Thanks a lot for your feedback. I've forgot to put the PR's URL here. I'll copy / past your comment there and we can discuss on it the PR