New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: exclude .git dir in ci build #7234
Conversation
With this change and with #7231, we've massively cut down the size of the context transferred here:
Without uploading these expensive directories, we now only need to transfer 56MB. |
e27ae07
to
0e1f4b8
Compare
Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs. However! A problem with this was that we needed to upload *the entire* git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB. Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such. Signed-off-by: Justin Chadwell <[email protected]>
The only thing this was being used for here was the connect job - we can just use `--help` to get this info to sanity check it. Signed-off-by: Justin Chadwell <[email protected]>
Signed-off-by: Justin Chadwell <[email protected]>
0e1f4b8
to
58c363a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Yeah I had wondered when doing this if we could have added a new API for getting a content hash, but using the hash already in the ID is even easier 🎉
* ci: exclude .git dir in ci build Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs. However! A problem with this was that we needed to upload *the entire* git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB. Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such. Signed-off-by: Justin Chadwell <[email protected]> * ci: remove dependency on go sdk The only thing this was being used for here was the connect job - we can just use `--help` to get this info to sanity check it. Signed-off-by: Justin Chadwell <[email protected]> * ci: run linting over ci itself Signed-off-by: Justin Chadwell <[email protected]> --------- Signed-off-by: Justin Chadwell <[email protected]> Signed-off-by: kpenfound <[email protected]>
* ci: exclude .git dir in ci build Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs. However! A problem with this was that we needed to upload *the entire* git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB. Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such. Signed-off-by: Justin Chadwell <[email protected]> * ci: remove dependency on go sdk The only thing this was being used for here was the connect job - we can just use `--help` to get this info to sanity check it. Signed-off-by: Justin Chadwell <[email protected]> * ci: run linting over ci itself Signed-off-by: Justin Chadwell <[email protected]> --------- Signed-off-by: Justin Chadwell <[email protected]>
* ci: exclude .git dir in ci build Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs. However! A problem with this was that we needed to upload *the entire* git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB. Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such. Signed-off-by: Justin Chadwell <[email protected]> * ci: remove dependency on go sdk The only thing this was being used for here was the connect job - we can just use `--help` to get this info to sanity check it. Signed-off-by: Justin Chadwell <[email protected]> * ci: run linting over ci itself Signed-off-by: Justin Chadwell <[email protected]> --------- Signed-off-by: Justin Chadwell <[email protected]>
* ci: exclude .git dir in ci build Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs. However! A problem with this was that we needed to upload *the entire* git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB. Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such. Signed-off-by: Justin Chadwell <[email protected]> * ci: remove dependency on go sdk The only thing this was being used for here was the connect job - we can just use `--help` to get this info to sanity check it. Signed-off-by: Justin Chadwell <[email protected]> * ci: run linting over ci itself Signed-off-by: Justin Chadwell <[email protected]> --------- Signed-off-by: Justin Chadwell <[email protected]>
Follow-up to #7231.
As a fun coincidence, this should fix the continued weirdness when doing this build in a git-worktree.
Previously, we were using this to generate a work-tree hash, so that new builds would have a different version tag, which helped to solve some weird caching issues with nested execs.
However! A problem with this was that we needed to upload the entire git directory every time we built - which is an expensive operation, my .git directory for dagger/dagger is 147MB.
Instead of relying on git for this, we can use the ID of the input directory! Because of our use of blob sources, a different ID represents a different directory. Additionally, we prefix these new builds with a special "dev-" string to specifically indicate that these are dev builds, with no specific version passed in - these aren't commit hashes, and cannot be treated as such.