Cached submodules during clone

[antonysouthworth-halter]

Hi,

We have quite a few submodules in one of our projects and at the moment it can take about 10 minutes just to clone the repo + submodules. Concourse workers are in AWS and we are cloning from GitHub (not self-hosted).

An idea we had would be if this resource could track and cache the submodules?

Example workflow of what I had in mind, how it might work:

• a job which get:s a git resource is triggered by any mechanism
• before doing the full checkout, this resource does the following for each submodule:
  • check if we have a cache entry for the submodule
  • if we do, compare the checked-out-commit recorded in .gitmodules with the checked-out-commit recorded in the cache
  • if those commits match -> cache hit: ln or cp or other fast-ish operation to populate the submodule directory
  • otherwise -> cache miss: clone the submodule from upstream
• checkout continues as normal

Keen to hear thoughts on that^ idea, whether it's technically possible or if I've missed something? Would such a feature be within the scope of this resource?

Also keen to hear any other ideas, e.g. if we are just using this resource wrong 😅 or in a way that was not originally intended, or if there are other methods we could use to speed up the clone.

I've also heard rumors that other CI systems support this out-of-the-box, but haven't been able to find any docs to support that after a quick Google search.