github action often exceeds the allowed cache.

[Tocktock]

Select Topic Area

Question

Body

My repository has 10 GB of cache storage limit. But sometimes usage of cache exceeds it. I've even seen usage reach 20 GB.

And github action workflow sometimes fail because of cache storage connection fail. I'm guessing this connection issue was caused by exceeding the cache limit.

How do I fix this?

[Akash1134]

Hi @Tocktock ,

GitHub will remove any cache entries that have not been accessed in over 7 days. There is no limit on the number of caches you can store, but the total size of all caches in a repository is limited. By default, the limit is 10 GB per repository, but this limit might be different depending on policies set by your enterprise owners or repository administrators.

If you exceed the limit, GitHub will save the new cache but will begin evicting caches until the total size is less than the repository limit.

For information on changing the policies for the repository cache size limit, see "Enforcing policies for GitHub Actions in your enterprise" and "Managing GitHub Actions settings for a repository."

[adamjstewart]

In our case (https://github.com/microsoft/torchgeo/actions/caches), we're seeing:

Approaching total cache storage limit (37.49 GB of 10 GB Used)

Is it possible that the cache is not being purged correctly? I just purged all cached files, but it still says we're using close to 40 GB. It might take a while to update. But I'm more concerned that ~40 GB > 10 GB.

[kny011]

We are seeing the same with our cache entries. I am not sure if someone is working on this issue. It seems like the cache eviction policy doesn't work. We have to keep deleting the caches using the CLI

[adamjstewart]

FWIW, our issue was that a single action was generating more than 10 GB of data. By default, pytest saves all temporary files, even if the test passed. You can change this using:

tmp_path_retention_policy = "failed"

We also started to purge pip's download cache after each run. This got us down from 10s of GBs to 100s of MBs. So this could be a similar issue for others as well.

[jsoref]

There are APIs to delete cache entries. And you could split your cache items into smaller units.