Actions/cache: Cache not being hit despite of being present

[RJ722]

Hi,

My workflow looks like this:

- name: Set Poetry Config
      run: |
        poetry config virtualenvs.create true
        poetry config cache-dir
- uses: actions/cache@v1
  name: Poetry Cache
  with:
    path: ~/.cache/pypoetry/virtualenvs
    key: poetry-${{ runner.os }}-${{ hashFiles('poetry.lock') }}
    restore-keys: |
      poetry-${{ runner.os }}-

- name: Install python dependencies
  run: make installdeps

The corresponding logs showed:

poetry config virtualenvs.create true
poetry config cache-dir
shell: /bin/bash -e {0}
env:
DEEPSOURCE_DSN: ***
pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64
/home/runner/.cache/pypoetry
Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

Run actions/cache@v1

with:

path: ~/.cache/pypoetry/virtualenvs

key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618

restore-keys: poetry-Linux-
env:

DEEPSOURCE_DSN: ***

pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

The post job logs show that cache was stored successfully:

Post job cleanup.
/bin/tar -cz -f /home/runner/work/_temp/d1cb4b2a-f400-44ba-ac17-8a863c548c95/cache.tgz -C /home/runner/.cache/pypoetry/virtualenvs .
Cache saved successfully

However, on the second run, the cache is missed again. Logs here:

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.
Run actions/cache@v1
  with:
    path: ~/.cache/pypoetry/virtualenvs
    key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618
    restore-keys: poetry-Linux-
env:

DEEPSOURCE_DSN: ***

pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

Does anyone have an idea about why the cache is being missed in the second run? Also, would it be apt to open an issue on https://github.com/actions/cache?

Thanks!

[BrightRan]

@rj722 ,

I tested and tried as the demo you shared, but I did not reproduce the problem. Not sure what causes this problem.

I have helped you report this ticket in the repository of the cache action:

actions/cache#231

You can follow this issue ticket, and add you comment here.

Of course, actually any user can directly open an issue this repository, that will allow you to directly interact with the appropriate engineering team, and make it more convenient for the engineering team to help you solve the problems occur when you use this action, and also can collect and categorize your suggestions for this action. I think this is why the team will set the repository to be public.

[BrightRan]

@rj722 ,

I noticed the appropriate engineer has provided some suggestions for troubleshooting, please check with these:

Cache’s are scoped to a key and a branch, so a cache created on branch-A won’t be accessible by branch-B unless branch-A is the default branch.

It’s recommended that you have a workflow run on your default branch (usually master ) so that worklows on other branches will have a cache to restore from.

[RonaldGalea]

Is there any solution to this? How are you supposed to get a cache for the default branch? Trigger workflows manually on it?

[david-molnar-oculai]

This just makes no sense. Please fix this.

[amirasyraf]

That's absurd. How would this work for pull requests?

What's the point of deciding this for the users as opposed to letting them decide? That's what the cache key is for.

[rene-hermenau]

It’s recommended that you have a workflow run on your default branch (usually master ) so that worklows on other branches will have a cache to restore from.

This makes no sense for us! We are building our dist of a branch, run workflow generated tests on the same branch, when they pass, we merge into master, not the opposite around.

[1ukastesar]

This is not working for any project that uses PRs and branching. In these projects, the cache hit almost never occurs, despite its key matching entirely.

It is just stupid; please add at least configurable option to turn that feature off.

[nroose]

This doesn’t make sense to me. Seems like the cache should be dependent on the repo and the keys. Not on the branch. And in my experience, it only gets a cache hit on the same commit when I re-run the workflow. It never seems to get a hit even on the same branch. This is pretty silly.

[partydrone]

Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues.

[mmbarness]

This can be prevented if the engineer dealing with this writes out a cache key specific enough to eliminate other branches. Why is, instead, github making it impossible for us to make our own decisions about this.

[matinzd]

Handling cache hits at the repository level seems like a more sensible approach. We can easily customise cache keys by incorporating the branch's reference name into the key. The current default behaviour can be a bit confusing and unconventional or at least there should be a way to override this behaviour.
I am wondering how tags work in this scenario.
This can significantly reduce build times but it seems that Github wants to charge us for it.

[nroose]

Oh, so it’s scoped to the specific tag for runs triggered by a tag push as well. This is even more silly. It means that no tag build can use cache. Why do they scope it to the branch???

[mihaerzen]

Hey. How did you figure out it is scoped to the tag as well?

[kostyaplis]

  <a href="https://github.com/actions/cache/issues/556" target="_blank" rel="noopener nofollow ugc">github.com/actions/cache</a>

using cache on tags does not work

<div class="github-info">
  <div class="date">
    opened <span class="discourse-local-date" data-format="ll" data-date="2021-03-27" data-time="11:31:57" data-timezone="UTC">11:31AM - 27 Mar 21 UTC</span>
  </div>


  <div class="user">
    <a href="https://github.com/hq9000" target="_blank" rel="noopener nofollow ugc">
      <img alt="hq9000" src="https://user-images.githubusercontent.com/53430368/181101539-7c55ed4c-5cad-4ed7-bdbd-ea6e43b09a88.jpeg" class="onebox-avatar-inline" width="20" height="20">
      hq9000
    </a>
  </div>
</div>

<div class="labels">
</div>

hi!

I have a cache created on a non-default branch, all works fine except wh…en the workflow is run on a tag, the cache isn't hit

when it's not a tagged commit, and my github.ref is "...heads" it starts working again.

I guess it's related to https://github.com/actions/cache#cache-scopes

Essentially, it looks like there is no way to use cache in runs on tagged revisions, or is it? (I don't have any caches on my master yet). Is it an intended behavior?

At least, it'd be good if it was described in readme in that "scopes" section.

[swushi]

Makes no sense to cause a cache miss if it's from a non-default branch. That makes the cache much less effective. This essentially causes any workflow that you develop and presumably cache in a PR immediately cache-miss when it's merged into the default branch.

The hash is guaranteeing a stable cache, regardless of branch.. I don't see the point for the branch protection as well.

[SvenStaehs]

well, I can see it as a failsafe measure. You say The hash is guaranteeing a stable cache, but is this really the case? Will you bet the stability of your delivery on the fact that nobody managed to break the cache for a given content of poetry.lock?

The thing is, the key does not guarantee anything. There are so many things that can go wrong, even if you don't take malice into account. First the key is generated, then the venv is filled, then some action runs using that venv, then the venv is published as cache. But the action can have modified the contents of what you want to cache, in this case something could have installed additional packages in the poetry venv, without any change in poetry.lock, thus not changing the key.
If two branches make the same changes in poetry.lock, but one of them breaks the venv in their action, the other branch (and even the main branch!) could get this broken cache.

As far as I understand, running the workflow on the same branch with the same poetry.lock file should restore the cache just fine, so it only has to run once and from then on will get the cached venv. It's a tradeoff, and confusing as heck because it's not prominently documented, but it's a tradeoff I can understand (with the power of hindsight)

I hope my explanation attempt wasn't too confusing, sorry 😉

[alambike]

Why not let the user tweak the cache scope? I mean, using the cache/save action, GitHub could allow you to define the scope of the cache to be stored.

In this way we can have safe defaults, but allow them to be altered for other use cases.

For example, when merging a PR against the default branch, I would like to be able to promote the last cache generated in the PR, to the scope of the default branch, so that it would be the most recent so new PRs could use it.

But the way the action currently behaves, I don't see a way to do it, without launching a new workflow that does the build again and publishes the cache, once the PR has been merged.

A waste of minutes and time since this new build is going to be exactly the same as the last build executed in the PR. Furthermore, if a new PR comes in while the default branch is being built, it will grab an older version from the cache (unless the cache key has been altered).

[nroose]

This is one is the main reasons GitHub actions didn't work for us. The cache key, assuming you can set it in a relatively dynamic/powerful way (hash, branch, etc.), is enough, without any additional scoping. And caching is important to make many workflows fast enough to be practical. Most of our workflows are not going to production.

[Cypher1]

This caching behavior should at minimum be configurable, if not straight up not branch specific.

[nroose]

Sure. That’s why we have cache keys. And fallbacks. Let the person configuring the cache do that.

…

On Fri, Mar 31, 2023 at 9:51 AM Andrew Porter ***@***.***> wrote: Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues. — Reply to this email directly, view it on GitHub <#27059 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLJIQBU5MIZMFMF6CIRJTW64DQXANCNFSM6AAAAAAQK2XZJI> . You are receiving this because you commented.Message ID: ***@***.***>

[btfshadow]

I really dont see any improvement to use a cache inside the same branch, and that don't work with the default branch either ... only bullshit.
If I make a cache I want to use every time I want ...

[RafidMuhymin]

Keeping things isolated by branch makes sense when you don't want the cache in one branch polluting the cache in another branch. For example, if I have a node project and I open a PR that adds, updates, or deletes a new node module to the project, then someone else on my team opens a PR that triggers the same workflow, a repo-level cache would likely cause issues.

@partydrone A repository-level cache would likely cause issues. But not all people use the action just to cache node modules. There are many valid use cases. For example, I might want to use a production cache for the main branch, and for all the other branches I might want to use another development cache.

And even if it's a security vulnerability, it's not so high that it won't be even configurable. The discussion is running for 3 years without any response.

[wind57]

We have a real use-case where again circleci got it right and github got it wrong.

We want to split tests by their times, so that each matrix runs more or less the same amount of time, think of it as a perfectly distributed tests per each instance of the matrix.

If I am on branch-A and I have 100 tests, this will generate some distribution of test-times per one matrix. I cache these results. If I add 10 more tests, and try to get the previous distribution... surprise! there is no cache. This leaves me speechless.

[cAstraea]

So what's the solution ? Is my understanding that for some arbitrary reason it won't run on

name: Deploy development

on:
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+-release*dev'

Is that correct ?

[MatMercer]

After doing everything fine, and even testing the cache manually in a test branch, the cache suddenly isn't being hit, what is going on?

The cache is saved in a first run:

Then, not hit for no reason:

The two caches exists:

And the action is configured like that:

      - name: Restore cached golangci-lint
        id: golangci-lint-restore
        uses: actions/cache/restore@v3
        with:
          path: ${{ env.GOLANGCI_LINT_CACHE }}
          # search for golangci-lint-{commit} OR restore latest golangci-lint-*
          # for the current branch or main branch
          key: golangci-lint-${{ github.sha }}
          restore-keys: |
            golangci-lint-

Anyone know what is going on? env.GOLANGCI_LINT_CACHE is a fixed value and won't change.

@BrightRan do you have any clues?

[BillyTom]

The cache is not hit because the cache keys are not the same. The cache/save action uses the key golangci-lint-79b.... The cache/restore action uses the key golangci-lint-5ce...

[MatMercer]

@BillyTom

No:

If cache key is not found on current branch it will search current branch cache using restore-keys. If there is also no match it will search default repository branch cache using key and then restore-keys. Here is some documentation

The cache is at my default branch, and the workflow is running from a pr event. The cache still isn't matching, and for the above comment I posted it started working randomly.

Here's another person that started working randomly: https://github.com/orgs/community/discussions/58583#discussioncomment-6235462

There's clearly a bug for how caches are matched in the GitHub action.

Cache was saved at the default branch

Workflow was triggered in a PR after the cache was saved

[MatMercer]

Now I'm just waiting for it to randomly work again, as before, but there's clearly something weird going on

[udondan]

About two weeks ago, I noticed it started working for me again. Not sure it always works everywhere but I have seen it working on some renovate PRs. I just did a manual test, for a branch that certainly never existed before and the cache was restored instantly.

My cache action: https://github.com/udondan/cdk-ec2-key-pair/blob/509e42c4fb9fd3d1667babffd11c3bfce0761b75/.github/workflows/test.yml#L35-L46

[MatMercer]

@udondan

Can it be because you are using actions/cache@v4 and I actions/cache/restore@v4? This is the only difference I see...

[pavelsaman]

Is there any solution to this? It seems like people would like to configure the scope for cache but it also seems it's not configurable in any way. I've recently encountered this limitation myself as well.

[OgulcanCelik]

@BrightRan as you guys can see many of us are pretty frustrated by not being able to share cache between branches. After 3 years, do you still not have a plan to add this functionality to the cache action? I should be allowed to set the branch restrictions, not you guys by default.

[ajfranzoia]

Amazing stubbornness

[nroose]

Seems clear to me that they are making money off of doing this and that is a hard thing to just give up.

…

On Wed, Sep 27, 2023 at 3:57 AM Oğulcan Çelik ***@***.***> wrote: @BrightRan <https://github.com/BrightRan> as you guys can see many of us are pretty frustrated by not being able to share cache between branches. After 3 years, do you still not have a plan to add this functionality to the cache action? I should be allowed to set the branch restrictions, not you guys by default. — Reply to this email directly, view it on GitHub <#27059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLJIRNOTWKP5JIKT2IG5DX4QBC7ANCNFSM6AAAAAAQK2XZJI> . You are receiving this because you commented.Message ID: ***@***.***>

[Wastrelli232]

Hi,

My workflow looks like this:

- name: Set Poetry Config
      run: |
        poetry config virtualenvs.create true
        poetry config cache-dir

• uses: actions/cache@v1
  name: Poetry Cache
  with:
  path: ~/.cache/pypoetry/virtualenvs
  key: poetry-${{ runner.os }}-${{ hashFiles('poetry.lock') }}
  restore-keys: |
  poetry-${{ runner.os }}-

• name: Install python dependencies
  run: make installdeps

The corresponding logs showed:

poetry config virtualenvs.create true
poetry config cache-dir
shell: /bin/bash -e {0}
env:
DEEPSOURCE_DSN: ***
pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64
/home/runner/.cache/pypoetry
Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.
Run actions/cache@v1
with:
path: ~/.cache/pypoetry/virtualenvs
key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618
restore-keys: poetry-Linux-

env: DEEPSOURCE_DSN: *** pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64 Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

The post job logs show that cache was stored successfully:

Post job cleanup.
/bin/tar -cz -f /home/runner/work/_temp/d1cb4b2a-f400-44ba-ac17-8a863c548c95/cache.tgz -C /home/runner/.cache/pypoetry/virtualenvs .
Cache saved successfully

However, on the second run, the cache is missed again. Logs here:

Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.
Run actions/cache@v1
  with:
    path: ~/.cache/pypoetry/virtualenvs
    key: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618
    restore-keys: poetry-Linux-

env: DEEPSOURCE_DSN: *** pythonLocation: /opt/hostedtoolcache/Python/3.8.2/x64 Cache not found for input keys: poetry-Linux-10e15406c5ba2cd9517ecdbad0c577bdaeb25da4c86ef997a3082522bfa0d618, poetry-Linux-.

Does anyone have an idea about why the cache is being missed in the second run? Also, would it be apt to open an issue on https://github.com/actions/cache?

Thanks!

[anderssonjohan]

What does debug logs say? (re-run with debug checked or create the ACTIONS_RUNNER_DEBUG and ACTIONS_STEP_DEBUG repository secrets)

I had the same result as you describe and found this in my debug output:

##[debug]No matching cache found for cache key 'pipx-Windows-3.11.5', version '4d538ab54b0848ab0bc2da5beb29e1222ffe934edc435065f6f947d317a3bb6c and scope refs/heads/dev. There exist one or more cache(s) with similar key but they have different version or scope. See more info on cache matching here: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key 
##[debug]Other caches with similar key:
##[debug]Cache Key: pipx-Windows-3.11.5, Cache Version: 5a3129aad8ab3c8bedeaf1c143c[63] d547ce3cf06db888f00b9691877c885cf48, Cache Scope: refs/heads/dev, Cache Created: 2023-09-29T19:42:55.73Z

In my case the version seemed to be based on the paths I provided to the action. To make it work on both ubuntu and windows I set up the cache action with two sets of paths:

path: |
  ~/.local/bin
  ~/.local/pipx/shared
  ~/.local/pipx/venvs
  C:\Program Files (x86)\pipx\shared
  C:\Program Files (x86)\pipx\venvs
  C:\Program Files (x86)\pipx_bin

Then, in the "Post" step of the cache action I noticed it created a cache with a version different to what it tried to restore, only matching the expected subset of paths provided to the action:

##[debug]Search path 'C:\Users\runneradmin\.local\bin'
##[debug]Search path 'C:\Users\runneradmin\.local\pipx\shared'
##[debug]Search path 'C:\Users\runneradmin\.local\pipx\venvs'
##[debug]Search path 'C:\Program Files (x86)\pipx\shared'
##[debug]Search path 'C:\Program Files (x86)\pipx\venvs'
##[debug]Search path 'C:\Program Files (x86)\pipx_bin'
##[debug]Matched: C:/Program Files (x86)/pipx/shared
##[debug]Matched: C:/Program Files (x86)/pipx/venvs
##[debug]Matched: C:/Program Files (x86)/pipx_bin
##[debug]Cache Paths:
##[debug]["C:/Program Files (x86)/pipx/shared","C:/Program Files (x86)/pipx/venvs","C:/Program Files (x86)/pipx_bin"]

This was of course not a good setup but when I started using the paths cached according to the post-step, the version matched up and I got a cache hit.
Albeit being a different setup than yours, perhaps you can get some insights from the debug logs to see if there are caches found but with non-matching versions?

So now after using the debug logs I learnt there are three moving parts: cache scope, key and version. 🤔

[matinzd]

Handling cache hits at the repository level seems like a more sensible approach. We can easily customise cache keys by incorporating the branch's reference name into the key. The current default behaviour can be a bit confusing and unconventional or at least there should be a way to override this behaviour.
I'm curious about how tags function in this situation, as each tag essentially acts as a unique branch each time.
This could make our builds much faster, but it looks like GitHub wants to charge us for it intentionally.

[david-molnar-oculai]

Classic GitHub stupidity. It doesn't work for releases as it's always has a different tag which means no cache hit. It makes no sense. I used the hash of package-lock.json in the cache key. It's safe to cache it and use it even for different branches.

[emschu]

Thank you for this gigantic resource waste by design

[vegidio]

Github clearly doesn't want cache to work in a way that makes sense because they can charge corporate users for using more minutes of Github Actions.

[rene-hermenau]

Due to this, We are seriously considering switching from GitHub to GitLab Ci or another CI.
We don't care about the costs, the CI minutes in GitHub are not much and are fair rated but we need a caching solution that makes sense and that saves us time. At the moment our tests eat up huge time because our builds are recreated on each of our 70 workflows and we are required to run them all for each release. It would save us tons of waiting time if we can share caches between different workflows.

If this decision was made by design it's not smart. You will loose customers Github! Once, we move our CI to another service, it's just a matter of time until we have to move our whole project. I don't want to, and love GH, but its starting to get a bad taste.

[bgdnvk]

interesting design choice!

[adrians5j]

Based on everything that's been discussed above, it does look like it might just be better not to use actions/cache, but implement your own, where you store caches in an S3 bucket or something. 🤷🏻

[nroose]

That’s bound to be much slower.

…

On Wed, Apr 3, 2024 at 6:25 AM Adrian Smijulj ***@***.***> wrote: Based on everything that's been discussed above, it does look like it might just be better not to use actions/cache, but implement your own, where you store caches in an S3 bucket or something. 🤷🏻 — Reply to this email directly, view it on GitHub <#27059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLJIRGJ3B536P6FRSOG73Y3P7LFAVCNFSM6AAAAAAQK2XZJKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSOJXGEZTI> . You are receiving this because you commented.Message ID: ***@***.***>

[ahmadabdul3]

ugh... why is it like this. I'm running into the same issue, but theres a difference.
I have a CI workflow when I create a PR which stores caches (per branch in some instances)
I have another workflow that runs manually when I want to deploy to specific staging environments.

The weird thing is, if I run the deploy workflow on the same branch that the PR workflow ran on, it still cant find the cache. I dont understand how its 'scoped to a branch' if both workflows are running on the same branch...

if the PR is branch-a -> master and the deploy workflow runs on branch-a directly, do those count as 2 different branches?
I do know that PR workflows in GH-actions run in a detached state. 🤦

this is very frustrating

[david-molnar-oculai]

...and of course, GitHub just ignores us, they have nothing to say.

[rene-hermenau]

I have a feeling that since this discussion was tagged as "Answered by BrightRan", no one from the core Github team will ever read our comments!

Guys, we need to open another discussion!

[ajfranzoia]

In my case, I found a workaround for my Yarn dependencies:

• create a compressed file of the cache dir and upload to a remote storage provider
• add the file public URL as an Actions secret
• in the first step of the workflow, download the file to the Yarn cache folder

This is sub optimal because it requires updating the cache file once in a while, but it's better than no cache at all

[mtsadler-branch]

If someone wants to make a universal action for this, I'm sure people would sponsor it

[rene-hermenau]

Definitely. Count me in with sponsoring.