cache (yarn,npm,...)

[strowi]

Hi,

i'm trying toget the caching of npm, yarn, etc to work while running in kubernetes.
I modified the config + mounted a persistent volume to the basedir.
This works fine for repositories. However caches are still stored in default locations e.g. /home7ubuntu/.cache/yarn.

I was under the impression renovate would use $cachedir automatically for yarn/npm etc. caches but the folders are pretty empty:

~> renovate
...
~> du -hd1 /renovate/cache/others/ /home/ubuntu/.cache
12K	/renovate/cache/others/yarn
4.0K /renovate/cache/others/pnpm
4.0K /renovate/cache/others/npm
24K	/renovate/cache/others/
89M	/home/ubuntu/.cache/yarn
89M	/home/ubuntu/.cache

~> /usr/local/node/14.15.3/bin/yarn cache dir
/home/ubuntu/.cache/yarn/v6

Also setting env var YARN_CACHE_FOLDER=/renovate/cache/others/yarn and running renovate still uses ~/.cache/yarn.
Do i still have to configure the yarnrc to use the different directory (this isn't clear to me from the docs)?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: renovate-config
data:
  config.json: |-
    {
      "logLevel" : "debug",
      "dryRun" : false,
      "persistRepoData": true,
      "onboardingConfig": {
        "extends": [
          "config:base",
          ":labels(bot)",
        ]
      }
    }

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: $CI_ENVIRONMENT_SLUG-renovate-bot
  labels:
    team: sys
  annotations:
    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
    config/revision: "${REVISION}"
spec:
  schedule: '@hourly'
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      parallelism: 1
      template:
        metadata:
          labels:
            team: sys
            cronjob: renovate-bot
          annotations:
            app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
            app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
        spec:
          # so we don't pull the 3,5GB image on all nodes
          affinity:
            nodeAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                    - node19
          initContainers:
            - image: busybox
              name: renovate-init
              command: ["sh", "-c", "chown -R 1000:1000 /renovate"]
              volumeMounts:
                - name: work-volume
                  mountPath: /renovate/

          containers:
            - image: renovate/renovate:24.11
              imagePullPolicy: IfNotPresent
              name: renovate-bot
              env:
                - name: RENOVATE_PLATFORM
                  value: 'gitlab'
                - name: RENOVATE_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: renovate-secret
                      key: gitlab-token
                # - name: RENOVATE_AUTODISCOVER
                #   value: 'false'
                - name: GITHUB_COM_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: renovate-secret
                      key: github-token
                - name: RENOVATE_BASE_DIR
                  value: '/renovate'
                - name: RENOVATE_ENDPOINT
                  value: "https://XYZ/api/v4"
                - name: RENOVATE_CONFIG_FILE
                  value: '/opt/renovate/config.json'
              volumeMounts:
                - name: config-volume
                  mountPath: /opt/renovate/
                - name: work-volume
                  mountPath: /renovate/
                - name: ssh-key-volume
                  readOnly: true
                  mountPath: '/home/ubuntu/.ssh'
          restartPolicy: Never

          volumes:
            - name: config-volume
              configMap:
                name: renovate-config
            - name: work-volume
              persistentVolumeClaim:
                claimName: renovate-work

            - name: ssh-key-volume
              secret:
                secretName: ssh-config
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: renovate-work
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

regards,
strowi

[rarkins]

The cache location is determined by each package manager individually. Typically Renovate needs to pass some environment variable to customize that location, like the YARN_CACHE_FOLDER you identified. It should be getting passed to the child process:

renovate/lib/manager/npm/post-update/index.ts

Lines 465 to 466 in 5d99ff9

	env.YARN_CACHE_FOLDER =
	env.YARN_CACHE_FOLDER || upath.join(config.cacheDir, './others/yarn');

and

renovate/lib/manager/npm/post-update/index.ts

Line 459 in 5d99ff9

'YARN_CACHE_FOLDER',

[strowi]

@rarkins thx for the reply, but that is what i tried. It seems the variable is not being passed through..

~> export YARN_CACHE_FOLDER=/renovate/cache/others/yarn
~> renovate
...
~>  du -hd1 /home/ubuntu/.cache/ /renovate/cache/others/
29M	/home/ubuntu/.cache/yarn
29M	/home/ubuntu/.cache/
12K	/renovate/cache/others/yarn
4.0K	/renovate/cache/others/pnpm
4.0K	/renovate/cache/others/npm
24K	/renovate/cache/others/

Meanwhile i just mounted another pvc on /home/ubuntu/.cache. Not sure though if this will solve the problem for all cache-dirs..

[rarkins]

Thanks, I found the missing code and pushing a fix

[strowi]

Wow that was fast. Thank you ;)
Looks like its working as intended now. Even without manually setting YARN_CACHE_FOLDER:

~> du -hd1 /home/ubuntu/.cache/ /renovate/cache/others/
4.0K	/home/ubuntu/.cache/
162M	/renovate/cache/others/yarn
4.0K	/renovate/cache/others/pnpm
4.0K	/renovate/cache/others/npm
162M	/renovate/cache/others/

[rarkins]

Yes, it shouldn’t require any manual configuration

[rarkins]

I needed to revert the previous change, so this now requires #8343 to be implemented