Problem with Retention and Forget

[bauerbyter]

Hey,
I have this following profile running for some time and absolute love how easy it is. Thanks for the good work :-)

But one thing is not working as expected (but probably a mistake from my side): It is not forgetting snapshots.
I thought that this "keep-daily: 7 keep-weekly: 4" means that it will keep the last 7 days and one from each of the last 4 weeks. Am I Wrong?
But if I see the list of snapshots not one is "missing".
What do I do wrong?

Thanks in advance :-)

default-command: snapshots
  # initialize a repository if none exist at location
  initialize: false
  # resticprofile won't start a profile if there's less than 100MB of RAM available
  min-memory: 250

# a group is a profile that will call all profiles one by one
groups:
  # when starting a backup on profile "full-backup",
  # it will run the "root" and "src" backup profiles
  default:
    - omv
    - synology

omv:
  lock: "/tmp/resticprofile-profile-src.lock"
  initialize: true
  repository: "****
  password-file: "password.key"
  status-file: "omv_backup-status.json"
  extended-status: true
  run-after-fail: 'mail -s "[Restic][ERROR] - $PROFILE_NAME " ***** <<< $ERROR_MESSAGE'

  backup:
    one-file-system: true                                   # Don't leave the file system via mount points
    check-after: true
    source:
      - "/home/yunohost.app"
      - "/root/"
      - "/home/yunohost.backup"
      - "/home/bauerbyter"
    schedule: "01:00"
    schedule-permission: system
    schedule-lock-wait: 10m
    schedule-log: '{{ tempFile "omv_backup.log" }}'             # Create log file "backup.log" in temporary directory and delete it when resticprofile is done
    verbose: 2                                                  # Write details about each processed file to the log
    send-before:
      - method: HEAD
        url: https://hc-ping.com/*****/start

    send-after:
      - method: HEAD
        url: https://hc-ping.com/*******

    send-after-fail:
      method: POST
      url: https://hc-ping.com/*****/fail
      body: "${ERROR}\n\n${ERROR_STDERR}"
      headers:
        - name: Content-Type
          value: "text/plain; charset=UTF-8"

  retention:
    before-backup: false
    after-backup: true
    keep-daily: 7
    keep-weekly: 4
    prune: true
    
synology:
  inherit: omv
  repository: "sftp*****"
  status-file: "syn_backup-status.json"

  backup:
    schedule: "03:00"
    schedule-log: '{{ tempFile "syn_backup.log" }}'
    send-before:
      - method: HEAD
        url: https://hc-ping.com/***/start

    send-after:
      - method: HEAD
        url: https://hc-ping.com/*****

    send-after-fail:
      method: POST
      url: https://hc-ping.com/*****/fail
      body: "${ERROR}\n\n${ERROR_STDERR}"
      headers:
        - name: Content-Type
          value: "text/plain; charset=UTF-8"

  retention:
    before-backup: false
    after-backup: true
    keep-daily: 7
    keep-weekly: 4
    keep-monthly: 6
    prune: true

[creativeprojects]

Hi!
To be honest I've had some funny issues with the restic forget command recently. It turns out I didn't fully understand the parameters.

Would you be able to post a --dry-run to see all the commands that would be running?

Thanks 😉

[bauerbyter]

Thanks! Sure, here is the dry-run:

root@bauerbyter:~/restic# resticprofile --dry-run --name synology backup
2023/09/15 08:04:45 using configuration file: profiles.yaml
2023/09/15 08:04:45 profile 'synology': initializing repository (if not existing)
2023/09/15 08:04:45 dry-run: /usr/local/bin/restic init --password-file=password.key --repo=sftp:**********:/homes/bauerbyter/backup/yunohost
2023/09/15 08:04:45 dry-run: webhook request method=HEAD url="https://hc-ping.com/******/start" headers:
User-Agent:  resticprofile/0.22.0

2023/09/15 08:04:45 profile 'synology': starting 'backup'
2023/09/15 08:04:45 dry-run: /usr/local/bin/restic backup --one-file-system --password-file=password.key --repo=sftp:******:/homes/bauerbyter/backup/yunohost --verbose=2 /home/yunohost.app /root/ /home/yunohost.backup /home/bauerbyter
2023/09/15 08:04:45 profile 'synology': finished 'backup'
2023/09/15 08:04:45 profile 'synology': cleaning up repository using retention information
2023/09/15 08:04:45 dry-run: /usr/local/bin/restic forget --keep-daily=7 --keep-monthly=6 --keep-weekly=4 --password-file=password.key --path=/home/yunohost.app --path=/root/ --path=/home/yunohost.backup --path=/home/bauerbyter --prune --repo=sftp:*****:/homes/bauerbyter/backup/yunohost
2023/09/15 08:04:45 profile 'synology': checking repository consistency
2023/09/15 08:04:45 dry-run: /usr/local/bin/restic check --password-file=password.key --repo=sftp:bauerb*****:/homes/bauerbyter/backup/yunohost
2023/09/15 08:04:45 dry-run: webhook request method=HEAD url="https://hc-ping.com/********" headers:
User-Agent:  resticprofile/0.22.0

[creativeprojects]

Yeah it does look good to me; I'll try and see if that works on one of my repos 👍🏻

[jkellerer]

@bauerbyter, when you list the remaining snapshots, do all of them have the same paths as specified in the forget command line?

--path=/home/yunohost.app 
--path=/root/ 
--path=/home/yunohost.backup 
--path=/home/bauerbyter

E.g. if you add paths later, existing snapshots with less paths would remain in the repo because they will then not satisfy the condition. You can prevent this by using tags or fix it by temporarily disabling the paths filter (path: false in retention) until the history is correct.

To use tags instead of paths:

backup:
  tag: "{{ .Profile.Name }}"

retention:
  path: false
  tag: true

Note: If you don't use any tag or path filter, restic would still apply forget to groups of snapshots. By default it groups them by "host,paths" this means it keeps a unique set of snapshots matching the retention policy for every unique value of the group-by condition. Grouping can be disabled.. but it is safer to keep it enabled, especially when repositories are reused with multiple profiles and hosts.