Downward compatibility and garbage collection

[mbunse]

We have been utilizing DVC in our projects for a considerable time. Each project establishes its own Python environment along with its respective DVC setup, and we consistently update to the latest DVC version as needed.

Recently, we wanted to leverage the garbage collection feature to remove data, with exceptions for the data utilized for models in production (with the commit being tagged) and data used in ongoing developments (head of a branch). The command employed is dvc gc -f --all-branches --all-tags.

This approach functions well if we maintain the same version throughout the entire project history and utilize the corresponding DVC version to execute the garbage collection. However, if we upgrade to DVC 3 at any point, we get the following error:

'file.pkl.dvc' validation failed in revision '8ac726': 2 errors.

extra keys not allowed, in cmd

extra keys not allowed, in outs -> 0 -> metric, line 13, column 3
  12 outs:
  13 - md5: XXXXXXXXXXX
  14   path: files.pkl

Any help on how to solve our use case would be appreciated.

[pmrowla]

This is the same issue as #9767 and #9878, the problematic files were generated in very old DVC releases and we dropped support for those files entirely (and intentionally broke backwards compatibility) in DVC 3.0.

The workaround for now would be to remove the tags you have which point to the old DVC files containing the unsupported entries:

outs:
  - ...
    metric: ...

If for some reason you still need that tagged data, you can make a new commit with the metric entry removed, and then generate a new tag to replace the old one

something along the lines of

git checkout old-tag
# <edit .dvc file to remove the old metric entry>
git add file.dvc
git commit
git tag new-tag
git tag -d old-tag