Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to force re-upload harbor existing layers? Harbor Image Layer Re-upload Mechanism: Forced Updates for Data Recovery #20424

Closed
will-mei opened this issue May 14, 2024 · 5 comments
Assignees
Labels

Comments

@will-mei
Copy link

Background: Our Harbor service utilizes a Ceph cluster acting as S3 storage. Unfortunately, an unexpected data center shutdown due to AC malfunctions caused overheating and data loss on some hard drives.

Current Situation: We've successfully restored the object storage and service functionality. However, data recovery for lost objects is not possible. As a result, we need to re-upload existing images.

The Challenge: During image re-uploading, some image layers fail to update and are recognized as already existing. Is there a way to force Harbor to re-upload all layers of existing images?

@MinerYang
Copy link
Contributor

Hi @will-mei ,
What the lost is objects in S3 storage while data in postgres is not affected? Correct Me if I understood wrong

@will-mei
Copy link
Author

will-mei commented May 14, 2024

@MinerYang

Hi @will-mei , What the lost is objects in S3 storage while data in postgres is not affected? Correct Me if I understood wrong

Right, PostgreSQL is healthy and fine. However, the objects stored on S3 are corrupted, and the consistency of the sharded data has been compromised. The sharded data of objects is no longer functioning as intended (it seemed that they are now empty blobs).

@MinerYang
Copy link
Contributor

MinerYang commented May 20, 2024

Hi @will-mei ,

I would like to suggest you following steps to re-pushing images while some of the objects missing in S3 filesystem

  • Step 1 Run GC to clean the unused blobs from storage and registry redis cache
  • Step 2 re-push images to harbor

Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Jul 20, 2024
Copy link

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants