How to force re-upload harbor existing layers? Harbor Image Layer Re-upload Mechanism: Forced Updates for Data Recovery

[will-mei]

Background: Our Harbor service utilizes a Ceph cluster acting as S3 storage. Unfortunately, an unexpected data center shutdown due to AC malfunctions caused overheating and data loss on some hard drives.

Current Situation: We've successfully restored the object storage and service functionality. However, data recovery for lost objects is not possible. As a result, we need to re-upload existing images.

The Challenge: During image re-uploading, some image layers fail to update and are recognized as already existing. Is there a way to force Harbor to re-upload all layers of existing images?

[MinerYang]

Hi @will-mei ,
What the lost is objects in S3 storage while data in postgres is not affected? Correct Me if I understood wrong

[will-mei]

@MinerYang

Hi @will-mei , What the lost is objects in S3 storage while data in postgres is not affected? Correct Me if I understood wrong

Right, PostgreSQL is healthy and fine. However, the objects stored on S3 are corrupted, and the consistency of the sharded data has been compromised. The sharded data of objects is no longer functioning as intended (it seemed that they are now empty blobs).

[MinerYang]

Hi @will-mei ,

I would like to suggest you following steps to re-pushing images while some of the objects missing in S3 filesystem

• Step 1 Run GC to clean the unused blobs from storage and registry redis cache
• Step 2 re-push images to harbor

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.