bug: cache does not get invalidated on resource deletion?

[nazarewk]

TLDR: I believe that with before-hook-creation under heavy load our ArgoCD (2.3.3 on EKS) processes a PreSync phase as Completed before fully processing resource (Job) deletion.

This is a followup from argoproj/argo-cd#10077

Probably watch event arrives late or gets stuck in a queue for a while and I could not track down any cache invalidation during deletion in argo-cd or gitops-engine itself.

gitops-engine/pkg/sync/sync_context.go

Lines 1085 to 1112 in 2bc3fef

	// delete anything that need deleting
	hooksPendingDeletion := createTasks.Filter(func(t *syncTask) bool { return t.deleteBeforeCreation() })
	if hooksPendingDeletion.Len() > 0 {
	ss := newStateSync(state)
	for _, task := range hooksPendingDeletion {
	t := task
	ss.Go(func(state runState) runState {
	sc.log.WithValues("dryRun", dryRun, "task", t).V(1).Info("Deleting")
	if !dryRun {
	err := sc.deleteResource(t)
	if err != nil {
	// it is possible to get a race condition here, such that the resource does not exist when
	// delete is requested, we treat this as a nop
	if !apierr.IsNotFound(err) {
	state = failed
	sc.setResourceResult(t, "", common.OperationError, fmt.Sprintf("failed to delete resource: %v", err))
	}
	} else {
	// if there is anything that needs deleting, we are at best now in pending and
	// want to return and wait for sync to be invoked again
	state = pending
	}
	}
	return state
	})
	}
	state = ss.Wait()
	}

[nazarewk]

After brief look into the code seems like the fix would be simple, but still an architectural change:

1. in addition to waiting for event: add cache invalidation at https://github.com/argoproj/gitops-engine/blob/master/pkg/sync/sync_context.go#L1009-L1016
2. pass and store cache on syncContext
3. make deleteAPIResource public

WDYT?

[nazarewk]

TODO: try to find a place to improve/introduce "has the deletion completed?" or "is the object fresh enough after deletion policy: before-hook-creation?" kind of checks.

[nazarewk]

looks like a known issue that was only partially remedied by introducing sync wave delays:

• Argo CD might progress through waves/PostSync prematurely argo-cd#4669
• https://github.com/argoproj/argo-cd/blob/9fac0f6ae6e52d6f4978a1eaaf51fbffb9c0958a/controller/sync.go#L465-L485
• Argo CD considers PreSync phase finished even though the Job was just created argo-cd#10077 (comment)