Slack notification delay after new issue arrived to sentry

[maylow]

Self-Hosted Version

24.4.1

CPU Architecture

x86_64

Docker Version

20.10.24

Docker Compose Version

2.23.3

Steps to Reproduce

1. Create alert to send slack notification when new issue is created.
2. Send issue via sentry-cli.
3. Sentry gets an issue but not mark it as new issue for 5 minutes.

4. After 5 minutes, it gets mark new issue and send issue to slack.

Expected Result

1. Create alert to send slack notification when new issue is created.
2. Send issue via sentry-cli.
3. Sentry gets an issue, mark it as new issue and send it to slack.

Actual Result

It is regression of the issue: getsentry/self-hosted#2034
The issue showed up after upgrade from 24.3.0 -> 24.4.0 with continue to 24.4.1

I also find the same reason where kafka has LAG on post-process events:

GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                  HOST            CLIENT-ID
post-process-forwarder generic-events  0          478291          478291          0               rdkafka-b5bbee3d-650c-465f-aeb5-b1b2b78eb9a9 /172.18.0.43    rdkafka
post-process-forwarder events          0          9288607         9288608         1               rdkafka-a7f67d85-627b-4fae-ba46-c6e03a7bb3d4 /172.18.0.51    rdkafka
post-process-forwarder transactions    0          186570290       186570290       0               rdkafka-482098cd-d1ab-46fd-8ee6-b9ae3dcf3aea /172.18.0.42    rdkafka

Event ID

No response

[hubertdeng123]

It is expected for there to be some sort of lag between receiving an event and sending an event on slack. It is unlikely that this process will be in real time. I have a few questions:

1. Was the slack notification sent in real time before?
2. Is the slack notification sent only when a new event is ingested after the issue sent via sentry-cli?

[maylow]

We have 2 sentry instances and second has version 24.2.0 and this one is working without any issue. New issue sent to sentry is immediately marked as new which is trigger the alert and sent issue to slack.

1. Yes, before version 24.4.0 it was working as expected.
2. Slack notify should be sent once new issue is created in sentry. Unfortunately, every issue created in sentry has mark ongoing so it is not sent to slack. Once ongoing mark switched to new then slack shows notify.

[hubertdeng123]

Got it, you are saying that new issues that come into Sentry appear as Ongoing instead of New? Then, they're not triggering alert rules.

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-alerts for triage ⏲️

[getsantry]

Routing to @getsentry/product-owners-issues for triage ⏲️