Checks executed twice and no recovery notifications are sent

[danpicpic]

Describe the bug

We have observed a couple of times in the last 3 weeks a weird behaviour where the checks are performed twice, the notifications sent twice (at least the Problem one), but at the same time we also saw that no Recovery notifications were ever sent.
Every time it happened in a small time frame (for e.g. between 8am and 9am), on different number of servers/services with no common pattern between them.

The checker and notifications features are enable in HA on both master. On both of them, from the icinga2.log (is it normal that they log the same? are they doing the same action in parallel?) I see the following lines, where a Problem notification is sent but not the Recovery one:

[2024-02-03 08:20:18 +0100] information/Checkable: Checkable 'hostxxx!servicexxxx' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2024-02-03 08:20:18 +0100] information/Notification: Sending 'Problem' notification 'hostxxx!servicexxxx!state-notification-to-service' for user 'dummy_user'
[2024-02-03 08:20:18 +0100] information/Notification: Completed sending 'Problem' notification 'hostxxx!servicexxxx!state-notification-to-service' for checkable 'hostxxx!servicexxxx' and user 'dummy_user' using command 'state-notification'.
[2024-02-03 08:20:18 +0100] information/Checkable: Checkable 'hostxxx!servicexxxx' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2024-02-03 08:43:18 +0100] information/Checkable: Checkable 'hostxxx!servicexxxx' has 1 notification(s). Checking filters for type 'Recovery', sends will be logged.
[2024-02-03 08:43:18 +0100] information/Checkable: Checkable 'hostxxx!servicexxxx' has 1 notification(s). Checking filters for type 'Recovery', sends will be logged.

1. The first screenshot below is the one linked to the above log. All the messages regarding the notification not sent are weird, as the Problem notification was sent anyway, but not the Recovery.
2. From the two screenshots we can see how every check/action is done twice or multiple times (soft state, hard state, ok, notifications)

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): r2.14.1-1
• Operating System and version: RHEL 9.2
• Enabled features (icinga2 feature list): api-users api checker command graphite ido-mysql mainlog notification
• Icinga Web 2 version and modules (System - About): 2.11.4
• Config validation (icinga2 daemon -C): OK
• If you run multiple Icinga 2 instances, the zones.conf file:

object Endpoint "master1" {
}
object Endpoint "master2" {
  host = "master2"
}
object Endpoint "satellite1" {
  host = "satellite1"
}
object Endpoint "satellite2" {
  host = "satellite2"
}
object Zone "director-global" {
  global = true
}
object Zone "global-templates" {
  global = true
}
object Zone "master" {
  endpoints = [ "master1", "master2", ]
}
object Zone "satellite" {
  endpoints = [ "satellite1", "satellite2", ]
  parent = "master"
}

Additional context

• We have migrated our infrastructure from SLES12.5 (Icinga 2.10.3) to RHEL9 (Icinga 2.14.0) around 2 months ago
• We have also installed jemalloc-5.2.1-2.el9.x86_64
• At the beginning we only had test servers (of which ~1000 with active notifications) to validate the new Icinga2
• 3 weeks ago we started to monitor the remaining ~2000 Production servers and upgraded Icinga2 to v2.14.1

We have started to see the error in the last 3 weeks, but we don't know if it was introduced by the last minor update to 2.14.1, or if it was already present since the first migration, but as we had fewer servers and less important, it might have been ignored.

[Al2Klimov]

Every time it happened in a small time frame (for e.g. between 8am and 9am)

Do those frames correlate with your Director deployments?

[danpicpic]

Every time it happened in a small time frame (for e.g. between 8am and 9am)

Do those frames correlate with your Director deployments?

We don't use Director. But anyway no, and it also happened other few times with less servers (even one single event at the time)

[danpicpic]

We have experienced this additional few times. We have no clue how to debug it (db query?), or how to replicate it, therefore any suggestions are appreciated.

Just to sum up, and add 2 additional new info:

• In the icingaweb history we see the checks attempts are displayed twice
• eventually there are 2 notifications that are actually sent
• no ok notification is ever sent, showing the the message This notification was not sent out to any contact. what is weird is that is always systematically like this (twice the critical/warning are sent but no ok is sent)
• we have recently realised that we had both satellites communicating with each other. We have now set the communication only from satelliteA to satelliteB as suggested by the documentation (even though it should only cost in terms of performance and not of functionality). As the problem is not constant, we have no clue to know if this could be the solution