Dashboard randomly not showing available data

[sati-max]

Bug description

In our dashboards some charts are randomly not showing data when we change viewed time period to view earlier data. This happens randomly for different users - one user can see the correct data, when the other sees bugged charts, but after few days the user who saw bugged charts sees them correctly... Used browser doesn't matter - we checked Chrome, Edge, Firefox. All below screens are from the same chart.

One common thing is that it always bugs around 12:30AM.

UserA chart last week timeframe 5.05 - 8.05 (this was at 9AM, later at 10PM this chart worked correctly):

UserA yesterday:

UserB today:

UserB today, timeframe from last 10 days (so it should show the data that was on the 1st screen from UserA 5.05-8.05):

We noticed this in 2 separate dashboards, on total of 3 charts (2 charts in one dashboard, 1 chart in second dashboard). Screens are from the 1 chart in second dashboard.

We've setup metric retention for 6 months, and the change was saved correctly:

I've did some searching inside the signoz_metrics database using this guide https://signoz.io/docs/userguide/write-a-metrics-clickhouse-query/ and

time range from 15.05.2024 00:32:53 GMT+02:00 DST to 15.05.2024 00:34:53 GMT+02:00 DST shows 912 rows (should be around ~1 minute before the chart works correctly):
SELECT timestamp_ms, value FROM signoz_metrics.distributed_samples_v2 WHERE metric_name = 'XXXXXX' AND timestamp_ms > 1715725973000 AND timestamp_ms < 1715726093000;

time range from 15.05.2024 11:03:53 GMT+02:00 DST to 15.05.2024 11:04:53 GMT+02:00 DST shows 912 rows (the chart works correctly):
SELECT timestamp_ms, value FROM signoz_metrics.distributed_samples_v2 WHERE metric_name = 'XXXXXX' AND timestamp_ms > 1715763773000 AND timestamp_ms < 1715763893000;

I also tried to find the oldest record and it was from March 29th.

Expected behavior

The chart should be drawn correctly.

How to reproduce

1. Log into Signoz
2. Go to Dashboards
3. Open dashboard
4. Change the range to something like 12h, 24h, 48h
5. Some charts aren't drawn correctly

Version information

• Signoz version: 0.43.0 (also happened on 0.42.0)
• Browser version: Chrome, Firefox, Edge
• Your OS and version: Windows 10, Windows 11, Fedora
• Your CPU Architecture(ARM/Intel):

Additional context

Thank you for your bug report – we love squashing them!