Error on ingesting out-of-order samples

[ofirshtrull]

for the last 5+ days, I'm getting via Prometheus the error

component="scrape manager" scrape_pool=serviceMonitor/monitoring/prometheus-stackdriver-exporter/0 target=http://10.205.6.116:9255/metrics msg="Error on ingesting out-of-order samples" num_dropped=946 

I tried playing with the offsets and intervals, but I cannot fix this error,
the error I am getting in alertmanager is PrometheusDuplicateTimestamps
and I'm using chart version 4.4.1 (I upgraded it after the alert started showing up)

here are the values I pass to configure this exporter

stackdriver:
  projectId: '{{values.projectId}}'
  metrics:
    typePrefixes: >-
      logging.googleapis.com/user,
      pubsub.googleapis.com,
      bigtable.googleapis.com,
      cloudsql.googleapis.com,
      custom.googleapis.com,
      dataflow.googleapis.com,
      monitoring.googleapis.com/uptime_check,
      serviceruntime.googleapis.com/quota,
      file.googleapis.com,
      loadbalancing.googleapis.com
resources:
  requests:
    cpu: 100m
    memory: 128Mi
serviceMonitor:
  enabled: true
  interval: 20s
  additionalLabels:
    release: prometheus

and some time I get this in the logs

ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/innodb_buffer_pool_pages_dirty err="Get \"https://monit
oring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=20
23-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/open_table_definitions err="Get \"https://monitoring.go
ogleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Fopen_table_definitions%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08
%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/sqlserver/transactions/forwarded_record_count err="Get \"http
s://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fsqlserver%2Ftransactions%2Fforwarded_record_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&i
nterval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/replication/log_archive_failure_count err="Get \"https://moni
toring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Freplication%2Flog_archive_failure_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=
2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/postgresql/insights/perquery/lock_time err="Get \"https://mon
itoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fpostgresql%2Finsights%2Fperquery%2Flock_time%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.star
tTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.693Z caller=monitoring_collector.go:228 level=error msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"https://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric
.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"