[Grafana 8.4.4] Failed to load rules config from Loki : AccessDenied: Access Denied status code: 403 request id

[tahchiev]

Hi,
I have issue Failed to load rules config from [Loki] AccessDenied: Access Denied status code: 403, for 5 from 7 Loki servers on Alerting.All Loki servers are located in AWS account.
I have configured all of them to use same bucket to store rule files in loki.yaml
ruler:
storage:
type: s3
s3:
s3: loki-config
bucketnames: loki-config
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
For 2 of 7 it is working fine they are connecting to the bucket and sharing rules.Also they appear in create new cortex rule source drop down menu, but the rest 5 are giving access denied.
All 7 lokis are in separated vpcs which have peering and have same security group policy in/out rules on LBA. Target groups are also with same configuration.
S3 bucket is open - no policy or restrictions. Compare working with non working vpc endpoints >same with same configuration.

Inside Grafana logs is nothing new:
"2022-04-07 16:07:37
logger=context t=2022-04-07T13:07:37.91+0000 lvl=eror msg="AccessDenied: Access Denied\n\tstatus code: 403, request id: 3KCQMHWXE2QYJMPG, host id: 9AEwdzOdvIRXbpX223Qt6UHCAZAsroyj8G9wmD2cC2Oaw3GqRzc5twsWkKbu99P4oYro4VykbiQ=\n" error="AccessDenied: Access Denied\n\tstatus code: 403, request id: 3KCQMHWXE2QYJMPG, host id: 9AEwdzOdvIRXbpX223Qt6UHCAZAsroyj8G9wmD2cC2Oaw3GqRzc5twsWkKbu99P4oYro4VykbiQ=\n" remote_addr=x.x.x.x"

Any ideas why on 2 is working and on the rest 5 is not?What else i can check? i tried wget from grafana pod since we don't have curl
and same result for working and non working
/tmp $ wget https://loki.*.xyz
Connecting to loki.*.xyz (x.x.x.x:443)

[pracucci]

👋 Hi! This look an question related to Grafana Loki and not Grafana Mimir. Maybe you wanna move the discussion to https://github.com/grafana/loki instead?