Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thanos receive context deadline exceeded: Permanent error: Post \"http://obs-thanos-receive.xxxsvc.cluster.local:19291/api/v1/receive\": context deadline exceeded #7257

Open
yliu138repo opened this issue Apr 4, 2024 · 2 comments
Open

Thanos receive context deadline exceeded: Permanent error: Post \"http://obs-thanos-receive.xxxsvc.cluster.local:19291/api/v1/receive\": context deadline exceeded #7257

yliu138repo opened this issue Apr 4, 2024 · 2 comments

Comments

@yliu138repo
Copy link

yliu138repo commented Apr 4, 2024 •
edited

Thanos, Prometheus and Golang version used:
Thanos: v0.34.1

Object Storage Provider: MinIO object storage

What happened:
Our pipeline has a OpenTelemetry collector in front of the thanos Receiver stacks.

We performed load testing with TPS (Transaction Per Second)=2500 using k6 testing tool, we observed that there are significant data drops, and errors from Opentelemetry collector shows some context deadline timeout issue, e.g.

2024-04-04T07:14:46.107Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded; Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded", "errorCauses": [{"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.xxx.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}, {"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}], "dropped_items": 304868}

However, we are not able to identify any errors from the receiver logs except some out-of-order samples, which is expected.

We don't have any limits configured on Thanos receiver, and as far as we are concerned, By default all the limits and gates are disabled on receivers.

Receiver config:

        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--receive.hashrings-algorithm=ketama'
        - '--objstore.config=$(OBJSTORE_CONFIG)'
        - '--tsdb.path=/var/thanos/receive'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--tsdb.retention=2h'
        - >-
          --receive.local-endpoint=$(NAME).obs-thanos-receive-headless.$(NAMESPACE).svc.cluster.local:10901

Receiver distributor config

        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--receive.hashrings-file=/var/lib/thanos-receive/hashrings.json'
        - '--receive.hashrings-file-refresh-interval=3s'
        - '--receive.replication-factor=1'

Hashring

[{"hashring":"default","endpoints":["obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901","obs-thanos-receive-1.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901"]}]

It works perfectly with TPS=1500

What you expected to happen:
There should be minimal data drops with TPS=2500 as we have similar load testing against Prometheus before, which works fine.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy the pipeline with Opentelmetry collector and Thanos receiver stacks
  2. Load test with k6 using following script together with command
import { group, check } from 'k6';
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  systemTags: ['status', 'proto', 'method', 'name', 'group', 'error', 'error_code', 'scenario', 'expected_response'],
  insecureSkipTLSVerify: true,
  discardResponseBodies: true,

  scenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
	  // Requests per iteration = 200
      // Start iterations per `timeUnit`
      startRate: 1,
      // Start `startRate` iterations per second
      timeUnit: '1s',
      // Pre-allocate necessary VUs.
      preAllocatedVUs: 5000,
      stages: [
        // Start 300 iterations per `timeUnit` for the first minute.
        { target: 25, duration: '15m' },
        // Linearly ramp-up to starting 600 iterations per `timeUnit` over the following two minutes.
        { target: 25, duration: '45m' },
        // Continue starting 600 iterations per `timeUnit` for the following four minutes.
        //{ target: 200, duration: '1m' },
        // Linearly ramp-down to starting 60 iterations per `timeUnit` over the last two minutes.
        //{ target: 10, duration: '1m' },
      ],
    },
  },
};

export default function () {
  
  // Generate a random number between 1 and 100000 (adjust the range as needed)
  //const randomNumber = Math.floor(Math.random() * 100000) + __VU + __ITER;

  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName1 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName1 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName1 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName1 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName1 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 07 Link Entity'},});
    
    // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName3 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName3 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName3 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName3 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName3 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 07 Link Entity'},});
    
    // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName5 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName5 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName5 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName5 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName5 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 07 Link Entity'},});
  
    // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName7 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName7 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName7 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName7 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName7 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 07 Link Entity'},});
    
    	
    // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName9 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName9 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName9 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName9 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName9 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 07 Link Entity'},});
    
   // Execute the HTTP request
  http.get(`https://appau101mel1108.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName11 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName11 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName11 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName11 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName11 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 07 Link Entity'},});
    
    // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName13 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName13 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName13 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName13 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName13 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 07 Link Entity'},});
    
    // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName15 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName15 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName15 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName15 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName15 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 07 Link Entity'},});
  
    // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName17 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName17 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName17 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName17 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName17 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 07 Link Entity'},});
    
    	
    // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName19 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html`, {tags: { name: 'AppOrModuleName19 02 Load Beta Website'},});
 
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName19 03 Load Gamma Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName19 01 Load Alpha Website'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 04 Load Delta Website'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 05 Load Epsilon Website'},});
  
  // Execute the HTTP request
  http.get(`https://appau101mel1107.globaltest.blabla.com/websites/Google.html`, {tags: { name: 'AppOrModuleName19 07 Download File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 01 Load Login Page'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 02 Login'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 03 View Dashboard'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 04 Search'},});
   
    // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 04 Search'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 05 Modify Entity'},});
  
  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 06 Create New Entity'},});

  // Execute the HTTP request
  http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 07 Link Entity'},}); 

}

Full logs to relevant components:

Anything else we need to know:
@GiedriusS
Copy link
Member

Maybe you could try the version from main in your tests and check whether daa34a5 helps? 🤔

@yliu138repo
Copy link
Author

yliu138repo commented Apr 5, 2024 •
edited

@GiedriusS I have tried with main-2024-04-03-f80fd94, I am still seeing following error context deadline exceeded. I've also tried to configure otel collector timeout. Does it mean I will need to increase aysnc workers --receive.forward.async-workers=x with number of async workers higher than 5?

2024-04-05T04:19:49.099Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.       {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n", "errorCauses": [{"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n"}], "dropped_items": 461230}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GiedriusS @yliu138repo