Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: update retry loop #2032

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

chore: update retry loop #2032

wants to merge 3 commits into from

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented Sep 5, 2024

No description provided.

@kohlisid
chore: update retry loop
b06944c 
Signed-off-by: Sidhant Kohli <[email protected]>
@codecov Codecov
Copy link

codecov bot commented Sep 5, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 94.52055% with 8 lines in your changes missing coverage. Please review.

Project coverage is 61.95%. Comparing base (9f13068) to head (dc5608b).

Files with missing lines Patch % Lines
rust/monovertex/src/forwarder.rs 94.52% 8 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2032      +/-   ##
==========================================
+ Coverage   61.78%   61.95%   +0.17%     
==========================================
  Files         318      318              
  Lines       28977    29075      +98     
==========================================
+ Hits        17902    18013     +111     
+ Misses      10126    10105      -21     
- Partials      949      957       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kohlisid
Copy link
Contributor Author

kohlisid commented Sep 5, 2024 

    retryStrategy:
     backoff:
       interval: "1s"
       steps: 10
     onFailure: "fallback"

2024-09-05T22:42:29.528225Z  WARN monovertex::forwarder: Retry attempt 1 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:30.533403Z  WARN monovertex::forwarder: Retry attempt 2 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:31.305908Z  INFO monovertex::metrics: Pending messages (1m): 0
2024-09-05T22:42:31.305922Z  INFO monovertex::metrics: Pending messages (default): 0
2024-09-05T22:42:31.305925Z  INFO monovertex::metrics: Pending messages (5m): 0
2024-09-05T22:42:31.305927Z  INFO monovertex::metrics: Pending messages (15m): 0
2024-09-05T22:42:31.537288Z  WARN monovertex::forwarder: Retry attempt 3 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:32.540987Z  WARN monovertex::forwarder: Retry attempt 4 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:33.545641Z  WARN monovertex::forwarder: Retry attempt 5 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:34.304933Z  INFO monovertex::metrics: Pending messages (1m): 0
2024-09-05T22:42:34.304945Z  INFO monovertex::metrics: Pending messages (default): 0
2024-09-05T22:42:34.304947Z  INFO monovertex::metrics: Pending messages (5m): 0
2024-09-05T22:42:34.304949Z  INFO monovertex::metrics: Pending messages (15m): 0
2024-09-05T22:42:34.549736Z  WARN monovertex::forwarder: Retry attempt 6 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:35.552577Z  WARN monovertex::forwarder: Retry attempt 7 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:36.556006Z  WARN monovertex::forwarder: Retry attempt 8 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:37.304488Z  INFO monovertex::metrics: Pending messages (1m): 0
2024-09-05T22:42:37.304506Z  INFO monovertex::metrics: Pending messages (default): 0
2024-09-05T22:42:37.304508Z  INFO monovertex::metrics: Pending messages (5m): 0
2024-09-05T22:42:37.304510Z  INFO monovertex::metrics: Pending messages (15m): 0
2024-09-05T22:42:37.561006Z  WARN monovertex::forwarder: Retry attempt 9 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:38.565121Z  WARN monovertex::forwarder: Retry attempt 10 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:38.565141Z  WARN monovertex::forwarder: Moving messages to fallback after 10 attempts. Errors: {"Pushing to error": 500}    
2024-09-05T22:42:38.610130Z  INFO monovertex::forwarder: Forwarded 500 messages at time 2024-09-05 22:42:38.610127496 UTC
2024-09-05T22:42:39.734909Z  WARN monovertex::forwarder: Retry attempt 1 due to retryable error. Errors: {"Pushing to error": 500}

@kohlisid
Copy link
Contributor Author

kohlisid commented Sep 5, 2024 

   sink:
    retryStrategy:
     backoff:
       interval: "1s"
       steps: 10
     onFailure: "drop"

2024-09-05T22:57:08.175000Z  INFO monovertex::metrics: Pending messages (5m): 0
2024-09-05T22:57:08.175003Z  INFO monovertex::metrics: Pending messages (15m): 0
2024-09-05T22:57:08.737880Z  WARN monovertex::forwarder: Retry attempt 7 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:57:09.742265Z  WARN monovertex::forwarder: Retry attempt 8 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:57:10.746158Z  WARN monovertex::forwarder: Retry attempt 9 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:57:11.174678Z  INFO monovertex::metrics: Pending messages (1m): 0
2024-09-05T22:57:11.174695Z  INFO monovertex::metrics: Pending messages (default): 0
2024-09-05T22:57:11.174698Z  INFO monovertex::metrics: Pending messages (5m): 0
2024-09-05T22:57:11.174700Z  INFO monovertex::metrics: Pending messages (15m): 0
2024-09-05T22:57:11.750976Z  WARN monovertex::forwarder: Retry attempt 10 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:57:11.750993Z  WARN monovertex::forwarder: Dropping messages after 10 attempts. Errors: {"Pushing to error": 500}

@kohlisid
Copy link
Contributor Author

kohlisid commented Sep 5, 2024 

  sink:
    retryStrategy:
     backoff:
       interval: "1s"
       steps: 10

2024-09-05T22:58:53.289889Z  WARN monovertex::forwarder: Retry attempt 16 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:54.295514Z  WARN monovertex::forwarder: Retry attempt 17 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:55.299598Z  WARN monovertex::forwarder: Retry attempt 18 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:56.302252Z  WARN monovertex::forwarder: Retry attempt 19 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:57.306426Z  WARN monovertex::forwarder: Retry attempt 20 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:58.311239Z  WARN monovertex::forwarder: Retry attempt 21 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:58:59.315561Z  WARN monovertex::forwarder: Retry attempt 22 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:00.318776Z  WARN monovertex::forwarder: Retry attempt 23 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:01.323207Z  WARN monovertex::forwarder: Retry attempt 24 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:02.326654Z  WARN monovertex::forwarder: Retry attempt 25 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:03.330671Z  WARN monovertex::forwarder: Retry attempt 26 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:04.334063Z  WARN monovertex::forwarder: Retry attempt 27 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:05.337481Z  WARN monovertex::forwarder: Retry attempt 28 due to retryable error. Errors: {"Pushing to error": 500}    
2024-09-05T22:59:06.341816Z  WARN monovertex::forwarder: Retry attempt 29 due to retryable error. Errors: {"Pushing to error": 500}

vigith
vigith reviewed Sep 6, 2024
View reviewed changes
rust/monovertex/src/forwarder.rs
@@ -223,50 +224,52 @@ impl Forwarder {
// we will overwrite this vec with failed messages and will keep retrying.
let mut messages_to_send = messages;

// check what is the failure strategy in the config
let strategy = config().sink_retry_on_fail_strategy.clone();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not clone in new since it is literally one time and can be reused?

rust/monovertex/src/forwarder.rs
&mut fallback_msgs,
&mut messages_to_send,
strategy,
self.common_labels.clone(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not pass reference?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigith vigith vigith left review comments

@whynowy whynowy Awaiting requested review from whynowy whynowy will be requested when the pull request is marked ready for review whynowy is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kohlisid @vigith