Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mito): Stall write requests and add more flush tests #2322

Merged
merged 14 commits into from
Sep 6, 2023

Conversation

evenyag
Copy link
Contributor

@evenyag evenyag commented Sep 4, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements the following feature for flush

  • Rejecting write requests
    • reject_write_requests()
    • should_reject_write()
  • Stalling write requests if WriteBufferManager::should_stall() returns true
    • StalledRequests holds stalled write requests
    • Once a flush job is finished, stalled requests can proceed. (In handle_flush_finished())

It also refactors tests and adds more test cases:

  • It moves common functions to test_util.rs.
    • column_metadata_to_column_schema()
    • build_rows()
    • rows_schema()
    • delete_rows_schema()
    • put_rows()
    • build_rows_for_key()
    • build_delete_rows_for_key()
    • delete_rows()
  • It moves flush tests to flush_test.rs.
  • It adds an EventListener to listen to flush and stall events.
  • Adds methods to set write buffer manager and event listener for the engine

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@evenyag evenyag marked this pull request as ready for review September 4, 2023 09:34
@codecov
Copy link

codecov bot commented Sep 4, 2023

Codecov Report

❗ No coverage uploaded for pull request base (migrate-region-server@854033d). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files
@@                   Coverage Diff                    @@
##             migrate-region-server    #2322   +/-   ##
========================================================
  Coverage                         ?   78.51%           
========================================================
  Files                            ?      726           
  Lines                            ?   117808           
  Branches                         ?        0           
========================================================
  Hits                             ?    92494           
  Misses                           ?    25314           
  Partials                         ?        0           

@evenyag evenyag requested a review from v0y4g3r September 6, 2023 02:50
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@evenyag evenyag added this pull request to the merge queue Sep 6, 2023
Merged via the queue into GreptimeTeam:migrate-region-server with commit 9dc3748 Sep 6, 2023
11 checks passed
@evenyag evenyag deleted the feat/mito2-stall branch September 6, 2023 08:08
waynexia pushed a commit that referenced this pull request Sep 12, 2023
* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
waynexia pushed a commit that referenced this pull request Sep 12, 2023
* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
waynexia pushed a commit that referenced this pull request Sep 12, 2023
* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Sep 13, 2023
…am#2322)

* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
…am#2322)

* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants