New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Ingestion TaskQueue #3136
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
taimingl
force-pushed
the
feat/input-buffer
branch
from
April 2, 2024 23:13
2ebee7c
to
153939c
Compare
…Queues for TaskQueueManager
…e reaches max number of workers allowed
…estable before enqueued into ingest buffer
…configure ingest buffer
- Memory: add `ZO_INGEST_BUFFER_QUEUE_CNT` to configure max_worker_count, which limits total memory usage - Latency: added an estimated running average request size that is used to estimate wait time to ensure latency is within the constraint
add current time as timestamp value for buffered ingestion requests. This value will be used when request without timestamp in its payload is being ingested. This timestamp also helps maintain the order of the logs.
compress/decompress serialized/deserialized data to/from wal when persisting requests to disk.
/ok-to-test sha=578578a |
taimingl
force-pushed
the
feat/input-buffer
branch
from
May 13, 2024 22:51
f6d5824
to
4b4608f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
impl #3099
Add a TaskQueue that serves as IngestBuffer to improve ingestion endpoints responsiveness in the cases where ingestion request spikes.
Basic flow:
Producers, ingestion endpoint handlers,
/_json
or/_bulk
Consumers, async workers, managed by TaskQueue
Components
TaskQueue
: mpmc of a single channel implemented with 'async-channels'Main methods:
Configured by:
DEFAULT_CHANNEL_CAP
: bounds channel capacityDEFAULT_WORKER_CNT
: default initial & incremental workers countSEARCHABLE_LATENCY
: acceptable latency between a request is accepted and searchableWorkers
: consumer side of the TaskQueueMain methods:
Configured by:
WORKER_DEFAULT_WAIT_TIME
: default wait time between each pull from channelWORKER_BATCH_PROCESSING_SIZE
: max number of requests a worker processes in one batchWORKER_MAX_IDLE_TIME
: max idle time in seconds before a worker shuts itself downOther util functions: