-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: Add optional commit commit to kafka in s3 plugin #92
Labels
enhancement
New feature or request
Comments
Why S3 plugin needs to write something to another destination? AFAIK convention is that there is one output plugin for one destination If this PR is not a 2-phase commit we should not be using this term |
ansakharov
changed the title
Feature: Add 2 phase commit to s3 plugin
Feature: Add optional commit commit to kafka in s3 plugin
May 30, 2022
I removed |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue
Current implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage.
Desired implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage and commits metadata of upload to kafka.
Feature must be optional.
Solution
Add optional embedding kafka output plugin in s3 plugin and add commitment logic.
Afterwards commits to other sources will be provided.
The text was updated successfully, but these errors were encountered: