Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add optional commit commit to kafka in s3 plugin #92

Open
ansakharov opened this issue May 23, 2022 · 2 comments
Open

Feature: Add optional commit commit to kafka in s3 plugin #92

ansakharov opened this issue May 23, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@ansakharov
Copy link
Contributor

ansakharov commented May 23, 2022

Issue
Current implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage.
Desired implementation: file plugin commits event after written to file, s3 plugin writes zip files to storage and commits metadata of upload to kafka.
Feature must be optional.

Solution
Add optional embedding kafka output plugin in s3 plugin and add commitment logic.
Afterwards commits to other sources will be provided.

@andrewmed
Copy link
Contributor

Why S3 plugin needs to write something to another destination? AFAIK convention is that there is one output plugin for one destination

If this PR is not a 2-phase commit we should not be using this term

@ansakharov ansakharov changed the title Feature: Add 2 phase commit to s3 plugin Feature: Add optional commit commit to kafka in s3 plugin May 30, 2022
@ansakharov
Copy link
Contributor Author

I removed 2pc term to avoid ambiguity.

@ansakharov ansakharov mentioned this issue Aug 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants