Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve MySQL replication processing #1382

Open
altmannmarcelo opened this issue Sep 26, 2024 · 0 comments
Open

Improve MySQL replication processing #1382

altmannmarcelo opened this issue Sep 26, 2024 · 0 comments
Assignees
Labels
Customer Created by Linear-GitHub Sync High priority Created by Linear-GitHub Sync
Milestone

Comments

@altmannmarcelo
Copy link
Contributor

Description

Our current MySQL replication flow can be improved to process replication events faster. I Propose we do the following:

  1. Group events from the same transaction and dispatch them as a batch to Readyset. Currently we read one event and dispatch a single event. We should take into consideration compressed events.
  2. Pass table_filter object to replicator and skip events as soon as we read them. Currently we build the whole TableOperation and only discard the event in the noria code.

Change in user-visible behavior

Requires documentation change

@altmannmarcelo altmannmarcelo added the Customer Created by Linear-GitHub Sync label Sep 26, 2024
@altmannmarcelo altmannmarcelo added this to the v.44 milestone Sep 26, 2024
@altmannmarcelo altmannmarcelo added the High priority Created by Linear-GitHub Sync label Sep 26, 2024
@altmannmarcelo altmannmarcelo self-assigned this Sep 26, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.44, v.45 Sep 30, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.45, v.46 Oct 7, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.46, v.47, v.48 Oct 14, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.48, v.49 Oct 28, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.49, v.50 Nov 4, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.50, v.51, v.52 Nov 11, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.52, v.53, v.54 Nov 25, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.54, v.55, v.56 Dec 9, 2024
@altmannmarcelo altmannmarcelo modified the milestones: v.56, v.57 Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Customer Created by Linear-GitHub Sync High priority Created by Linear-GitHub Sync
Projects
None yet
Development

No branches or pull requests

1 participant