Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use invalidations for performing check for intersecting periods being processed #22394

Open
wants to merge 5 commits into
base: 5.x-dev
Choose a base branch
from

Conversation

sgiehl
Copy link
Member

@sgiehl sgiehl commented Jul 12, 2024

Description:

When archiving is running, we are basically processing invalidations that were previously created.
To ensure that every processed invalidation already has all required data available, we perform certain checks to avoid that e.g. a week archive is processed when a day archives of that week is still processed. Similar checks are applied for segment processing.

Currently the check for other archiving jobs that might process an invalidation for an intersecting period (or segment) is performed by checking the running multicli processes. This might work fine if a single machine is performing the archiving. But as soon as there are multiple servers performing the archiving, this detection would fail, causing intersecting periods being archived in parallel.

By looking into the invalidation table instead of checking running processes, we can ensure that running jobs are detected across multiple machines.

It though carries the risk of making detection of failed archiving jobs bit worse. We will work on further improvements for this separately.

Refs DEV-18290

Review

@sgiehl sgiehl force-pushed the fixintersectingcheck branch 6 times, most recently from 5022d59 to 9178d93 Compare July 15, 2024 15:22
@sgiehl sgiehl added the Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. label Jul 15, 2024
@sgiehl sgiehl added this to the 5.2.0 milestone Jul 15, 2024
@sgiehl sgiehl added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Jul 15, 2024
@sgiehl sgiehl marked this pull request as ready for review July 15, 2024 16:03
@sgiehl sgiehl added the Needs Review PRs that need a code review label Jul 16, 2024
@sgiehl sgiehl requested a review from a team July 16, 2024 12:34
@sgiehl sgiehl force-pushed the inprogressinvalidation branch from 90808ff to 7806cd6 Compare July 16, 2024 13:01
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from acb5b40 to a516546 Compare July 16, 2024 13:09
Copy link
Contributor

@michalkleiner michalkleiner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks ok, but would prefer another pair of 👀 to have a look as well.

caddoo
caddoo previously approved these changes Jul 19, 2024
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from 7728544 to 5311faf Compare July 19, 2024 10:34
Base automatically changed from inprogressinvalidation to 5.x-dev July 19, 2024 13:54
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from 5311faf to 607f032 Compare July 19, 2024 13:58
michalkleiner
michalkleiner previously approved these changes Jul 22, 2024
@sgiehl sgiehl force-pushed the fixintersectingcheck branch 2 times, most recently from 60902ed to 7fd600b Compare July 28, 2024 15:56
Copy link
Contributor

github-actions bot commented Aug 5, 2024

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions github-actions bot added the Stale The label used by the Close Stale Issues action label Aug 5, 2024
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from 7fd600b to e1fed6a Compare August 5, 2024 08:36
@sgiehl sgiehl dismissed stale reviews from michalkleiner and caddoo via e1fed6a August 9, 2024 03:22
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from d88dcc7 to f27a0fb Compare August 19, 2024 18:14
@sgiehl sgiehl removed the Needs Review PRs that need a code review label Sep 3, 2024
Copy link
Contributor

This PR was last updated more than one month ago, maybe it's time to close it. Please check if there is anything we still can do or close this PR. ping @matomo-org/core-reviewers

@github-actions github-actions bot added the Stale for long The label used by the Close Stale Issues action label Oct 16, 2024
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from 4e261e3 to 81ebc39 Compare October 25, 2024 08:52
@sgiehl sgiehl force-pushed the fixintersectingcheck branch from 81ebc39 to 2d18aeb Compare October 25, 2024 09:23
@sgiehl sgiehl modified the milestones: 5.2.0, 5.3.0 Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Stale for long The label used by the Close Stale Issues action Stale The label used by the Close Stale Issues action
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants