-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
alt stage-out for unified queues #152
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… errors ending up in stderr
…cked and verified
PalNilsson
reviewed
Nov 22, 2024
@@ -1072,14 +1072,15 @@ class StageOutClient(StagingClient): | |||
|
|||
mode = "stage-out" | |||
|
|||
def prepare_destinations(self, files: list, activities: list or str) -> list: | |||
def prepare_destinations(self, files: list, activities: list or str, alt_exclude: list = []) -> list: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pylint will complain with "W0102: Dangerous default value [] as argument (dangerous-default-value)" here. I will change it to alt_exclude: list = None, and then add "if alt_exclude is None: alt_exclude = []"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enable alt stage-out for unified queues.
to be tested;
Few remarks:
unified
queue: Pilot makes decision to choose properdestination
(prepare_destinations
, in previous implementation Pilot silently followed PanDA decisions) by consideringwrite_lan_analysis
,write_lan
activities for analysis job, or simplywrite_lan
for production jobs.job.nucleus
is excluded as a possible alt-stageout destination;stageout.py
script (used to apply stageout in a container) has not been yet updated (since it's a bit weird because applies stage-out always towrite_lan
storage even forlog
data)P.S.
I used recent Paul's next branch as a base for updates to avoid conflicts, so that given MR includes also all other Paul's updates not related to alt stageout. Once Paul updates will be merged into pilot3/next, this MR should be auto cleaned.
Essential part of Pilot
stage-out
logic to determine destination (for review):unified queue
andanalysis
jobs it uses first defined entry in activities=[write_lan_analysis
,write_lan
, 'w'], in other cases relies on activitites=[write_lan
, 'w']astorages[activity][0]
and alt location ddm_alt=next(ddm) in corresponding astorages list (excludingnucleus
if any)astorages
(not a local storage) then Pilot will use given destination as alternative location if it's not ignored (nucleus), for primary desitnation pilot will use defaultddm
(astorages[0])