Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alt stage-out for unified queues #152

Merged
merged 28 commits into from
Nov 22, 2024

Conversation

anisyonk
Copy link
Contributor

@anisyonk anisyonk commented Nov 8, 2024

Enable alt stage-out for unified queues.

to be tested;

Few remarks:

  • For unified queue: Pilot makes decision to choose proper destination (prepare_destinations, in previous implementation Pilot silently followed PanDA decisions) by considering write_lan_analysis, write_lan activities for analysis job, or simply write_lan for production jobs.
  • job.nucleus is excluded as a possible alt-stageout destination;
  • the logic of standalone stageout.py script (used to apply stageout in a container) has not been yet updated (since it's a bit weird because applies stage-out always to write_lan storage even for log data)

P.S.
I used recent Paul's next branch as a base for updates to avoid conflicts, so that given MR includes also all other Paul's updates not related to alt stageout. Once Paul updates will be merged into pilot3/next, this MR should be auto cleaned.

Essential part of Pilot stage-out logic to determine destination (for review):

  • Pilot considers allowed destination storages by checking appropriate activities; for unified queue and analysis jobs it uses first defined entry in activities=[write_lan_analysis,write_lan, 'w'], in other cases relies on activitites=[write_lan, 'w']
  • if job does not specify default destination for a file, then Pilot will consider primary destination as ddm=astorages[activity][0] and alt location ddm_alt=next(ddm) in corresponding astorages list (excluding nucleus if any)
  • if job requests file destination which is not in astorages (not a local storage) then Pilot will use given destination as alternative location if it's not ignored (nucleus), for primary desitnation pilot will use default ddm (astorages[0])
  • Otherwise (when job requests local destination) Pilot will respect job preferences and use it as primary destination and choose alternative location as next(primary) in cycled manner of astorages skipping any ignored destinations (nucleus)

@@ -1072,14 +1072,15 @@ class StageOutClient(StagingClient):

mode = "stage-out"

def prepare_destinations(self, files: list, activities: list or str) -> list:
def prepare_destinations(self, files: list, activities: list or str, alt_exclude: list = []) -> list:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pylint will complain with "W0102: Dangerous default value [] as argument (dangerous-default-value)" here. I will change it to alt_exclude: list = None, and then add "if alt_exclude is None: alt_exclude = []"

@PalNilsson PalNilsson merged commit 3440240 into PanDAWMS:next Nov 22, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants