HPC workflow

HPC workflow in Pilot 2

The Pilot 2 HPC workflow is a special mode where the application works without a remote connection to PanDA server or other remote facilities. All intercommunications in this case are managed by the Harvester application. Also, in this mode Pilot 2 acts like a simple MPI application, which performs execution of multiple jobs on the computing nodes of the HPC.

How to launch HPC workflow

To launch the HPC workflow in Pilot 2 the command line parameter '-w generic_hpc' (Pilot option) should be specified.

HPC PlugIns

Different HPCs may require special treatments for the preparation of launching the payload. To cover this, implementations were placed in HPC specific plugins. The particular plugin should be specified using the command line parameter: '--hpc-resource' followed by the resource name.

Mandatory functions in the HPC Plugins

get_job(communication_point) - retrieve job description and fill Job object (usually from a json file)

set_job_workdir(job, communication_point) - set job/pilot working directory

get_setup() - return the list of setup commands, which may be required by the infrastructure

set_scratch_workdir() - setup the working directory on the transient high-speed storage (RAM disk, local SSD etc)

command_fix() - adapt some payload parameters for execution on the particular infrastructure

process_jobreport() - shrink the job report file and copy to the working directory from scratch if needed

postprocess_workdir() - some post processing of the working directory (if needed)