HPC workflow

HPC workflow in Pilot 2

The Pilot 2 HPC workflow is a special mode where the application works without a remote connection to PanDA server or other remote facilities. All intercommunications in this case are managed by the Harvester application. Also, in this mode Pilot 2 acts like a simple MPI application, which performs execution of multiple jobs on the computing nodes of the HPC.

How to launch HPC workflow

To launch the HPC workflow in Pilot 2 the command line parameter '-w generic_hpc' (Pilot option) should be specified.

HPC PlugIns

Different HPCs may require special treatments for the preparation of launching the payload. To cover this, implementations were placed in HPC specific plugins. The particular plugin should be specified using the command line parameter: '--hpc-resource' followed by the resource name.

Mandatory functions in the HPC Plugins

Get job

get_job(communication_point)

Retrieve job description and fill Job object (usually from a json file).

Set working directory for job

set_job_workdir(job, communication_point)

Set job/pilot working directory.

Get setup

get_setup()

Return the list of setup commands, which may be required by the infrastructure.

Set working directory for scratch space

set_scratch_workdir()

Setup the working directory on the transient high-speed storage (RAM disk, local SSD etc).

Payload command fix

command_fix()

Adapt some payload parameters for execution on the particular infrastructure.

Job report processing

process_jobreport()

Shrink the job report file and copy to the working directory from scratch if needed.

Post processing of working directory

postprocess_workdir()

Some post processing of the working directory (if needed).