You are in the main area:Organization
Headerimage for: MPI-Start Documentation

MPI-Start Documentation

Overview MPI-Start is a set of scripts that allow the execution of MPI programs on clusters. The main advantage of MPI-Start is the possibility to detect and use site-specific configuration features - like the batch scheduler and the file system on the site. Also, different MPI implementations on a site are supported. MPI-Start has a core which is always executed and uses the available frameworks at different stages of the execution. MPI-Start in its current version has three different frameworks:
  • scheduler framework
  • MPI framework
  • file distribution framework
All of those frameworks use well-defined interfaces which allow different plugins to be loaded and used for each framework.
Workflow The workflow of MPI-Start is as follows:
  • check for scheduler plugin
  • activate scheduler plugin
  • get machinefile
  • check for MPI plugin
  • check I2G/EGEE environment
  • parameter adjustments depending on scheduler/MPI
  • activate MPI plugin and set MPI implementation specific parameters
    • if necessary, set additional specific parameters for tool support
  • check user pre-run hooks
  • check file system
  • choose file distribution plugin if FS is non-shared
  • distribute files using the file distribution plugin
  • check for external MPI tools (runtime settings)
  • execute MPI application with generated command
  • check user post-run hook
  • check for external MPI tools (post-processing)
  • clean up files using the file distribution plugin
  • return the value of the mpirun/mpiexec command
Scheduler framework Every plugin for this framework defines a method scheduler_available() which returns 0 if and only if the scheduler is available on the site. A second method scheduler_get_machinefile() sets the variable MPI_START_MACHINEFILE to point to the machine file. The format of this file has to be generic and if necessary the plugin has to guarantee that. The assumption is that each site has exactly one scheduler. The first suitable scheduler is chosen. Supported schedulers:
  • SGE
  • PBS
  • LSF
MPI framework Every plugin for MPI has mpi_exec() and mpi_start() methods. mpi_start() is always called and calls a generic flow of running pre_run_hook/mpi_exec/post_run_hook. mpi_exec() is the MPI-implementation specific method of the plugin. It sets special parameters that are used to parametrize the generic_mpiexec() in the common /mpi directory. This plugin is *not* automatically detected. The user or the broker must set I2G_MPI_TYPE to the MPI plugin. Supported MPI implementations are:
  • Open MPI
  • MPICH
  • MPICH2
  • LAM-MPI
  • PACX-MPI as an MPI implementation on top of one of those implementations
File distribution framework Following file distribution mechanisms are currently supported:
  • If the current working directory is shared, no file distribution
  • passwordless SSH
  • copying to a shared directory different from the current working directory
  • using OSC mpiexec (see here)
  • mpi_mt - a tool for letting the MPI implementation distribute the files without SSH
Environment variables
  • I2G_MPI_APPLICATION The application binary to execute.
  • I2G_MPI_APPLICATION_ARGS The command line parameters for the application
  • I2G_MPI_TYPE The name of the MPI implementation to use. So far defined values:
    • openmpi
    • lam
    • mpich
    • mpich2
    • pacx
  • I2G_MPI_VERSION Specifies the version of the MPI implementation specified by I2G_MPI_TYPE. If not specified the default version will be used.
  • I2G_MPI_PRE_RUN_HOOK This variable can be set to a script which must define the pre_run_hook function. This function will be called after the MPI support has been established and before the internal pre-run hooks. This hook can be used to prepare input data or compile the program.
  • I2G_MPI_POST_RUN_HOOK This variable can be set to a script which must define the post_run_hook function. This function will be called after the mpirun has finished.
  • I2G_MPI_PRECOMMAND This variable allows the run mpirun "inside" another program.
    The resulting command is : $I2G_MPI_PRECOMMAND mpirun/mpiexec ...
  • I2G_MPI_FLAVOUR Specifies which "sub-mpi" to use. In the case of a PACX-MPI jobs this variable specifies the local MPI implementation to use. Else this variable is equal to I2G_MPI_TYPE.
  • I2G_MPI_JOB_NUMBER If a MPI Job runs across multiple clusters this variable specifies which sub-job should be started on this cluster. The values are 0,1,2,... . In the case of a MPI job that runs only inside the local cluster this variable is always 0.
  • I2G_MPI_STARTUP_INFO This variables provides additional information for the start of the MPI programm. In the case of the PACX-MPI this variable specifies the connection information to the startup server (FQDN:PORT).
  • I2G_USE_MARMOT Activate Marmot log file generation, only tested in I2G
  • I2G_USE_MPITRACE Activate MPITrace trace file generation, only tested in I2G
Debugging For debugging purpose the I2G_MPI_START_DEBUG variable can be set to 1 to enable the debugging output. The I2G_MPI_START_VERBOSE variable can be set to 1 to turn on the additional output. The variable I2G_MPI_START_TRACE can be set to 1 to trace every operation that is performed by mpi-start (goes to stdout).