MPI-SIM User's Guide

MPI-SIM provides direct execution-driven, parallel simulation for performance prediction of parallel computation-, communication-, and I/O-intensive programs written using the MPI 1.1 message-passing library. This document describes how to install and use the MPI-SIM software.
  1. You will want to download the MPI-SIM software. It is currently available as two separate packages for the IBM SP2 and the SGI Origin 2000. Due to irreconcilable differences between the libraries from these two vendors, we had no choice but to split the MPI-SIM distribution between these two. The package is available as a tar'ed, gzip'ed file.

  2. Un-tar the package to find a variety of C source files. A list of these files is available here.

  3. You will need to provide your own C program written with the MPI v1.1 library to simulate. Now comes the fun part. MPISIM simulates 24 core MPI library calls (a list of these), but currently we support only the Fortran bindings to MPI calls, not the C bindings. The crucial reason for this is that most of our MPI applications have been real-world scientific programs that were originally written in Fortran and translated to C via the program f2c. This translator converts all function calls into their Fortran calling-convention equivalents, namely all parameters are passed by reference. MPISIM thus uses MPI's Fortran calls as its entry API. Note further that the C and Fortran bindings have different parameters in many of the calls. For instance:

    • C calls:
      MPI_Init(int *argc, char ***argv);
      MPI_Comm_rank(MPI_Comm comm, int *rank)
      MPI_Send(char *buffer, int count, MPI_Datatype type, int recipient, int tag, MPI_Comm comm);

    • Fortran call bindings written in C (this is what MPISIM supports):
      mpi_init(int *err_val);
      mpi_comm_rank(MPI_Comm *comm, int *rank, int *err_val);
      mpi_send(char * buffer, int *count, MPI_Datatype *type, int *recipient, int *tag, MPI_Comm *comm, int *err_val);

    Futhermore, just to make things interesting, all mpi_* calls need to have two underscores appended to them (e.g. mpi_init__(), mpi_comm_size__()). Again, the reason for this is that f2c appends two underscores to all functions it comes across, so MPISIM's use of this form is an artifact of this necessity. A quick script is available here to do this final conversion.

  4. Rename your MPI program's main() function to be SimTargetProg() with no arguments. Command-line arguments to your program are available in the variable extern int parm[10].

  5. In order to have MPISIM produce its statistical output, your program must complete normally and call mpi_finalize__().

  6. MPISIM provides its own threading library and can simulate more virtual processors than your architecture has real processors. For example, your architecture may have only 8 processors, but MPISIM can simulate 256 processors for your program; that is, mpi_comm_size__() will return 256. However, our multithreading technique is subject to the need for localization. If, for now, you are not planning to run your MPI program with more simulated processors than real processors, you can skip to the next step. Otherwise, read on.

    Multiple copies of the same program can be concurrently run in the same address space without interference. Since multiple copies of a program are executed in the same address space using multi threading (this is the only case we handle), each copy has its own stack. Consequently, references to the same automatic variable in a function, by different threads, are translated to different addresses, i.e. on the referencing thread's very own very private stack. Hence automatic variables of different threads *never* interfere. However, "permanent" variables are not allocated on the stack, but in a common data area (the heap). Hence, a reference to the same permanent variable by different threads produces a reference to the very same physical address, and hence threads may interfere with each other.

    We have created a program we appropriately call the "localizer" to solve this problem by adding a dimension to each permanent variable (hence if it were a scalar (0 dimension), it now becomes a 1 dimensional array of size total-number-of- threads), so that each thread can use its id to reference the variable at its (the thread's) unique index. So, a reference to variable "a" should now become "a[MYID]", where MYID contains the thread's unique and unchanging id. MYID clearly has to be changed whenever a new thread is swapped in by the scheduler.

  7. Name your application as target.c and place it in the same directory as the distribution files. The Makefile has already been configured to compile target.c and link it with the other files.

  8. Compile-time options for simulation algorithms: MPI-SIM supports a variety of conservative simulation algorithms. You can set them with the SIMFLAGS variable in your Makefile if you want:

    # Compile Localized and Simulator ready file for Slow Simulation
    #SIMFLAGS = -DSLOW_SIMULATION

    # Compile Localized and Simulator ready file for Fast Simulation
    #SIMFLAGS = -DFAST_SIMULATION

    # Compile Localized and Simulator ready file for Super Slow Simulation
    # (Only conditional event algorithm, no null messages)
    #SIMFLAGS = -DSLOW_SIMULATION -DNULLIFY_NULL_MESSAGE_ALGORITHM

    # Compile Localized and Simulator ready file for Super Duper Slow Simulation
    # (Only null mesg, no conditional event algorithm)
    #SIMFLAGS = -DSLOW_SIMULATION -DNULLIFY_CONDITIONAL_EVENT_ALGORITHM

    #OPTS=-O2 -DPROGRAM_WILL_SWITCH_ON_INSTRUMENTATION
    OPTS=-O2

    CFLAGS = $(INCS) $(LIBS) $(OPTS) $(SIMFLAGS)

  9. Compile-time options for disk simulation: MPI-SIM also supports two methods to simulate disk usage in your program if you are using the MPI-IO extensions. In the Makefile, you can specify either -DUSE_FS_SIM or -DUSE_DISK_MODEL in the USES variable. The former will give a very detailed, albeit slow-executing, simulation of each MPIO call, while the latter uses a simpler, abstracted disk model.

  10. Execute make, and everthing will be compiled into the application target.

  11. To run the simulated program, the executable supports the following command line arguments:

    target < c > < i > < d > -ttp < your_args > -ttp
    where:

    • c is the number of virtual compute nodes your program will run on
    • i is the number virtual I/O nodes provided in the system
    • d is the number virtual disks provided in the system
    • your_args is the arguments provided to your program. (currently, they must be integer numbers)

    Note that your vendor's MPI implementation supports its own MPI command-line arguments. For instance, to run MPI programs with the implementation on the IBM SP, you will execute programs like so: a.out -procs n arguments. On the SGI Origin, the format will be: mpirun -np n a.out arguments. In both cases, n is the number of real physical processors the program will run on.

    Here are some examples of running an MPI-SIM simulation:

    • IBM: target -procs 4 4
      SGI: mpirun -np 4 target 4

      This will run the simulation on 4 physical processors while simulating 4 processors to your MPI program (i.e. mpi_comm_size will return 4).

    • IBM: target -procs 4 16
      SGI: mpirun -np 4 target 16

      This will run the simulation on 4 physical processors while simulating 16 processors to your MPI program (i.e. mpi_comm_size will return 16).

    • IBM: target -procs 4 16 8 4
      SGI: mpirun -np 4 target 16 8 4

      This will run the simulation on 4 physical processors while simulating 16 processors, 8 I/O nodes, and 4 disks to your MPI-IO program.

    • IBM: target -procs 4 16 -ttp 1 2 3 -ttp
      SGI: mpirun -np 4 target 16 -ttp 1 2 3 -ttp

      Same as the second above, while providing your program with the arguments 1, 2, and 3 in the int parm[] array.

  12. MPISIM will produce a wide variety of results upon completion of your MPI program. Please see the MPISIM output guide for a thorough explanation of the results.

Back to the MPI-SIM homepage.