                      MPE (Multi-Processing Environment)
                      ----------------------------------

                          Version 1.9.2. July, 2004

                    Mathematics and Computer Science Division
                           Argonne National Laboratory

I.  INTRODUCTION
----------------

The Multi-Processing Environment (MPE) attempts to provide programmers with 
a complete suite of performance analysis tools for their MPI programs based
on post processing approach.  These tools include a set of profiling libraries, 
a set of utility programs, and a set of graphical tools.

The first set of tools to be used with user MPI programs is profiling libraries
which provide a collection of routines that create log files.  These log files
can be created manually by inserting MPE calls in the MPI program, or 
automatically by linking with the appropriate MPE libraries, or by combining 
the above two methods.  Currently, the MPE offers the following 3 profiling 
libraries.

   1) Tracing Library - Traces all MPI calls.  Each MPI call is preceded by a 
      line that contains the rank in MPI_COMM_WORLD of the calling process, 
      and followed by another line indicating that the call has completed.
      Most send and receive routines also indicate the values of count, tag,
      and partner (destination for sends, source for receives).  Output is to
      standard output.

   2) Animation Libraries - A simple form of real-time program animation 
      that requires X window routines.  

   3) Logging Libraries - The most useful and widely used profiling libraries 
      in MPE.  These libraries form the basis to generate log files from 
      user MPI programs.  There are several different log file formats
      available in MPE.  The default log file format is CLOG.  It is a low
      overhead logging format, a simple collection of single timestamp events.
      ALOG, which is provided for backward compatibility reason, is not
      being developed, users are strongly urged not to use ALOG if
      possible.  SLOG-2, which stands for Scalable LOGfile format version II,
      is a total redesign of the original SLOG format.  SLOG-2 allows for 
      much improved scalability for visualization purpose.  CLOG file can be
      easily converted to SLOG-2 file through the new SLOG-2 viewer,
      Jumpshot-4.

The set of utility programs in MPE includes log format converter (e.g. 
clogTOslog2), logfile print (e.g. slog2print) and logfile viewer and
convertor (e.g. jumpshot).  These new tools, clogTOslog2, slog2print and
jumpshot(Jumpshot-4) replace old tools, clog2slog, slog_print and logviewer
(i.e. Jumpshot-2 and Jumpshot-3).  For more information of various
logfile formats and their viewers, see 

http://www.mcs.anl.gov/perfvis



II. CONFIGURATION
-----------------

MPE can be configured and installed as an extension to most MPI standard
compliant MPI implementations, e.g. MPICH-2, MPICH, LAM, SGI's MPI,
HP-UX's MPI and IBM's MPI.  It has been integrated seamlessly into 
MPICH and MPICH-2 distributions, so MPE will be installed automatically
during MPICH-X's installation process.  



II. a)  Configuration Model
---------------------------

MPE is designed to be used as an extension to an existing MPI implementation,
so its configuration model assumes a general MPI development environment.  
Here are the some of the variables that MPE configure reads, some are read 
as environmental variables and some are read from the command line arguments 
to configure.

CC          : C compiler used to create serial executable,
              e.g. xlc for IBM MPI.
MPI_CC      : C compiler used to compile MPI program and to create parallel 
              executable,
              e.g. mpcc for IBM MPI, or mpicc for MPICH.
MPE_CFLAGS  : CFLAGS for CC and MPI_CC.

F77         : F77 compiler used to create serial executable,
              e.g. xlf for IBM MPI.
MPI_F77     : F77 compiler used to compile MPI program and to create parallel
              executable,
              e.g. mpxlf for IBM MPI, or mpif77 for MPICH.
MPE_FFLAGS  : FFLAGS for F77 and MPI_F77.

MPI_INC     : compiler's include flag (with prefix "-I") for MPI_CC/MPI_F77, 
              e.g. "-I/usr/include" for mpi.h on IRIX64.
MPI_LIBS    : compiler's library flag (with prefix "-L" for library path and
              prefix "-l" for each library name) needed by MPI_CC/MPI_F77,
              e.g. "-L/usr/lib -lmpi" for libmpi.a on IRIX64.
F2CMPI_LIBS : compiler's library flag for Fortran to C MPI profiling wrapper
              library, e.g. "-lfmpich" when MPI_CC=mpicc & MPI_F77=mpif77
              for MPICH.

Among above listed variables, CC, MPI_CC, F77 and MPI_F77 are usually set
by the corresponding environmental variables.  The rest can be set through 
command line arguments to configure.  In some MPI implementations, like 
HP-UX's, MPI_CC and MPI_F77 are reserved for use by the MPI implementation,
use the configure options to set MPI_CC and MPI_F77 instead.

For those who requires cross-compilation support, there are 5 or 8 
more environmental variables are needed depends on if one needs to use the
MPE's internal Fortran to C MPI profiling wrapper library.  If your MPI
implementation does NOT provide any Fortran to C MPI profiling wrapper
library, then you need to specify at least following 5 environmental
variables regarding the properties of running environment in the executable
platform.

CROSS_SIZEOF_CHAR      : sizeof(char), e.g. 1
CROSS_SIZEOF_SHORT     : sizeof(short), e.g. 2
CROSS_SIZEOF_INT       : sizeof(int), e.g. 4
CROSS_SIZEOF_LONG_LONG : sizeof(long long), e.g. 8
CROSS_BIGENDIAN        : true if it is a big endian machine, false otherwise.

The following 3 environmental variables specify properties of MPI
implementation if F2CMPI_LIBS is defined, i.e MPE's internal Fortran
to C MPI profiling library is used.

CROSS_MPI_STATUS_SIZE  : value of MPI_STATUS_SIZE defined in mpif.h, 
                         i.e. sizeof(MPI_STATUS_SIZE)/sizeof(MPI_Fint), e.g. 4
CROSS_FORTRAN2C_TRUE   : value of fortran's .TRUE. in C program, e.g. 1
CROSS_FORTRAN2C_FALSE  : value of fortran's .FALSE. in C program, e.g. 0

II. b)  Build Options and Features
----------------------------------

MPE's configure is written using autoconf 2, and supports VPATH style install  
process.  It means the actual source directory and the building directory
can be in 2 different locations.  This allows the same source directory to be
used to build multiple versions of MPE with different options and still won't 
mess up the original source.  It is highly recommended that user should do a 
VPATH build.  Also MPE involves several different independent packages, in 
order to create a tightly integrated environment for user, it is recommended 
that user should do a "make install" to install the MPE in a separate directory
after the build is done.  The benefit is that all utility programs will be in
bin/, all libraries will be in lib/ and all graphic tools will be nicely 
organized in share/ ...

There are 2 types of configure options.

	1)  MPI implementation and User options
	2)  Generic configure flags supplied by autoconf 2

For a list of flags/switches for type 1 (not type 2) in MPE, use the script
"configure--help".

The following is not a complete list but some of the more important ones.
Generic flags:
--prefix=INSTALL_DIR            Specifies the final install directory for 
                                "make install".  All libraries, utility
                                programs, graphic programs and examples
                                are installed in a standard directory
                                structure without files created in the
                                building process.

--x-includes=X_INC              Specifies the directory where X include
                                files are located.  This is used when
                                configure has trouble in locating X in
                                user system.

--x-libraries=X_LIBS            Specifies the directory where X libraries
                                are located.  This is used when configure
                                has trouble in locating X in user system.
	

MPI implementation Options:
Besides the following command line options, the serial C and Fortran compilers
can be specified through typical environmental variables CC and F77.

--with-mpicc=MPI_CC             Specify MPI C compiler to generate parallel
                                executable, e.g. mpcc for AIX.  This can be
                                specified through environmental parameter
                                MPI_CC.

--with-mpif77=MPI_F77           Specify MPI F77 compiler to generate parallel
                                executable, e.g. mpxlf for AIX.  This can be
                                specified through environmental parameter
                                MPI_F77.

--with-cflags=MPE_CFLAGS        Specify extra CFLAGS to the C and MPI_CC 
                                compilers, e.g. "-64" for IRIX64 C compiler

--with-fflags=MPE_FFLAGS        Specify extra FFLAGS to the F77 and MPI_F77
                                compilers, e.g. "-64" for IRIX64 F77 compiler

--with-mpiinc=MPI_INC           Specify compiler's include flag for MPI 
                                include directory,
                                e.g. "-I/pkgs/MPI/include" for mpi.h 

--with-mpilibs=MPI_LIBS         Specify compiler's library flag for MPI 
                                libraries,
                                e.g. "-L/pkgs/MPI/lib -lpmpich -lmpich"

--enable-f77                    Enable the compilation of routines that 
                                require a Fortran compiler.  If configuring 
                                with MPICH, the configure in the top-level 
                                MPICH directory will choose the appropriate 
                                value for you.  However, it can be overridden.
                                The default is yes, --enable-f77.

--enable-f2cmpilib              Enable the building of MPE's internal Fortran 
                                to C MPI wrapper library.  The default is yes,
                                --enable-f2cmpilib

--with-f2cmpilibs=F2CMPI_LIBS   Specify compiler's library flags for Fortran
                                to C MPI wrapper library.  Using this option 
                                will force --disable-f2cmpilib.
                                e.g. "-lfmpich" when configuring MPE for MPICH

Other User Options:
--enable-echo                   Turn on strong echoing.  The default is no,
                                --disable-echo.

--with-mpelibname=MPE_LIBNAME   Specify the MPE library name instead of the 
                                default 'mpe'.  e.g. if MPE_LIBNAME="MPE",
                                then the libraries generated will be libMPE.a,
                                liblMPE.a, libtMPE.a, libaMPE.a and
                                libMPE_f2cmpi.a.  This option is necessary
                                when configuring MPE for a existing and older
                                version of MPICH which has MPE installed.

--enable-mpe_graphics           Enable the building of MPE graphics routines.
                                If disabled, then the MPE routines that make 
                                use of X11 graphics will not be built.  This 
                                is appropriate for systems that either do not 
                                have the X11 include files or that do not 
                                support X11 graphics.  The default is 
                                enable=yes.

--enable-viewers                Enable the build of all the available log 
                                viewers.  The default is enable=yes

--with-java=JAVA_HOME           Specify the path of the top-level directory 
                                of the Java, JDK, installation.  If this 
                                option or --with-javaX is not given, configure
                                will try to locate JDK for you to build 
                                Jumpshot-2 and Jumpshot-3.  JDK 1.1.6 to 
                                JDK 1.1.8 can be used to build both Jumpshots.
                                It is recommended that user should use this 
                                option when there is only 1 version of JDK 
                                installed on the machine.

--with-java1=JAVA_HOME          Specify the path of the top-level directory
                                of the Java, JDK, installation for Jumpshot-2
                                only.  If this option or --with-java is not 
                                given, Jumpshot-2's configure will try to 
                                locate JDK for you to build Jumpshot-2.  For
                                performance reason, it is recommended to use
                                the latest Java 1, i.e. JDK-1.1.8, to build
                                Jumpshot-2.

--with-java2=JAVA_HOME          Specify the path of the top-level directory
                                of the Java, JDK, installation for Jumpshot-3
                                only.  If this option or --with-java is not 
                                given, Jumpshot-3's configure will try to 
                                locate JDK for you to build Jumpshot-3.  For
                                performance reason, it is recommended to use
                                the latest Java 2, i.e. JDK-1.4.x, to build
                                Jumpshot-3.

--with-wishloc=WISHLOC          This switch specifies the name of Tcl/Tk wish
                                executable.  If this switch is omitted, 
                                configure will attempt to locate a version.  
                                This is used only for upshot.  
                                Note: Because Tcl and Tk keep changing in 
                                incompatible ways, we will soon be dropping 
                                support for any tool that uses Tcl/Tk.

In order to achieve maximum performance benefit, it is recommended to
use latest Java 1, i.e. JDK-1.1.8, as well as latest Java 2, i.e. JDK-1.4.x
to build Jumpshots.  So instead of using --with-java=<JDK-1.1.8> option, 
--with-java1=<JDK-1.1.8> and --with=java2=<JDK-1.4.x> should be used instead.




III. INSTALLATION INSTRUCTIONS
-------------------------------  

As noted earlier, the MPE library can be installed as part of the MPICH
configure or as an extension of an existing MPI implementation.  Below are
instructions and examples for typical installation of MPE on popular MPI 
implementations. 



III. a)  Configuring as part of the MPICH-x configure
---------------------------------------------------

The configure in the MPICH directory will try to determine the necessary 
information and pass it to the MPE configure.  If no options are given, 
the MPE will automatically be configured by default.  However, the user 
can provide extra configuration information to MPE through MPICH 
configure with the following options:

	-mpe_opts=MPE_OPTS

where MPE_OPTS is one or more of the choices in section II.  Multiple
instances of -mpe_opts are allowed to specify different options for 
the MPE configure.

The MPICH-2 configure works seamlessly with MPE's, so MPE configure
options can be passed down from MPICH-2 without -mpe_opts.


III. b) Configuring as part of an existing MPI implementation
-------------------------------------------------------------

The following are some examples for configuring MPE for an existing
MPI implementation.

For SGI MPI, e.g. denali.mcs.anl.gov, do the following for default ABI, -n32

    setenv MAKE gmake
    ${MPE_SRC_DIR}/configure --with-mpilibs=-lmpi \
                             --with-java=/usr/java-1.1.6/usr/java
    make
    make install PREFIX=${MPE_INSTALL_DIR}

for 64 bits ABI, add options --with-cflags=-64 and --with-fflags=-64 to the
configure options.


For IBM MPI, e.g. quad.mcs.anl.gov, do

    setenv MPI_CC mpcc
    setenv MPI_F77 mpxlf
    ${MPE_SRC_DIR}/configure --with-java=/homes/chan/pkgs/java/J1.1.8
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For SUN MPI, do

    setenv MPI_CC mpcc
    setenv MPI_F77 mpf90
    ${MPE_SRC_DIR}/configure --with-mpilibs=-lmpi
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For HP-UX's MPI implementation,

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L"
    make
    make install PREFIX=${MPE_INSTALL_DIR}


MPE's Fortran support on HP-UX's MPI is NOT working yet.  So to get MPI 
Fortran code to generate logfile, you could use HP-UX's libfmpi.a if it
is there.  Here is the configure options.

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L" \
                             --with-f2cmpilibs=-lfmpi
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For LAM, do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-java1=/sandbox/jdk117_v3 \
                             --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}

Using MPE with LAM for fortran MPI program is not working until recently.  
Configure options listed above enable MPE's internal Fortran to C MPI library.
To use LAM's Fortran to C MPI library in LAM 6.3.3 or later, liblamf77mpi.a, 
do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-f2cmpilibs=-llamf77mpi \
                             --with-java1=/sandbox/jdk117_v3 \
                             --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}

LAM 6.5.6 to 6.5.9 has a bug that interferes with MPE's configure and make
in enabling byte swapping on little endian machine, e.g. intel box.  For 
details, see

http://www.lam-mpi.org/MailArchives/lam/msg04894.php
http://www.lam-mpi.org/MailArchives/lam/msg04896.php

Solution: Upgrade to newer version of LAM.

LAM 7.0 has included libpmpi.a into libmpi.a, so --with-mpilibs may not be
needed.


For prebuilt version of MPICH-1.1.2 or MPICH-1.2.0, e.g. donner, do

    setenv MPI_CC ${MPICH_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${MPICH_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure  --with-f2cmpilibs=-lfmpich \
                              --with-mpelibname=newMPE \
                              --with-java1=/sandbox/jdk117_v3 \
                              --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}

It is important to use the configure option --with-mpelibname to specify a 
different MPE library name than the default "mpe" when configuring MPE for
older MPICH.  Without this option, the linkage tests in MPE would most likely
use the old MPE libraries in the MPICH instead of the newly built MPE 
libraries in resolving the MPE symbols.  Also the option --with-f2cmpilibs
forces MPE to use the Fortran to C MPI wrapper library in previous
version of MPICH.  Not using --with-f2cmpilibs would cause MPE to use
its own Fortran to C MPI profiling wrapper library. 


For cross-compilation with an existing version of MPICH(or LAM). e.g. ASCI-Red.

*** If the Fortran to C MPI profiling wrapper library exists (it is called
    libfmpich.a in MPICH and liblamf77mpi.a in LAM), do 

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false 
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --with-f2cmpilibs=-lfmpich \
                             --disable-mpe_graphics --disable-viewers
    make
    make install PREFIX=${MPE_INSTALL_DIR}

Here assume mpicc/mpif77 scripts exist.  If not, mpicc should include lines:

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


mpif77 can be composed by including the following line

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


<-lmpi> refers to various MPI libraries included your MPI implementation.


*** If the Fortran to C MPI profiling wrapper library does NOT exist,
    then the MPE's own Fortran to C MPI profiling wrapper library needs
    to be built.  3 additional environmental variables are needed.  They
    are CROSS_MPI_STATUS_SIZE, CROSS_FORTRAN2C_TRUE & CROSS_FORTRAN2C_FALSE

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false 
    setenv CROSS_MPI_STATUS_SIZE 4 
    setenv CROSS_FORTRAN2C_TRUE  1 
    setenv CROSS_FORTRAN2C_FALSE 0
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --disable-mpe_graphics --disable-viewers
    make
    make install PREFIX=${MPE_INSTALL_DIR}




IV.  EXAMPLE PROGRAMS
----------------------

As previously noted, the MPE library is composed of 3 different profiling
libraries.  During configure, the compiler's library linkage flags and 
appropriate libraries are determined.  These variables are first substituted
in the Makefile in the directory mpe/contrib/test.  The Makefile is then 
installed into directory share/examples during the final installation process.
The following is a list of these variables:

LOG_LIBS   = compiler's library flag needed to link with the logging library
TRACE_LIBS = compiler's library flag needed to link with the tracing library
ANIM_LIBS  = compiler's library flag needed to link with the animation library

The variable FLIB_PATH is the compiler's library path needed to link fortran 
MPI programs with the logging library.

During make, small test programs cpi.c and fpi.f will be linked with 
each of the above libraries.  In the output from Make, a message will be 
written regarding the success of each attempted link test.  The success 
of these linkage tests will also be included in the Make output.  If the
linkage tests are successful, then these library linkage flags can be used
for your programs as well.   

The following example programs are also included in the mpe/contrib directory:

    mpe/contrib/mandel is a Mandelbrot program that uses the MPE 
    graphics package.  

    mpe/contrib/mastermind is a program for solving the Mastermind
    puzzle in parallel.

These programs should work on all MPI implementations, but have not been
extensively tested.





V.  MPEINSTALL
--------------

A 'mpeinstall' script is created during configuration.  If configuring with
MPICH, then the 'mpiinstall' script will invoke the 'mpeinstall' script.  
However, 'mpeinstall' can also be used by itself.  This is only optional and 
is of use only if you wish to install the MPE library in a public place so 
that others may use it.  Final install directory will consist of an include,
lib, bin, sbin and share subdirectories.  Examples and various logfile viewers
will be installed under share.





VI. USAGE
---------

The final install directory contains the following subdirectories.

    include/ contains all the include files that user program needs to read.
    lib/     contains all the libraries that user program needs to link with.
    bin/     contains all the utility programs that user needs to use.
    doc/     contains available MPE documentation, e.g. Jumpshot-4's userguide.
    sbin/    contains the MPE uninstall script to uninstall the installation.
    share/   contains user read-only data.  Besides share/examples/, user 
             usually does NOT need to know the details of other subdirectories.

In terms of usage of MPE, user usually only need to know about the files
that have been installed in include/, lib/ and bin/.  



VI. a) CUSTOMIZING LOGFILES
---------------------------

In addition to using the predefined MPE logging libraries to log all MPI 
calls, MPE logging calls can be inserted into user's MPI program to define
and log states.  These states are called User-Defined states.  States may 
be nested, allowing one to define a state describing a user routine that 
contains several MPI calls, and display both the user-defined state and 
the MPI operations contained within it.

The routine MPE_Log_get_event_number() has to be used to get unique
event numbers (this is important if you are writing a library that uses
the MPE logging routines) from the MPE system.  The routines
MPE_Describe_state() and MPE_Log_event() are then used to describe 
user-defined states.

\begin{verbatim}

int eventID_begin, eventID_end;
...
eventID_begin = MPE_Log_get_event_number();
eventID_end   = MPE_Log_get_event_number();
...
MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
...
MyAmult( Matrix m, Vector v )
{
    /* Log the start event along with the size of the matrix */
    MPE_Log_event( eventID_begin, m->n, NULL );
    ... Amult code, including MPI calls ...
    MPE_Log_event( eventID_end, 0, NULL );
}

\end{verbatim}

For Fortran program, the last argument NULL in MPE_Log_event()
should be replaced by Fortran empty string, i.e. ''.

The logfile generated by this code will have the MPI routines within the
routine MyAmult() indicated by a containing bluegreen rectangle.

If the MPE logging library, liblmpe.a, are NOT linked with the user program, 
MPE_Init_log() and MPE_Finish_log() need to be used before and after all 
the MPE calls.   Sample programs cpilog.c and fpi.f are available in MPE 
source directory contrib/test or the installed directory share/examples to 
illustrate the use of these MPE routines.  

For undefined user-defined state, i.e. corresponding MPE_Describe_state() 
has not been issued, new jumpshot (Jumpshot-4) may display the legend name as
"UnknownType-INDEX" where INDEX is the internal MPE category index.



VI. b) EXAMPLE MAKEFILE 
-----------------------

share/examples contains some very useful and simple example programs and
Makefile which illustrates the usage of MPE routines and the linkage of MPE 
libraries to generate logfiles. In most cases, users can simply copy the 
share/examples/Makefile to their home directory, and do a "make" to compile 
the suggested targets.  Users don't need to copy the .c and .f files when 
MPE has been compiled with a MAKE that has VPATH support.  The created 
executables can be launched with mpirun from the MPI implementation to 
generate sample logfiles.



VI. c) ENVIRONMENTAL VARIABLES
------------------------------

There are 3 environmental variables, MPE_LOG_FORMAT, MPE_LOGFILE_PREFIX
and TMPDIR, that user can use to set the enviroment for the generation of
logfiles :

MPE_LOG_FORMAT : determines the format of the logfile generated from the 
                 execution of application linked with MPE logging libraries.  
                 The allowed value for MPE_LOG_FORMAT are CLOG, SLOG and 
                 ALOG. When MPE_LOG_FORMAT is NOT set, CLOG is assumed.
                 If slog_api/ is not shipped with your MPE, SLOG is
                 NOT an available option for MPE_LOG_FORMAT.

MPE_LOGFILE_PREFIX : specifies the filename prefix of the output logfile.
                     The file extension will be determined by the output
                     logfile format, i.e. MPE_LOG_FORMAT.

TMPDIR : specifies a directory to be used as temporary storage for each
         process.  By default, when TMPDIR is NOT set, /tmp will be used.
         When user needs to generate a very large logfile for long-running
         MPI job, user needs to make sure that TMPDIR is big enough to
         hold the temporary logfile which will be deleted if the
         merged logfile can be created successfully.  In order to
         minimize the overhead of the logging to the MPI program, it is
         highly recommended user to use a *local* file system for TMPDIR.

         Note : The final merged logfile will be written back to the 
                file system where process 0 is.



VI. d) UTILITY PROGRAMS
-----------------------

In bin/, user can find several useful utility programs when manipulating 
logfiles.  These includes log format converters, e.g. clogTOslog2, log format
print programs, e.g. slog2print, and a script to launch display program,
jumpshot.


Log Format Converters
---------------------

clogTOslog2 : a CLOG to SLOG-2 logfile convertor.  For more details,
              do "clogTOslog2 -h".

rlogTOslog2 : a RLOG to SLOG-2 logfile convertor.  For more details,
              do "rlogTOslog2 -h".  Where RLOG is an internal MPICH2 logging
              format.

logconvertor : a standalone GUI based convertor that invokes clogTOslog2
               or rlogTOslog2 based on logfile extension.  The GUI also
               shows the progress of the conversion.  The same convertor
               can be invoked from within the logfile viewer, jumpshot.

clog2alog : a CLOG to ALOG logfile converter.
            It is provided here for backward compatibility purpose.
( DEPRECATED )

clog2slog : a CLOG to SLOG logfile converter.  Since the automatic generation
            of SLOG file through setting of environmental variable 
            MPE_LOG_FORMAT to SLOG may NOT work for some non well-behaved 
            MPI programs, using the logfile format converter can generate
            extra diagnostic information about the condition of the logfile.
            Also the converter allows one to adjust certain parameters of
            the logfile, like frame size which is the segment of the logfile
            to be displayed by Jumpshot-3's time line window.  For non
            well-behaved MPI program, one may need to increase the frame
            size from the default 64KB to a bigger value.  For more
            information about the converter, do "clog2slog -h".
( DEPRECATED )
	    

Log Format Print Programs
-------------------------

clog_print : a stdout print program for CLOG file.  Java version is clogprint.

rlog_print : a stdout print program for SLOG-2 file.

slog2print : a stdout print program for SLOG-2 file.

slog_print : a stdout print program for SLOG file.  It serves to check the
             content of the logfile.  If the SLOG file is too big, it may
             NOT be useful to use slog_print.  Also, when slog is NOT complete,
             slog_print won't work.  So it serves as a simple test to check
             if the SLOG file is generated completely.
( DEPRECATED )


Display Program Selector
------------------------

jumpshot : the Jumpshot-4 launcher script.  Jumpshot-4 does logfile
           conversion as well as visualization.

logviewer : the script which involves appropriate viewer based on the file
            extension of logfile.  For instance, if the logfile is foo.slog,
            logviewer will invoke jumpshot-3 to display the logfile.
            Jumpshot-3 resides in share/.  For more information of
            logviewer, do "logviewer -help" to list all available options.
( DEPRECATED )





VII. Using MPE in MPICH
-----------------------

MPE has been seamlessly integrated into MPICH distribution, so user may
find it easier to use MPE when using it with MPICH.  Here are the differences
of using MPE with MPICH and with other MPI implementations.



VII. a) Compilation and Linkage
-------------------------------
MPICH provides scripts to help users to compile and link C/C++ and
F77/F90 programs.  They are mpicc for C programs, mpiCC for C++ programs,
mpif77 for F77 and mpif90 for F90 programs.  In addition, these 4 scripts
allows special options to be used to link with MPE profiling libraries.
These options are :

-mpitrace   - to compile and link with tracing library.
-mpianim    - to compile and link with animation libraries.
-mpilog     - to compile and link with logging libraries.

For instance, the following command creates executable, {\tt fpilog}, which
generates logfile when it is executed.

mpif77 -mpilog -o fpilog fpilog.f 

For other MPI implementations, user needs to compile and link their application
with MPE profiling libraries explicitly as shown in the example makefile.



VII. b) Inheritance of Environmental Variables
----------------------------------------------
MPE relies on certain environmental variables (e.g. TMPDIR).  These
variables determine how MPE behaves.  It is important to make sure that
all the MPI processes receive the intended value of environmental variables.
The complication of this issue comes from the fact that different MPI
implementations have different way of passing environmental varaiable.  For
instance, MPICH contains many different devices for different platforms,
some of these devices have their unique way of passing of environmental
variables to other processes.  The often used devices, like ch_p4 and ch_shmem,
do not require special attention to pass the value of the environmental 
variable to spawned processes.  The spawned process inherits the value from
the launching process when the environmental variable in the launching
process has been set.  But this is NOT true for all the devices, for instance,
the ch_p4mpd device requires special option of mpirun to set environmental
variables to all processes.

mpirun -np N fpilog -MPDENV- MPE_LOGFILE_PREFIX=fpilog

In this example, the option -MPDENV- is needed to make sure
that all processes have their environmental variable, MPE_LOGFILE_PREFIX,
set to the desirable output logfile prefix.

In MPICH-2 using MPD as a launcher, passing MPE_LOGFILE_PREFIX and
TMPDIR can be done as follows:

mpiexec -default -env MPE_LOGFILE_PREFIX <output-logname-prefix> \
                 -env TMPDIR <local-tmp-dir> : -n 32 <executable-name>

For other MPI implementations, how environmental variables are passed
remains unchanged.  User needs to get familar with the environment and
set the environmental variables accordingly.



VII. c) Viewing Logfiles
------------------------
MPE's install directory structure is the same as MPICH's and MPICH-2's.
So all MPE's utility programs will be located in the bin/ directory of 
MPICH and MPICH-2.  To view a logfile, say fpilog.slog2, do

jumpshot fpilog.slog2

The command will select and invoke Jumpshot-4 to display the content
of SLOG-2 file if Jumpshot-4 has been built and installed successfully.
One can also do

jumpshot fpilog.clog2

or

jumpshot barrier.rlog

Both will invoke the logfile convertor first before visualization.



VIII.  Java Requirement
-----------------------
The only component in MPE that requires Java is slog2sdk which provides
all SLOG-2 related tools.  slog2sdk is distributed with prebuilt
jar files, i.e. precompiled byte code, so users do not have to build them.
For MPICH user, the minimum Java requirement to use SLOG-2 tools is
Java Runtime Environment (JRE).  However some of the SLOG-2 tools
require jni.h which is not necessarily available in JRE on
some platforms, e.g. AIX.  With JRE, tools that are related to RLOG
cannot be built.  Since only MPICH2 has RLOG support, it would be fine
to use JRE for MPICH.
                                                                                
slog2sdk is developed based on Java2, so any Java2 runtime enviroment
(i.e. j2re 1.2.2. and above) will be enough to run SLOG-2 tools.  To rebuild
all slog2sdk tools, it is recommended to use the latest and stable
release of Java2 Development Kit (JDK).  For optimal performance
of slog2 tool, like jumpshot, it is best to use the latest and stable
JRE.  At of this writing, j2sdk1.4.2 provides best performance for
Jumpshot-4.
