Time stamps

When printing from an MPI-program the output need not come in the correct order. If the program takes a short time to run, all the output from process 0 may come first, for example, even though, process 1 has printed in the middle. One way to order the output in the correct order is to use so called time stamps. When we print something, we always put the time at the beginning of the printed line. After the run we sort all the printed lines with respect to this time. For this construction to work we need a clock that is synchronized between CPUs.

The following solution seems to work well on the Sun:

...
double precision fsecond
...
print*, fsecond(), my_rank, ' the actual message ', variables, etc.

To know which process is responsible for a particular line I always print out the rank as well. It can look something like this:

% mpirun c0-3 ./a.out | sort -n
921243855.264121 1 before MPI_Recv
921243855.264373 0 before MPI_Send
921243855.265793 1 after MPI_Recv 1000.000000
921243855.265832 1 before MPI_Send
921243855.265921 1 after MPI_Send
921243855.265943 1 before MPI_Recv

| sort -n takes the output from a.out and sorts it. -n means sorting with respect to numbers in the beginning of the lines.