|
|
| (32 intermediate revisions not shown) |
| Line 1: |
Line 1: |
| - | This page is mostly for my own notes (to myself) regarding the implementation of Parallel SpGEMM library.
| |
| | | | |
| - | [http://gauss.cs.ucsb.edu/~aydin http://gauss.cs.ucsb.edu/~aydin]
| |
| - |
| |
| - | ----
| |
| - |
| |
| - |
| |
| - | == Project Description ==
| |
| - | The project is to write a 2D parallel implementation of the sparse matrix times sparse matrix routine. It is projected to be implemented in C++ using MPI. The project will have multiple iterations:
| |
| - |
| |
| - | 1) The simplest decomposition is the [http://www.parallab.uib.no/projects/molecul/matrix/main/node12.html regular block checkerboard partitioning] and to use a SUMMA-like algorithm on top of that. Here we assume the existence of a fast sequential algorithm for sparse matrix multiplication that is used as a block box.
| |
| - |
| |
| - | 2) The second way to decompose to data is to try assigning equal number of edges to each processor. The initial idea is to map each non-zero of the matrix to a point in the rectangular (m x n) region and use a quadtree [http://en.wikipedia.org/wiki/Quadtree] data structure. Here one may or may not use a sequential matrix multiplication as block box [Yet to be explored].
| |
| - |
| |
| - | 3) The third and the hardest way is to assign edges to processors such that the following metric is minimized.
| |
| - | <math>min(max_i(communication(i) + computation(i)))</math>
| |
| - |
| |
| - | In other words, we try to load balance across the processors. The load balancing is generally done using graph partitioning [http://www.math.ruu.nl/publications/preprints/1238.ps] [http://medicine.osu.edu/informatics/DIGC/pubs/umit_irr01.pdf] but it is unclear how to do it for Parallel SpGEMM. Whether this load balancing is going to be done statically or dynamically is yet to be determined.
| |
| - |
| |
| - |
| |
| - | ----
| |
| - |
| |
| - | == Resources ==
| |
| - |
| |
| - | Boost.MPI [http://www.generic-programming.org/~dgregor/boost.mpi/doc/mpi/getting_started.html#mpi.mpi_impl]
| |
| - | Note that the data pointed by the pointers in a class are transmitted automatically due to the 4th requirement of the
| |
| - | [http://www.boost.org/libs/serialization/doc/ serialization library], which is "Deep pointer save and restore. That is, save and restore of pointers saves and restores the data pointed to"
| |
| - |
| |
| - | Boost Build v2 [http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Build_V2]
| |
| - |
| |
| - | [http://www.boost.org/libs/serialization/doc/serialization.html#derivedpointers Serizalization of Derived Pointers]
| |
| - |
| |
| - | SDSC Resources [http://www.sdsc.edu/us/resources/datastar/references.html]
| |
| - |
| |
| - | ----
| |
| - |
| |
| - | == Caveats ==
| |
| - |
| |
| - | 1) When implementing MPI with threads, you will be oversubscribing (running more processes than processors) the nodes without OpenMPI figuring this out. For example, if each of your processes has 3 threads, where 2 of these contain MPI_SEND, MPI_RECV operators, you would want your thread to yield() if it blocks on a MPI_RECV or MPI_SEND. However, by default, MPI will run on aggressive mode when the number of slots = the number of mpi_processes [http://icl.cs.utk.edu/open-mpi/faq/?category=running#oversubscribing].
| |
| - |
| |
| - | So, you wanna set ''mpi_yield_when_idle'' manually to 1 when oversubscribing implicitly.
| |
| - |
| |
| - | ''mpirun --mca mpi_yield_when_idle 1 -np 16 ./testpar''
| |
| - |
| |
| - |
| |
| - | Warning: This may degrade the performance when processors = threads ! [not always though]
| |
| - |
| |
| - | mpi_yield_when_idle 1 --> 4.13 sec
| |
| - |
| |
| - | mpi_yield_when_idle 0 --> 2.95 sec
| |
| - |
| |
| - |
| |
| - | 2) OpenMPI installed on Neumann has the following default:
| |
| - |
| |
| - | '' ompi_info | grep Thread''
| |
| - |
| |
| - | ''Thread support: posix (mpi: no, progress: no) ''
| |
| - |
| |
| - | Meaning that the build '''doesn't''' support MPI_THREAD_MULTIPLE !
| |
| - |
| |
| - | Solution: OpenMPI has thread-support disabled on default. You have to re-build OpenMPI from source and use '--enable-mpi-threads' during 'configure'.
| |
| - |
| |
| - | However, OpenMPI's thread support is only lightly tested and there are many complaints about it on the web such as "it hangs". Therefore, I am installing MPICH2 for my own sake using:
| |
| - |
| |
| - | ''./mpich2-1.0.6/configure -prefix=/usr/local/bin/mpich2 --enable-threads=multiple --with-thread-package''
| |
| - |
| |
| - | Now, the system has two different MPI implementations (thus two different mpirun, mpicc, etc) we need to tell the system which one we want to use by:
| |
| - |
| |
| - | ''export PATH=/usr/local/bin/mpich2/bin:$PATH'' --> this will give precedence to MPICH2
| |
| - |
| |
| - | However, MPICH system has the following problem (http://www.mpi-softtech.com/article.php?id=r1037051037):
| |
| - |
| |
| - | "MPICH and its derivatives rely on polling for synchronization and notification, which leaves little space for exploiting programming techniques that would achieve computation and communication overlapping and optimal resource utilization [13]. Although polling can ultimately achieve the lowest message passing latency, it wastes CPU cycles for spinning on notification flags. A user thread that polls for synchronization does not yield the CPU until this thread is de-scheduled by the OS, thus reducing the CPU time for useful computation. Latency is generally not considered as a realistic measure of performance for applications that can overlap computation and communication."
| |
| - |
| |
| - | Again, this might not be true either. MPICH2 installation guide says:
| |
| - |
| |
| - | "build the channel of your choice. The options are sock, shm, and ssm.
| |
| - | The shm channel is for small numbers of processes that will run on a
| |
| - | single machine using shared memory. The shm channel should not be
| |
| - | used for more than about 8 processes. The ssm (sock shared memory)
| |
| - | channel is for clusters of smp nodes. This channel should not be used
| |
| - | if you plan to over-subscribe the CPU���¢�¯�¿�½�¯�¿�½s. If you plan on launching
| |
| - | more processes than you have processors you should use the default
| |
| - | sock channel. The ssm channel uses a [[polling progress engine]] that
| |
| - | can perform poorly when multiple processes compete for individual
| |
| - | processors."
| |
| - |
| |
| - | This sounds as if sock doesn't use a polling progress engine?
| |
| - |
| |
| - |
| |
| - |
| |
| - |
| |
| - | 3) Forcing the processes to be launched on a subset of processors only:
| |
| - |
| |
| - | ''mpirun -np 16 taskset -c 12-15 ./testpar '' --> 100 sec.
| |
| - |
| |
| - | [http://www.open-mpi.org/faq/?category=tuning#using-paffinity This] might also be helpful in the case of processor affinity.
| |
| - |
| |
| - | So, combine the two: "''mpirun -np 16 --mca mpi_yield_when_idle 1 taskset -c 12-15 ./testpar ''" --> 7.7 sec. only
| |
| - |
| |
| - | MPICH2's MPD's solution to the problem:
| |
| - | "If you tell the mpd how many cpus it has to work with by using the --ncpus argument, as in
| |
| - |
| |
| - | ''mpd --ncpus=2''
| |
| - |
| |
| - | then the number of processes started the first time the startup message
| |
| - | circles the ring will be determined by this argument.
| |
| - |
| |
| - | 4) Check to see the compiler options for mpic++ by:
| |
| - |
| |
| - | '' mpic++ --showme:compile
| |
| - |
| |
| - | mpic++ --showme:link''
| |
| - |
| |
| - |
| |
| - | 5) Difference between boost::timer and boost::mpi::timer
| |
| - |
| |
| - | boost::mpi::timer is a wrapper around the MPI_TIMER so it provides much finer grain timings up to 5 digits precision whereas boost::timer only provides 2 digits precision such as "0.04 sec".
| |
| - |
| |
| - | 6) Using '''gprof2dot''' to get call graphs of your program:
| |
| - |
| |
| - | - Compile with -pg option
| |
| - |
| |
| - | - Run that executable with whatever input you want (./exp p256), that'll create the gmon.out file
| |
| - |
| |
| - | - ''gprof expnogas > explite.txt''
| |
| - |
| |
| - | - '' python gprof2dot.py -e 0.00 -n 0.00 explite.txt > gvexplite '' (includes all the nodes and edges)
| |
| - |
| |
| - | - ''python xdot.py gvexplite''
| |
| - |
| |
| - |
| |
| - |
| |
| - | == Regular Installation of Boost with MPICH2 ==
| |
| - |
| |
| - | 0) If you have another boost installed before and you want to keep the installation, first remove "boost/bin.v2/libs" contents
| |
| - |
| |
| - | 1) Add the following lines to $(HOME)/user-config.jam:
| |
| - |
| |
| - | ''using gcc : : : <cxxflags>-DMPICH_IGNORE_CXX_SEEK ;''
| |
| - |
| |
| - | ''using mpi : $(PATH_MPICH2)/mpicxx ;''
| |
| - |
| |
| - | 2) Then execute the following command from the top boost directory:
| |
| - |
| |
| - | sudo ./bjam --with-mpi --with-thread
| |
| - |
| |
| - | 3) Finally the installation:
| |
| - |
| |
| - | sudo ./bjam --prefix=/usr/local/boostmpich2 --with-mpi --with-thread install
| |
| - |
| |
| - | 4) If you still insist on keeping the old MPI installation, then you either explicitly point out the path to new mpicxx/mpirun for each call, or
| |
| - | put the following line to your .bashrc file so that it finds the new mpicxx/mpirun during calls.
| |
| - |
| |
| - | ''export PATH=/usr/local/bin/mpich2/bin:$PATH''
| |
| - |
| |
| - |
| |
| - | == Threading Issues ==
| |
| - |
| |
| - | Data copying from SparseDColumn<T> ** M[i][j] to local matrix by each thread takes the following times:
| |
| - |
| |
| - | Without Hoard. Using kinner=i (no contention avoidance) :
| |
| - |
| |
| - | Data copy took 0.134991 seconds
| |
| - |
| |
| - | Data copy took 0.074193 seconds
| |
| - |
| |
| - | Data copy took 0.083097 seconds
| |
| - |
| |
| - | Data copy took 0.088695 seconds
| |
| - |
| |
| - |
| |
| - | ''For a total of 0.379 seconds.''
| |
| - |
| |
| - |
| |
| - |
| |
| - | Without Hoard. Using kinner= (i + ((dimx+dimy)% gridy)) % gridy (contention avoidance) :
| |
| - |
| |
| - | Data copy took 0.090011 seconds
| |
| - |
| |
| - | Data copy took 0.059552 seconds
| |
| - |
| |
| - | Data copy took 0.069522 seconds
| |
| - |
| |
| - | Data copy took 0.072168 seconds
| |
| - |
| |
| - | ''For a total of 0.290 seconds.''
| |
| - |
| |
| - |
| |
| - | With Hoard. Using kinner=i (no contention avoidance) :
| |
| - |
| |
| - | Data copy took 0.217126 seconds
| |
| - |
| |
| - | Data copy took 0.125259 seconds
| |
| - |
| |
| - | Data copy took 0.142138 seconds
| |
| - |
| |
| - | Data copy took 0.175438 seconds
| |
| - |
| |
| - |
| |
| - | ''For a total of 0.659 seconds''
| |
| - |
| |
| - |
| |
| - |
| |
| - | With Hoard. Using kinner= (i + ((dimx+dimy)% gridy)) % gridy (contention avoidance) :
| |
| - |
| |
| - | Data copy took 0.217977 seconds
| |
| - |
| |
| - | Data copy took 0.054324 seconds
| |
| - |
| |
| - | Data copy took 0.124619 seconds
| |
| - |
| |
| - | Data copy took 0.107576 seconds
| |
| - |
| |
| - |
| |
| - | ''For a total of 0.502 seconds''
| |
| - |
| |
| - |
| |
| - |
| |
| - |
| |
| - | == Building GASNET C++ Clients ==
| |
| - |
| |
| - | Change ''.../smp-conduit/smp-par.mak'' file (and any ''.../xxx-conduit/xxx-par.mak'' file you wanna use):
| |
| - |
| |
| - | - In two places, there are ''/usr/bin/gcc'', make them ''g++''
| |
| - |
| |
| - | - In one place, there is ''-lgcc'', make it ''-lstdc++''
| |
| - |
| |
| - | - Remove the ''-Winline'' flag used in compilation.
| |
| - |
| |
| - |
| |
| - | '''Starting them:'''
| |
| - |
| |
| - | > ''export PATH=$PATH:/home/aydin/localinstall/gasnet/bin''
| |
| - |
| |
| - | > ''amudprun -spawn 'L' -np 4 ./comp input1 input2_1''
| |
| - |
| |
| - |
| |
| - | '''Checking memory errors in them:'''
| |
| - |
| |
| - | > ''export GASNET_SPAWNFN='L'''
| |
| - |
| |
| - | > ''valgrind --trace-children=yes ./testgas 4''
| |
| - |
| |
| - |
| |
| - | '''To see the details:'''
| |
| - |
| |
| - | > ''GASNET_VERBOSEENV=1''
| |
| - |
| |
| - |
| |
| - | ''' LONESTAR (VAPI/IBV) Details:'''
| |
| - |
| |
| - | > GASNET_VAPI_SPAWNER (set to "mpi" or "ssh") can override the value set at configuration time.
| |
| - |
| |
| - | > GASNET_TRACEFILE - specify a file name to recieve the trace output may also be "stdout" or "stderr", (or "-" to indicate stderr) each node may have its output directed to a separate file, and any '%' character in the value is replaced by the node number at runtime (e.g. GASNET_TRACEFILE="mytrace-%")
| |
| - | unsetting this environment variable (or setting it to empty) disables tracing output (although the trace code still has performance impact)
| |
| - |
| |
| - | > For some reason, mpi-spawner of gasnetrun_ibv do not get along well with the batch environment of lonestar. It requires a machinefile list. Luckily, ssh-spawner automatically looks at $LSB_HOSTS after if cannot find $GASNET_SSH_NODEFILE variable.
| |
| - |
| |
| - | ''bsub -I -n 4 -W 0:05 -q development gasnetrun_ibv -n 4 -spawner=ssh ./testgas''
| |
| - |
| |
| - | == Installing BOOST.MPI to DataStar ==
| |
| - |
| |
| - | 1) Download boost into Project directory.
| |
| - |
| |
| - | 2) Make sure mpCC works.
| |
| - |
| |
| - | 3) Go to "../boost/tools/jam/src" folder
| |
| - |
| |
| - | 4) Type "./build.sh vacpp"
| |
| - |
| |
| - | 5) Go to "../boost/tools/jam/src/bin.aixppc" folder and copy the "bjam" executable to "../boost" directory. (i.e. top-level boost directory)
| |
| - |
| |
| - | 6) Copy "../boost/tools/build/v2/user-config.jam" to $HOME and add line "using mpi; "
| |
| - |
| |
| - | 7) "using mpi; " will probably fail. Thus you might need to configure MPI yourself. In order to do that, you need to know which libraries are related to mpi. Such libraries are inside the PE (parallel environment) folder of dspoe: "/usr/lpp/ppe.poe/lib"
| |
| - |
| |
| - | using mpi : : <find-shared-library>library1 <find-shared-library>library2 <find-shared-library>library3 ;
| |
| - |
| |
| - | 8) Type "bjam --with-mpi --toolset=vacpp" in your top-level boost directory.
| |
| - |
| |
| - | to see what is going on: "bjam --with-mpi --toolset=vacpp --debug-configuration 2 > debugconferr.txt"
| |
| - |
| |
| - |
| |
| - |
| |
| - | == C++ Lessons Learned ==
| |
| - |
| |
| - | 1) Don't use operator BT() inside the composition closure object MMul. SparseDColumn (or whatever template is used for BT) is going to take care of implementing the necessary operation through the assignment & constructor accepting MMult<T> &
| |
| - |
| |
| - | 2) Distinguish between assigning a pointer or assigning the value of a pointer very clearly. If you're gonna malloc a pointer inside a function, you should pass it as a reference to a pointer:
| |
| - |
| |
| - | ''void (int * & array)''
| |
| - |
| |
| - | But if you're gonna just change the contents (for example write NULL to the memory location)
| |
| - |
| |
| - | ''int * array = malloc(100 * sizeof(int));''
| |
| - |
| |
| - | ''*((MemoryPool**) array) = NULL;''
| |
| - |
| |
| - | 3) Optional parameters for members of a class:
| |
| - |
| |
| - | Inside header:
| |
| - | SparseDColumn (const SparseTriplets<T> & rhs, bool transpose, MemoryPool * mpool = NULL);
| |
| - |
| |
| - | Inside cpp:
| |
| - | SparseDColumn<T>::SparseDColumn(const SparseTriplets<T> & rhs, bool transpose, MemoryPool * mpool){...}
| |
| - |
| |
| - | The reason for this is natural. Other parts of the code that uses this class should know that mpool is an optional argument, and it is OK not to supply it. We can only achieve that effect by putting it inside the header file.
| |
| - |
| |
| - |
| |
| - |
| |
| - | == CUDA Stuff ==
| |
| - |
| |
| - | Running decuda:
| |
| - |
| |
| - | >> make data/dlur.cubin
| |
| - |
| |
| - | >> python decuda dlur.cubin > mygemm.decuda
| |
| - |
| |
| - |
| |
| - |
| |
| - | Start-up script:
| |
| - |
| |
| - | Inside ''/etc/rc.local'', you'll see the following line:
| |
| - |
| |
| - | ./root/nvidia.sh start
| |
| - |
| |
| - |
| |
| - | CUDA SDK 1.1 caveat:
| |
| - |
| |
| - | >> make USECUBLAS=1 verbose=1
| |
