ARPACK
ARPACK
From the ARPACK website:
The package is designed to compute a few eigenvalues and corresponding eigenvectors of a general n by n matrix A. It is most appropriate for large sparse or structured matrices A where structured means that a matrix-vector product w <- Av requires order n rather than the usual order n2 floating point operations. This software is based upon an algorithmic variant of the Arnoldi process called the Implicitly Restarted Arnoldi Method (IRAM). When the matrix A is symmetric it reduces to a variant of the Lanczos process called the Implicitly Restarted Lanczos Method (IRLM). These variants may be viewed as a synthesis of the Arnoldi/Lanczos process with the Implicitly Shifted QR technique that is suitable for large scale problems. For many standard problems, a matrix factorization is not required. Only the action of the matrix on a vector is needed.
Using ARPACK on HPC cluster
The parallel version of ARPACK was compiled for the GNU, Intel and PGI compilers for both openmpi and mvapich2.
The corresponding libraries are located in
/opt/hpc/[gnu|intel|pgi]/[openmpi|mvapich2]/lib64
For example, to use libarpack.so and libparpack.so built from gnu-openmpi Fortran compiler, add the following options to your command line:
-L/opt/hpc/gnu/openmpi/lib64 -lparpack -larpack
Note that arpack is based upon the blas and lapack libraries, and parpack calls for arpack. Therefore, we have to link to both the lapack and blas packages and specify the path to these two packages according to the compiler you are using.
For the GNU compiler, add the options
-L/usr/lib64 -lblas -llapack
For the Intel compiler, we use the lapack and blas packages from the intel-mkl library. Add the options
-L/gpfs/research/software/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lm -lmkl_blas95_lp64 -lmkl_lapack95_lp64
For the PGI compiler, add the options
-L/gpfs/research/software/pgi/linux86-64/14.3/lib -llapack -lblas
To run some simple examples provided by the ARPACK developers, first create two directories to hold the serial and parallel example files, respectively. Run the following commands:
# copy serial and parallel examples to your home directories
$ cd $HOME
$ mkdir arpack
$ cd arpack
$ mkdir SERIAL
$ mkdir PARALLEL
$ cp -r /opt/rcc/share/doc/arpack-doc/EXAMPLES SERIAL
$ cp -r /opt/rcc/share/doc/arpack-doc/MPI PARALLEL
Below is an example of compiling and running a serial example program (e.g., sssimp.f, a serial f77 program) using the GNU compiler. If you want to run a different program, also replace the sssimp argument in the second command with whatever you would like to name your executable file. Make sure to change that executable name in the last command as well.
$ cd $HOME/arpack/SERIAL/EXAMPLES/SIMPLE
$ module load gnu
$ gfortran sssimp.f -L/opt/rcc/gnu/lib64 -larpack -lblas -llapack -o sssimp
$ ./sssimp
Below is an example of compiling and running a parallel program (e.g., pssdrv1.f)on the HPC cluster using the GNU compiler. Make sure to create your Slurm script before you run the last command, which submits the script. If you want to run a different program, replace the sub_parpack.sh argument with the name of the script you submit to the Slurm scheduler. Your script file must use the .sh extension.
$ cd $HOME/arpack/PARALLEL/MPI
$ module load openmpi
$ mpif77 pssdrv1.f -L/opt/rcc/gnu/openmpi/lib64 -larpack -lparpack -lblas -llapack -o pssdrv1
$ sbatch sub_parpack.sh
If you want to use Intel mvapich2 with a parallel program (e.g., pssdrv1.f), run the commands:
$ module load intel mvapich2
$ mpif77 pssdrv1.f -L/opt/rcc/intel/mvapich2/lib64 -larpack -lparpack -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lm -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -L/gpfs/research/software/intel/mkl/lib/intel64 -o pssdrv1
Below is an example Slurm script called sub_parpack.sh to run the executable created with the above commands. Replace $SLURM_SUBMIT_DIR with the directory where your script is located.
$ cat sub_parpack.sh # displays content of file
#!/bin/bash
#SBATCH --job-name="parpack_test"
#SBATCH -N 4
#SBATCH -p backfill
#SBATCH --mail-type="ALL"
#SBATCH -t 00:05:00
module purge
module load gnu openmpi
cd $SLURM_SUBMIT_DIR
mpirun -np 4 ./pssdrv1
You will see the following output upon a successful run:
_saupd: number of update iterations taken
-----------------------------------------
1 - 1: 5
_saupd: number of "converged" Ritz values
-----------------------------------------
1 - 1: 4
_saupd: final Ritz values
-------------------------
1 - 4: 6.350E-01 3.985E-01 3.985E-01 1.620E-01
_saupd: corresponding error bounds
----------------------------------
1 - 4: 1.381E-13 8.283E-12 1.017E-10 3.955E-21
==========================================
= Symmetric implicit Arnoldi update code =
= Version Number: 2.1 =
= Version Date: 3/19/97 =
==========================================
= Summary of timing statistics =
==========================================
Total number update iterations = 5
Total number of OP*x operations = 81
Total number of B*x operations = 0
Total number of reorthogonalization steps = 81
Total number of iterative refinement steps = 0
Total number of restart steps = 0
Total time in user OP*x operation = 0.001000
Total time in user B*x operation = 0.000000
Total time in Arnoldi update routine = 0.001999
Total time in p_saup2 routine = 0.001999
Total time in basic Arnoldi iteration loop = 0.001999
Total time in reorthogonalization phase = 0.000000
Total time in (re)start vector generation = 0.000000
Total time in trid eigenvalue subproblem = 0.000000
Total time in getting the shifts = 0.000000
Total time in applying the shifts = 0.000000
Total time in convergence testing = 0.000000
Ritz values and direct residuals
--------------------------------
Col 1 Col 2
Row 1: 1.62029E-01 1.44471E-06
Row 2: 3.98507E-01 1.35166E-06
Row 3: 3.98508E-01 1.75542E-06
Row 4: 6.34986E-01 1.71304E-06
_SDRV1
======
Size of the matrix is 100
The number of processors is 4
The number of Ritz values requested is 4
The number of Arnoldi vectors generated (NCV) is 20
What portion of the spectrum: SM
The number of converged Ritz values is 4
The number of Implicit Arnoldi update iterations taken is 5
The number of OP*x is 81
The convergence criterion is 5.96046448E-08
Using ARPACK on Spear nodes
The serial version of ARPACK was compiled for the Intel, GNU, and PGI compilers, respectively. To compile applications based on arpack, add the following options to your command line:
-L/opt/rcc/intel/lib64 -larpack -lblas -llapack
Below is an example of using these options with a file called myprog.f:
$ ifort -o a.out -L/opt/rcc/intel/lib64 -larpack -lblas -llapack myprog.f
Serial example files provided by the ARPACK developers are located in the directory:
/opt/rcc/share/doc/arpack-serial-3.1.5/EXAMPLES
For detailed usage information, refer to the official users' guide on the ARPACK website.