Submit Batches of Serial Jobs to HPC

Suppose you have serial code--a bash script, for example--and you want it to operate on different sets of data (i.e., with different input data) and generate different sets of output. We strongly recommend you to use the Condor cluster which supports large batches of serial jobs where overall throughput is preferred over performance, but if you want to test it on the HPC cluster (say, before you parallelize your code, or you have access to some dedicated queue), you can do so by creating a script that submits jobs with different input parameters.

Below is an example:

First, one important useful MOAB command line flag is the -F flag:

-F  "\"arg1 arg2 ... argN\"" 

...where arg1 arg2 ... argN (separated by spaces, no commas) are the parameters to pass to the MOAB submit script.

For example, you can submit your script as follows:

  $  msub -F  "\"par1\""  submit.sh 

where submit.sh is your MOAB submit script with takes one parameter par1.

Below, for example, is a simple bash script, moab_cp.sh. It takes two parameters; the name of the file to be copied, and the name of the copy.

#!/bin/bash
# moab_cp.sh  
# usage:  moab_cp.sh file1 file2
if [ "$#" -ne "2" ]
then
echo "Usage: `basename $0` inputfile outputfile"
exit 1
elif [ ! -e "$1" ]
then
echo "file $1 doesnot exist"
exit 1
else
cp $1 $2
exit 0
fi

Here is the job submit script, sub_moab_cp.sh:

#!/bin/bash
#  sub_moab_cp.sh: submit script for moab_cp.sh
#  usage: msub -F "\"file1 file2\"" sub_moab_cp.sh

#MOAB -N "moab_script_job"
#MOAB -l nodes=1:ppn=1
#MOAB -j oe
#MOAB -q backfill
#MOAB -m abe
#MOAB -l walltime=00:10:00

cd $PBS_O_WORKDIR
./moab_cp.sh $1 $2

Now suppose you want to run moab_cp.sh several times but with different input and output file names. To do so, you will create a third script to submit multiple jobs with different parameters:

#!/bin/bash
# test_sub_moab_cp.sh
# submit sub_moab_cp.sh jobs with different input/output data
if [ ! -e input ]
then
mkdir input
fi
if [ ! -e output ]
then
mkdir output
fi
for i in {1..5}
do
touch input/file.${i}
echo "$i" >>  input/file.${i}
# submit same job script but passing different parameters
msub -F "\"input/file.${i} output/file.${i}.cp\"" sub_moab_cp.sh
#   wait for 2 secs to reduce the burden of the scheduler       
sleep 2
done
exit 0 

You will run the above script directly on the submit node. It will submit one job to the MOAB job scheduler every two seconds, each with different data.

  $  ./test_sub_moab_cp.sh

 7650482

 7650483

7650484

7650485

7650486

The script first creates the input and output directories in your working directory, then creates 5 input data files. It then submits the sub_moab_cp.sh submit script to MOAB 5 times with two input parameters file.$i and file.$i.cp. Each job above uses one processor (since each is a serial job), and copies ./input/file.$i to ./output/file.$i.cp.

In the above example, we have a single submit script, sub_moab_cp.sh, and we pass parameters to it using the -F flag of the MOAB msub command. Another option is to write a script generating N different submit scripts. For example, each submit script may be titled something like sub_moab_cp_$i.sh, each containing different input parameters. Inside each one, the last line of sub_moab_cp.sh is different.

  ./moab_cp.sh $1 $2

by

  ./moab_cp.sh   file.$i  file.$i.cp

, and submit sub_moab_cp_$i.sh separately. The following script, create_sub.sh, will do this job for you:

#!/bin/bash
#  create_sub.sh
#  Create submit scripts running moab_cp.sh with different input 
#+  and submit all of them.

njobs=5
if [ ! -e input ]
then
mkdir input
fi

if [ ! -e output ]
then
mkdir output
fi

for (( i=1; i<=$njobs; i++ ))
do
touch input/file.${i}
echo "$i" >> input/file.${i}
done

# generate an array of submit scripts.
for (( i=1; i<=$njobs; i++ ))
do
echo "#!/bin/bash"   > sub_moab_cp_$i.sh   # overwrite if exists
echo "# sub_moab_cp_$i.sh" >> sub_moab_cp_$i.sh
echo "# usage: msub -F \"\\\"file1 file2\\\"\" sub_moab_cp_$i.sh" >> sub_moab_cp_$i.sh
echo "#MOAB -N \"script_job_test\"" >> sub_moab_cp_$i.sh
echo "#MOAB -l nodes=1:ppn=1" >> sub_moab_cp_$i.sh
echo "#MOAB -j oe" >> sub_moab_cp_$i.sh
echo "#MOAB -q backfill" >> sub_moab_cp_$i.sh
echo "#MOAB -m abe" >> sub_moab_cp_$i.sh
echo "#MOAB -l walltime=00:01:00" >> sub_moab_cp_$i.sh
echo "cd \$PBS_O_WORKDIR" >> sub_moab_cp_$i.sh
echo "./moab_cp.sh input/file.$i output/file.$i.cp" >> sub_moab_cp_$i.sh
done

echo "done with generating the submit scripts, now submit them:"
for (( i=1; i<=$njobs; i++ ))
do
msub sub_moab_cp_$i.sh
sleep 2
done

exit 0

We do not encourage you to submit many serial jobs to the queue simultaneously, since this will exhaust the job scheduler. Furthermore, the jobs you submit must comply with the HPC queue policy.

Note: If your code is serial--for example, a bash script--then submitting your job to multiple processors in your submit script will probably not speed up execution of your code. For example, adding the line #MOAB -l nodes=8 to your script will not automatically cause your code to utilize eight processors. Only one processor will be used, even though eight are allocated to you.