How to submit HPC Jobs

An HPC job usually consists of a compiled C, C++, or Fortran program. Refer to our tutorial to learn more about compiling code for HPC, and to our software list for what compilers and libraries are available.

Some jobs may actually be Python, Java, or even simple BASH scripts, which our HPC also supports.

Creating a MOAB submission script

The FSU HPC uses the MOAB Job Management (e.g. batch processing) system to schedule jobs on the HPC cluster. MOAB connects to a backend system called Torque which will run your code on our hardware. You will need to submit your job to the MOAB queue in order for it to be scheduled and eventually run in our HPC cluster.

Before submitting a job to the HPC, you must decide a few things:

  • Which queue to submit your job to - If you have purchased HPC nodes, you will have your own dedicated queue. For all others, you can choose between the genacc_q for long-running jobs, and the backfill queue for shorter running jobs that may use more resources at a time.
  • How long your job needs to run - Choose this value carefully. If you choose a value too short, the system may kill your job before it completes. If you choose a value too long, the scheduler may delay the start of your job for longer than necessary. This value is limited to the maximum length of time allowed for each queue.
  • How many compute nodes your job will need - This is the level of parallelization, and is also limited per user and per queue.

Simple Example Job

Submitted jobs typically consist of compiled C, C++, or Fortran code. They sometimes also consist of Python or Java Code. You can, however, submit Linux bash scripts as jobs. In this example, we will create a simple BASH script, submit it to the HPC general access queue, and view the output.

Start by logging into the HPC via SSH. In your home directory, create or upload a file named moab_example.sh with the following contents:

#!/bin/bash
echo "------------------------------------------------------"
echo "-n 'Job is running on node '; cat $PBS_NODEFILE"
echo "------------------------------------------------------"
echo "MOAB: the queue manager (qsub) is running on $PBS_O_HOST"
echo "MOAB: this job was submitted to the queue: $PBS_O_QUEUE"
echo "MOAB: this job is actually running on the queue: $PBS_QUEUE"
echo "MOAB: working directory is: $PBS_O_WORKDIR"
echo "MOAB: execution mode is: $PBS_ENVIRONMENT"
echo "MOAB: job identifier is: $PBS_JOBID"
echo "MOAB: job name is: $PBS_JOBNAME"
echo "MOAB: node file is: $PBS_NODEFILE"
echo "MOAB: current home directory is: $PBS_O_HOME"
echo "MOAB: PATH = $PBS_O_PATH"
echo "------------------------------------------------------"
echo "-n 'Job is running on node '; cat $PBS_NODEFILE"
echo "------------------------------------------------------"
echo ' '
echo ' '

When submitted, this example BASH script will simply output some environment variables that exist in the HPC cluster. If you were to attempt to run this script in the terminal directly (chmod +x moab_example && ./moab_example.sh, it will most likely hang. This is because the environment variables ($PBS_...) are only available while the job is running in MOAB.

To submit the job, use the msub command. We will pass in two flags, -N to give the job a name, and -q to specify which queue we wish to submit to:

msub -q genacc_q -N mytestjob moab_example.sh

If the job submits successfully, you will see a number printed to the output. This is the JOB ID number. You can use this number to track the status of the job. Refer to our MOAB Job Management for commands you can run to monitor the status of the job in MOAB. For this example, we will wait a few moments for the job to execute and finish. You will know that the job has finished when you see two new files appear in your home directory that look something like the following:

-rw------- 1 cam02h cam02h   0 Dec  2 11:50 mytest.e6852974
-rw------- 1 cam02h cam02h 782 Dec  2 11:50 mytest.o6852974

The file that starts with mytest.e... contains any errors that occured during script execution. The mytest.o... contains the standard output. If we look at the contents of this file (more mytest.o...), we will see something similar to the following:

------------------------------------------------------
Job is running on node hpc-23-20
------------------------------------------------------
MOAB: the queue manager (qsub) is running on moab.local
MOAB: this job was submitted to the queue: genacc_q
MOAB: this job is actually running on the queue: genacc_q
MOAB: working directory is: /panfs/storage.local/genacc/home/cam02h
MOAB: execution mode is: PBS_BATCH
MOAB: job identifier is: 6852974.moab.local
MOAB: job name is: mytest
MOAB: node file is: /opt/torque/aux//6852974.moab.local
MOAB: current home directory is: /panfs/storage.local/genacc/home/cam02h
MOAB: PATH = /sbin:/usr/sbin:/bin:/usr/bin
------------------------------------------------------
Job is running on node hpc-23-20
------------------------------------------------------

If this were an actual program (C, C++, Fortran, Python, Java), the output file would contain any output that the program generated during execution.

Useful MOAB command line flags

In the example above, we used the -N and -q command line flags to specify the name and queue for the job, respectively. There are other flags that may be useful for your job as well:

Flag What it Means
-N [name] The name to give the job. This will affect what the output and error files are named
-q [queue] Which queue to submit the job to
-j oe Join the output and error messages into a single file
-m abe Send an email* when the job is aborted, or begins and ends (use any combination of a, b, or e)
-l walltime=00:00:00 How long you think the job will take to run**. This helps schedule your job more efficiently
-l nodes=xx How many nodes you would like your job to run on simultaneously**
-F "\"arg1 arg2 ... argN\"" flags to pass parameters arg1, arg2, ... argN to your submit script***

* The system will use the email address registered with your RCC account
** You are limited to the nodes and maximum run time allocated to whichever queue you submitted to
*** This is particularly useful when you want to submit multiple jobs using the same submit script but with different input parameters. See here for a simple example

For a complete list of command line flags, refer to the qsub manpage (man qsub).

Creating a submission script

Often times, you will want to use the same job name and parameters to run your job. For this reason, it is best to create a submission script for your job. Refer to the example file (moab_example_submit.sh) below:

#!/bin/bash

#MOAB -N "my_super_job"
#MOAB -l nodes=8
#MOAB -j oe
#MOAB -q genacc_q
#MOAB -m abe
#MOAB -l walltime=00:00:10

$PBS_O_WORKDIR/moab_example.sh

This submission script tells MOAB to name the job my_super_job, run it on eight nodes, combine the error and output streams into a single file, use the General Access queue, send an email when the job starts and stops, and that we think the job will take 10 seconds to run.

Save and execute this script in your home directory, and set it to be executable: chmod +x moab_example_submit.sh. Then you can queue the job anytime you wish by running:

msub moab_example_submit.sh

This will run the job with the same parameters every time.