MATLAB

Version: 
2014b
A high-level language and interactive environment for numerical computation, visualization, and programming

Availability of MATLAB on RCC resources

MATLAB is installed on all spear, hpc, and condor nodes and login nodes. However, your MATLAB job may fail due to unavailability of MATLAB licenses due to usage. This can be avoided by checking the number of licenses available (inside MATLAB) using the command,

license('checkout','MATLAB')

Non-interactive jobs submitted using the scripts shown in this page will check for available MATLAB licenses before running. However, if you want to use large number of MATLAB jobs simultaneously, use MATLAB Compiler as described below to create executables from your codes. You will not be using any MATLAB licenses this way.

Using MATLAB at RCC

MATLAB can be used in different modes at different RCC resources.

MATLAB Profiles

Creating a MATLAB profile on HPC

To submit MATLAB jobs to HPC cluster by logging in to HPC, you have to create a profile. This needs to be done only ONCE per user per MATLAB version. Copy the profile file to your MATLAB working folder using the following command.

cp /panfs/storage.local/opt/userfiles/hpc.settings .

Now, follow these steps to import this file to MATLAB.

  • From the HOME tab in MATLAB select "Manage Cluster Profiles" from the "Parallel" pull down menu to open "Cluster Profile Manager" window
  • Click "Import" and browse to the location where you downloaded "hpc.settings" file and click "Open"
  • Close the "Cluster Profile Manager" window

If you cannot open MATLAB GUI, you can use following MATLAB command to import the profile.

hpc = parallel.importProfile('/panfs/storage.local/opt/userfiles/hpc.settings');

The procedure to submit jobs to HPC cluster is explained in this section

Creating a MATLAB profile on your local machine

This discussion only applies if you are planning to submit MATLAB jobs directly to HPC from your own workstation without first logging into hpc-login.rcc.fsu.edu. If you are off campus, you still have to use VPN. Note that you need the MATLAB R2013b version installed in your machine for this. The profile used here is slightly different and can be created on your LOCAL machine (desktop or laptop) using the following steps.

  • Open "ClusterProfile Manager" as shown in the previous section
  • Click "Add" and select Custom>>Generic from the menus
  • Right click on the "GenericProfile1" on the left pane and rename it to "hpc"

The actual job submission procedure is explained here.

Interactively Running MATLAB

Following discussion applies to running MATLAB on SPEAR nodes. Click here to find how to connect to SPEAR cluster. This is mainly for running short jobs or for testing and debugging of production runs. You can interactively run MATLAB on SPEAR and any login node as you would normally do on your own laptop simply by typing,

module load matlab
matlab

in the terminal window. You can use MATLAB Parallel Computing Toolbox (PCT) to utilize more than one cores as login nodes contain multiple cores. To use any parallel features of MATLAB (such as parfor), there are two modes : pmode and mathlabpool. pmode is an interactive mode where you see the individual workers (labs) in a GUI. The mathlabpool is the mode where the labs will run in the background. These modes are specially useful during the development stage of a complicated parallel code. The maximum number of workers that can be used in either mode is limited to 12 by MATLAB.

Interactive parallel computing using `pmode`

As an example, you can invoke interactive pmode with 4 workers by,

pmode start local 4

which will open a Parallel Command Window (PCW). The workers then receive commands entered in PCW (at P>> prompt), process them, and send the command output back to the PCW. Variables can be transferred between the MATLAB client and the workers. To copy the variable x of lab 2 to xc on client use

pmode lab2client x 2 xc

and to copy from client to lab 2 use

pmode client2lab xc 2 x

Plotting and all other operations can be done inside PCW. One can distribute an array x among workers using the command codistributed(x,'convert') and use numlabs, labindex, labSend, labReceive, labProbe, labBroadcast, labBarrier functions similar to MPI commands for parallelizing. Please refer to MATLAB manual for a full discussion of commands. Entering the command,

pmode quit

in the PCW will end the session and return the licenses.

Interactive parallel computing using matlabpool

The mathlabpool mode is invoked with 4 workers by,

matlabpool local 4

and exited by typing,

matlabpool close

It is also possible to invoke mathlabpool using batch command. An example is shown below.

batch('test1','matlabpool', 2);

where test1.m is a MATLAB script and we want to run it with 2 workers. The program runs in the background and will have to direct the output to be written into files for longer running jobs. The advantage of this approach is that you can create set of jobs using same function with different input parameters by,

batch('matlabpool', 'test1', number of output arguments, {x1,...,xn});

where x1, ..., xn are different function inputs.

Back to Top

Interactive parallel computing using parpool

The matlabpool utility will be replaced by the parpool utility for future versions of Matlab. The syntax of parpool is

parpool
parpool(poolsize)
parpool('profile',poolsize)   
parpool('cluster',poolsize)
ph = parpool(...)

where poosize, profile, and cluster are respectively the size of the matlab pool of workers, the profile, or the cluster you created, and the last one creates a handle ph for the pool you create.

parpool enables the full functionality of the parallel language features (parfor and spmd) in MATLABĀ® by creating a special job on a pool of workers, and connecting the MATLAB client to the parallel pool.

The following example creates a pool of 4 workers, and runs a parfor-loop using this pool:

>>parpool(4)
>> parfor i = 1:10
        feature getpid;
        disp(ans)
     end
     1172
     1172
     1171
     1171
     1169
     1169
     1170
     1172
     1171
     1169
 >> delete(gcp)

The next example creates a pool of 4 workers, and runs a simple spmd code block using this pool:

  >> ph = parpool('local',4)       % ph is the handle of the pool
  >> spmd
  >> a = labindex
  >> b = a.^2
  >> end
 Lab 1: 
a = 1 
b = 1
Lab 2: 
a = 2
b = 4
Lab 3: 
a = 3
b = 9  
Lab 4:
a = 4
b = 16
>> delete(ph)

Note. You can not run more than one interactive parpool sessions. You have to delete the current parpool session before starting a new one. To delete the current session, use

  delete(gcp)
  delete(ph)

where gcp utility returns the current pool, and ph is the handle of the pool.

Interactive parallel computing using GPUs

Some Spear nodes (Spear-1 to Spear-8) have GPU cards. MATLAB is capable of using Nvidia GPGPUs to accelerate calculations and most of the built in functions do come with GPU versions as well. When logging into a Spear node using the IQS, note that the gpu option will be set to 0, but the GPUs will still be available on the first 8 Spear nodes. It is possible to SSH to another GPU-capable Spear node if one isn't automatically assigned. First we should verify that a GPU is available on the currently used Spear node. This should return information with the name Tesla M2050:

gpuDevice

If the device (GPU) is found, we can try an example. We'll write a mandelbrot program, f_mandelbrot.

function [mbset, t] = gpu_mandelbrot(niter, steps, xmin, xmax, ymin, ymax)
t0 = tic();
x = gpuArray.linspace(xmin, xmax, steps);
y = gpuArray.linspace(ymin, ymax, steps);
[xGrid,yGrid] = meshgrid(x, y);
c = xGrid + 1i * yGrid;
z = zeros(size(c));
mbset = zeros(size(c));
for ii = 1:niter
    z = z.*z + c;
    mbset(abs(z) > 2 & mbset == 0) = niter - ii;
end
t = toc(t0);

Run the program and display the results with the commands

[mandelSet, time] = gpu_mandelbrot(3600,100,-2,1,-1.5,1.5)
surface(mandelSet)

The arrays x and y are generated on the GPU, utilizing its massively parallel architecture, which also handles the remainder of the computations that involve these arrays.

Non-interactive job submission

Single core jobs

Following discussion applies to deploying MATLAB jobs on HPC nodes. Click here to find how to connect to HPC cluster. After your code is successfully tested, the production runs should be submitted through a batch scheduler (MOAB). Following is a sample submit script to run the MATLAB program test1.m that uses a single core. Note that test1.m should be a function and NOT a script.

#!/bin/bash
#MOAB -W x=GRES:MATLAB
#MOAB -j oe
#MOAB -l nodes=1:ppn=1
#MOAB -q genacc_q
#MOAB -l walltime=01:00:00 #Change the walltime as necessary
cd $PBS_O_WORKDIR
module load matlab
matlab -nosplash -nojvm -nodesktop -r "test1 exit"

Copy this script and save it (eg. test1.pbs). This example is using one hour of wall time and change this to suit for your jobs requirements. Then, submit this job as,

msub test1.pbs

Note that PCT cannot be used within test1.m. i.e. you cannot use parfor or any other command that use more than one core.

Using MATLAB with Condor

Following discussion applies to running MATLAB on condor nodes. Click here to find how to connect to condor cluster. Your Matlab job will consist of three files:

  • YOUR_SCRIPT.m - Your primary Matlab script, which you will upload to Condor
  • matRun.sh - A simple BASH script that will execute your MATLAB script on Condor
  • YOUR_SCRIPT.submit - A text file with parameters that instruct Condor how to execute your script

None of the file names matter, but it is good that the second ends with .sh and the third with .submit to keep consistent with general convention.

Create a simple matRun.sh BASH script to execute your MATLAB file on Condor:

#!/bin/bash
/opt/matlab/current/bin/matlab -nosplash -nodesktop -nojvm -r "run('./YOUR_SCRIPT.m'); quit"

Create a YOUR_SCRIPT.submit file. Here is an example to guide you:

universe = Vanilla
Executable = matRun.sh
Arguments =
input =
requirements = (OpSys =="LINUX" && Arch =="X86_64" && Matlab == "true")
Rank = memory
should_transfer_files = YES
transfer_input_files = YOUR_SCRIPT.m
when_to_transfer_output = ON_EXIT
notification = Complete
notify_user = YOUR_EMAIL@fsu.edu
Output = $(Process).out
Log = $(Process).log
Queue 1
  • Line 2: The executable declaration should point to your BASH script, which will be run inside the Condor cluster when you execute the job
  • Line 3: The arguments declaration allows you to specify arguments (e.g. --some-option) to your BASH script if your script allows them. These would be the same arguments used when executing the program on the command line. Note that our example does not make use of arguments
  • Line 4: The input declaration allows you to specify a input file that your BASH script will use, if it uses one. Our example does not
  • Line 5: The requirements declaration instructs Condor to send our job to a machine in the cluster with certain prerequisites. In this case, we are instructing Condor to send this job to a 64-bit Linux computer with Matlab installed
  • Line 7: The should_transfer_files directive is necessary and instructs Condor to copy the necessary files for execution to the machine that will do the work
  • Line 8: The transfer_input_files directive tells Condor which files to copy; include the YOUR_SCRIPT.m file and any necessary input files here, separated by a comma. We only have one file, since we're not using an input file
  • Line 9: The when_to_transfer_output directive tells Condor to copy the output files back to your home directory when the job completes
  • Lines 10, 11: The notification and notify_user directives instruct Condor when and whom to notify. Change notification to NEVER to not receive an email upon job completion. If you set notification to YES, Condor will send you emails when your job completes
  • Lines 12, 13: The Output and Log directives are files that Condor will generate for output. In this example, we specified $(Process), which refers to the process identifier for the job. In other words, the first execution of the program will output "0.out" and "0.log", the second will output "1.out" and "1.log", etc. You can use a static filename (e.g. "output.out") if you want. The files will be written to the same directory that the BASH script is run from

Copy all of your files to the Condor submit node (condor-login.rcc.fsu.edu). Login to the Condor submit node, and browse to the location of your files. Set the permissions on your BASH script to executable:

chmod 700 matRun.sh

Execute your script:

./matRun.sh

The output from successful execution of your script should appear when the script completes.

Multiple core jobs

Following discussion applies to deploying MATLAB jobs on HPC nodes. Click here to find how to connect to HPC cluster. You can use multiple cores in a single processor by changing the "ppn" in the script. The maximum number of cores available is currently 8.

#!/bin/bash
#MOAB -W x=GRES:MATLAB:4
#MOAB -j oe
#MOAB -l nodes=1:ppn=4 #to use 4 cores in a SINGLE processor
#MOAB -q genacc_q
#MOAB -l walltime=01:00:00 #Change the waltime as necessary
cd $PBS_O_WORKDIR

module load matlab_dcs
matlab -nosplash -nodesktop <<EOF
matlabpool open local 4

test2 %This is your matlab FUNCTION

matlabpool close
exit
EOF

Back to Top

Submitting Multiple Node Jobs From HPC Login Node

Following discussion only applies to deploying MATLAB jobs from HPC login nodes to HPC compute nodes. Click here to find how to connect to HPC cluster. We have only used the resources within a single node so far. To use more than one node, we need to use Torque scheduler along with MATLAB Distributed Computing Engine (MDCE) properly. First, load the matlab_dcs module as,

module load matlab_dcs
matlab

You should have created the hpc profile as described in here. This needs to be done ONLY once. After creating the profile, you can use the following as a template to submit jobs from MATLAB commandline on any HPC login node. Make sure that you created the matlab_work folder or change line 3 to point to the folder where you store all your codes. Also, change $USER in line 3 to your user name and change the name of the queue in line 4.

cluster = parcluster('hpc');
%REPLACE $USER in the following line with your user name and create the folder "matlab_work"
set(cluster, 'JobStorageLocation', '/panfs/storage.local/genacc/home/$USER/matlab_work');
%change the name of the queue (I'm using "genacc_q") as necessary
set(cluster, 'SubmitArguments', '-j oe -q genacc_q');
%Change the number of workers below as necessary. I have 2 nodes x 2ppn = 4 workers
set(cluster, 'ResourceTemplate', '-l nodes=2:ppn=2 -l walltime=00:10:00');
set(cluster, 'NumWorkers', 4);
%Note that MATLAB needs n+1 workers assigned to it as 1 would become the "master"
%For example, if you assign 4 workers, only 3 will do actual work

%creating an independent job
j = createJob(cluster, 'Profile', 'hpc');
%creating tasks. My test3 FUNCTION takes one input (100 in this example) and generates 1 output
createTask(j, @test3, 1, {100});
%To create two independant tasks with input values 100 and 200,
%createTask(j, @test3, 1, {{100},{200}});

submit(j)

%Collect outputs. You need to wait until the job is done!
%If you use the command wait(j), you will not get the MATLAB prompt back until all jobs are finished

%fetch outputs (if any). However, it is a good idea to have the outputs save into files within your code
taskoutput = fetchOutputs(j);

%VERY IMPORTANT! delete job object
delete(j)

You can create a communicating job simply by replacing the createJob command by,

j = createCommunicatingJob(cluster, 'Profile', 'hpc');

Using MATLAB Compiler

If you need to use large number of MATLAB workers simultaneously or not enough licenses available at runtime, you can make an executable of your code and run as many instances of it as you want. You can use the MATLAB compiler, mcc, on Spear or any of the HPC Submit nodes in order to create a binary of the code. This way, unless MATLAB Distributed Computing Engine toolbox features are used, no licenses will be consumed by running mcc generated binaries. Be sure to compile in whichever environment you intend to run the code.

For a non parallel code test1.m you can use the MATLAB command:

mcc -R -nodisplay -R -nojvm -R -nosplash -R -singleCompThread -m test1.m

Note that this only works with matlab module, not matlab_dcs module. Above command will create the script run_test1.sh and the executable test1 (names of actual files will be different depending on the name of your script). A brief description of these compiler flags follows.

  • -R: Specifies runtime options, and must be used with the other runtime flags (nodisplay, nosplash, etc)
  • -R -nodisplay: Any functions that provide a display will be disabled
  • -R -nojvm: Disable the Java Virtual Machine
  • -R -nosplash: Starts MATLAB, but does not display the splash screen
  • -R -singleCompThread: Runs only a single thread in the runtime environment
  • -m: Generates a C binary (-p would generate a C++ binary)

After successful completion, mcc creates the binary file, a script to load necessary environment variables and run the binary, a readme.txt, and a log file. This binary can be run via the generated script with the following command, or this command can be added to a MOAB script to submit to the HPC.

<full-path-to-script>run_test1.sh /opt/matlab/current <input-arguments>

Note that input arguments will be read in as strings, so any code that utilizes these arguments must convert these strings to the correct data type within the code if necessary. A concern with this method of compiling and running a MATLAB program is that the binaries generated will contain all of the toolboxes available in the user's MATLAB environment, resulting in large binary files. To avoid this, simply use the -N compiler flag. This will remove all but essential toolboxes, and other tools or .m files required can be attached with the -a option. A recommended way to generate serial binaries is

mcc -N -v -R -nodisplay -R -nojvm -R -nosplash -R -singleCompThread -m test1.m

Here the -v option enables verbose mode. To run a MATLAB program in parallel using the Parallel Computing Toolbox, the -p distcomp option must be added to the command line arguments,

mcc -N -v -p distcomp -m test1.m

To run this binary, a parallel profile should be provided:

<full-path-to-script>run_test1.sh /opt/matlab/current -mcruserdata ParallelProfile:/opt/matlab/r2013b/toolbox/distcomp/parallel.settings

Here, the default local profile is used. This can be used in any MOAB submit script without any modification, and no MATLAB licenses will be used. The generation of binaries is the same for MATLAB Distributed Computing Engine jobs, but a different MOAB aware profile needs to be provided. As described in the RCC MATLAB documentation, users should always used the supplied hpc profile for these types of jobs. See the section on parallel computing to find what lines should be added to a MATLAB script when using the Distributed Computing Engine. The submit command for a distributed job is

<full-path-to-script>run_test1.sh /opt/matlab/current -mcruserdata ParallelProfile:/panfs/storage.local/opt/userfiles/hpc.settings

Back to Top

Remote Job Submission From Local Workstations

This section describes how to submit MATLAB jobs from your own workstation (desktop/laptop) to HPC cluster. Use VPN if you are off campus.

The following figure illustrates the scheme of remote job submission.

Remote Submission VS Login-Node Submission

To submit MATLAB job to the HPC cluster from your local machine which is NOT on a shared file system with the cluster of compute nodes, a generic scheduler interface must be used. This is significantly different from the case when you submit job to the HPC cluster from the login node (i.e., hpc-login.rcc.fsu.edu) for which the client session and the compute nodes are all on the same shared file system (i.e., 'panasas').

A generic cluster object must be created before you can create and submit communicating or independent jobs to the HPC cluster. The following functions must be provided before you can create a generic cluster object:

function description
CancelJobFcn Function to run when cancelling job
CancelTaskFcn Function to run when cancelling task
CommunicatingSubmitFcn Function to run when submitting communicating job
DeleteJobFcn Function to run when deleting job
DeleteTaskFcn Function to run when deleting task
GetJobStateFcn Function to run when querying job state
IndependentSubmitFcn Function to run when submitting independent job

We have provided these functions in the tar ball matlab.tar located at

/panfs/storage.local/opt/userfiles/matlab.tar

, and these functions should be copied to

local_matlab_root/toolbox/local/ 

where local_matlab_root is the matlabroot on your local workstation.

Following are the steps you need to follow:

1. Setting up password-less login to HPC

The first step is to configure password-less login to HPC if you have not already done so. This is essential so that MATLAB can connect from your local machine to HPC cluster. The procedure depends on the operating system you are using and please refer to "Using SSH Keys" section here for instructions. Make sure that you do NOT enter a password when prompted during this process.

2. Copying necessary files to local machine

Open a terminal window and copy the "matlab.tar" file from HPC to your machine and extract it,

$ scp you@hpc-login.rcc.fsu.edu:/panfs/storage.local/opt/userfiles/matlab.tar .
$ tar -xvf matlab.tar
matlab/
matlab/communicatingSubmitFcn.m
matlab/independentSubmitFcn.m
matlab/getSubmitString.m
matlab/communicatingJobWrapper.sh
matlab/createSubmitScript.m
matlab/extractJobId.m
matlab/deleteJobFcn.m
matlab/getRemoteConnection.m
matlab/getJobStateFcn.m
matlab/independentJobWrapper.sh
matlab/getCluster.m

Now, move all the files except getCluster.m to the toolbox/local folder on YOUR machine. For example, if you installed MATLAB to /usr/local folder in YOUR LOCAL machine, this can be done by,

mv matlab/!(getCluster.m) /usr/local/MATLAB/R2013b/toolbox/local/

Make sure the path is correct before you copy and you have administrative privileges on the local machie.

3. Update the `getCluster.m` file

matlab.tar provides several functions you need to create a generic cluster object. The function getCluster() use these functions to configure a cluster object for you.

Here is the content of the getCluster.m

function [ cluster ] = getCluster(ppn, queue, rtime, LocalDataLocation, RemoteDataLocation)

%Find the path to id_rsa key file in YOUR system and update the following line
 username = 'YOUR HPC USER NAME'
 keyfile = '/home/USER/.ssh/id_rsa'; %Your actual path may be DIFFERENT!

%Do not change anything below this line

 if (strcmp(username, 'YOUR HPC USER NAME') == 1)
    disp('You need to put your RCC user name in line 4!')
    return
 end

 if (exist(keyfile, 'file') == 0)
    disp('Key file path does not exist. Did you configure password-less login to HPC?');
    return
 end

 ClusterMatlabRoot = '/opt/matlab/current';
 clusterHost='submit.hpc.fsu.edu';

 cluster = parcluster('hpc');
 set(cluster,'HasSharedFilesystem',false);
 set(cluster,'JobStorageLocation',LocalDataLocation);
 set(cluster,'OperatingSystem','unix');
 set(cluster,'ClusterMatlabRoot',ClusterMatlabRoot);
 set(cluster,'IndependentSubmitFcn',{@independentSubmitFcn,clusterHost, ...
     RemoteDataLocation,username,keyfile,rtime,queue});
 set(cluster,'CommunicatingSubmitFcn',{@communicatingSubmitFcn,clusterHost, ...
     RemoteDataLocation,username,keyfile,rtime,queue,ppn});
 set(cluster,'GetJobStateFcn',{@getJobStateFcn,username,keyfile});
 set(cluster,'DeleteJobFcn',{@deleteJobFcn,username,keyfile});

The five input arguments of the function getCluster are

input-args description
ppn processor/core per node
queue queue you want to submit job to (e.g., backfill, genacc_q)
rtime wall time
LocalDataLocation directory to store job data on your workstation
RemoteDataLocation directory to store job data on your HPC disk space

The first time you download, edit the getCluster.m file to,

  • Include your RCC user name in line 4
  • Include the correct path to your id_rsa key file in line 5 (see section "Setting up password-less login to HPC")

Before you call this function, create a separate folder (to be used as "RemoteDataLocation" in the getCluster.m script) in your HPC disk space to store runtime MATLAB files, for example,

mkdir -p  $HOME/matlab/work

, and create a folder in your local machine to be used as the LocalDataLocation. Clean these folders regularly after finishing a job.

4. Submitting jobs

Following lines can be used as a template to create a generic cluster object which you will use to submit jobs to HPC

processors = 4;        %Number of processors used. MUST BE LESS THAN OR EQUAL TO 32
ppn = 4;                   %Number of cores used per processor
queue = 'genacc_q'; %Replace this with your choice of queue
time = '01:00:00';      %Run time
LocalDataLocation = ''; % Full path to the matlab job folder on YOUR machine
% the following is the Full path to a matlab scratch folder on HPC
RemoteDataLocation = '/panfs/storage.local/genacc/home/USER/matlab/work'; 

cluster = getCluster(ppn, queue, time, LocalDataLocation, RemoteDataLocation);

Following is an example to create a communicating job for the cluster:

j1 = createCommunicatingJob(cluster); %This example creates a communicating job (eg:parfor)
j1.AttachedFiles = {'testparfor2.m'}; %Send all scripts and data files needed for the job
set(j1, 'NumWorkersRange', [1 processors]);
set(j1, 'Name', 'Test'); %Give a name for your job
t1 = createTask(j1, @testparfor2, 1, {processors-1});
submit(j1);

%wait(j1); %MATLAB will wait for the completion of the job
%o=j1.fetchOutputs; %Collect outputs after job is done

Note. Only use the last two steps for testing of small jobs. Production jobs will probably wait in the queue for some time and therefore MATLAB on your workstation will keep looking for the job to be finished if you used this.

Following is an example to create an independent job for the cluster:

 j2  = createJob(cluster);     % create an independent job
 t2 = createTask(j2, @rand, 1, {{10,10},{10,10},{10,10},{10,10}});  % create an array of 4 tasks
 submit(j2) 
 wait(j2)
 o2 = fetchOutputs(j2)      % fetch the results 
 o2{1:4}                           % display the results

Note. A communicating job contains only one task. However, this task can run on multiple workers. Correspondingly the task can contain parfor-loop or spmd code block to improve the performance. On the other hand, an independent job can contain multiple tasks. These tasks do not communicate with each other and each task run on a single worker.

Back to Top

MATLAB Workshops

We regularly offer MATLAB workshops and following are the slides for the workshop on 01/29/2015. parallel_matlab.pdf

Category: 
Applications and Tools
Available On: 
HPC
Condor
Spear