Supercomputer Facility

Queues

 

Contents:

Overview

This document describes the Portable Batch System available on the MSU Chemistry Department supercomputer. It provides only a brief introduction to help users get started using PBS. For more details, consult the appropriate man pages. The pbs man page (type "man pbs") lists all of the PBS commands and the corresponding man pages.

Queue Structure

There are two queues on hbar. There are no memory, disk or time limits on any of the queues. However, the queues do have different limits on the maximum number of processors that each user may use. Also, the queue system will limit the total number of running jobs so that no more than 26 processors are used by the queue system.

workq

The name of the default queue is workq. This is the queue where jobs will be placed if no other queue is specified. All users except for those in the theory group must use this queue. Theory group users may NOT use this queue. Workq has a limit of 2 CPUs per user, and a total limit of 8 CPUs. Each user may run either 2 single-CPU jobs, or one 2-CPU job in workq. If a user tries to submit a job to workq requesting more than 2 CPUs, the request will be rejected. If a user submits more than 2 single-CPU jobs, the first 2 will run (if the queue is not already full). However, the remaining jobs will be held in the queue until one of the first jobs completes. As each job completes, the queue system will begin executing the next job in line, never allowing more than 2 CPUs per user.

theory

The second queue is named theory, and as its name suggests, it is reserved for theory group users only. The limit on this queue is 4 CPUs per user, and a total of 18 CPUs. If workq is not using all 8 of its allocated CPUs, then the theory queue may "borrow" some CPUs from workq. However, workq has priority on these processors, so they will be given back as soon as workq needs them. When the borrowed CPUs are given back, any theory jobs running on them will be suspended until more processors become available.

Submitting a Job

Jobs are submitted to PBS using the qsub command. Since this command can require extra work to use, a script to make the process easier is available; see "An Easier Way" below.

Qsub accepts a script which contains the shell commands to be executed when the job runs. You can also instruct qsub to modify the characteristics of the job by embedding switches in the script or by placing them on the command line. There are many switches available; see the qsub man page for details.

A script file can be as simple as a single line of text containing the command to run. Here is an example script file:

 a.out <file.input >file.output

This job could be submitted to the default queue with the following command (assume the script file is named "scriptfile"):

 qsub scriptfile

It is important to note that PBS will start a new login session for your script. One implication of this is that your script will have its working directory set to your home directory. If your script needs to be in a different directory, you will need to add the appropriate "cd" command to your script.

Jobs can also be submitted interactively with qsub. For example, instead of putting "a.out ..." in a file and then submitting that file, the job could be submitted interactively as follows:

 qsub <ENTER> 
 a.out <file.input >file.output <ENTER>
 <CONTROL-D>

An Easier Way

The qsub command can be cumbersome to use at times, so a locally developed shell script named "submit" is available to make the job submission procedure easier. This script copies its arguments to a temporary file, then submits the file to the queue system using qsub. To allow for qsub arguments, the script looks for a "-qsub" argument, and passes all following arguments to the qsub command. For convenience, this script gets the current working directory, and adds a command in the qsub script to change to this directory before running the job.

The syntax of submit is:

 submit command [command_args] [-qsub qsub_args]

To include file redirection, the command must be enclosed in quotes. For example:

 submit "a.out <infile >outfile" -qsub -q workq

Submitting Parallel Jobs

To submit parallel jobs, you must tell the queue system how many processors your job will use. This is done with the "-l ncpus=" option to the qsub command (the "l" character is a lower case L, not the number one). For example, to submit a job that will use 4 processors, type:

 qsub -l ncpus=4 scriptfile

Submitting Aces Jobs

To submit an Aces job to the queue system, type the following command:

 aces2sub [-64] [-m email] [additional_qsub_arguments] filename

where "filename" is the name of your input file without the ".inp" extension. The arguments in brackets are optional. The -64 option will invoke the 64-bit version of the program. If an email address is given, the output file will be sent to that address upon completion of the job.

Submitting Gaussian Jobs

Submitting Gaussian jobs can be a complicated process, so a separate script for this purpose has already been constructed. It will accept qsub switches in addition to the input file on the command line. Use the following command to submit a Gaussian job: 

 g98sub [-m email] [qsub_arguments] filename

where "filename" is the name of your input file without the ".inp" extension. The optional qsub arguments will be passed to PBS. If an email address is given, the output file will be sent to that address upon completion of the job. To submit a Gaussian job to the default workq queue, use this command: 

 g98sub filename

Note for parallel use: the g98sub command will look inside your input file for the %NProc= line, and it will automatically add the correct qsub options for a parallel job. You do NOT have to use the "-l ncpus=" option with g98sub.

Submitting GAMESS Jobs

Submitting GAMESS jobs is much like submitting Gaussian jobs. Use the following command to submit a GAMESS job: 

 gmssub [-b basisfile] [-m email] [-n ncpus] [qsub_args] file_name

where "filename" is the name of your input file without the ".inp" extension. The optional qsub arguments will be passed to PBS. If an email address is given, the output file will be sent to that address upon completion of the job. GAMESS can run in parallel on hbar, and you must specify the number of processors to use on your job. You MUST add the "-l ncpus=" qsub option for parallel jobs. This may seem redundant with the -n option, but it is not. GAMESS actually starts 2*n processes, however only n of them are computationally intensive. You might be able to specify a number smaller than 2*n in the ncpus option.

If you do not wish to run your job in parallel, specify 1 processor.

Submitting Molpro Jobs

Use the following command to submit a Molpro job: 

 m00sub [molpro options] [-q queuename] filename

where "filename" is the name of your input file. The arguments in brackets are optional.

Interactive Jobs

PBS allows interactive programs to be run in the queue system. To start an interactive job, type:

 qsub -I

You should see a couple of messages similar to:

 qsub: waiting for job 824.hbar.cem.msu.edu to start
 qsub: job 824.hbar.cem.msu.edu ready

Then you should get a prompt back. At this point, your shell and every command you run will be executed under control of the queue system. This can be used to run short Spartan jobs in the PBS queues, for example. However, when you logout of this shell, your queue session will end and all processes you started in the session will be terminated.

Getting the Status of a Job

To check the status of a job, use the qstat command. For example:

 qstat

will list all jobs running on hbar.

Deleting a Job

The qdel command is used to delete a job from a queue. First get the job ID number by using the qstat command, then type qdel followed by the job ID.

For example, to delete job number 28, type:

 qdel 28

Summary

To do this: Type this:
Submit a script to the default queue (workq) 
qsub shell_script
Submit a command to the default queue (workq) 
submit command
Submit a job to the theory queue 
qsub -q theory shell_script
Submit a Gaussian job 
g98sub filename
Check the status of your jobs 
qstat
Delete job number 26 
qdel 26