Last Modified: Thursday, 15-Jun-2017 18:52:15 EDT

Using Batch Queues in the CSB Core

Summary of Changes

• 2017.06.15 -- Latest time limits are:
  • short: 4 hr
  • medium: 24 hr
  • long: 4 days
  • sponge: 28 days
  • cryo: 7 days
• 2011.08.19 -- "short" and "medium" time limits have been (temporarily?) increased to 4 hours and 24 hours, respectively. Try using these higher-priority queues for your shorter jobs.
• 2011.02.14 -- "short" and "medium" time limits have been increased to 2 hours and 12 hours, respectively.
• 2007.09.13 --
  • The batch queuing software has been switched from DQS to Sun Grid Engine (SGE).
  • Job submission has been modified. You cannot submit to a particular batch computer any longer. Instead, use "qsub -q [job length]" as documented below.
• 2007.03.29 -- Set long queues to 3 days time limit.
• 2007.01.17 -- Queues removed from Alpha computers, leaving only Linux queues. Documentation updated.
• 2006.07.13 -- Queues have been added on crunch3, a four-processor Linux system which supports cns big.
• 2006.06.03 -- Obsolete queues eliminated from thor and emperor. Documentation updated.
• 2004.09.17 -- General-purpose queues are being removed from the slow alpha processors -- bhima, hercules, atlas -- and from darwin. This is because there is sufficient capacity on the fast alphas. There are now no queues on SGI computers.
• 2004.04.14 -- Queues have been added on crunch1 and crunch2, dual-processor Linux systems which support cns big.
• 2003.03.20 --
  • Your desired working directory must be specified in your command script with "cd [path to your directory]" at the beginning. Furthermore, [path to your directory] must be in the form /csb/$groupname/$N/somewhere, not /amd/bunter/csb/$groupname/$N/somewhere. Otherwise, your job must be run in your /srv/people home directory, /srv/temp or some other permanently-mounted filesystem. Likewise, if using "-cwd", your job may not be located in a /csb/$groupname/$N filesystem. ("-cwd" will still work in /srv/people and other such filesystems, but it's best to get into the habit of explicitly changing directories.)
  • Queues have been added on Thor, an EV6 Alpha (like Ajax).
  • Sponge queues have been removed from all slower, EV5 computers.
• 2001.05.17 -- Upgrade to DQS version 3.3.2
  • The categories fast and safe have been eliminated.
  • Categories ev5 and ev6 have been added. The fastest Alpha computers, janus and ajax, are ev6, all others are ev5. The category alpha matches any Alpha computer.
  • The qdel command works.

What are Batch Queues?

Batch queues are implemented in the CSB Core through the Sun Grid Engine (SGE). Batch queues allow one or more computer programs to be combined into a job, which is executed on one of several computers. Batch queues provide the following features:

What Types of Queues are Available?

Four types of queue are available based on maximum clock time, and one queue based on type of job (GPU-enabled cryo-EM applications). The resources short, medium, long and sponge specify that your job will take no more than 4 hours, 24 hours, 4 days and 28 days respectively on all servers. On a given host, short jobs will be allocated more time than medium jobs, which will get more time than long jobs. Sponge jobs, however, will generally get no time if any other category is also running. The queuing system will kill jobs which exceed the stated time limit.

There is also a separate 7-day "cryo" queue for running on a multi-GPU server. This is primarily for cryoEM jobs (hence the name), but molecular dynamics will soon be available as well.

To submit a job named test.com to a medium queue on the most-available computer, you type:

qsub -q medium test.com

For other examples, see the section, Common SGE Commands.

If you incorrectly specify the resource(s) that you need, SGE will complain. For example, if you type, qsub -q med ... instead of qsub -q medium ..., your job submission will fail with an error message.

Back to Top

Where Should I Run My Job?

Usually, you will want to run your job in a queue with the shortest time limit, and therefore the highest priority. IE, if you know your job will take about 1 day, you will want to run it in a long queue. The simplest thing is to use the command,

qsub -q long (jobname)

At this writing (2/2011), the SGE system will select which computer will run your job based on current job load, with the more powerful systems given greater weight. If this is a problem, see Miscellaneous Information below.

Back to Top

What Specific Queues are Available?

These queues are available, as of 15 June 2017. However, the configuration changes faster than the documentation. For a list of active queues, use the command, qstat -f.
 

Back to Top

Setting up a SGE Job

1. Create a SGE script file for your job. This is simply a script for your favorite shell, containing the commands that you want to execute in batch. Unless you specifically include a line to "cd" to a particular directory at the beginning of your script, your job will be run from your home directory, so define your file paths accordingly.

2. Submit your job to the appropriate queue, using the qsub command.

Common SGE commands

Summary Examples

qsub  -q medium test.com
qstat -f -q medium
qdel 67

Submitting a job -- qsub

qsub -q medium test.com
qsub -e test.err -o test.out -q short-crunch3 test.com

• qsub -- This command submits a job to a DQS queue.
• test.com -- this is the name of the script you are submitting to DQS to be executed.
• DO NOT USE -cwd -- this flag says to execute your job from your current working directory instead of from your home directory. THIS WILL NO LONGER WORK; directly specify the directory to use by including "cd [path to your directory]" at the beginning of your script.
• -q medium -- run the job on a medium queue on the computer host with the lowest adjusted load average.
• -e test.err -- direct error output (stderr) to the file ~/test.err instead of the default (~/[script filename].e[job number]). To send somewhere other than your home directory, specify the full path.
• -o test.out -- direct standard output (stdout) to the file ~/test.out instead of the default (~/[script filename].o[job number]). To send somewhere other than your home directory, specify the full path.

See the SGE documentation for a list of all possible options. Generally, options can be included on the command line, or in your command file, as shown in the example in the section on Setting up a SGE Job.

Discovering queue status -- qstat

qstat
qstat -f -q long 

• qstat -- reports status of queues and jobs. With no arguments, lists all jobs running in all queues.
• -f -- list all queuues, along with any jobs running in them.
• -q long -- shows status of long queues only.

Deleting a job -- qdel

Each DQS job has a unique job-id. The job id is reported when you submit the job, and can be discovered at any time using the command qstat. To delete a job, use the command

qdel jobid

Back to Top

Miscellaneous Information

Trouble with specific batch computers

With the current SGE setup, it is no longer possible to select a particular computer in the batch queuing system for running your job. Since all of the batch computers are supposed to be configured the same, this should not normally be a problem. However, if you find that your job doesn't run when submitted to a particular machine, contact the Core Staff for assistance.

Back to Top

How to Get More Information

The documentation that came with the SGE system is available in PDF format. SGE is a large and complex system, of which we use only a small part, so the full document set can be overwhelming.

A User Guide describes the concepts and workings of SGE.

The Installation and Administration manuals might be of some help to the staff.

Documentation for the old DQS software is also available: User Guide, Reference, and Installation/Maintainance.

Last Modified: Thursday, 15-Jun-2017 18:52:15 EDT