Submitting and managing jobs with Torque and Moab#

Submitting your job: qsub#

Once your job script is finished, you submit it to the scheduling system using the qsub command to place your job in the queue:

$ qsub <jobscript>
205814.leibniz

When qsub successfully queues your job, it responds with a job ID, 205814.leibniz in the example above. This is a unique identifier for your job, and can be used to manage it. In general, the number will suffice for this purpose.

As explained on the pages on Specifying job resources and Specifying job name, output files and notifications, there are several options to inform the scheduler about the resources your jobs requires, or whether you want to be notified on events related to your job.

These options can be specified

  • at the top of your job script, or/and

  • as additional command line options for qsub.

In case both are used, options given on the command line take precedence over those in the job script. For example, suppose the job script has the following directive:

#PBS -l walltime=2:00:00

However, when submitting it with qsub, you specify -l walltime=1:30:00, the maximum walltime for your job will be 1 hour, 30 minutes.

Starting interactive jobs#

Though our clusters are mainly meant to be used for batch jobs, there are some facilities for interactive work:

  • The login nodes can be used for light interactive work. They can typically run the same software as the compute nodes. Some sites also have special interactive nodes for special tasks, e.g., scientific data visualization. See the “ Infrastructure” section where each site documents what is available. Examples of work that can be done on the login nodes :

    • running a GUI program that generates the input files for your simulation,

    • a not too long compile,

    • a quick and not very resource intensive visualization.

    We have set limits on the compute time a program can use on the login nodes.

  • It is also possible to request one or more compute nodes for interactive work. This is also done through the qsub command. Interactive use of nodes is mostly meant for

    • debugging,

    • for large compiles, or

    • larger visualizations on clusters that don’t have dedicated nodes for visualization.

To start an interactive job, use qsub’s -I option. You would typically also add several -l options to specify for how long you need the node and the amount of resources that you need. For instance, to use a node with 20 cores interactively for 2 hours, you can use the following command:

qsub -I -l walltime=2:00:00 -l nodes=1:ppn=20

qsub will block until it gets a node and then you get the command prompt for that node. If the wait is too long however, qsub will return with an error message and you’ll need to repeat the command.

If you want to run graphical user interface programs (using X) in your interactive job, you have to add the -X option to the above command. This will set up the forwarding of X traffic to the login node, and ultimately to your terminal if you have set up the connection to the login node properly for X support.

Note

  • Please be reasonable when requesting interactive resources. On some clusters, a short walltime will give you a higher priority, and on most clusters a request for a multi-day interactive session will fail simply because the cluster cannot give you such a node before the time-out of qsub kicks in.

  • Please act responsibly, interactive jobs are by definition inefficient: the systems are mostly idling while you type.

Viewing your jobs in the queue#

Two commands can be used to show your jobs in the queue:

  • qstat show the queue from the resource manager’s perspective. It doesn’t know about priorities, only about requested resources and the state of your job: Still idle and waiting for resources, running, completed, …

  • showq shows the queue from the scheduler’s perspective, taking priorities and policies into account.

qstat#

On the VSC clusters, users will only receive a part of the information that qstat offers. To protect the users’ privacy, output is always restricted to the user’s own jobs.

To see your jobs in the queue, enter:

$ qstat

This will give you an overview of all jobs including their status, possible values are listed in the table below.

status

meaning

Q

job is queued, i.e., waiting to be executed

S

job is starting, i.e., its prologue is executed

R

job is running

E

job is exiting, i.e., its epilogue is executed

C

job is completed, i.e., finished.

H

job has a hold in place

Several command line options can be specified to modify the output of qstat:

  • -i will show you the resources the jobs require.

  • -n or -n1 will also show you the nodes allocated to each running job.

showq#

The showq command will show you information about the queue from the scheduler’s perspective. The showq output is split up according to the following categories:

  • Active jobs are actually running (or just about to get started or terminated). Their state in the showq output will be “Running” and they are sorted according to their expected end time.

  • Eligible jobs are queued and considered eligible for scheduling. Their state in the showq output will be “Idle” and they are sorted according to their current priority.

  • Blocked jobs are ineligible to run or to be queued for scheduling. These jobs can have various states (see list below).

Blocked jobs will have one of the following states, depending on the reason for the block:

Idle

Job violates a fairness policy, i.e., you have used too many resources lately.

UserHold

A user hold is in place. This may be caused by job dependencies.

SystemHold

An administrative or system hold is in place. The job will not start until that hold is released.

BatchHold

A scheduler batch hold is in place, used when the job cannot be run because

  • the requested resources are not available in the system, or

  • because the resource manager has repeatedly failed in attempts to start the job. This typically indicates a problem with some nodes of the cluster, so you may want to contact user support.

Deferred

A scheduler defer hold is in place (a temporary hold used when a job has been unable to start after a specified number of attempts. This hold is automatically removed after a short period of time).

NotQueued

Job is in the resource manager state NQ (indicating the job’s controlling scheduling daemon in unavailable).

If your job is blocked, you may want to run the checkjob command to find out why.

There are some useful options for showq:

  • -r will show you the running jobs only, but will also give

    more information about these jobs, including an estimate about how efficiently they are using the CPU.

  • -i will give you more information about your eligible jobs.

  • -p <partition> will only show jobs running in the specified partition.

A note on queues#

Both qstat and showq can show you the name of the queue (qstat) or class (showq) which in most cases is actually the same as the queue.

All VSC clusters have multiple queues that are used to define policies. E.g., users may be allowed to have many short jobs running simultaneously, but may be limited to a few multi-day jobs to avoid long-time monopolization of a cluster by a single user.

This would typically be implemented by having separate queues with specific policies for short and long jobs. When you submit a job, qsub will put the job in a particular queue based on the resources requested automatically.

Warning

The qsub command does allow to specify the queue to use, but unless explicitly instructed to do so by user support, we advise strongly against the use of this option.

Putting the job in the wrong queue may actually result in your job being refused by the resource manager, and we may also chose to change the available queues on a system to implement new policies.

Getting detailed information about a job#

qstat#

To get detailed information on a single job, add the job ID as argument and use the -f or -f1 option:

$ qstat -f <jobid>

The -n or -n1 will just show you the nodes allocated to each running job in addition to regular output.

checkjob#

The checkjob command also provides details about a job, but from the perspective of the scheduler, so that you get different information.

The command below will produce information about the job with jobid 323323:

$ checkjob 323323

Adding the -v option (for verbose) gives you even more information:

$ checkjob -v 323323

For a running job, checkjob will give you an overview of the allocated resources and the wall time consumed so far. For blocked jobs, the end of the output typically contains clues about why a job is blocked.

Deleting a queued or running job: qdel#

This is easily done with qdel, e.g., the following command will delete the job with ID 323323:

$ qdel 323323

If the job is already running, the processes will be killed and the resources will be returned to the scheduler for another job.

Getting a start time estimate for your job: showstart#

This is a very simple tool that will tell you, based on the current status of the cluster, when your job is scheduled to start:

$ showstart 20030021
job 20030021 requires 896 procs for 1:00:00
Earliest start in       5:20:52:52 on Tue Mar 24 07:36:36
Earliest completion in  5:21:52:52 on Tue Mar 24 08:36:36
Best Partition: DEFAULT

Note

This is only an estimate, based on the jobs that are currently running or queued and the walltime that users gave for these jobs.

  • Jobs may always end sooner than requested, so your job may start sooner.

  • On the other hand, jobs with a higher priority may also enter the queue and delay the start of your job.

Checking free resources for a short job: showbf#

When the scheduler performs its task, there is bound to be some gaps between jobs on a node. These gaps can be back filled with small jobs. To get an overview of these gaps, you can execute the command showbf:

$ showbf
backfill window (user: 'vsc30001' group: 'vsc30001' partition: ALL) Wed Mar 18 10:31:02
323 procs available for      21:04:59
136 procs available for   13:19:28:58

To check whether a job can run in a specific partition, add the -p <partition> option.

Note

There is however no guarantee that if you submit a job that would fit in the available resources, it will also run immediately. Another user might be doing the same thing at the same time, or you may simply be blocked from running more jobs because you already have too many jobs running or have made heavy use of the cluster recently.