{"id":346,"date":"2017-07-28T11:05:20","date_gmt":"2017-07-28T15:05:20","guid":{"rendered":"https:\/\/carleton.ca\/rcs\/?page_id=346"},"modified":"2026-04-09T10:46:13","modified_gmt":"2026-04-09T14:46:13","slug":"background-queue-of-jobs","status":"publish","type":"page","link":"https:\/\/carleton.ca\/rcs\/rcdc\/background-queue-of-jobs\/","title":{"rendered":"Background queue of jobs"},"content":{"rendered":"\n
\n
\n\n
\n
\n
\n

\n Background queue of jobs\n <\/h1>\n \n \n <\/header>\n\n <\/div>\n\n <\/div>\n\n <\/div>\n<\/section>\n\n\n\n

Overview<\/h2>\n\n\n\n

Task Spooler (tsp<\/strong>) is not installed on every Linux server. However, we mention it here because if it happens to be available, it is a straightforward way to run many jobs.<\/p>\n\n\n\n

Task Spooler is a type of job queuing software. There are many more complicated job queueing software packages. Task Spooler is a particularly simple tool. It only runs on one server for one user.<\/p>\n\n\n\n

The tsp<\/strong> command will run a job in the background; this lets you run other commands while tsp<\/strong> is busy running your program. The tsp<\/strong> command will even keep running your jobs after you logout or disconnect from the server.<\/p>\n\n\n\n

In its normal setup, only one job will run with tsp<\/strong> at a time.  Once the first job finishes, tsp<\/strong> will automatically start the next job. This list of waiting jobs is called queuing<\/em>, and is analogous to the lineup at a grocery store.<\/p>\n\n\n\n

Let’s start taking a more concrete look at how to use the tsp command.<\/p>\n\n\n\n

Running Jobs<\/h2>\n\n\n\n

Just put tsp<\/strong> in front of the command you want to run. Note that tsp <\/strong>can only directly run a single command, but itself does not handle the special functionality of the Unix shell. So for example if you use a command containing pipes or output redirects, then it is easiest to put those shell commands into a file. This file is called a shell script, but is out of the scope of our current discussion.<\/p>\n\n\n\n

Let’s run a fictitious program called `run_simulation` using tsp<\/strong>:<\/p>\n\n\n

\n
\n
tsp .\/run_simulation -t 10\ntsp .\/run_simulation -t 3\ntsp .\/run_simulation -t 20\ntsp<\/strong>\nID   State      Output               E-Level  Times(r\/u\/s)   Command [run=1\/1]\n0    running    \/tmp\/ts-out.Smoz2i                           .\/run_simulation -t 10\n1    queued     (file)                                       .\/run_simulation -t 3\n2    queued     (file)                                       .\/run_simulation -t 20<\/pre>\n<\/blockquote>\n<\/div>\n\n\n
We have queued up 3 jobs, each running a run_simulation<\/strong>. We’ll move on now to ways of managing jobs and getting output.<\/p>\n\n\n\n
Listing Jobs<\/h2>\n\n\n\n
You can view your list of tsp jobs:<\/p>\n\n\n
\n
\n
tsp<\/strong>\nID   State      Output               E-Level  Times(r\/u\/s)   Command [run=1\/1]\n1    running    \/tmp\/ts-out.IiCgZe                           .\/run_simulation -t 10\n2    queued     (file)                                       .\/run_simulation -t 10\n0    finished   \/tmp\/ts-out.QUFZ40   0        2.00\/0.00\/0.00 .\/run_simulation -t 2<\/pre>\n<\/blockquote>\n<\/div>\n\n\n
The number from the ID column will be used to refer to particular jobs. The State column tells you the progress tsp is making in your job queue.<\/p>\n\n\n\n
One important caveat – the list of jobs is kept in memory.  So if your server reboots then you will lose any existing jobs.  So it is important to collect any needed information about completed jobs.<\/p>\n\n\n\n
Inspecting Jobs<\/h2>\n\n\n\n
Let’s assume you have some finished tsp<\/strong> jobs that look like this:<\/p>\n\n\n
\n
\n
tsp<\/strong>\nID   State      Output               E-Level  Times(r\/u\/s)   Command [run=1\/1]\n0    finished   \/tmp\/ts-out.Smoz2i   0        10.04\/0.00\/0.00 .\/run_simulation -t 10\n1    finished   \/tmp\/ts-out.1UDMiN   0        3.02\/0.00\/0.00  .\/run_simulation -t 3\n2    finished   \/tmp\/ts-out.twnIHU   0        20.11\/0.00\/0.00 .\/run_simulation -t 20<\/pre>\n<\/blockquote>\n<\/div>\n\n\n
Note that tsp gives each job an ID.  You can see the output of job with ID 1:<\/p>\n\n\n
\n
\n
tsp -c 1<\/strong>
\nThis is sample output from
\na program!<\/p>\n<\/blockquote>\n<\/div>\n\n\n
And to see detailed information about the job:<\/p>\n\n\n
\n
\n
tsp -i 1<\/strong>\nExit status: died with exit code 0\nCommand: .\/run_simulation -t 3\nSlots required: 1\nEnqueue time: Fri Jul 14 10:45:28 2017\nStart time: Fri Jul 14 10:45:38 2017\nEnd time: Fri Jul 14 10:45:41 2017\nTime run: 3.021296s<\/pre>\n<\/blockquote>\n<\/div>\n\n\n
Cleaning up Jobs<\/h2>\n\n\n\n
To remove a queued job with ID 2:<\/p>\n\n\n
\n
\n
tsp -r 2<\/strong><\/pre>\n<\/blockquote>\n<\/div>\n\n\n
To clear the list of all finished jobs:<\/p>\n\n\n
\n
\n
tsp -C<\/strong><\/pre>\n<\/blockquote>\n<\/div>\n\n\n
To kill (stop) a running job with ID 2:<\/p>\n\n\n
\n
\n
tsp -k 2<\/strong><\/pre>\n<\/blockquote>\n<\/div>\n\n\n
Concurrent Jobs<\/h2>\n\n\n\n
The tsp command can run more than one job at the same time.  In the tsp<\/strong> queue list, the header indicates how many jobs are running, and how many are allowed to run.<\/p>\n\n\n\n
However, if your job already uses multiple cores or if your job uses most of the server’s RAM, then you should not run multiple tsp jobs at the same time.<\/p>\n\n\n\n
For example, to run 4 jobs at the same time:<\/p>\n\n\n
\n
\n
tsp<\/strong>\nID   State      Output               E-Level  Times(r\/u\/s)   Command [run=0\/1<\/strong>]\n\ntsp -S 4<\/strong>\n\ntsp<\/strong>\nID   State      Output               E-Level  Times(r\/u\/s)   Command [run=0\/4<\/strong>]<\/pre>\n<\/blockquote>\n<\/div>","protected":false},"excerpt":{"rendered":"
Overview Task Spooler (tsp) is not installed on every Linux server. However, we mention it here because if it happens to be available, it is a straightforward way to run many jobs. Task Spooler is a type of job queuing software. There are many more complicated job queueing software packages. Task Spooler is a particularly […]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":96,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_cu_dining_location_slug":"","footnotes":"","_links_to":"","_links_to_target":""},"cu_page_type":[],"class_list":["post-346","page","type-page","status-publish","hentry"],"acf":{"cu_post_thumbnail":""},"_links":{"self":[{"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/pages\/346","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/comments?post=346"}],"version-history":[{"count":4,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/pages\/346\/revisions"}],"predecessor-version":[{"id":1142,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/pages\/346\/revisions\/1142"}],"up":[{"embeddable":true,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/pages\/96"}],"wp:attachment":[{"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/media?parent=346"}],"wp:term":[{"taxonomy":"cu_page_type","embeddable":true,"href":"https:\/\/carleton.ca\/rcs\/wp-json\/wp\/v2\/cu_page_type?post=346"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}