From: 

dave safford
director, computing and information services,
texas a&m university
d-safford@tamu.edu 


PATCHES FOR NQS-3.36 FOR IRIX-6.0

The IRIX6 specific patches (kmem stuff) are ifdef'ed with "IRIX6",
and the TAMU extensions (processor set and TMPDIR) are ifdef'ed
with "TAMU".


OTHER INFORMATION:

0. IRIX6.0 interactive file commands (rm -i, mv -i, ln -i, cp -i)
   are broken and will cause infinite loops and spool overflow
   if used in an nqs job. I ported the gnu versions, and they
   work fine. This is NOT an NQS bug, but needs to be fixed
   if nqs is to operate reliably.

1. How the processor set handling is done:

In the "keep-it-simple" mode, all I did was change the init.d/network
script to invoke inetd using "pset -c" to set inetd, (and all
descendants, including all telnetd based logins...) into the
"interactive" processor set. This does mean that all of an interactive
user's shells, subshells, commands, etc. are executed in the same
processor set. (The root user is able to change his process's sets
any he wants.)

As for nqs, I modified the code that execs the batch program to set
the processor set to the same name of the batch queue. That is, a job
in the "small" nqs queue will try to set the processor set to the processor
set "small".  So I have to create processor sets with the same names
as the nqs queues, but then I don't have to have any fancy nqs parameters
or set mapping files.  Definition of processor sets was added to /etc/rc2
(at the end of the startup from single-user) with a "/sbin/pset -i"
command. This assumes that the file /etc/psettab has been established.
Our psettab looks like:
	interactive 100 0,1,2,3
	small 101 4,5,6,7
	medium 102 4,5,6,7
	large 103 8,9,10,11,12,13,14,15
This defines an "interactive" processor set with cpu 0 thru 3,
nqs batch processor sets "small" and "medium", which share cpu 4 - 7,
and nqs "large" processor set with cpu 8 - 15.

It's simple, and it works great.

2. How interactive limits are controlled:

Nqs has the full ability to set various process limits. In our case,
it is desirable to put limits on interactive work, so that users are
encouraged (actually pretty much required) to queue large jobs.
The question is how best to put limits on interactive work?
We did two things. First, we put hard limits in /etc/profile and
/etc/cshrc, so that logins inherit the limits.  For example,
/etc/profile has:
        limit -h cputime 20:00
        limit -h vmemory 100m

Second, we wrote a short wrapper "setlimit" for use in /etc/inetd.conf,
to set limits on invoked services.  Currently we wrap only rsh,
as rlogin, and telnet limits are set by the /etc/profile and /etc/cshrc
files. The setlimit wrapper code is:

/* setlimit time(sec) memory(bytes) program args .... */
#include <stdlib.h>
#include <unistd.h>
#include <sys/resource.h>
main(int argc, char *argv[])
{
        struct rlimit rlim;

        rlim.rlim_cur = atol(argv[1]);
        rlim.rlim_max = atol(argv[1]);
        setrlimit(RLIMIT_CPU, &rlim);

        rlim.rlim_cur = atol(argv[2]);
        rlim.rlim_max = atol(argv[2]);
        setrlimit(RLIMIT_VMEM, &rlim);

        execv(argv[3], &argv[3]);
}       


which is put into inetd.conf as:

	shell   stream  tcp     nowait  root    /bin/setlimit \
		setlimit 1200 104857600 /usr/etc/rshd


which sets the cputime to 1200 seconds, and the vmem limit to 100mb.

3. TMPDIR usage:

One problem we have with batch jobs is guaranteeing users adequate
scratch space.  /tmp is notorious for filling up, and there is
no correct way to force cleaning. The TMPDIR modification to nqs
builds a unique scratch directory of the form /work/<queue>/<job_id>
which is owned and writeable only by the job's user. These scratch
directories can be located to guarantee available space, as they
are deleted by nqs upon completion of the job.  The job has
the environment variable TMPDIR set to point to the scratch directory,
so jobs can easily utilize it.  For example, the following job
script demonstrates a typical usage:
	#!/bin/sh
	cd $TMPDIR
	~/bigjob > bigscratchfile
	~/bigjob2 <bigscratchfile >littleoutfile
	cp littleoutfile ~

	
