Grun is a tool to start jobs on a Globus-based Grid.
It is comparable to the standard Globus job submission tool
"globus-job-run", but has numerous additional features to make
the life of a Grid user easier:

- Support for co-scheduling, with timeouts in case insufficient
  resources are available;

- Grun may be interrupted at any stage, resulting in a clean job
  termination at all the remote sites;

- Explicit support for distributed Ibis jobs by means of a remote job
  script and a local startup script for support services;

- On the remote resources, jobs are started within a "sandbox":
  a separate job-specific directory in which inputs, temporaries
  and output files are stored;

- User preferences about Grid sites, e.g., the default job manager
  to be used, can be specified in a separate configuration file;

- Convenient output and status redirection to a job-specific subdirectory
  on the submission site for further processing;

- Support to work around network connectivity limitations (e.g., for
  clusters with firewalls or private networks).  Actually, the real
  work is all done in the Ibis runtime system, but in grun it is
  easy to integrate functionality like this.

Since Grun is written in Python, it can quickly and easily be
extended as the need arises.  We are thinking of adding support
for the following in the near future:

- Support for (dynamic) scheduling within Grun itself.  Most of our
  current users prefer explicit allocation of resources (e.g., to obtain
  reproducible performance characteristics), but for production use
  of Grids some built-in automatic scheduling would also be useful.

- Support for non-Globus based Grids, e.g., PlanetLab.  This will
  probably mean adding an "ssh" backend, in addition to the current
  Globus Resource Allocation Manager (GRAM) protocol.


==============================================================================
Using Grun
  

Grun is best introduced by means of a few practical examples,
going from simple to more complex scenarios.  A complete listing
of the options supported by grun is included below, but can also
be obtained using
---
$ grun --help
---


Example 1: simple command, no output redirection:
---
$ grun -c '/bin/uname -ns' -m fs0
Linux node006
---
Note that the argument to the "-c" option specifying the command has
to be quoted, in order to include the command arguments (otherwise
they would be interpreted as another grun option).

The Grid resources on which the command is run is specified by means of
one or more "-m" options.  In this case "fs0" is specified, which is really
a short alias for the full version "fs0.das2.cs.vu.nl/jobmanager-pbs".
Aliases like this are defined in the file griddefs.py (a default version
is in $GRUN_ROOT/bin, but the user can override this by putting a modified
version in $HOME).  By default only a single process is run on the
allocated resource.


Example 2: like Example 1, but verbose, and with multiple resources:
---
$ grun -v -c '/bin/uname -ns' -N 4 -m fs0 -m fs1
grun: starting job 468
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job pending
grun: fs1.das2.liacs.nl/jobmanager-pbs: job pending
Linux node131
Linux node131
Linux node130
Linux node130
Linux node065
Linux node065
Linux node064
Linux node064
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job completed
grun: fs1.das2.liacs.nl/jobmanager-pbs: job completed
grun: Total running time: 14.1 sec
grun: job 468 done
---
The -v option causes grun to print diagnostics about changing job
states to standard output (grun gets upcalls from the GRAM protocol
about this).  This provides useful feedback in cases where it is
unclear why a distributed application cannot be started succesfully.
Occasionally, insufficient resources will be unavailable at some of
the sites, and it is currently up to the user to decide whether it
makes sense to retry the same job (possibly after some delay)
or whether it is more useful to try using different resources.
In this case, grun exits with a status of 3 to differentiate
it from other error conditions.

The -N option specifies the number of cpus to be allocated.
On DAS-2, which is consists of dual processor nodes, this causes two
processes to be run per node allocated.  To allocate a separate
compute node for each process, use the -H option instead of -N.


Example 3: running a script and redirecting job output:
---
$ cat >example.script
#!/bin/sh
uname -ns
printenv IBIS_ROOT
$ chmod +x example.script 
$ grun -v -o -c "@GRUN_GASS@/./example.script" -E IBIS_ROOT=ibis -m fs0 -m fs1
grun: starting job 1125
grun: fs1.das2.liacs.nl/jobmanager-pbs: job stagein
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job stagein
grun: fs1.das2.liacs.nl/jobmanager-pbs: job pending
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job pending
grun: fs1.das2.liacs.nl/jobmanager-pbs: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job completed
grun: Total running time: 21.3 sec
grun: job 1125 done

$ cat grun.out/job-1125/site-0/stdout 
https://fs0.das2.cs.vu.nl:54166
Linux node065
ibis
$ cat grun.out/job-1125/site-1/stdout 
https://fs0.das2.cs.vu.nl:54167
Linux node130
ibis
$ cat grun.out/job-1125/site-*/stderr 
$ head -3 grun.out/job-1125/site-*/status
==> grun.out/job-1125/site-0/status <==
host = fs0.das2.cs.vu.nl
rm = fs0.das2.cs.vu.nl/jobmanager-pbs
rsl = &(executable=@GRUN_GASS@/./example.script)

==> grun.out/job-1125/site-1/status <==
host = fs1.das2.liacs.nl
rm = fs1.das2.liacs.nl/jobmanager-pbs
rsl = &(executable=@GRUN_GASS@/./example.script)
---

Often it is useful to run the commands at the Grid sites from a script,
without first having to transfer the script manually (since the
script may have to be updated regularly during development, doing
the transfers by hand is also error-prone).  In addition, it is
convenient to pass some information as environment parameters
rather than options (e.g., since the latter require some additional
amount of argument processing).

As the above example shows, this can be done by using the '@GRUN_GASS@'
construct in the -c option.  What happens is that grun starts several
Globus "GASS" I/O servers locally, which can be contacted by the remote
jobs to do transfers to and from the job submission site.
In this example, the GASS servers are used in two ways:

- to transfer the job script to the sites used; this happens in
  the initial job "stagein" phase shown.

- to redirect the standard output and error of the remote job back to
  files in a job/site-specific sub-directory of "grun.out"
  (this is due to the "-o" option; note that the first line of the stdout
  files starting with https:// are output from the GASS server themselves,
  and should be ignored).

Furthermore, as shown above, grun itself writes some state information
to "status" files in the job/site-specific subdirectories.  This is mainly
for diagnostic purposes in case of runtime errors.


Example 4: Ibis application run, using existing Ibis trees:
---
$ port1=61234
$ port2=61235
$ grun_etc=/usr/local/package/grun/etc

$ grun -v -o -s "$grun_etc/runsvrs.ibis -name $port1 -pool $port2" \
  -c "@GRUN_GASS@/$grun_etc/runit.ibis ../../build Main 2000 2000" \
  -E IBIS_NAMESVR_PORT=$port1 -E IBIS_POOLSVR_PORT=$port2          \
  -E IBIS_ROOT="../../ibis" -H 2 -m fs0 -m fs1
grun: starting job 1135
PoolInfoServer: starting run, waiting for a host to connect...
grun: fs1.das2.liacs.nl/jobmanager-pbs: job stagein
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job stagein
grun: fs1.das2.liacs.nl/jobmanager-pbs: job pending
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job pending
PoolInfoServer: host 1 has connected, total hosts = 4
PoolInfoServer: Host 2 has connected
PoolInfoServer: Host 3 has connected
PoolInfoServer: Host 4 has connected
PoolInfoServer: all hosts have connected, now broadcasting host info...
PoolInfoServer: broadcast done, run is finished
ibis@node062.das2.cs.vu.nl_1088435065225 JOINS  pool key1135 (1 nodes)
ibis@node061.das2.cs.vu.nl_1088435065226 JOINS  pool key1135 (2 nodes)
ibis@node128.das2.liacs.nl_1088435065239 JOINS  pool key1135 (3 nodes)
ibis@node127.das2.liacs.nl_1088435065267 JOINS  pool key1135 (4 nodes)
grun: fs1.das2.liacs.nl/jobmanager-pbs: job active
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job active
ibis@node061.das2.cs.vu.nl_1088435065226 LEAVES pool key1135 (3 nodes)
ibis@node062.das2.cs.vu.nl_1088435065225 LEAVES pool key1135 (2 nodes)
ibis@node127.das2.liacs.nl_1088435065267 LEAVES pool key1135 (1 nodes)
ibis@node128.das2.liacs.nl_1088435065239 LEAVES pool key1135 (0 nodes)
grun: fs1.das2.liacs.nl/jobmanager-pbs: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job completed
grun: Total running time: 39.4 sec
grun: job 1135 done

$ cat grun.out/job-1135/site-*/stdout | grep seconds
Application: SOR 2000 x 2000; Ncpus: 4; time: 10.052 seconds
---

To support running Ibis jobs, grun employs two scripts:

- runsvrs.ibis, which is used to start the local daemons required
  for startup (e.g., Ibis name server, and pool server in case of
  closed world jobs); this is done by means of the -s option above.

- runit.ibis, which is run on the Grid sites to actually start the
  Ibis job by using the remotely available JVMs.

The ports used by the servers are passed by argument (in the case
of runsvrs.ibis) and by environment parameter IBIS_*_PORT
(in the case of runit.ibis). 

Other information that is required for Ibis jobs is the location of
the Ibis installation (IBIS_ROOT) and of the application.  In this first
Ibis example, the assumption is that the right Ibis distribution is
already available on all participating sites as $HOME/ibis, and that
the application "Main" class is in $HOME/build.  The reason why "ibis"
and "build" are referred to by means of "../.." is twofold:

- in general the name of the home directory of a grid user is not necessarily
  the same on every Grid site, so absolute paths should be avoided;
  
- although a grid job is started in the home directory of the user
  by default, the runit.ibis script changes directory into the "sandbox"
  directory "grun.dir/job-<jobid>" to avoid jobs from interfering each other.
  So from the sandbox directory, the home directory is available as "../..".

Note that because the home directory is the initial destination of file
transfers done during Globus job submits, it is advisable *NOT* to run
grun from your home directory itself, since this might cause naming
conflicts for jobs submitted to the site itself.

We will now show how the "runit.ibis" script can be used for different
Ibis application run scenarios as well.

Example 5: a fully-staged Ibis application run:
---
$ grun -v -o -s "$grun_etc/runsvrs.ibis -name $port1 -pool $port2"      \
  -c "@GRUN_GASS@/$grun_etc/runit.ibis build Main 2000 2000"		\
  -E IBIS_NAMESVR_PORT=$port1 -E IBIS_POOLSVR_PORT=$port2		\
  -p -E GRUN_INFILES="ibis-tree.jar:build-tree.jar"			\
  -P -E GRUN_CLEANUP=1							\
  -E IBIS_ROOT="ibis" -H 2 -m fs0 -m fs1

grun: starting job 1137
PoolInfoServer: starting run, waiting for a host to connect...
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job stagein
grun: fs1.das2.liacs.nl/jobmanager-fork: job stagein
grun: fs1.das2.liacs.nl/jobmanager-fork: job active
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job active
grun: fs1.das2.liacs.nl/jobmanager-fork: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job completed
grun: fs1.das2.liacs.nl/jobmanager-pbs: job stagein
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job stagein
grun: fs1.das2.liacs.nl/jobmanager-pbs: job pending
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job pending
PoolInfoServer: host 1 has connected, total hosts = 4
PoolInfoServer: Host 2 has connected
PoolInfoServer: Host 3 has connected
PoolInfoServer: Host 4 has connected
PoolInfoServer: all hosts have connected, now broadcasting host info...
PoolInfoServer: broadcast done, run is finished
ibis@node059.das2.cs.vu.nl_1088497897983 JOINS  pool key1137 (1 nodes)
ibis@node060.das2.cs.vu.nl_1088497897983 JOINS  pool key1137 (2 nodes)
ibis@node129.das2.liacs.nl_1088497898042 JOINS  pool key1137 (3 nodes)
ibis@node128.das2.liacs.nl_1088497898069 JOINS  pool key1137 (4 nodes)
grun: fs1.das2.liacs.nl/jobmanager-pbs: job active
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job active
ibis@node060.das2.cs.vu.nl_1088497897983 LEAVES pool key1137 (3 nodes)
ibis@node059.das2.cs.vu.nl_1088497897983 LEAVES pool key1137 (2 nodes)
ibis@node128.das2.liacs.nl_1088497898069 LEAVES pool key1137 (1 nodes)
ibis@node129.das2.liacs.nl_1088497898042 LEAVES pool key1137 (0 nodes)
grun: fs1.das2.liacs.nl/jobmanager-pbs: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job stagein
grun: fs1.das2.liacs.nl/jobmanager-fork: job stagein
grun: fs1.das2.liacs.nl/jobmanager-fork: job active
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job active
grun: fs1.das2.liacs.nl/jobmanager-fork: job completed
grun: fs0.das2.cs.vu.nl/jobmanager-fork: job completed
grun: Total running time: 85.2 sec
grun: job 1137 done

$ cat grun.out/job-1137/site-*/stdout | grep seconds
Application: SOR 2000 x 2000; Ncpus: 4; time: 10.055 seconds
---

New in this example are the -p and -P options, with associated environment
settings by means of -E.  The -p option causes a (single-cpu) pre-execution
stage to be run, to take care of setting up the execution environment
(in this case consisting of the Ibis tree and application directory,
but in general this can also include other input files).
In the output above this can be observed as the initial "jobmanager-fork"
lines.  In this stage, the files specified by means of the environment
parameter GRUN_INFILES are transferred by runit.ibis; since their name
ends with "-tree.jar", they are then also unpacked using "jar xvf"
into the sandbox.  For that reason the "ibis" and "build" directories
are also no longer in "../.." .

The -P option is used to run post-execution stage.  In this case
it is only used to delete the entire sandbox, since "GRUN_CLEANUP=1"
is specified.  Since an entire Ibis tree was transferred in this case,
it makes sense to remove it afterwards, or your disk space would quickly
run out. Another use of the post-execution stage is to transfer back some
application output files to the originating site.  This can be done
by specifying GRUN_OUTFILES in the environment, similar to GRUN_INFILES
in the example above.

Notice that due to the additional transfers, unpacking, etc,
the overall completion time has increased significantly for the job,
compared with the previous example.  Although the job is now completely
self-supporting, it does come with a price, so in practice it might be
more convenient to keep a consistent Ibis tree at all remote sites,
and use that for most of your application runs.

However, an attractive alternative is to transfer just the
ibis.jar and application.jar file in every run.  In most cases
this is all you really need, and it is done as follows.

Example 6: a quick self-supporting Ibis run:
---
$ grun -v -o -s "$grun_etc/runsvrs.ibis -name $port1 -pool $port2"	\
  -c "@GRUN_GASS@/$grun_etc/runit.ibis . Main 1000 1000"		\
  -E IBIS_NAMESVR_PORT=$port1 -E IBIS_POOLSVR_PORT=$port2		\
  -E IBIS_ROOT="../../ibis.jar:../../sor_grid.jar"			\
  -I ibis.jar -I sor_grid.jar -N 1 -m fs1 -m fs2
grun: starting job 183
PoolInfoServer: starting run, waiting for a host to connect...
grun: fs2.das2.nikhef.nl/jobmanager-pbs: job stagein
grun: fs1.das2.liacs.nl/jobmanager-pbs: job stagein
grun: fs2.das2.nikhef.nl/jobmanager-pbs: job pending
grun: fs1.das2.liacs.nl/jobmanager-pbs: job pending
PoolInfoServer: Key key183 Host 0 has connected
PoolInfoServer: starting run, waiting for a host to connect...
PoolInfoServer: Key key183 Host 1 has connected
PoolInfoServer: Key key183: All hosts have connected, now broadcasting host info...
PoolInfoServer: Key key183: Broadcast done
ibis@node231.das2.nikhef.nl_1089793630353 JOINS  pool key183 (1 nodes)
ibis@node119.das2.liacs.nl_1089793630469 JOINS  pool key183 (2 nodes)
grun: fs2.das2.nikhef.nl/jobmanager-pbs: job active
grun: fs1.das2.liacs.nl/jobmanager-pbs: job active
ibis@node119.das2.liacs.nl_1089793630469 LEAVES pool key183 (1 nodes)
ibis@node231.das2.nikhef.nl_1089793630353 LEAVES pool key183 (0 nodes)
grun: fs2.das2.nikhef.nl/jobmanager-pbs: job completed
grun: fs1.das2.liacs.nl/jobmanager-pbs: job completed
grun: Total running time: 31.2 sec
grun: job 183 done
---

The file "ibis.jar" in this example is the one from the $IBIS_ROOT/build
directory, and file "sor_grid.jar" was built by collecting the application
class files using:
---
$ jar cvf sor_grid.jar *.class
---
in the $IBIS_ROOT/ibis/apps/rmi/sor_grid/build directory.
The "-I" options cause both files to be transfered to the
remote sites in a "stagein" phase.  But since the jar files don't
need additional processing (e.g., unpacking the -tree.jar files
as in the previous example), a seperate stagein phase using
the fork jobmanagers is no longer necessary.
Since only the essential jar files are transferred, total running
time is significantly reduced.
Notice that IBIS_ROOT in this case contains both the ibis.jar
and application.jar file from the user's home directory; this
will get both of them in the Java classpath as is needed here.


On the Grid, compute resources are often hidden behind firewalls,
NAT routers, etc.  Normally this would make it quite hard to
connect these resources for truely distributed computing.
However, Ibis has special support in its connection setup mechanism
that will allow it to work around this limitation in many cases.

Example 7: Requesting alternative connection setups in Ibis:
---
$ grun -v -o -s "$grun_etc/runsvrs.ibis -name $port1 -pool $port2 -hub $port3"\
  -c "@GRUN_GASS@/$grun_etc/runit.ibis . Main 1000 1000"		\
  -E IBIS_NAMESVR_PORT=$port1 -E IBIS_POOLSVR_PORT=$port2		\
  -E IBIS_NETWORK=firewall -E IBIS_HUB_PORT=$port3			\
  -E IBIS_ROOT="../../ibis.jar:../../sor_grid.jar"			\
  -I ibis.jar -I sor_grid.jar -N 1 -m fs0 -m n0 
grun: starting job 184
PoolInfoServer: starting run, waiting for a host to connect...

# ControlHub: listening on fs0.das2.cs.vu.nl:61241
# ControlHub: new connection from fs0.das2.cs.vu.nl:40821
# ControlHub: 1 nodes currently connected
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job stagein
grun: n0.hpcc.sztaki.hu/jobmanager-condor: job stagein
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job pending
grun: n0.hpcc.sztaki.hu/jobmanager-condor: job pending
PoolInfoServer: Key key184 Host 0 has connected
PoolInfoServer: starting run, waiting for a host to connect...
PoolInfoServer: Key key184 Host 1 has connected
PoolInfoServer: Key key184: All hosts have connected, now broadcasting host info...
PoolInfoServer: Key key184: Broadcast done
# ControlHub: new connection from node062.das2.cs.vu.nl:53453
# ControlHub: 2 nodes currently connected
ibis@node062.das2.cs.vu.nl_1089795375928 JOINS  pool key184 (1 nodes)
# ControlHub: new connection from n1.hpcc.sztaki.hu:4338
# ControlHub: 3 nodes currently connected
ibis@n1.hpcc.sztaki.hu_1089795376168 JOINS  pool key184 (2 nodes)
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job active
grun: n0.hpcc.sztaki.hu/jobmanager-condor: job active
ibis@n1.hpcc.sztaki.hu_1089795376168 LEAVES pool key184 (1 nodes)
# ControlHub: EOF detected for n1.hpcc.sztaki.hu:4338
# ControlHub: 2 nodes currently connected
ibis@node062.das2.cs.vu.nl_1089795375928 LEAVES pool key184 (0 nodes)
# ControlHub: EOF detected for node062.das2.cs.vu.nl:53453
# ControlHub: 1 nodes currently connected
grun: fs0.das2.cs.vu.nl/jobmanager-pbs: job completed
grun: n0.hpcc.sztaki.hu/jobmanager-condor: job completed
grun: Total running time: 39.4 sec
grun: job 184 done
---

The compute nodes of the n0 machine are hidden behind the headnode,
which also acts as a router. They cannot be reached directly from
the outside, but they are able to make connections to the outside
world themselves.  Ibis is able to use this aspect in its connection
setup procedures to make full communication possible.

The above example shows how this is done using grun.  New are the "-hub"
option to the runsvrs.ibis script, and the associated IBIS_NETWORK and
IBIS_HUB_PORT environment settings used in the runit.ibis script.
The "hub" is an additional local daemon taking care of connection
setup negotiations.
Alternative setting of IBIS_NETWORK are "new_firewall" and "routedmsg".
Value "new_firewall" is an optimization over the "firewall" setting:
often only one of two sites trying to connect is behind a firewall,
so in that case using a standard client/server connection in the
right direction will work fine, which is often quicker than the
default protocol used by "firewall".
In case IBIS_NETWORK is set to "routedmsg", all communication will go
through the central daemon, which may be necessary in case the routing
restrictions imposed on one of the sites is very harsh.
Usually the setting "firewall" or "new_firewall" should work fine.

Other grun options that may be useful for everyday use are -t and -T
that specify maximum startup time and total running time respectively.
The -x option specifies how long the application is allowed to
terminate properly after the first subjob exits.  If it takes longer
than that, grun assumes that the application is hanging, and it
will terminate the other subjobs explicitly.

For more details, e.g., on how the JVM is started remotely, have a
look at the runit.ibis script.  There really is no magic involved,
and the script can usually be extended or tailored to the application
if needed.  Extensions in functionality that could be useful for other
applications as well may be sent to versto@cs.vu.nl.  If possible,
they may be included in the standard runit.ibis script for a
following release.


==============================================================================
Usage: grun [options]
Options:
    -c '<cmdline>'
        Specifies grid job command line to be used.
        Should precede subjob(s) specified with -m option.
        Use    -c '@GRUN_GASS@/./<script> <args>' to transfer file
        <script> from local directory to remote system and run it there.
    -r <rsl>
        Same as '-c' option, only specifies grid job using Globus RSL syntax.
        Example:   -r '(executable=/bin/uname)(arguments=-a)'
    -m <resource manager contact> 
        Create subjob on the grid resource specified, e.g.,
            fs0.das2.cs.vu.nl/jobmanager-pbs
        Grid resource may also be an alias, e.g., 'fs0', as defined in
        file griddefs.py.
    -N <cpucount>
        Set default number of cpus used for all subsequent subjobs
    -n <cpucount>
        Set non-default number of cpus used for single subjob
    -H <hostcount>
        Set default number of hosts used for all subsequent subjobs
        (for fork jobmanagers, -H is interpreted as -N)
    -h <hostcount>
        Set non-default number of hosts used for single subjob
        (for fork jobmanagers, -H is interpreted as -N)
    -E <var>=<val>
        Add environment setting for each job
    -e <var>=<val>
        Add environment setting for single subjob
    -s <cmdline>
        Run local command prior to grid job, e.g., to start name server
        or startup barrier.
    -t <minutes>
        Sets the maximum collective job start-up time in minutes.
        The subjobs already started are terminated if some other site
        is not starting its job within the specified time.
        The default is 3 minutes.
        Use -t 0 to disable the collective startup time check.
    -T <minutes>
        Sets the maximum job 'wall' time in minutes.
        The default is 10 minutes.
        -T 0 causes the use of (site-dependent) standard maximum walltimes.
    -x <minutes>
        Sets the maximum job 'exit' time in minutes, i.e., the time
        other subjobs have to complete termination after the first subjob
        exits.  After that, remaining subjobs are cancelled explicitly.
        The default is 3 minutes.
    -X <maxretry>
        Set maximum retry count for failed job submissions due to timeout.
        The default is 5 retries.
    --pre '<cmdline>'
        Specifies grid job pre-execution command line to be used,
        that are used to stage files in, and possibly do additional
        processing.  Hosts for which this applies are specified
        with subsequent -m opions.
    --pre-all '<cmdline>'
        Like --pre, but run the pre-execution cmdline on all resources
        used for regular execution.
    -p
        Like --pre-all, only the cmdline is assumed to be the same as for
        the -c option (environment variable GRUN_PHASE is set to 'stagein'
        rather than 'run', so the job can see the difference).
    --post '<cmdline>'
        Specifies grid job post-execution command line to be used,
        that are used to stage files out.  Hosts for which this
        applies are specified with subsequent -m opions.
    --post-all '<cmdline>'
        Like --post, but run the post-execution cmdline on all resources
        used for regular execution.
    -P
        Like --post-all, only the cmdline is assumed to be the same as for
        the -c option (environment variable GRUN_PHASE is set to 'stageout'
        rather than 'run', so the job can see the difference).
    --output-enable
        Use the Globus GASS server module to redirect standard output and
        standard error to grun (this is the default)
    --output-disable
        Do NOT use the Globus GASS server module to redirect standard out and
        standard error to grun (default is enabled)
    -o
        Create files stdout/stderr/status in job-specific directories:
           grun.out/job-<jobnumber>/site-<sitenumber>/
    --stage-in <file>|<remotefile>=<localfile>
        Copy file <localfile> to <remotefile> in single subjob
        Single argument <file> is a shortcut for <file>=<file>
    [--stage-in-all | -I] <file>|<remotefile>=<localfile>
        Same as --stage-in, only for all subjobs
    --stage-in-shared[-all] <file>|<remotefile>=<localfile>
        Same as --stage-in[-all], only creates a cache with symlinks
    --stage-out <file>|<localfile>=<remotefile>
        Copy back file <remotefile> to <localfile> in single subjob.
        File <localfile> will be placed in the job-specific directories
        (see -o option).
    --stage-out-all <file>|<localfile>=<remotefile>
        Same as --stage-out, only for all subjobs
    -l <level>
        specifies the level of diagnostics printed:
        -l CRITICAL: only print critical errors
        -l ERROR:    print any errors
        -l WARN:     also print warnings
        -l INFO:     also print informational diagnostics (e.g. job scheduling)
        -l DEBUG:    be very verbose
        The default level is WARN.
    -v
        An alias for '-l INFO'
    
