PE(1)



NAME

     pe - pipeline execution


SYNOPSIS

     pe  [-v]  [-d]  [-L]   [-A]   [touch=#]   [db=database_name]
     [rcp=/dir] [pds=host_name] map=map_file


DESCRIPTION

     pe executes a series of strategy level servers  in  a  pipe-
     line.

     pe is launched with the name of the  map_file  that  defines
     its  processing.  The  map file specifies the strategy level
     servers to be run, the hosts they are  to  be  run  on,  the
     sequence  in which to run them, the dataset they are to pro-
     cess, and any DSDS support for dataset  name  resolution  or
     data archiving.

     pe is run as a PVM (Parallel Virtual Machine)  task.  A  pvm
     daemon is spawned if necessary on each host specified in the
     map file.


OPTIONS

     -v   The verbose flag.

     -d   The debug flag. Brings up a dbx window on each server.

     -L   The local flag. Run local dsds_svc/ampex_svc. Don't use
          pe_rpc. This is for development use only and requires a
          password.

     -A   The  ampex  flag.  Access  the  ampex_svc  if  need  to
          retrieve data that is not on-line.

     db=  The name of the database to  connect  to.  If  this  is
          missing  then  the  env vrbl DBNAME specifies the data-
          base. If this is missing then the default is the mdi_2X
          database.

     rcp= The local dir to rcp the input datasets to before  run-
          ning a module.

     pds= The host name to assign local /PDS  storage  from.   If
          not specified, will use the env vrbl PDS_SET_HOST, else
          will use the local host that pe is executing on. If the
          host  has  no  local  /PDS storage, it is assigned from
          sonar.

     touch=
          The #of days to retain input datasets before deletable.
          The  value  "keep" will retain the ds until findkeep is
          run to change it.


EXAMPLES

     #This is a text file that specifies a pipeline to be run by PE.
     #Notice that it operates on "keyword=value" pairs. In the case
     #of flags, any non-zero value is treated as true.
     #
     #Control Statements:
     #!
     #    A shell escape command. Any setenv is intercepted and interpreted
     #    locally for the PE environment. No metachars for a setenv.
     #    (see below for the use of setenv for directory mapping.)
     #    Note that a map file is not a shell script.
     #    Pe validates the entire map file and executes any shell
     #    escape commands before any servers are started.
     #DSDSOUT=n
     #    DSDS will be asked to allocate n megabytes for the output
     #    of this pipe. The output directory that DSDS assigns can be
     #    used in the name template env variable "prog:" by specifying
     #    {dbase}. Notice that if you do not specify d=1 (see below)
     #    for a server, but did specify DSDSOUT then you must set a
     #    default {dbase} so that the name template will also map the
     #    input.
     #HOST=name
     #    The host name that all the servers are to run on. If the
     #    fsn-lsn split feature is reimplemented then multiple hosts
     #    can be given [,name,...]. Also multiple copies of the
     #    servers can then be run on the same host by specifying the
     #    host name more than once.
     #NOWARN=1
     #    Omit pe warning messages arising from analysis of the map
     #    file. This is used by map building programs that use pe
     #    features that may result in a warning. A user building a
     #    map file should never use NOWARN.
     #START_GROUP=1
     #END_GROUP=1
     #    Any servers between a START_GROUP/END_GROUP pair will be
     #    started at the same time by pe. Normally a server does not
     #    start until the previous server has completed. Pe will check
     #    for any mismatched pairs.
     #START_ARCHIVE=1
     #END_ARCHIVE=1
     #    Any servers between a START_ARCHIVE/END_ARCHIVE pair will
     #    be archived only according to the archive flag of the last
     #    server of the group. This allows the same server
     #    (e.g. gather_info_svc) to be run more than once with the
     #    same output dirs with only one archive occuring according
     #    to the last run.  Pe will check for any mismatched pairs.
     #Line Tokens:
     #p=  The name of the server process to run.
     #d=  A non-zero indicates that this servers input will be
     #    obtained from DSDS. Otherwise a name template, found in the
     #    env variable of the name of the "prog:" in the ds name, will
     #    be used to determine the input directory.
     #a=  Determines the type of archiving for this server's output.
     #       This will be a nop unless DSDSOUT=n is specified. Examples:
     #    a=t5 Temporary dataset. Do not archive and do not mark delete
     #         pending for 5 days.
     #    a=n5 Normal. Archive but do not mark delete pending for 5 day.
     #    a=p  Permanent. Archive but never mark delete pending.
     #    a=a  Appendable. The output dataseries will be appended to.
     #         The dataseries must be defined in the ds_naming table.
     #         The data is not archived and not deleted. No DSDSOUT=n
     #         is required as the storage is in seperate disk partitions
     #         assigned for append.
     #l=  The default is for pe to copy the history log file and the map
     #    file that was run, into the dataset output working dirs. This
     #    is overridden by a log flag of l=0. This is used for data that
     #    is appended to a dir outside of DSDS storage and you don't
     #    want the history log and map file to show up for every run.
     #    NOTE!! The l= is obsolete with the V3R0B0 release (29Apr98).
     #    It is replaced by COPY_HISTORY=no to override the default.
     #x=  The default action, if a module send back an abortflg and is
     #    not part of a group (i.e. START_GROUP/END_GROUP), is to abort
     #    further execution of the map file and to not archive the output
     #    wd of the module. The x=1 will override this default abort
     #    action and case the map file to continue and any output wd to
     #    be archived.
     #    NOTE!! The x= is obsolete with the V2R7B0 release (15Dec97).
     #    It is replaced by ABORT_ACTION=continue to override the default.
     #in= The datacollection for the pipe's input data. This isn't
     #    necessarily called "in", but whatever the server's input ds
     #    argument is called.
     #out=     The datacollection for the pipe's output data. This isn't
     #    necessarily called "out", but whatever the server's output ds
     #    argument is called.
     #arg=     Optional argument name and value to override any default
     #    argument values for a server.
     #history= The name of the history log output file. The default is
     #    /tmp/modulename_username_pvmtid.log.
     #errlog=  The name of the error log output file. The default is the
     #    history log file.
     #
     #Any control statement must be on a separate line.
     #Any non-control line must start with p= ,and must be a separate line.
     #Any p= line may optionally contain an a=[t,n,p][#].
     #Any p= line may optionally contain an d=1.
     #Any p= line may optionally contain an l=0 and an x=1.
     #Any p= line may optionally contain any argument value for the server.
     #Any line, except a "!" line, may be extended by ending it with " \".
     #The "\" line extention cannot be used between a "key=value" pair.
     #Any string argument may contain imbedded spaces by quoting the
     #string, e.g. name="my string". However no quoted string may contain
     #the "*" character.
     #
     !setenv mdi_raw wd:{dbase}/{level}/{series}/{#%06d#series}
     DSDSOUT=300
     HOST=fault.Stanford.EDU
     START_GROUP=1
     p=merge_hrsc d=1 a=n \
      in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
      out=prog:mdi_raw,level:merge,series:merge160k[5209]
     p=merge_hrsc d=1 a=n \
      in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
      out=prog:mdi_raw,level:merge,series:merge160k[5210]
     p=merge_hrsc d=1 a=n \
      in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
      out=prog:mdi_raw,level:merge,series:merge160k[5211]
     p=merge_hrsc d=1 a=n \
      in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
      out=prog:mdi_raw,level:merge,series:merge160k[5212]
     END_GROUP=1

     A setenv variable of the name of the "prog:" specification in the
     dataset names, must be set to map the dataset names into a directory
     structure. This variable is used for input dataset name resolution when
     dsds_svc is not used for input name resolution, i.e d=1 is not specified.
     The variable is used for output name resolution in the case of both DSDS
     assigned output storage and local output storage. The {} tokens are taken
     from the dataset name, except {dbase} which is specified, or resolved by
     dsds_svc when DSDS output is requested, i.e. when DSDSOUT=n is given.
     The request for DSDS output will override any specified {dbase}.
     The {dbase} is basically the root of the output directory tree.


FILES

     none


SEE ALSO

     SOI-TN-104 "Naming of Datasets for SOI" SOI-TN-107 "Program-
     ming  in  the  SOI  Analysis  Environment"  SOI-TN-109 "SSSC
     Implementation Version 1.0"


DIAGNOSTICS


BUGS


AUTHOR

     Jim Aloise


HISTORY

     1996-06-14     SOI Version 1.3.0 1995-06-20     SOI  Version
     1.0.1