PE(1)
NAME
pe - pipeline execution
SYNOPSIS
pe [-v] [-d] [-L] [-A] [touch=#] [db=database_name]
[rcp=/dir] [pds=host_name] map=map_file
DESCRIPTION
pe executes a series of strategy level servers in a pipe-
line.
pe is launched with the name of the map_file that defines
its processing. The map file specifies the strategy level
servers to be run, the hosts they are to be run on, the
sequence in which to run them, the dataset they are to pro-
cess, and any DSDS support for dataset name resolution or
data archiving.
pe is run as a PVM (Parallel Virtual Machine) task. A pvm
daemon is spawned if necessary on each host specified in the
map file.
OPTIONS
-v The verbose flag.
-d The debug flag. Brings up a dbx window on each server.
-L The local flag. Run local dsds_svc/ampex_svc. Don't use
pe_rpc. This is for development use only and requires a
password.
-A The ampex flag. Access the ampex_svc if need to
retrieve data that is not on-line.
db= The name of the database to connect to. If this is
missing then the env vrbl DBNAME specifies the data-
base. If this is missing then the default is the mdi_2X
database.
rcp= The local dir to rcp the input datasets to before run-
ning a module.
pds= The host name to assign local /PDS storage from. If
not specified, will use the env vrbl PDS_SET_HOST, else
will use the local host that pe is executing on. If the
host has no local /PDS storage, it is assigned from
sonar.
touch=
The #of days to retain input datasets before deletable.
The value "keep" will retain the ds until findkeep is
run to change it.
EXAMPLES
#This is a text file that specifies a pipeline to be run by PE.
#Notice that it operates on "keyword=value" pairs. In the case
#of flags, any non-zero value is treated as true.
#
#Control Statements:
#!
# A shell escape command. Any setenv is intercepted and interpreted
# locally for the PE environment. No metachars for a setenv.
# (see below for the use of setenv for directory mapping.)
# Note that a map file is not a shell script.
# Pe validates the entire map file and executes any shell
# escape commands before any servers are started.
#DSDSOUT=n
# DSDS will be asked to allocate n megabytes for the output
# of this pipe. The output directory that DSDS assigns can be
# used in the name template env variable "prog:" by specifying
# {dbase}. Notice that if you do not specify d=1 (see below)
# for a server, but did specify DSDSOUT then you must set a
# default {dbase} so that the name template will also map the
# input.
#HOST=name
# The host name that all the servers are to run on. If the
# fsn-lsn split feature is reimplemented then multiple hosts
# can be given [,name,...]. Also multiple copies of the
# servers can then be run on the same host by specifying the
# host name more than once.
#NOWARN=1
# Omit pe warning messages arising from analysis of the map
# file. This is used by map building programs that use pe
# features that may result in a warning. A user building a
# map file should never use NOWARN.
#START_GROUP=1
#END_GROUP=1
# Any servers between a START_GROUP/END_GROUP pair will be
# started at the same time by pe. Normally a server does not
# start until the previous server has completed. Pe will check
# for any mismatched pairs.
#START_ARCHIVE=1
#END_ARCHIVE=1
# Any servers between a START_ARCHIVE/END_ARCHIVE pair will
# be archived only according to the archive flag of the last
# server of the group. This allows the same server
# (e.g. gather_info_svc) to be run more than once with the
# same output dirs with only one archive occuring according
# to the last run. Pe will check for any mismatched pairs.
#Line Tokens:
#p= The name of the server process to run.
#d= A non-zero indicates that this servers input will be
# obtained from DSDS. Otherwise a name template, found in the
# env variable of the name of the "prog:" in the ds name, will
# be used to determine the input directory.
#a= Determines the type of archiving for this server's output.
# This will be a nop unless DSDSOUT=n is specified. Examples:
# a=t5 Temporary dataset. Do not archive and do not mark delete
# pending for 5 days.
# a=n5 Normal. Archive but do not mark delete pending for 5 day.
# a=p Permanent. Archive but never mark delete pending.
# a=a Appendable. The output dataseries will be appended to.
# The dataseries must be defined in the ds_naming table.
# The data is not archived and not deleted. No DSDSOUT=n
# is required as the storage is in seperate disk partitions
# assigned for append.
#l= The default is for pe to copy the history log file and the map
# file that was run, into the dataset output working dirs. This
# is overridden by a log flag of l=0. This is used for data that
# is appended to a dir outside of DSDS storage and you don't
# want the history log and map file to show up for every run.
# NOTE!! The l= is obsolete with the V3R0B0 release (29Apr98).
# It is replaced by COPY_HISTORY=no to override the default.
#x= The default action, if a module send back an abortflg and is
# not part of a group (i.e. START_GROUP/END_GROUP), is to abort
# further execution of the map file and to not archive the output
# wd of the module. The x=1 will override this default abort
# action and case the map file to continue and any output wd to
# be archived.
# NOTE!! The x= is obsolete with the V2R7B0 release (15Dec97).
# It is replaced by ABORT_ACTION=continue to override the default.
#in= The datacollection for the pipe's input data. This isn't
# necessarily called "in", but whatever the server's input ds
# argument is called.
#out= The datacollection for the pipe's output data. This isn't
# necessarily called "out", but whatever the server's output ds
# argument is called.
#arg= Optional argument name and value to override any default
# argument values for a server.
#history= The name of the history log output file. The default is
# /tmp/modulename_username_pvmtid.log.
#errlog= The name of the error log output file. The default is the
# history log file.
#
#Any control statement must be on a separate line.
#Any non-control line must start with p= ,and must be a separate line.
#Any p= line may optionally contain an a=[t,n,p][#].
#Any p= line may optionally contain an d=1.
#Any p= line may optionally contain an l=0 and an x=1.
#Any p= line may optionally contain any argument value for the server.
#Any line, except a "!" line, may be extended by ending it with " \".
#The "\" line extention cannot be used between a "key=value" pair.
#Any string argument may contain imbedded spaces by quoting the
#string, e.g. name="my string". However no quoted string may contain
#the "*" character.
#
!setenv mdi_raw wd:{dbase}/{level}/{series}/{#%06d#series}
DSDSOUT=300
HOST=fault.Stanford.EDU
START_GROUP=1
p=merge_hrsc d=1 a=n \
in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
out=prog:mdi_raw,level:merge,series:merge160k[5209]
p=merge_hrsc d=1 a=n \
in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
out=prog:mdi_raw,level:merge,series:merge160k[5210]
p=merge_hrsc d=1 a=n \
in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
out=prog:mdi_raw,level:merge,series:merge160k[5211]
p=merge_hrsc d=1 a=n \
in=prog:mdi_raw,level:tlm,series:sci160k[1001] \
out=prog:mdi_raw,level:merge,series:merge160k[5212]
END_GROUP=1
A setenv variable of the name of the "prog:" specification in the
dataset names, must be set to map the dataset names into a directory
structure. This variable is used for input dataset name resolution when
dsds_svc is not used for input name resolution, i.e d=1 is not specified.
The variable is used for output name resolution in the case of both DSDS
assigned output storage and local output storage. The {} tokens are taken
from the dataset name, except {dbase} which is specified, or resolved by
dsds_svc when DSDS output is requested, i.e. when DSDSOUT=n is given.
The request for DSDS output will override any specified {dbase}.
The {dbase} is basically the root of the output directory tree.
FILES
none
SEE ALSO
SOI-TN-104 "Naming of Datasets for SOI" SOI-TN-107 "Program-
ming in the SOI Analysis Environment" SOI-TN-109 "SSSC
Implementation Version 1.0"
DIAGNOSTICS
BUGS
AUTHOR
Jim Aloise
HISTORY
1996-06-14 SOI Version 1.3.0 1995-06-20 SOI Version
1.0.1