GATHER_QUAL(1)
NAME
gather_qual
SYNOPSIS
gather_qual in={level 1 dataset spec}
DESCRIPTION
Gather_qual produces a quality summary file for each input
dataset. The summary file is stored as a record data file
containing headers only in an "Appendable Dataset" within
the SSSC system. The file is an rdb file containing a set
of keyword-value pairs for each record in the input dataset.
Gather_qual takes input from both explicit and implicit
sources. The primary input is the input dataset specified
with the "in" argument. The "in" argument may specify a
single dataset or a data collection. Each dataset in a data
collection is processed one at a time. The description that
follows is for each input dataset.
The input dataset specifies a
prog,level,series,series_number. The prog,level,series is
used as the specifier of the output dataset. The input
series_number is used as the record_number in the output
dataset. The file containing the output data will be
referred to as the "output record file". It is expected
that after gather_qual is executed another program will be
run which enables opertor modifications to the output record
file based upon operator inspection of the data. A prior
version of the output record file will be read before any
other processing to preserve any operator input.
Gather_qual uses unix file locking to reserve the output
record file during processing.
In addition to reading the input dataset and a possible
prior output record file, gather_qual reads any level0
record file that might exist. The level0 record file is
found in the SSSC soidata "_rec" dataset collection.
In addition to these inputs, gather_qual reads the mission
long MDI status dataset. This is also an appendable dataset
named:
prog:{prog}_log,level:lev0,series:MDI_log_01d[{day_number}].
All the mdi_log datasets are stored in the same directory,
which is allowed for SSSC appendable dataset but not for
normal SSSC datasets.
If the input dataset name is:
prog:prog_name,level:level_name,series:series_name[series_number]
then the output record is:
prog:prog_name_log,level:level_name,series:series_name[0],sel:[series_number]
and the level0 record is:
prog_prog_name_rec,level:level_name,series:series_name[0],sel:[series_number]
and the implied templates are:
setenv prog_name_log
wd:/soidata/info/{prog}_log/{level}/{series},bn:{series}
setenv prog_name_rec
wd:/soidata/info/{prog}_rec/{level}/{series},bn:{series}
setenv mdi_log
wd:/soidata/info/{prog}_log/lev0/MDI_log_01d,bn:{#%06d#series}
The "protocol" of the output dataset is "RDB.RDB" which
means that the output file data record is itself an rdb
file. The filename is: {#%06d#series}.record.rdb
Processing:
Gather_qual does the following:
Build dataset names and pathnames for the output data file,
the level0 record and the mdi_log (or mdi_eof_log) dataset.
Read the level0 record, if present.
Abort if either the mdi_log or the level1 records are absent
for the times needed. Continue with a warning if
mdi_eof_log is absent.
If the output dataset does not exist, create it using
T_BLOCK, T_EPOCH, CONFORMS from input dataset.
Make a lock on the output record file.
Decode the DPC and warn in the cases where quality depends
on more than one time. gather_qual gathers quality for one
T_REF only.
Fill the output overview rdb file as follows:
DSNAME input dataset overview
DPC_STR input dataset overview
T_EPOCH input dataset overview
T_FIRST input dataset overview
T_LAST input dataset overview
TBLXYUPD input dataset overview
T_G_QUAL runtime of gather_qual
Fill the output record as follows:
T_REF input dataset
T_OBS input dataset
QUALITY quality
QUAL_VLD quality valid
T_QUAL runtime of gather_qual
DATA_SW data status word
DATA_SSW data status word valid
T_DATA_SW runtime of gather_qual
PROBLEM problem status
PROB_VLD problem status valid
T_PROB runtime of prob_upd
After the above list down to T_G_QUAL is filled in, read the
observable limits file described below and check all the
keywords specified there for limits and/or bits set as
specified in the limits file.
Clear the bit 31 (MDI_PROB_VALID) in PROB_VLD as an indica-
tion that gather_qual has been rerun and that operator
inspection may be necessary.
Write the output record file and remove the lock.
Free all malloced regions and loop to next input dataset.
NOTES ON STATUS WORDS:
DATA_SW and PROBLEM contain bits as defined in
http://soi.stanford.edu/production/QUALITY/QUALITY.html
DATA_SSW and PROB_VLD contain validity bits for DATA_SW and
PROBLEM.
QUALITY and QUAL_VLD are composite status words where
QUALITY = (DATA_SW & DATA_SSW) | (PROBLEM & PROB_VLD) and
QUAL_VLD = DATA_SSW | PROB_VLD
OBSERVABLE SPECIFIC TESTS:
Note that some of the keywords set rely on observable
specific info that is found in a file based on observable
name, derived from the DPC. The observable control and
limit file is named:
/home/soi/CM/tables/observables/XXX_limits.rdb where XXX is
the observable name, e.g. fd_V.
The format of the limits file is:
KEY Keyword to be tested.
EXPECT Typical value
MAX Max allowed value
MIN Minimum allowed value
RANGE Alternate form to MAX,MIN, gives +- factor to allow from EXPECT.
MASK_OK Bit mask with required bits.
MASK_BAD Bit mask with forbidden bits.
BADNESS value to put into QUALITY if limits exceeded.
DSW_BIT Bit to set in DATA_SW if limit exceeded. Set in DATA_SSW in any case.
A missing MASK_OK, MASK_BAD, MAX, MIN, or RANGE means do not
do that test. The value to test is first searched in the
output record file then in the input data.
FILES
/soidata/info/mdi_log/lev1/{series}/{#%06d#series}.record.rdb
/soidata/info/mdi_log/lev1/{series}/overview.{#%06d#series}.rdb
/soidata/info/mdi_rec/lev[01]/{series}/*
SEE ALSO
http://soi.stanford.edu/production/QUALITY/QUALITY.html
gather_info(1)
prob_upd(1)
DIAGNOSTICS
Writes informational messages to the history log.
Writes warning and error messages to the error log.
Returns non-zero error code on failure.
BUGS
gather_qual gathers quality for one T_REF only. Either this
module should be extended when quality depends on more than
one T_REF or another module should handle the cases where
the result of MDI_dpc_decode is dpc.observable ==
DPC_OBS_FDMAGSUM or dpc.cadence == DPC_SMPL_TWT.
The limits file and observable specific tests are not imple-
mented.