MDI QUALITY and the DATA_SW and PROBLEM Bits
A method has been implemented to summarize the qualities of MDI data.
The keyword QUALITY is set for each image or data element. If bits are
set in QUALITY there may be something noteworthy or bad in the data.
QUALITY bits are determined using both automatic criteria
and by manual inspection.
Automatically determined indicator bits are
first set in the data status word, DATA_SW (e.g. the fraction of missing).
Those set subjectively are contained in the keyword PROBLEM. These are
combined to determine QUALITY.
This page is intended to summarize how these keywords are to be used and understood.
These keywords appear at level 1 and higher.
The DEFINITIVE values are in the level 1
Derivative values appear in the higher level log
files, the record files at higher levels, in the mission long
and in various Data_Good files. Each quantity is actually
described by 3 keywords indicating the value, the validity, and the
time it was set:
QUALITY, QUAL_VLD, & T_QUAL;
DATA_SW, DATA_SSW, & T_DATASW;
PROBLEM, PROB_VLD, & T_PROB.
Order of Quality Propagation
- DATA_SW (Data Status Word) bits are determined automatically at level 1 by
- PROBLEM bits are determined manually and maintained using
- QUALITY is computed from DATA_SW and PROBLEM and is defined for each MDI data element
- Binlog files collect the QUALITY and QUAL_VLD keywords for each data series
- A BADMASK is applied to binlog values to select data in Data_Good files
General Properties of QUALITY, DATA_SW and PROBLEM
The keywords QUALITY, DATA_SW, and PROBLEM are 32 bit quantities.
Bits are assigned specific individual meanings.
Certain bits will have the same meaning for all (or many) MDI data
These common bits will generally be in the lower 16 bits of the keyword.
Other bits are specific to the context: observable, data series, or level.
These bits will generally be in the upper half of the word.
Individual bits in QUALITY, DATA_SW, and PROBLEM have the same meaning for a given context.
Each quantitity has accompanying keywords indicating which bits are valid and
the time it was last modified.
Quality Bit Assignments
Context Specific Bits
For programming usage, appropriate #DEFINEs for the bits can be
found in ~CM/include/data_qual.h
This should be updated as new definitions are created.
- Bits have been defined for the following observables:
DATA_SW - the Automatically Determined Data Status Word
The DATA_SW keyword contains bitwise information derived during data processing.
- Bits in DATA_SW will only be set automatically
at level 1 by gather_qual.
- DATA_SSW indicates bit for bit which bits of DATA_SW are valid.
- The T_DATASW will be set whenever DATA_SW or DATA_SSW change.
PROBLEM - Manually Determined Properties of the Data
- Bits in PROBLEM will be determined manually and only set using
Exception: Bit 31
- Bits in PROB_VLD will be set if the corresponding bits in
PROBLEM are valid.
- The prob_upd program will normally update PROBLEM and PROB_VLD
by `|'ing the new values with the existing values in the level 1 log files.
- prob_upd will normally set T_PROB to the current time when executed.
QUALITY - Summary of the Qualities of an Image
When deciding whether to use data, the bits of QUALITY must be evaluated.
For convenience a mission-long fits file containing QUALITY and QUAL_VLD
will be created for each data series that can be exported along with a data set.
Anyone can then specify a way to evaluate the various bits to make a selection.
The program binlog_upd will update these files.
QUALITY is computed from the DATA_SW, PROBLEM and the validity keywords as follows:
QUALITY = ( DATA_SW & DATA_SSW ) | ( PROBLEM & PROB_VLD)
QUALITY is updated whenever DATA_SW or PROBLEM are changed.
QUAL_VLD is computed at the same time from DATA_SSW and PROB_VLD.
T_QUAL is set to the current time whenever QUALITY is set.
BINLOGs and Data_Good files
BINLOGs and Data_Good files collect QUALITY information into a more
compact and useable form.
QUALITY and QUAL_VLD are collected in on-line mission-long log files called
There are two classes of binlogs: Observable and Data Series BINLOGs.
Binlogs are fits files.
- The observable binlogs are named
qual_observable.fits in the directory /soidata/info/mdi_log/lev1/
Observable binlogs have an entry for each mission minute (or half minute).
- There could be such a binlog for each observable.
In practice they will
likely be generated only for high-rate data products.
- Observable BINLOGs are based on the level 1 log files.
Data_series binlogs are named
qual_series.fits in the directory
- They contain the QUALITY and QUAL_VLD for each expected element of a
data series. The data series binlogs are derived from level 1.5 (or level 1.4) logs.
binlogs are created and maintained by binlog_upd.
The structure of each binlog is defined in a file named
observable.rdb or series.rdb
in the directory ~CM/tables/observables/status_logs/
Functions to access binlog information are available in the
MDI software library. They are described on the
binlog man page.
The Data_Good stuff is not very well defined yet.
A fits file containing a single value for each record
may be generated for any DPC or data series for any processing thread.
The value will be the result of a comparison of the QUALITY keyword with
a BADMASK. The value must be non-zero only if the data for that record are OK.
The fits file header will specify the start time and time step.
Currently these are defined only for ts_eq datasets.
The QUALITY comparison used to generate values in the Data_Good file will
be specified in a BADMASK table that may be unique to each analysis.
The responsibilty to oversee data quality has been divided. Each responsible
scientist is listed below, together with the data products he will examine
on an ongoing basis.
Some Historical Notes
Author JTH; Last modified by KRL on 23 Jun, 1998