MDI QUALITY and the DATA_SW and PROBLEM Bits

A method has been implemented to summarize the qualities of MDI data. The keyword QUALITY is set for each image or data element. If bits are set in QUALITY there may be something noteworthy or bad in the data. QUALITY bits are determined using both automatic criteria and by manual inspection. Automatically determined indicator bits are first set in the data status word, DATA_SW (e.g. the fraction of missing). Those set subjectively are contained in the keyword PROBLEM. These are combined to determine QUALITY.

This page is intended to summarize how these keywords are to be used and understood.

These keywords appear at level 1 and higher.

The DEFINITIVE values are in the level 1 log files.
Derivative values appear in the higher level log files, the record files at higher levels, in the mission long summary binlogs, and in various Data_Good files. Each quantity is actually described by 3 keywords indicating the value, the validity, and the time it was set:

QUALITY, QUAL_VLD, & T_QUAL;
DATA_SW, DATA_SSW, & T_DATASW;
PROBLEM, PROB_VLD, & T_PROB.

• DATA_SW (Data Status Word) bits are determined automatically at level 1 by gather_qual
• PROBLEM bits are determined manually and maintained using prob_upd
• QUALITY is computed from DATA_SW and PROBLEM and is defined for each MDI data element
• Binlog files collect the QUALITY and QUAL_VLD keywords for each data series
• A BADMASK is applied to binlog values to select data in Data_Good files

General Properties of QUALITY, DATA_SW and PROBLEM

• The keywords QUALITY, DATA_SW, and PROBLEM are 32 bit quantities.
• Bits are assigned specific individual meanings.
• Certain bits will have the same meaning for all (or many) MDI data products.
  These common bits will generally be in the lower 16 bits of the keyword.
• Other bits are specific to the context: observable, data series, or level.
  These bits will generally be in the upper half of the word.
• Individual bits in QUALITY, DATA_SW, and PROBLEM have the same meaning for a given context.
• Each quantitity has accompanying keywords indicating which bits are valid and the time it was last modified.

Quality Bit Assignments

Common bits

• Recommended definitions for the first 11 bits are suggested that apply to many image type data sets.
  Standard QUALITY bits.

  Note: The concept of a missing time step has no meaning at level 1. Thus bit 1 will be set only in the higher level log and record files by gather_15qual.

Context Specific Bits

• Bits have been defined for the following observables:
  • fd_M - Full Disk Magnetograms
  • fd_V - Full Disk Dopplergrams
  • fd_Ic - Full Disk Continuum Intensity
  • Limb Figure
  • rwbin_Ic - Flux Budget
  • rwbin_Ld - GOLF Proxy
  • loi_Ic - LOI Proxy Continuum
  • loi_V - LOI Proxy Velocity

DATA_SW - the Automatically Determined Data Status Word

• Bits in DATA_SW will only be set automatically at level 1 by gather_qual.
• DATA_SSW indicates bit for bit which bits of DATA_SW are valid.
• The T_DATASW will be set whenever DATA_SW or DATA_SSW change.

PROBLEM - Manually Determined Properties of the Data

• Bits in PROBLEM will be determined manually and only set using prob_upd. Exception: Bit 31
• Bits in PROB_VLD will be set if the corresponding bits in PROBLEM are valid.
• The prob_upd program will normally update PROBLEM and PROB_VLD by `|'ing the new values with the existing values in the level 1 log files.
• prob_upd will normally set T_PROB to the current time when executed.

QUALITY - Summary of the Qualities of an Image

• QUALITY is computed from the DATA_SW, PROBLEM and the validity keywords as follows:
  QUALITY = ( DATA_SW & DATA_SSW ) | ( PROBLEM & PROB_VLD)
• QUALITY is updated whenever DATA_SW or PROBLEM are changed. i.e. whenever gather_qual, gather_15qual, or prob_upd are run.
• QUAL_VLD is computed at the same time from DATA_SSW and PROB_VLD.
• T_QUAL is set to the current time whenever QUALITY is set.

BINLOGs and Data_Good files

BINLOGs

• The observable binlogs are named qual_observable.fits in the directory /soidata/info/mdi_log/lev1/
• Observable binlogs have an entry for each mission minute (or half minute).
• There could be such a binlog for each observable.
  In practice they will likely be generated only for high-rate data products.
• Observable BINLOGs are based on the level 1 log files.

• Data_series binlogs are named qual_series.fits in the directory /soidata/info/mdi_log/level/
• They contain the QUALITY and QUAL_VLD for each expected element of a data series. The data series binlogs are derived from level 1.5 (or level 1.4) logs.

• binlogs are created and maintained by binlog_upd.
• The structure of each binlog is defined in a file named observable.rdb or series.rdb in the directory ~CM/tables/observables/status_logs/
• Functions to access binlog information are available in the MDI software library. They are described on the binlog man page.

Data_Good

• A fits file containing a single value for each record may be generated for any DPC or data series for any processing thread.
• The value will be the result of a comparison of the QUALITY keyword with a BADMASK. The value must be non-zero only if the data for that record are OK.
• The fits file header will specify the start time and time step. Currently these are defined only for ts_eq datasets.
• The QUALITY comparison used to generate values in the Data_Good file will be specified in a BADMASK table that may be unique to each analysis.

DATA EVALUATION

The responsibilty to oversee data quality has been divided. Each responsible scientist is listed below, together with the data products he will examine on an ongoing basis.

Rick Bogart

• rwbin_Ic_01d
• rwbin_Ld_01d
• limb
• fd_Ic_01h

Rock Bush

• fd_Ic_bin2x2_30s_01h
• fd_V_bin2x2_30s_01h

Tom Duvall

• hr_I0_01h
• hr_I0_1024x850_01h
• hr_M_1024x850_01h

Todd Hoeksema

• fd_M_96m_01d
• hr_M_01h
• hr_M_1024x850_01h
• fd_M_01h

Phil Scherrer

• fd_I0_01h
• loi64_V_01d
• loi64_Ic_01d
• fd_V_bin2x2_30s_01h

Jesper Schou

• fd_V_01h