Archival of Mt. Wilson 60-Foot Solar Tower Data at the MDI Science Center

SOI TN 03-146
P. Rose & R.S. Bogart
2003.10.20

Introduction

This document provides a design and interface specification for the archiving and processing of full-disc, high-resolution (1024x1024 pixel array), and low-resolution (512x512 pixel array) solar Doppler data from the Magneto-Optical Filter (MOF) on the Mt. Wilson 60-ft Solar Tower. The aim of the project is to provide an ongoing publicly-accessible archive of original and processed data from the instrument, ultimately covering the duration of the observing series from 1987 to present date.

Data Product Descriptions

Level-0 data

The Level-0 filtergrams are obtained with an automated data acquisition system using the Magneto-Optical-Filter (MOF). The MOF allows for the selection of a right-circularly polarized component of a very narrow portion of the wavelength being sampled, or a left-circularly polarized component, shifted to a slightly lower wavelength. A pair of solar images are taken every minute in these doppler-shifted wavelengths producing a red-shifted component, and a blue-shifted component of the primary wavelength of the cells being used, in this case either potassium or sodium.

RED-filtergram image taken Sep 25,2000	BLUE-filtergram image taken Sep 25,2000

Observation histories for all 17 years of the Level-0 filtergrams are broken down into two categories; the pre-MDI era, and the MDI-era. The MDI instrument on the SOHO spacecraft became operational in early 1996, so observation histories are broken down into year ranges of 1987 through 1995 and 1996 to current date.

Pre-MDI-era Observation History covering the period 1987-1995



MDI-era Observation History covering the period 1996-present

Level-1 data

Level-1 Dopplergrams are computed from each pair of raw filtergrams on a per-minute basis. Each Dopplergram is a difference of the filtergrams in the sense of red - blue, and the difference is divided by the sum of the filtergrams to normalize the image. Likewise, a Total-Intensity image is simply the sum of the red and blue filtergrams. The purpose of the 60-Foot Solar Tower Project is to collect as many raw filtergrams for the computation of Level-1 Dopplergrams, by utilizing the 2-cell MOF. Unfortunately, two working filtercells were not always available because of their finite lifetime, and a limited amount of single-cell images were taken. By itself, a single cell has the affect of producing total-intensity images. This document is mainly concerned with the computation, conversion, and archiving of Level-1 Dopplergrams; the computation of total-intensity images is not being planned at this moment.

Dopplergram computed from above two filtergrams (red-blue)/(red+blue)	Total Intensity image computed from above two filtergrams; (red+blue)

The Level-1 Dopplergrams consist of FITS files organized in directories containing all images observed within a given day. These data sets will be organized under dataset names prog:mwo,level:lev1,series:vel_01d[day-number], with an epoch for the day number started early enough to assure non-negative values in the archive; i.e. 1987.01.01_00:00:00_TAI. Individual files will be named uyymmdd.hhmm.fits, where yymmdd is the 6-digit year, month and day of the data set, and hhmm is the 4-digit hour and minute, corresponding to the nominal observation time.

The FITS record attached to each of the images contains the following keywords.

The following keywords will be included in FITS records in all other years except for 1996;

FITS conversion notes

The FITS conversion process is verified for correctness by generating data tables of the keyword values for each minute of the day. Plots are generated from these tables and archived at USC, but they are not being made available at Stanford, or on the USC 60-Ft Solar Tower web site. Plots can be requested from USC through email; erhodes@usc.edu. A sample of the plots available are found in the links below:

1. Ephemeris data: including P-angle, B-angle, and Carrington Longitude
2. Heliographic coordinates: including major-axis, minor-axis, ellipse angle, etc.
3. Image Scaling: including major, minor, and average image scales, and time checks
4. Velocity and Calibration: including doppler signal, scaled velocity, velocity residuals, and quality of the images

Data Flow

1. Collection of Level-0 data: raw data has been collected on 8mm Exabyte tapes in original (sunio) format; data collection will continue to be carried out by the Mt. Wilson, 60-Foot Tower staff, with the tapes permanently archived at USC.
2. Creation of Level-1 data: raw data has been processed into Dopplergrams and stored to either 8mm tape or CD-R in the original sunio format. Future processed Dopplergrams will be stored only on CD-R.
3. Calibration: the Level-1 Dopplergrams will be re-calibrated to the Sun_Earth radial velocity component with a higher resolution than previously used and allows for computation of velocity errors and quality of the image.
4. Conversion: the calibration information, and additional meta-data that comes from the raw-to-doppler processing step will be used to create a FITS record for each image in the doppler time-series. Procedure was designed at USC and uses tools obtained from Stanford.
5. Transfer: the Level-1 FITS images are transfered to Stanford and stored on their tape-archive system at the MDI Science Center.
6. Helioseismology processing: the Level 1 data will be processed through the same pipeline modules in use at Stanford for processing MDI data to such products as spherical-harmonic amplitudes, mode frequencies, ring-diagram and time-distance data sets, and archived at Stanford.

Data flow notes

1. The processing of Mt. Wilson data from native format takes about 4 hours per tape, which contains one day's worth of data, primarily due to tape read time; apart from observation, this is the principle limitation on overall data throughput.
2. Tests conducted on 8 Oct., 1998 suggested that under good conditions the transfer rate from USC to Stanford was on the order of 13.5 2-MB files per minute, or a rate of roughly 27MB per minute. More recent tests conducted on 9 Oct., 2003 revealed a faster transfer rate of about 60MB per minute. At this rate, about 30 files per minute can be transfered to Stanford; about twice what it was in 1998. It should thus require about 15 to 20 minutes to transfer one day's worth of data. The data will be pulled from USC to Stanford by ftp. A file will be generated upon completion of the transfer of a full day of data. This file will include a total count of images received; a tabulation of the number of images of correct file length and number of images of incorrect file length. Images of incorrect file length will be re-sent.
3. Level-1 Doppler data located on temporary disks at Stanford will be "ingested" into the MDI archive via a cron job developed by Rick Bogart. The data will be immediately accesible through the MDI data archive located at: http://soi.stanford.edu/sssc/progs/mwo
4. The Level 1 data are expected to be functionally equivalent to MDI Level 1.5 data, i.e. capable of being processed directly by the level 2 pipeline modules; this requires that they have all required ancillary data described under appropriate keywords, and that they be organized in sets of equal time steps, with gaps noted in the descriptor file.
5. It is our aim and expectation that minimal changes will be required to the post-Level-1 processing modules: the only modifications that should be needed are fixes to bugs that may be uncovered in the processing of Doppler data with problems not encountered or successfully solved previously.

Estimated data volume

• Level 0: 250 MB / hr (dataset); 2 GB / day; 0.4 TB / yr; 6+ TB to date
• Level 1: 125 MB / hr (dataset); 1.0 GB / day; 0.2 TB / yr; 3+ TB to date

• 1024-Na-MOF: accounts for more than 90% of accumulated data
• 512-Na-MOF: most common in 1992 and 1993, when original JPL-built CCD camera failed.
• 1024-Na-INT: collected when there was only one working cell
• 1024-K-MOF: most common in latter half of 1993, when Na-cells were unavailable.
• 512-Na-INT: rare
• 512-K-INT: rare
• 1024-K-INT: extremely rare
• 512-K-MOF: extremely rare

Below, is a summary of the days observed from 1987 to the current date. The data products are broken down into hi-res data and low-res data, and may contain any of the wavelength and filter types outlined above.

Below is a graphical representation of the amount of Level-1 doppler data that has been processed each year and is available for FITS conversion and archive to Stanford. The lower panel plot shows the amount of data that has already been archived at Stanford. At the time of this writing, there remains 126 days of pre-MDI, Level-1 data to be processed, and 153 days of Level-0 filtergrams from 2002, that require a spatial correction and further processing to Level-1 Dopplergrams.

Data product availability at the University of Southern California is broken down here. For purposes of comparison and standardizing the way helioseismic data is counted and blocked, MWO data is broken down into GONG months. Named after the Global Oscillation Network Group, operated by the National Solar Observatory, a GONG month is a series of 36 calendar days, and the first GONG month started on May 7, 1995. The plot below shows the amount of data product processed for each GONG month at USC and The Mt. Wilson Observatory for the MDI-era. The MDI-era is defined as starting, here, on May 1, 1996, or at the beginning of GONG month 11.

Prior to the beginning of the GONG project, the 60-Foot Solar Tower data was also broken down into 36-day periods to keep in sync with the GONG months. We named the periods prior to the GONG project as MWO-months. These are numerically increasing months starting from June 12, 1988 and running through May 6, 1995, covering a total of 70 MWO-months. Note: data from 1987 has also been collected but it was not originally counted in the creation of the MWO-months. Ultimately, the MWO-months will be renumbered to accomodate for the 1987 data, but that re-numbering will occur later.

The bar-graph plot below represents the data collected and processed for the pre-MDI era. MWO months 71 through 80 are equivalent to GONG months 1 through 10, and are plotted this way for convenience only. The bars with open lines represent the amount of original raw data collected and the potential amount of doppler (or intensity) data that can be processed for that month, but not yet completed. The filled in dark bars are representative of the amount of complete doppler data processed for that month.

Estimated data transfer time

The remaining time to complete the level 1 processing on the unprocessed level 0 images is currently estimated to total 299 days. Given our current staffing levels, we expect that this processing will be completed in roughly ten months, or by mid-August, 2004.

At our current staffing levels, we estimate that the number of days of Dopplergrams which will be converted and transferred to Stanford between June 1, 2004, and May 31, 2005, will be 1560. Combined with the 176 days previously transferred to Stanford, and combined with the 310 additional days which we expect to have transferred while we are completing the level 0 processing of our pre-MDI filtergrams, we anticipate that the total number of days of Dopplergrams which will be transferred to Stanford prior to May 31, 2005, will equal 2046.

If we are forced to reduce the number of our data analysts from two to one in the middle of 2004, then we will not be able to transfer as many Dopplergrams to Stanford by May 31, 2005. In that case, we estimate that the number of days of Dopplergrams which will be transferred to Stanford by that date will only equal 310+780+176 or 1266.

Subtracting the 176 days of data which have already been transferred to Stanford from the total number of observing days of 3270, we get a total of 3094 days of observations to be processed and transferred. If we also subtract the total of 101 of these days during which we only had total intensity images rather than Dopplergrams available for processing, we arrive at a total of 2993 days of Dopplergrams which remain to be processed and transferred by the end of the third year of our current LWS grant. With the higher estimate of 2046 days of Dopplergrams which we think might be transferred to Stanford prior to May 31, 2005, we would be left with an additional total of 947 days of Dopplergrams, which would have to be transferred to Stanford prior to the May 31, 2006, termination of our current LWS grant. That number of Dopplergrams which would have to be transferred during the third year of our LWS grant is a reasonable number assuming that we can continue to obtain the services of all of our existing data analysts.

Using the reduced estimate of the number of days of Dopplergrams which will be transferred to Stanford by May 31, 2004, we will only have processed and transferred a total of 1266 Dopplergrams prior to May 31, 2005. This reduced level of processing would leave a total of 1727 days of observations to be processed and transferred during the third year of our current grant. This total is substantially more than one data analyst working alone would be able to process and transfer during that year.