Aa>;SHH SKI`R     dFootnote TableFootnote*e* . e. / - u)]#*]0icLOT TableTitleLOFFigureTOC1Headingy7yz{|}~-. !O/01234-=5- )57317: AppendixHead: Appendix I fz2fits.c. H14549: AppendixHead: Appendix I OP Test Data Catalog (~soi/data/catalog)/ Aa>32541: AppendixHead: Appendix III f0 to FITS conversion script0 SKK43430: AppendixHead: Appendix II Sample OPT Series Logfile (~soi/data/logs/1 P34568: AppendixHead: Appendix V Script for moving data between disk file systems*2 739686: AppendixHead: Appendix VI Archive restore script3 #S25401: AppendixHead: Appendix VII Script for generating a sorted series of data for1H4 g432521: AppendixHead: Appendix VIII Register script% y{} ! )57317: AppendixHead: Appendix I fz2fits.cH14549: AppendixHead: Appendix I OP Test Data Catalog (~soi/data/catalog)>32541: AppendixHead: Appendix III f0 to FITS conversion script>32541: AppendixHead: Appendix III f0 to FITS conversion scriptAppK43430: AppendixHead: Appendix II Sample OPT Series Logfile (~soi/data/logs/TDag (P34568: AppendixHead: Appendix V Script for moving data between disk file systemsscSK739686: AppendixHead: Appendix VI Archive restore scripti~sogsH14549: AppendixHead: Appendix I OP Test Data Catalog (~soi/data/catalog)isste739686: AppendixHead: Appendix VI Archive restore scripts s3>32541: AppendixHead: Appendix III f0 to FITS conversion script ofS25401: AppendixHead: Appendix VII Script for generating a sorted series of data for}432521: AppendixHead: Appendix VIII Register script <$paratext[Title]>7: <$paratext[Heading]> <$curpagenum><$lastpagenum> A<$daynum> <$monthname> <$year>at (Continued)+ (Sheet <$tblsheetnum> of <$tblsheetcount>)I<$daynum> <$monthname> <$year>32"<$daynum>/<$monthnum>/<$shortyear> c5<$daynum> <$monthname> <$year> <$hour24>:<$minute00> ix <$daynum> <$monthname> <$year>a/"<$daynum>/<$monthnum>/<$shortyear>ea <$fullfilename>r <$filename>n <$marker1> <$marker2>39Pagepage<$pagenum>iHeading & Page <$paratext> on page<$pagenum>eaSee Heading & Page%See <$paratext> on page<$pagenum>. Table & Page'Table<$paranumonly> on page<$pagenum>s Figure & Page(Figure<$paranumonly> on page<$pagenum> tSection & Page%Section<$paranum> on page<$pagenum>aEquation NumbertortHeadingd <$paranum>32aAAestTOC e[T Aa >maAuaum<ea..oLOFed)00 LOTshe33sIXUqAa!A3!nam">32#$da$ont%hor&'num(nam)> <*<$m+ix 1.0num,ont-yea."//<$01>ea2$fu3e>r4 $fi5! 6"ker7% 8&>399:age;m>i<dinAppendix III> =ge>>ea? He@ageAparBn pCenuD TaEe'FaraGon HgenI FiJgeK<$pLy> M$paNO & PtioQum>R<$pSTn NUEquVberWHeX$paYZ32[\]e^_TO`a[Tb cd >efghiajkeal.med)n0oshep3qrqs3tuvwxyz{|}~numix =>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^ _ ` a b cdefghijklmnopqrstu v!w"x#y$z%{&|'}(~)*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      ! Appendix I"y#z${%|&}'~()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\Appendix IV]^Appendix VI_`abcdefghijklmnopq r!s AtIuvwxyz{|}~2.0;Appendix II>? Appendix Vpen]^AVI-;ix >?pen      !"#$%&'()*+,-./012345673.0?89:4.0;<5.0=>?@ABCDEFGHIJKLMNOPQRSTUVW XYZ[\]^_`abAppendix VIIcdefghijklmnopqrstuvwxyz{|}~3.04.05.0?@CDGJMPU X[^aix cdefghijklnopqrstuvwxyz{|}~Appendix VIII      !"#$%&'()*+,-./012rqSKI`e SKI`eSKI`e 6ixf 6ix`!0 6)n6) )6.1&f 6.1&h$6 yz 6A=pl6A=pA6m 6 6 I f 6 I h%Pq24Q ofP 30RQLessons from the Analysis of Early MDI Optics Package Test Data Hix f  Hix`#0 H) n H))H.1& f  H.1&h&6` {| HA l HAAiHm H)H I f H I fh'Ql}Lessons from the Analysis of Early MDI Optics Package Test Data~P7Q of P30 1cnQ\(f 1cnQ\(UU`(. Hhnyhf HhnyhH)n24H))LH0$yQf MDH0$yQfh)618 February 1993 H\hro#H\hr\hrHmHSKI`e1H~6f- H~6UU`"."<$paranum><$paratext><$pagenum> `,-"<$paranum><$paratext><$pagenum> )US UT`-S"<$paranum><$paratext><$pagenum> H I f H I ptckh*P 1Q of P29 SKI`e6ixf 6ixOOV6@/UU.mpseudo-user soi with a home directory :~soi. (:/home/soi. on quake,: /usr/people/soi. on UU.`umbra). The exported file system is the one on which the directory :~soi/data. is mounted ar#m#oUU.hrSon both systems. Typical transfer rates are on the order of  MB / min. with no m+#lUU.Wapparent strong dependence on time of day and little dependence on the number of files 7#kUU.`being transferred concurrently (see TFig.. 1). At one time the network connection to LockraC#jUU.$pSheed was down when files were urgently needed, and two alternative mechansism were "<$O#iUU.t>Sused: first, approximately 300 2MB files were restored from a VMS backup tape on a [#hUU.[VMS machine at SLAC and the files were transferred over the campus/SLAC net by ;ftp. g#gUU.T(by hand, in batches of 10) at an average rate of approximately 4 MB /min.; second, s#fUU.do_another 300 files of the same size were read directly from the tape on ;quake. using the . #eUU@.Fnewly installed ;vmsbackup. utility, at a rate of 8.7 MB / min. #dUU ;. i[The data are converted from the 5f0. format to a simple FITS format with the program #cUU;.nocfz2fits (see Appendix III; note that all code and scripts not otherwise noted may be found er #bUU;.kon K~rick/soi/optst.). In principle the conversion could be done on ;umbra. prior to transfer, ra#aUU;.$pYwith the output file on the NFS-mounted system, but there are some minor bugs in the proi#`UU;.usrgram running on ;umbra., the ;f0. data are in ;quake.s natural byte-ordering, and the time con#_UU;.fi]sumed in conversion is negligible in comparison with the file transfer times. As long as the #^UU;.atjfiles are converted in place, i.e. the ;f0. files removed after the FITS files are created, there #]UU;.ec]is no disk usage penalty. To convert all the images in particular series a script is invoked c#\UU;.t `that creates a log file specific to that series, appends a line to that file for each image sum a#[UU;.t Wmarizing all the important key values , and at the sucessful completion informs people l c#ZUU;.t linvolved in maintenance of the database (see Appendix IV). The log files reside on :~soi/ c#YUU;:;edata/logs., and are named :optXXX., where XXX is the 3-digit sereies identifier of the releu'#XUU;.inrvant series, ;e.g.. :~soi/data/logs/opt017.. Appendix II shows an example of one of these log 3#WUU;.inYfiles. The logging of the key word information was added to the conversion program after i?#VUU;.erZa number of series had already been converted. It was added at the same time that the conK#UUU;.d Uversion program was modified to convert observation times from Pacific Standard Time W#TUU@;. iP(the local time in which they have all been recorded so far) to Universal Time. a{x`7 tData Management en#bUU ].fiVThe data files obtained so far would occupy roughly 7 GB of disk space, far more than #aUU]. cVwould fit on the exported file system and substantially more than the total amount of #`UU].A\scratch space currently available to the project. (Of a total of eight 1-GB scratch sytsems re#_UU]..Vtwo are being reserved for analysis of La Palma data and one for analysis of Hawaii / #^UU].. ZAlaska data. Substantial pieces of some of the other systems are devoted to other smaller #]UU].fiWprojects, notably support of the onboard Image Processing tests, or to general scratch ?#V#\UU].a Zspace. Approximately 4 GB divided among five file systems are currently available for all #[UU].veXMDI data products, raw and derived.) It has thus been necessary to move data frequently #ZUU@]. lBbetween different file systems and to and from 8mm tape archives. #YUU [.ZOf the five scratch file systems being devoted in whole or in part to the O.P. Test data, #XUU[.diathree are mounted directly on ;quake. under a single SCSI controller and the other two are l'#WUU[.ta_on similar SCSI disc drives mounted on workstations on the local 10 Mb Ethernet (;autry. f e3#VUU[.sy`and ;griffin.). All scratch systems are cross-mounted among the various machines via NFS, ly?#UUU[.#^hand the data and catalog files invariably reside under the true file directories :/scrOX:/mdi_r K#TUU[:fihdata., 5X. being an identifier associated with the physical drive. In all cases the particular ]W#SUU[.xiasubdirectory names are linked to names under :~soi/data., so the actual location of a data Dc#RUU[.aw[directory on disk is virtually transparent. Thus the raw data files from series opt085 may weeo#QUU[.ysabe physically located on :/scr3/mdi_data/opt085., a file system mounted on a disk drive on gSKI`ee ,HA3K ^f reHHDnFootnoteleH-f aFootnoteimmorta (Hf"sy ! 1Heading Rule _ m!e riUU~!l  fes~ fiH(SNd ^"l# diH_jH_jDi 1Heading RuleH0 #f"%ed$$ Chapter Ruleas $l#[xi o nHJ ^%l#& H(2H(2Da Chapter RuleUUHc&f%)tu'( Table Topd_'m(&ee#Qoca~(l'&.a ~ve EZS(8 ^)l&* EZ\_jEZ\_jD Table TopH ,*f),++ TableFootnote+m*HHRb ^,l* H_H_D TableFootnoteHn(C ^-f. HwjHwjDTable of Contents SpecificationriH-.f-/ UUH-UU`.."<$paranum><$paratext><$pagenum> H ^/f.0 diHEHEDList of Figures Specification$H60f/1 H6UU`/."<$paranum><$paratext><$pagenum> HJh ^1f0 HEHEDList of Tables SpecificationTaSKI`2e34H~3f42 H~`08(1, 23 l`18Z$<$symbols><$numerics><$alphabetics> `28 Level3IX &`38Ta Level2IX 0`48m Level1IX C`56LSymbols[\ ];Numerics[0];A;B;C;D;E;F;G;H;I;J;K;L;M;N;O;P;Q;R;S;T;U;V;W;X;Y;Z L`68e <$pagenum> ^HlAmN ^4f32 jHuHuDeIndex SpecificationHSKI`5e66Hix6f5 UUHix8@> .UU[.nhost ;autry., but can also be found at the location :~soi/data/opt085. (itself physically located UU[.hon file systemK /scr6/mdi_data., mounted on ;quake.). File transfer rates among the various UU[.E_file systems have been regularly recorded (see TFig.. 1). For transfers from one disk to *UU[.f`another, both mounted on ;quake., the average transfer rate varies from about 10 MB / min l6UU[.ZZto about 23 MB / min, the faster rates usually but not always corresponding to larger volBUU[.[umes (> 100 MB) of data transfer. Transfer rates from the disks on the workstations to the ;M;NUU[;W;[quake. disks are comparable (ranging from 10 MB / min to 23 MB / min), but the transfer uZUU[.ecZrate of files onto disks mounted on the workstations is substantially lower, ranging from fUU[.[about 3 MB / min to about 8 MB / min. This strongly suggests that the influence of the net carUU[. tXwork speed is negligible, that transfer rates are determined by the disk and controller il~UU@[./maspeeds, and that these are about a factor of 3 higher on ;quake. than on the workstations. EUU .e YAs with FITS conversion, several scripts have been generated to manage the orderly transUUU. b]ferral, restoral from archive,, removal, and relinking of entire series of data. A good examUU.bo_ple of such a script is the one to move a series from one file system to another (Appendix [UUH.MBV). trҪUU .at[The data are archived to 8mm (Exabyte) tape by directories, using the ;tar. program. parުUU. 1YBecause the data sets are small compared to the capacity of an 8mm tape, multiple series sꪙUU.woWare archived to individual tapes. The TTi Exabyte tape device driver does not properly abUU.is\support the append feature of ;tar. (tar -r), a problem apparently common to all such ibUU.ra]devices; thus it is necessary to archive each directory to separate tape files, forward spaceUU. 3\ing the tape the appropriate file. Individual archive tapes are labeled by 2-digit numbers, ioUU. h[although there is at this point no way of automatically verifying that the correct tape is rch&UU. rVmounted. All archiving has been performed and verified by hand by Jayasree Suryanaray2UU.ovXanan. Data can be restored from archive to a particular disk file system with a control tr>UUH.atscript (Appendix VI). VUU 8.ir\As data are captured, moved, removed, archived, and restored, it is necessary to be able to albUU8.ca^keep track of the location(s) of the data. For this purpose I developed a data catalogue in a nUU8.pe_flat file system, each table entry representing an entire data series, the convenient unit for . zUU8.m Zaccessing the data. This table is described in the next section. The data catalog is mainUU8.toYtained by hand (by Jayasree and myself), based in part on automatically generated notifi.UU8.e ]cations and in part on inspection of the data. A relational database table under Oracle base oUU8.llXon the data catalog has also been prepared by Jayasree. The Oracle databas is currently eeUU8.riQmaintained by hand, by downloading information from the catalog file after major dUU8. pUupdates. Programs designed to update the catalog automatically should be designed to VªUU@8.UU%update the database simultaneously.. r`77anData Tables and Catalogs UU(9.kThe data catalog (K~soi/data/catalog.; Appendix I) grew out of the need to keep track of the a  UU9.peYlocations of data moving among various disk file systems, hosts, and archive tape files. tUU9.UU\The series identifiers optXXX[x] were developed as unique entry identifiers for this table, m"UU9.8Zthe descriptive series test names providing useful but essentially redundant information. .UU9.8\The format of the table is designed to be easily readable and editable by hand, and alsoeasba:UU9.8[ily readable by programs, including the program creating the relational database mirroring eeFUU9.ri\the data catalog. Thus individual entries consist of lines of key values,the indiviudal key UURUU9. PXvaluesbeing arranged in white-space separated columns and being filled in as completely 8^UU9.at_as possible, even if with null or missing identifiers (usually x). The header lines are indi dajUU@9.oi*cated as comments by an initial # mark. SKI`7ehe88pe6ix8f7 ar6ix6: t.leUU`C.9CFor each entry in the data catalog, the key values are as follows: idUU D. teUnique series identifier, of the form ;yyyXXX[z]., where ;yyy. is (so far) always opt, a$UUD;eXXX. is a 3-digit identifier, and ;z. is an optional letter, necessitated by the occasional l0UUD.UUSaccidental assignment of of the same numerical identifier to distinct series. (The aba<UU@D.5numerical identifier was assigned by the observers). rKUU`E.es4Series descriptive name, assigned by the observers. PZUU F.ngVFirst sequence number of the individual images in the series, so far always 0000, but fUUF. eTthe series could have started somewhere else. To the extent that it represents real rUUF.ca[data (as recorded in the log files) this entry is necessary, but strictly as a data identi~UU@F.3fier it (and the following entry) are superfluous. UU H.TLast sequence number of the individual images in the series. The assumption is that idUUH. tSevery image is numbered sequentially, with a stepsize of 1. So far this assumption ar)UUH.Phas never been violated. Occasionally, as with opt053, the last sequence number ttUUH.y Prefers to the largest known data set, in this case what was once mounted on the alUUH;tiXumbra. disk, and not necessarily what exists on our own archives; such discrepancies rsɪUUH.EQare noted. In the case of opt067, a zero-length file corresponding to sn0867 was iժUU@H.r found, but ignored. es䪙UU I. fTDate of first observation, as recorded in the FITS header, but in year-major format . UU@I. i (YY/MM/DD). eaUU J.FQTime of first observation, as recorded in the FITS header. As remarked above the a UUJ.Uoriginal f0 files are time- and date-tagged in local time (PST). Some of the earlier HUUJ.e Vconverted FITS files use these tags in their headers, and are so recorded in the cata#UUJ.isTlog; all other entries are in UT. Although the observation times are recorded to an H/UUJ.enYaccuracy of 0.01 sec in the headers, the catalog entries for series startting and ending e;UU@J.t )times are rounded to the nearest minute. mJUU`K.Date of last observation. YUU`L.ssTime of last observation. hUU M.scXThe file system number X on which the series is currently mounted on ;umbra.; the retUUM. wgdirectory name will be K/dataLXK/optest.. A - indicates that the series is not currently dedUUM.r,Sonline. Although inherently dynamic, there is no mechanism for updating this entry QTiUU@M.at#other than periodic hand checking. As UU N. aXThe file system number X on which the series is currently mounted on ;quake.; the e UUN.idirectory name will be K/scrLXK/mdi_data/LID.. A - indicates that the series is not curUUN.loWrently online. As noted above, any mounted series can be found under the link K~soi/UU@NKradata/LID.. hΪUU O.alMThe ID number of the 8mm tape on which the (most recent) archive of the data nڪUU@O. m@exists. An xx indicates that the data have not been archived. UU骅UU`P.laMThe file number on the archive tape on which the particular archive resides. iUU Q.unRComments; in order to facilitate scanning by the Oracle database tool, commas canUU@Q.. 1not be used as separators, even within comments. dUU R.r,^Starting Jan. 7, individual log files have been created automatically whenever the data files (UUR.r `of a series are converted to FITS format with the standard script in Appendix IV. These i4UUR.d ]are also flat file databases, one line per record, in which each record corresponds to a par/@UUR.in^ticular image in the series, keyed by its sequence number. I began creating these files after L~UUR.Vrealizing that the naming scheme we had worked out for identifying and characterizing X}UU@R.r 3images by instrument configuration was inadequate. thSKI`9ets::atHix:f9 UUHix8<he.veUU S. p^The tabulated entries correspond to all of the potentially interesting FITS key values in the UUS. tVheaders, except for comments. The entries are comma-separated for the purpose of scanUU@S.r,Mning into an Oracle database. For each entry, the key values are as follows: r0UU T.SUnique entry id, corresponding to the image sequence number. There is no provision n <UUT.. Lfor dealing with the case in which a sequence number has inadvertantly been d,HUU@T.orDrepeated in the original data series, as this has not yet occurred. ieWUU`U.qu9Observation date, in FITS (day-major) format (DD/MM/YY). UfUU`V.g Observation time. uUU W.ut[Number of pixels in the ;x. direction (number of colums). This is the more rapidly wasUU@W.(varying subscript in the FITS raw data. :UU`X.@Number of pixels in the ;y. direction (number of rows). UU`Y.SPosition (in units of 2 degrees) of the 1st Michelson Interferometer tuning wheel. FITUU`[.e SPosition (in units of 2 degrees) of the 2nd Michelson Interferometer tuning wheel. or UU`\.n?Position (in units of 2 degrees) of the 1st calibration wheel. y, ̪UU`]. a?Position (in units of 2 degrees) of the 2nd calibration wheel. to ۪UU`^. nEPosition (in units of 2 degrees) of the polarization analyzer wheel. tꪚUU Z. sWA character code denoting the light path in use. Valid values are F (low-resolution, ta UUZ.s Xfull-disc path), H (high-resolution path), and D (dark). Other values are possible, /MUU@Z.but not recognized. tiUU _.WOA 3-character code denoting the instrument observing mode. Possible values are he UU@_.KXXX (dark), CAL (calibration mode), and IMG (image mode or unknown). .,UU `. iUA 3-character code denoting the polarization analysis state of the instrument. Possis8UU`.thRble values are VLP (vertical linear polarization), HLP (horizontal linear DUU`. oSpolarization), LCP (left-hand circular polarization), RCP (right-hand circular n uPUU`. oRpolarization), and XXP (unknown; non-standard location of polarization analyzer \UU@`.brwheel). l.kUU a.^VA character code denoting the effective filter position of the instrument tuning. PoswUUa.teZsible values are 0, 1, 2, 3, 4, and X. Note that the filename convention used UUa.c [a character 5 to denote all else in filenames, but the original raw data files use the @ZUUa.gnPcharacter X to denote non-standard combinations of the Michelson tuning wheel mUUa.esQpositions. Although supprting the convention in the generated FITS-file names, I aUU@a.).Rchose to preserve the X designation in ther data log as a less misleading code. `:7t.Data Analysis 㪙UU ;.vaUThe OP Test series data have been primarily analyzed using the tools of the Pipeline U滛UU;.tiQBreadboard developed by Phil Scherrer and Luiz S (SOI Technical Note TN-XX-XXX; UU;.ri[lost, in preparation, or forgotten). These are designed to work with specially named direc.brUU;.]tories containing sets of related FITS files representing sequential datasets, with names of UU;.aithe form N;X., where ;X. is the sequence number. In order to simplify filename arithmetic in UUU;.teYshell scripts such as those used in the Breadboard, it was agreed to map sequence number +UU;.acZ0 of each test data series to sequence number 50000 in a pipeline. The first task is thus 7UU;.ltZsorting the raw images, which are all filtergrams, into the appropriate order and copying CUU;.e Ythem or linking them to files with the correct names. This is more trouble than it would yOUU;.;Sseem because the commanding computer would occasionally stick, taking two succeseli[UU;.;\sive filtergrams in the same configuration, throwing off the planned order of the observing gUU;., Vprogram and making a straight mapping from serial numbers impossible. This was solved sUU;.Xat first by inspection of the raw data filenames and later by building scripts based on s SKI`;e N<<6ix<f; r 6ix:>);UU;.s finspection of entries in the logfile (see Appendix VII for an example). There is still a probUU;.es[lem with temporal association; no attempt has been made to find closest acceptable data to thUU;.h Ya target time or to put derived Dopplergrams on a regular cadence. Some tools to support i*UU;.s Zthese functions in a sorting program based on the data log files have been written by Kay 6UU@;.co#Leibrand, but not yet implemented. kinNUU .[In order to create Dopplergrams from sets of filtergrams it has been necessary to register of ZUU.Ythem, as the data were taken without the Image Stabilization System in place. No working sfUU.[pipeline registration code having been in place, we are using registration procedures writbasrUU.KYten in IDL by Igor Zayer and put into shell commands by Peter Milford. The procedure for ~UUH.Musing them is illustrated by the register script in Appendix VIII. iUU #.XOnce the images are registered remaining elements of the pipeline processing can be sucsoUU#.t Ncessfully invoked using the standard commands (;ppSSTAT, ppVCALC, ppREBIN, UU@#;deGppSPFFT, ppKW, .etc.) which are all found on K~luiz/scripts.. UUƪUU $.nc[For some of the test data, taken with a lamp-illuminated reticle as the object in order to @;ҪUU$.t Zmeasure the image displacements associated with rotation of the various wheels, a special ުUU$.be[analysis suite was developed. This suite, to be described in a forthcoming note, was develageꪚUU$.teXoped outside the framework of the pipeline breadboard, using simple script calls to proacUU@$.giDgrams operating on the FITS format raw files with minimal overhead. ay`<7elCritique and Suggestions .2UU =. ZAs remarked in the introduction, most of the procedures for dealing with the OP Test data >UU=.UU]are ;ad-hoc.. The point was to get timely results, in which we were to some reasonable uJUU=.#\degree successful, and to get experience. This note is an attempt to record the experience. UUVUU@=. p\In doing so it is useful to reflect on problems or ideas for improvements that have arisen. onUU %.ak\We have attempted to put all executed commands into shell-script files. I believe that this thzUU%.ntZis not only essential for consistency of behaviour, but also extremely useful for documenUU%.. Vtation and in providing a starting point from which more automated and compiled proceUU@%.medures can be built. reUU &.plVThe FITS/gds model in which program data handling has been built has proved workable, UU&.erYbut not particularly simple. In general, it imposes more overhead than really necessary. nªUU&. mZBetter data models are under active development. The pipeline model is simplistic and not ΪUU&.s Ywell-designed in its handling of ancillary (header data). This was no impediment so far, cڪUU&.t Wbut needs to be substantially improved when derived datasets are incorporated into the doi檚UU@&. tdata management scheme. deUU '.s YThe Oracle-based relational databases mirroring the various flat-file databases has been t UU'.es^constructed, but there are as yet no generally accessible tools for using it. (In particular, UU'.lyYthere are keyboard problems associated with the Oracle tools, as they rely on particular c"UU'.ndUcombinations of programmable function keys which may not always be present.) We need I.UU'.icVeither substantial improvement in the provided tools are locally developed interfaces :UU@'.pl,before we can dispense with the flat files. eaRUU ?.RI have come up with a number of ideas for improvements or changes to the existing ^UU@?.+scheme. In no particular order, these are: s hSKI`=e. >> sHix>f= neHix<@ve&seUU (.d TRegularize the naming of data sets. I suggest extending the three-letter plus three-OrUU(.naRdigit plus optional letter series identifier together with the sequence number to UU(. tRinclude multiple levels of processing where applicable and also to include at the *UU(.e Lfront end a project or instrument designation (such as MDI or WSO or GONG). ul6UU(.'QExamples of additional program codes might be fst (full-sunlight tests), dyn WBUU(.UUP(dynamics program, of which there would presumably be only two or four series), edNUU(.Tstr, mag, gct (granule correlation tracking), and cam (general campaign, of veZUU(.mbTwhich there might be hundreds). There is of course nothing magic about 3 characters InfUU(.erO(outside NASA), but there might be an advantage in keeping the designations of irUU(.Sfixed length, and they should be reasonably mnemonic. There should also be similar ula~UU(. d^designations to describe the level of reduction, ;e.g.. raw, reg, dop, frq (freUU(.s Rquencies). In such a scheme what is currently designated in separate locations as UU(.ssOopt067 & 0034 might become MDIopt067raw0034. Of course for flat files it prUU(.t Yis useful to be able to abbreviate these designations. There is no reason to include the oUU(.e Tcharacters MDI in every entry in a table located in a directory named mdi_data, nor thUU@(.lyWto include the meaningless sequence number in an entry referring to the entire series. atiɪUU >.cWAutomate the archiving of data, at least to the extent of providing a script-based pros oժUUH>.ag>cedure analogous to the restore script (Appendix VI). 䪘UU @.n TAutomate the update of the data catalog or its successor, as well as the relational eyUU@.blUdatabase based on it. Eventually the relational database is expected to be authoritaiUU@.duRtive; meanwhile procedures should be developed for settling questions of database UU@@.wh authority. ly UU G.raRModify the main data catalog to give the number of images rather than (or in addi#UUG. fRtion to) the starting and ending sequence numbers, and to give the (maximum) size /UU@G.no per image. clu>UU`A.UUQProvide additional log information for reduced data sets, with appropriate keys. _MUU B.QProvide a mechanism for support of multiple formats of lower-level data catalogs hYUUB.tiSwithin higher-level data catalogs. Currently there is only one format of low-level dineUUB.roVdata catalog, appropriate to MDI filtergrams. Clearly others need to be provided, and qUUB.n Othere should be some mechanism for determining the appropriate format from the ona}UUB.@_master data catalog. The series ID could be used as a key (;e.g.. all opt data are MDI UUB.; Xfiltergrams), but only if the key is extended to refer to multiple levels of processing @UU@B.y of data series. raUU .daWDevelop efficient and accurate image registration code. This is a lower priority issue n tUU.d Tonce the Image Stabilization System is in place and working properly, but will nonee.UU@.AMtheless be of undoubted usefuleness. Lyle Bacon is actively working on this. e˪UU ).UUTProvide tools for organizing data based on queries of the various databases. As menloתUU@).B;tioned above, these are under development by Kay Leibrand. nlySKI`?e@@al6ix@f? 6ix>B3UUh!7ou-OP Test Data Catalog (~soi/data/catalog) rUU`#9he # List of OP tests as of 2/5/93 at+UU`$9ieL# (U: data on /data#/Mdi) (Q: data on /scr#/optXXX) (PST: TIME-OBS in PST) UU7UU`%9am CUU`&9y M# id name fsn lsn st_date time end_date time U Q Archiv notes fOUU`'9 [UU (9loLopt013 cracks 0000 0002 92.xx.xx xx:xx 92.xx.xx xx:xx 4 x xx xx raw - uegUU@(9can't fitsify sUU`)9 SFopt017 fdfocus 0000 0035 92.11.10 19:09 92.11.10 19:55 3 - 09 05 UU`*9f Jopt018 hrfocus 0000 0010 92.11.10 12:01 92.11.10 12:27 3 - 05 01 PST UU`+9toJopt022 calpos 0000 0003 92.11.13 13:36 92.11.13 13:39 4 - 06 01 PST UU`,9BJopt024 calpos 0000 0003 92.11.13 17:17 92.11.13 17:20 4 - 06 02 PST UU`-9Fopt028 exposerence 0000 0044 92.11.17 02:06 92.11.17 02:42 4 - 09 02 UU`.9Fopt029 exposercale 0000 0035 92.11.17 02:52 92.11.17 03:15 4 - 09 01 UU`/9 rFopt030 magdopfd 0000 0041 92.11.17 22:41 92.11.17 23:07 4 - 09 03 ǪUU`09onFopt030a nilinetune 0000 0013 92.11.19 21:09 92.11.19 21:17 4 - 09 03 ӪUU`19Fopt031 nilinetune 0000 0013 92.11.19 21:22 92.11.19 21:30 4 - 11 01 ߪUU`29otFopt034 sunlimbhr 0000 0015 92.11.18 19:57 92.11.18 20:15 4 - 11 03 모UU`39x:Fopt035 nilinetune 0000 0013 92.11.18 21:05 92.11.18 21:12 4 - 11 04 UU`49 SJopt036 dpmtest 0000 0000 92.11.17 08:15 92.11.17 08:15 4 - 07 01 PST UU`59opJopt037 irleak 0000 0002 92.11.19 13:43 92.11.19 13:44 4 - 07 02 PST UU`69opFopt040 nomove 0000 0020 92.11.20 02:21 92.11.20 02:32 3 - 11 05 UU`79BJopt041 sunir 0000 0002 92.11.20 14:26 92.11.20 14:27 4 - 07 03 PST 'UU`89Jopt042 sunmagdop 0000 0040 92.11.20 14:47 92.11.20 15:05 - - 01 02 PST 3UU`99opPopt043 sunmagdop 0000 0043 92.11.20 23:29 92.11.20 23:47 - 6 13 01 0-40 only /?UU`:9pfJopt044 sunir 0000 0006 92.11.20 15:57 92.11.20 16:00 4 - 07 04 PST KUU`;9neJopt045 covernair 0000 0010 92.11.24 16:39 92.11.24 16:44 3 - 07 05 PST WUU`<900Jopt046 covered 0000 0010 92.11.24 16:50 92.11.24 16:55 3 - 07 06 PST cUU`=9 0Jopt047 polcheck 0000 0009 92.11.24 17:13 92.11.24 17:23 4 - 07 07 PST oUU`>93 Jopt048 magdopser 0000 1694 92.11.25 12:45 92.11.xx xx:xx - - 02 01 PST {UU`?9.1Fopt049 m1rotser 0000 0301 92.11.29 01:17 92.11.29 03:45 - - 09 04 UU`@92 Fopt050 m2rotser 0000 0304 92.11.30 18:29 92.11.30 21:00 - - 10 01 UU`A9 0Jopt051 windows 0000 0003 92.12.01 15:58 92.12.01 16:15 4 - 07 08 PST UU`B92 Fopt052 sundopser 0000 0018 92.12.04 19:38 92.12.04 19:47 4 - 11 06 UU C9 0Mopt053 sundopser 0000 1163 92.12.04 12:14 92.12.04 15:47 - 5 01 01 PST; 0-nUU@C93 1130 only êUU`D9- Jopt054 lamponoff 0000 0008 92.12.07 15:06 92.12.08 09:11 3 - 07 09 PST ϪUU`E90 Jopt059 lasdop 0000 0013 92.12.09 16:14 92.12.09 16:27 4 - 10 03 PST ۪UU`F94 Fopt060 protser 0000 0205 92.12.10 00:52 92.12.10 02:32 - - 01 03 窓UU`G911Fopt062 tunerotser 0000 0202 92.12.10 20:04 92.12.10 21:46 3 - 11 02 UU`H9 9Fopt062a tdtrotser 0002 0002 92.12.23 02:26 92.12.23 02:26 4 - 11 02 UU`I92:Jopt063 sunobsser 0000 0842 92.12.12 10:26 92.12.12 14:09 - - 03 01 PST  UU`J91:Jopt065 sunmagtdt 0000 0044 92.12.12 15:26 92.12.12 15:39 4 - 07 10 PST UU`K9 9Jopt066 suncaltdt 0000 0128 92.12.12 15:49 92.12.12 16:24 4 - 08 01 PST #UU L912Kopt067 sunmixser 0000 0866 92.12.13 18:51 92.12.13 22:41 - 6 04 01 many 2.1/UU@L94 $files after 0800 are PST; Rearchive op;UU`M9Fopt068 lamscan 0000 0105 92.12.14 19:53 92.12.14 20:25 4 - 01 04 GUU`N9Fopt069 sunmix 0000 0055 92.12.14 20:41 92.12.14 21:07 4 8 01 05 SUU`O9T Gopt070 sunmixser 0000 0398 92.12.14 21:31 92.12.14 23:16 - 7 13 02 09 _UU`P9T Fopt071 sunmixser 0000 0723 92.12.15 17:05 92.12.15 20:17 4 8 13 03 kUU`Q93 Fopt072 lamscan 0000 0073 92.12.15 20:38 92.12.15 21:01 4 7 12 01 SKI`AeHBBr HixBfA 3 Hix@D63 ssUU`R9.1Fopt073 sundop2 0000 0047 92.12.15 21:05 92.12.15 21:18 4 3 12 02 UU`S94 Fopt074 sundop3 0000 0059 92.12.15 21:19 92.12.15 21:36 4 3 12 03 UU`T9 0Fopt075 sundop 0000 0029 92.12.15 21:37 92.12.15 21:45 4 - 12 04 *UU`U900Fopt076 sunmix 0000 0082 92.12.15 22:04 92.12.15 22:32 4 7 12 05 6UU`V908Fopt077 suncalblock 0000 0044 92.12.15 22:37 92.12.15 22:58 4 3 12 06 BUU`W9 9Fopt080 sundop 0000 0040 92.12.16 17:39 92.12.16 18:41 4 3 12 07 NUU`X9 9Fopt081 sundop 0000 0201 92.12.16 18:43 92.12.16 20:06 4 8 12 08 ZUU`Y9 9Fopt084 darks 0000 0071 92.12.16 23:31 92.12.17 00:22 4 3 12 09 fUU`Z97:Fopt085 darks 0000 0059 92.12.17 02:46 92.12.17 03:03 4 3 12 10 rUU`"90: 9SKI`CeKDDH6ixDfC 6ixBF,(7.Sample OPT Series Logfile (~soi/data/logs/@71:opt080) 5 -UU`9D0000,16/12/92,17:38:30.37, 512, 512,129, 49, 19,126,125,F,CAL,VLP,0 5 9UU`9D0001,16/12/92,17:38:55.57, 512, 512,111, 58, 19,126,125,F,CAL,VLP,1 5 EUU`9D0002,16/12/92,17:39:11.33, 512, 512, 93, 67, 19,126,125,F,CAL,VLP,2 5 QUU`9D0003,16/12/92,17:39:27.17, 512, 512, 75, 76, 19,126,125,F,CAL,VLP,3 5 ]UU`9D0004,16/12/92,17:39:43.10, 512, 512, 57, 85, 19,126,125,F,CAL,VLP,4 6 iUU`9D0005,16/12/92,17:51:14.89, 512, 512,129, 49, 19,126,125,F,CAL,VLP,0 6 uUU`9D0006,16/12/92,17:51:30.97, 512, 512,111, 58, 19,126,125,F,CAL,VLP,1 7 UU`9D0007,16/12/92,17:51:46.90, 512, 512, 93, 67, 19,126,125,F,CAL,VLP,2 7 UU`9D0008,16/12/92,17:52:06.59, 512, 512, 75, 76, 19,126,125,F,CAL,VLP,3 UU`9D0009,16/12/92,17:52:22.67, 512, 512, 57, 85, 19,126,125,F,CAL,VLP,4 UU`9D0010,16/12/92,17:59:01.02, 512, 512,129, 49, 19,126,125,F,CAL,VLP,0 1:UU`9D0011,16/12/92,17:59:16.94, 512, 512,111, 58, 19,126,125,F,CAL,VLP,1 F,UU`9D0012,16/12/92,17:59:32.79, 512, 512, 93, 67, 19,126,125,F,CAL,VLP,2 F,ɪUU`9D0013,16/12/92,17:59:48.79, 512, 512, 75, 76, 19,126,125,F,CAL,VLP,3 F,ժUU`9D0014,16/12/92,18:00:04.88, 512, 512, 57, 85, 19,126,125,F,CAL,VLP,4 F,᪪UU`9D0015,16/12/92,18:18:33.96, 512, 512,129, 49, 19,126,125,F,CAL,VLP,0 F,UU`9D0016,16/12/92,18:18:49.97, 512, 512,111, 58, 19,126,125,F,CAL,VLP,1 F,UU`9D0017,16/12/92,18:19:06.05, 512, 512, 93, 67, 19,126,125,F,CAL,VLP,2 F,UU`9D0018,16/12/92,18:19:25.27, 512, 512, 75, 76, 19,126,125,F,CAL,VLP,3 F,UU`9D0019,16/12/92,18:19:41.11, 512, 512, 57, 85, 19,126,125,F,CAL,VLP,4 F,UU`9D0020,16/12/92,18:31:08.18, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,)UU`9D0021,16/12/92,18:31:23.30, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,5UU`9D0022,16/12/92,18:31:38.43, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,AUU`9D0023,16/12/92,18:32:31.15, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,MUU`9D0024,16/12/92,18:32:52.35, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,YUU`9D0025,16/12/92,18:33:28.59, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,eUU`9D0026,16/12/92,18:33:57.07, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,qUU`9D0027,16/12/92,18:34:14.27, 512, 512, 57, 85, 19,126,125,D,XXX,VLP,4 F,}UU`9D0028,16/12/92,18:36:24.38, 512, 512,129, 49, 55, 17,125,F,IMG,VLP,0 F,UU`9D0029,16/12/92,18:36:48.63, 512, 512,111, 58, 55, 17,125,F,IMG,VLP,1 F,UU`9D0030,16/12/92,18:37:04.55, 512, 512, 93, 67, 55, 17,125,F,IMG,VLP,2 F,UU`9D0031,16/12/92,18:37:20.16, 512, 512, 75, 76, 55, 17,125,F,IMG,VLP,3 D,UU`9D0032,16/12/92,18:37:35.84, 512, 512, 57, 85, 55, 17,125,F,IMG,VLP,4 D,UU`9D0033,16/12/92,18:38:41.94, 512, 512,129, 49, 19,126,125,F,CAL,VLP,0 D,ŪUU`9D0034,16/12/92,18:38:57.55, 512, 512,111, 58, 19,126,125,F,CAL,VLP,1 D,ѪUU`9D0035,16/12/92,18:39:13.56, 512, 512, 93, 67, 19,126,125,F,CAL,VLP,2 D,ݪUU`9D0036,16/12/92,18:39:29.40, 512, 512, 75, 76, 19,126,125,F,CAL,VLP,3 D,骔UU`9D0037,16/12/92,18:39:45.01, 512, 512, 57, 85, 19,126,125,F,CAL,VLP,4 D,UU`9D0038,16/12/92,18:40:49.11, 512, 512,129, 49, 55, 17,125,F,IMG,VLP,0 D,UU`9D0039,16/12/92,18:41:04.71, 512, 512,111, 58, 55, 17,125,F,IMG,VLP,1 F, UU`9D0040,16/12/92,18:41:20.39, 512, 512, 93, 67, 55, 17,125,F,IMG,VLP,2 F,UU`9 USKI`Ee5,FF 1HixFfE HixDH 73 5h<7P,fz2fits.c UU`>912/* 8:3+UU`?9 5+ * fz2fits.c R S Bogart 92.12.04 `7UU`@92, * :41CUU`A9 4H * Convert a LPARL "fz" format file to FITS by translating the header. 5,OUU`B9 1E * The raw data is assumed FITS compatible (big-endian, IEEE or 2's ,[UU`C9 1 * complement). [see Bugs] gUU`D9,1 * 92,sUU`E9516 * CCD Image File Format is defined in OpTest Notes. UU`F9:3 * 01,UU`G9 1 * Modifications: D,UU`H9J * Write important keyword values to logfile& R S Bogart 93.01.06 UU`I900 * convert TIME-OBS to UT ,11UU`J9F, * LP,UU`K9 * Bugs: ǪUU`L99,E * GDS structure can only handle up to rank 3 data, while fz data ӪUU`M9 *can be up to rank 16. ߪUU`N99 * Neither GDS nor FITS supports 12-bit packed format 모UU`O9( * FITS Keyword COMMENT doesn't work UUUU`P9E * Assumes that extra pixels and line count are of same data type UU`Q941@ *as data pixels; will not strip if data type is 12-bit packed bUU`R9heA * 3-character MDI configuration code plus .fits extension is aUU`S9 I- *unconditionally added to output file name e'UU`T99 * Assumes that Image File was written little-endian. 3UU`U9inF * Doesn't quite match on little-endian machine because of swapped ?UU`V9D,$ *NULL character at end of COMMENT rtKUU`W9 t) * ~ doesn't expand on directory name WUU`X9 : * Assumes that time is Pacific Standard Time (UT - 8) cUU`Y9 * */ : oUU`Z99, *{UU`[9ca#include ankUU`\9da#include 9UU`]9ra#include UUU`^9it#include UU`_9d  mUU``9O#define CAL0INCALIB(19) MêUU`a9UU#define CAL1INCALIB(126) ϪUU`b9 p#define PAWVERTICAL(125) ۪UU`c9#define PAWHORIZONTAL(89) pix窓UU`d9p #define PAWLCP(17) pUU`e9UU#define PAWRCP(53) cUU`f9io#define MTM0CONT(129) on  UU`g9S#define MTM1CONT(49) UU`h9il#define MTM0M120(111) 9#UU`i9at#define MTM1M120(58) /UU`k9#define MTM0M40(93) ;UU`l9#define MTM1M40(67) GUU`m9d #define MTM0P120(75) SUU`n9 e#define MTM1P120(76) _UU`o9 #define MTM0P40(57) kUU`p9#define MTM1P40(85) SKI`GeTiHHUU6ixHfG 9,6ixFJh>4UU`q9#i/* e <UU`r9H * Magic number is 0xaaaa5555 interpreted according to whether machine UU`s9 m * is big- or little-endian. IN*UU`t9 */ a6UU`u9IN#define MAGIC_EL(1431677610) BUU`v9CA#define MAGIC_EB(-1431677611) finNUU`w99) xZUU`x9p main (int argc, char **argv) fUU`y9in{ rUU`z9FILE *in, *logfile; MT~UU`{9 GDS *gds; UU`|9CO float sec; UU`}9in int i, j, k; )UU`~9i int *length; TUU`9Aint magic, datum_size, rank, data_count, read_count, data_start; TUU`93int img_line_length, day, year, seqnum, hour, min; 9 eƪUU`9(int strip, same_end; ҪUU`9(Jshort overscan_pixels, overscan_rows, bls_pixels, engineering_line_lngth; ުUU`9Hshort camera_hdr_size, stemp; ꪗUU`9short *bls, *overscan; FJUU`9Fchar line[256], keyname[8], c, header_block_ct, data_type, videodisk; UU`9edNchar date[12], time[12], site[21], instrument[21], pointing[36], exp_time[8], UU`9UUEsum_mode[4], frame[11], frame_type[5], pc_command[28], comment[81], UU`9#dseq_name[17], month[4]; )&UU`9wNstatic char *month_name[] = {"JAN", "FEB", "MAR", "APR", "MAY", "JUN", "JUL", 2UU`9z)"AUG", "SEP", "OCT", "NOV", "DEC", ""}; {>UU`9Jstatic int molen[] = {0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}; JUU`9*l*char *fname, in_name[256], out_name[256]; VUU`9a_char *data, *img_line; tarbUU`9#union {int i; short s[2];} swabuf; senUU`9 e zUU`9in/* ip,UU`9$ * Process command line arguments: veUU`9xe8 *fz2fits in=name [indir=name out=name outdir=name -s] meUU`9;  * defaults: UU`9*o *indir = outdir = . UU`9e[ *out constructed from in locªUU`9id) *strip extra pixels unless -s flag set ,ΪUU`9], */ umڪUU`936if (argc < 2) 檂UU`9UU{ UU`911 fprintf (stderr, comUU`981O "usage: fz2fits in=name [indir=name out=name outdir=name log=stdout -s]\n"); cha UU`9{" exit (1); MA~UU`9"J} "}UU`9UUsetargs (argc, argv); .|UU`9C",strcpy (in_name, nameparam ("indir", ".")); []:{UU`9 3.strcpy (out_name, nameparam ("outdir", ".")); FzUU`9amstrcat (in_name, "/"); [25RyUU`9strcat (out_name, "/"); e;^xUU`9fname = nameparam ("in", 0); }jwUU`9 if (!fname) SKI`IeJJesHixJfI HixHLme4amUU`9me{ UU`9 * fprintf (stderr, UUUU`9 =O "usage: fz2fits in=name [indir=name out=name outdir=name log=stdout -s]\n"); 9id*UU`9ix exit (1); fl6UU`9UU} BUU`9strcat (in_name, fname); NUU`9UU-strcat (out_name, nameparam ("out", fname)); mZUU`981fname = nameparam ("log", 0); fUU`9ut if (fname) stdrUU`9 logfile = fopen (fname, "a"); ~~UU`9} else UU`9rg logfile = stdout; UU`9py)strip = (is_flag ("-s")) ? FALSE : TRUE; {UU`9st yUU`9rain = fopen (in_name, "r"); UUUU`9in/* , "ƪUU`9UUF * Check that file type is correct and use magic number to establish ҪUU`9wH * whether system reading data header is little_endian or big_endian ުUU`9 */ ꪗUU`9magic = getw (in); UU`9if (magic == MAGIC_EL) `UU`9UU same_end = 1; f UU`9else if (magic == MAGIC_EB) s UU`9e  same_end = 0; me&UU`9);else 2UU`9 { >UU`9UU> fprintf (stderr, "fz2fits error: wrong input file type\n"); JUU`9st# fprintf (stderr, "%x\n", magic); me)VUU`9 exit (1); nabUU`9; } nUU`9if nzUU`9UUgds = gds_getnew (); oUU`9~ UU`9elfor (i=0; i<2; i++) rgUU`9ut getc (in); UUU`9(iheader_block_ct = getc (in); {UU`9stdata_type = getc (in); inªUU`9 "'datum_size = (data_type == 1) ? 2 : 4; ΪUU`9Chswitch (data_type) coڪUU`9c { 檂UU`9 case (1): UU`9r  gds->type = gds_SHORT; iUU`9 b break;  UU`9 case (2): ~UU`9ma gds->type = gds_LONG; "}UU`9ma break; C.|UU`9UU case (3): :{UU`9 gds->type = gds_FLOAT; mFzUU`9s break; URyUU`9nd default: ^xUU`9el gds->type = gds_ANY; { jwUU`9UU strip = FALSE; ,SKI`KetyLL6ixLfK ag6ixJPna4UU`9} UU`9/* UUUU`9s_H * Get main values from header and write into keyvalues as appropriate ++*UU`9 */ 6UU`9gds->rank = rank = getw (in); BUU`9if (!same_end) a_tNUU`9in{ ZUU`9da swab (&rank, &swabuf, 4); 2 fUU`9UU stemp = swabuf.s[0]; pe)rUU`9 swabuf.s[0] = swabuf.s[1]; a~UU`9UU swabuf.s[1] = stemp; ds_UU`9UU gds->rank = rank = swabuf.i; `UU`9~} UU`9s-if (rank > 3) UU`9{ UU`9I fprintf (stderr, "fz2fits unable to convert data of rank %d\n", rank); zƪUU`9 exit (1); ҪUU`9ul} ުUU`9el/length = (int *) malloc (rank * sizeof (int));  ꪗUU`9for (i=0; i<3; i++) UU`9L getc (in); UU`9videodisk = getc (in); xUU`9for (i=0; i<8; i++) UU`9 getc (in); &UU`9UU1fread (&overscan_pixels, sizeof (short), 1, in); 2UU`9ap/fread (&overscan_rows, sizeof (short), 1, in); UU>UU`9k ,fread (&bls_pixels, sizeof (short), 1, in); !sJUU`98fread (&engineering_line_lngth, sizeof (short), 1, in); ufVUU`9UU1fread (&camera_hdr_size, sizeof (short), 1, in); bUU`9] if (!same_end) ~nUU`9 { zUU`9s_' swab (&overscan_pixels, &swabuf, 2); = sUU`9! overscan_pixels = swabuf.s[0]; -UU`9% swab (&overscan_rows, &swabuf, 2); UU`9r, overscan_rows = swabuf.s[0]; of UU`9 z" swab (&bls_pixels, &swabuf, 2); UU`9ul bls_pixels = swabuf.s[0]; thªUU`9(r. swab (&engineering_line_lngth, &swabuf, 2); ΪUU`9( engineering_line_lngth = swabuf.s[0]; ڪUU`9od' swab (&camera_hdr_size, &swabuf, 2); fo檂UU`9! camera_hdr_size = swabuf.s[0]; UU`9UU} UU`9elfseek (in, 192, 0); in UU`9data_count = (rank) ? 1 : 0; z~UU`9);for (i=0; idims[i] = length[i] = getw (in); li:{UU`9sh if (!same_end) FzUU` 9fr { caRyUU` 9UU# swab (&length[i], &swabuf, 4); 9] ^xUU` 9 stemp = swabuf.s[0]; jwUU` 9  swabuf.s[0] = swabuf.s[1]; 2);MgOanr#ouf'plus[num[1.5,"1.5"],pm[num[0.7,"0.7"]]]ersSKI`NeUUPPowP! OfEMMelHixPfN ulHixLRUU4(rUU` 9ri swabuf.s[1] = stemp; UU`9) gds->dims[i] = length[i] = swabuf.i; UU`9od } ab*UU`9,  data_count *= length[i]; `6UU`9_s} BUU`9if (data_type == 5) NUU`9fs data_count *= 0.75; ZUU`9da if (strip) ankfUU`9{ rUU`9; 4 gds->dims[0] -= overscan_pixels + bls_pixels + 1; ~UU`9i]! gds->dims[1] -= overscan_rows; UU`9 (- gds_info_newval (gds, "BLS_PIX", "0", ""); UU`9. gds_info_newval (gds, "EXT_COLS", "0", ""); UU`9sw. gds_info_newval (gds, "EXT_ROWS", "0", ""); UU`9);} UU`9else ƪUU`9uf{ ҪUU`9pm$ sprintf (line, "%d", bls_pixels); eުUU`9. gds_info_newval (gds, "BLS_PIX", line, ""); ꪗUU` 9M) sprintf (line, "%d", overscan_pixels); UU`!9/ gds_info_newval (gds, "EXT_COLS", line, ""); ` UU`"9[1' sprintf (line, "%d", overscan_rows); ->dUU`#9 =/ gds_info_newval (gds, "EXT_ROWS", line, ""); UUUU`$9ou} &UU`%9/* UU2UU`&92 * Write string info as necessary to FITS header >UU`'9 *H * The strings in the FZ file appear to be terminated by cr and are sp JUU`(9s- * filled. VUU`)9pi */ 1bUU`*9fseek (in, 256, 0); = nUU`+9fread (date, 1, 12, in); ozUU`,9_Pfread (time, 1, 12, in); UUU`-9fofread (site, 1, 21, in); 0UU`.9UUfread (instrument, 1, 21, in); "EUU /9;  /* Not used UU@/9 */ UU`09fread (pointing, 1, 36, in); nªUU`19s)fread (exp_time, 1, 8, in); s_ΪUU`29"Bfread (sum_mode, 1, 4, in); UUڪUU`39f fread (frame, 1, 11, in); 檂UU`49!fread (frame_type, 1, 5, in); UU`59);fread (pc_command, 1, 28, in); tf UU`69cafread (comment, 1, 81, in); = UU`79l fread (seq_name, 1, 17, in); U~UU`89ou: /* Replace final cr with string terminator */ "}UU`99 nGdate[11] = time[11] = site[20] = instrument[20] = pointing[35] = '\0'; ile.|UU`:9inOexp_time[7] = sum_mode[3] = frame[10] = frame_type[4] = pc_command[27] = '\0';  *:{UU`;9*#comment[80] = seq_name[16] = '\0'; UUFzUU`<9at: /* Convert to UT and rewrite date in FITS format */ RyUU`=92sscanf (date, "%2d-%3s-%4d", &day, month, &year); ^xUU`>9t,year = year % 100; jwUU`?9!molen[2] = (year % 4) ? 28 : 29; USKI`QeRR366ixRfQ d 6ixPW"B4d UU`@9n)0sscanf (time, "%2d:%2d:%f", &hour, &min, &sec); ; UU`A9!j = 0; fraUU`B9; (while (strcmp (month_name[j++], month)) , *UU`C9 ; 66UU`D9nt hour += 8; =BUU`E9l if (hour > 23) , 1NUU`F9{ ZUU`G9  hour -= 24; fUU`H9er day++; rUU`I9 n if (day > molen[j]) ~UU`J9en { = UU`K9'; day = 1; UUU`L9[7 j++; eUU`M9fr if (j > 12) omUU`N9 * { UU`O9en j = 1; eƪUU`P9z year++; ҪUU`Q9to if (year > 99) ުUU`R9y year = 0; ꪗUU`S93s } UU`T9;  } UU`U9 =} UU`V90sprintf (date, "%02d/%02d/%02d", day, j, year); UU`W94sprintf (time, "%02d:%02d:%05.2f", hour, min, sec); f&UU`X9date[8] = '\0'; 2UU`Y9"Btime[11] = '\0'; U>UU`Z9ti-gds_info_newstr (gds, "DATE-OBS", date, ""); JUU`[90;H/* Set other keywords, stripping trailing space where appropriate */ VUU`\96-gds_info_newstr (gds, "TIME-OBS", time, ""); UbUU`]9 >j = 19; nUU`^9{ while (site[j] == 0x20) urzUU`_9UU site[j--] = '\0'; UU``9 )gds_info_newstr (gds, "SITE", site, ""); UU`a9Kj = 19; daUU`b9UUwhile (instrument[j] == 0x20) UU`c9j  instrument[j--] = '\0'; UU`d9UU3gds_info_newstr (gds, "INSTRUME", instrument, ""); earªUU`e9Qj = 34; ΪUU`f9while (pointing[j] == 0x20) =ڪUU`g9S pointing[j--] = '\0'; T檂UU`h91gds_info_newval (gds, "MKSHNEXP", exp_time, ""); ,UU`i9 d0gds_info_newstr (gds, "SUMMING", sum_mode, ""); e,UU`j9",j = 9; n,  UU`k9UUwhile (frame[j] == 0x20) ~UU`l9"B frame[j--] = '\0'; "}UU`m9gd+gds_info_newstr (gds, "FRAME", frame, ""); J.|UU`n9/0gds_info_newstr (gds, "FTYPE", frame_type, ""); er:{UU`o9j = 79; UUFzUU`p9_nwhile (comment[j] == 0x20) ""RyUU`q9] comment[j--] = '\0'; UU^xUU`r9it/gds_info_newstr (gds, "COMMENT", comment, ""); = jwUU`s9UUj = 15; gdSKI`Se""WWKSKI`TewhUV0x1cnQ\(UfVT me1cnQ\(Vin st  jN, 'Lessons from the Analysis of Early MDI j $ @jNUUOptics Package Test Data ]L`70 R. S. Bogart jUU`8M0'SOI Technical Note TN-93-101 nUU 9.KSSFrom Nov 10 through Dec. 17, 1992 data from approximately fifty series of tests of de,UU9.UURthe MDI Optics Package were taken and made available for analysis at Stanford. In UU9. =Zsupport of desired analysis suites a number of 5ad-hoc. procedures and definitions, UU9.stPtables, and naming schemes were developed for management of the data. This note pUU9.ntPdocuments those procedures both for continued use and development as additional itȪUU9.(gVtest data are received and analyzed and as a basis for understanding requirements and ժUU@9.Fdeveloping specifications for the analysis pipeline and data servers. HhnyhVfUT HhnyhU@`+7j Data Capture rUU :. E[The Optics Package (OP) test data were initially written from the MDI camera to disk files t +UU:.0'Pand to 8mm data tapes using a VAX VMS system. The primary archive tapes were in 17UU:.oxRVMS Backup format. Individual files represent CCD camera readouts from individual CUU:.vabexposures (Filtergrams). The data files are written in a special format, the 5f0. format, OUU:. Wcontaining both header information and the actual data. The data themselves consist of ed [UU:.thW1024*1024 (or fewer when camera block summing is used) shorts (2-byte signed integers) se gUU@:. adims[0], gds->dims[1]); UU`9/ UU`9scif (videodisk) braUU`9 t. gds_info_newval (gds, "VIDEODSK", "T", ""); ƪUU`9else ҪUU`9. gds_info_newval (gds, "VIDEODSK", "F", ""); ުUU`9:/* Get appropriate keyvalues from PC Command line */ ꪗUU`9 { UU`9to+int exp_time, mtm0, mtm1, cal0, cal1, paw; SEQUU`9""char exp_type; `wUU`9 G2sscanf (pc_command, "S%5dM%3dm%3dC%3dc%3dP%3d%c", UU`9st:&exp_time, &mtm0, &mtm1, &cal0, &cal1, &paw, &exp_type); &UU`9zsprintf (line, "%d", mtm0); {2UU`9na,gds_info_newval (gds, "MKM1POS", line, ""); sp>UU`9 ssprintf (line, "%d", mtm1); gdJUU`9, ,gds_info_newval (gds, "MKM2POS", line, ""); VUU`9rdsprintf (line, "%d", cal0); e bUU`9,gds_info_newval (gds, "MKC1POS", line, ""); asnUU`9, sprintf (line, "%d", cal1); mszUU`9; ,gds_info_newval (gds, "MKC2POS", line, ""); ifUU`9sprintf (line, "%d", paw); newUU`9K",gds_info_newval (gds, "MKPAPOS", line, ""); UU`9if (exp_type == 'F') OUU`96 gds_info_newstr (gds, "MKSHPATH", "FULL DISK", ""); UU`9/ else if (exp_type == 'H') ªUU`9in2 gds_info_newstr (gds, "MKSHPATH", "HIREZ", ""); ΪUU`9 eelse if (exp_type == 'D') ڪUU`9om1 gds_info_newstr (gds, "MKSHPATH", "DARK", ""); 檂UU`9&melse tUU`9&p2 gds_info_newstr (gds, "MKSHPATH", "UNDEF", ""); UU`9: /* Record configuration parameters to logfile */  UU`9-fprintf (logfile, ",%3d,%3d,%3d,%3d,%3d,%c", U~UU`9_n)mtm0, mtm1, cal0, cal1, paw, exp_type); "}UU`9ntH /* Modify output file name based on physical configuration */ OS.|UU`9if (exp_type == 'D') t:{UU`91){ FzUU`9;  strcat (out_name, "d"); RyUU`9if fprintf (logfile, ",XXX"); l^xUU`9ew} jwUU`9gdelse nSKI`XeYYif6ixYfX 6ixW["F4SKUU`9UU{ UU`9_t2 if (cal0 == CAL0INCALIB && cal1 == CAL1INCALIB) UU`9"H { "*UU`9 if (exp_type == 'H') '6UU`9 strcat (out_name, "C"); BUU`9"" else NUU`9 t strcat (out_name, "c"); ZUU`9HP fprintf (logfile, ",CAL"); `fUU`9ec } nfrUU`9rs else le ~UU`9 { fpUU`9%3 if (exp_type == 'H') ~UU`9m strcat (out_name, "I"); UU`9UU else UU`9ut strcat (out_name, "i"); UU`9OS fprintf (logfile, ",IMG"); e =ƪUU`9UU } 1)ҪUU`9} ުUU`9me dꪗUU`9if (paw == PAWVERTICAL) ",UU`9UU{ UU`9UU strcat (out_name, "s"); UU`9 fprintf (logfile, ",VLP"); UU`9} &UU`9else if (paw == PAWHORIZONTAL) 2UU`9{ >UU`9 strcat (out_name, "p"); JUU`91I fprintf (logfile, ",HLP"); VUU`9} bUU`9 =else if (paw == PAWLCP) nUU`9t_{ zUU`9UU strcat (out_name, "l"); UU`9  fprintf (logfile, ",LCP"); UU`9fp} UU`9);else if (paw == PAWRCP) UU`9{ UU`9 strcat (out_name, "r"); ªUU`9if fprintf (logfile, ",RCP"); UΪUU`9rc} ڪUU`9else U檂UU`9 { UU`9  strcat (out_name, "X"); UU`9OS fprintf (logfile, ",XXP"); ; UU`9} ~UU`9UU "}UU`9UU*if (mtm0 == MTM0CONT && mtm1 == MTM1CONT) .|UU`9{ :{UU`9 strcat (out_name, "0"); FzUU`9 fprintf (logfile, ",0"); logRyUU`9} ^xUU`9/else if (mtm0 == MTM0M120 && mtm1 == MTM1M120) jwUU`9{ SKI`Zet_[[UUHix[fZ P"HixY]4elUU`9CP strcat (out_name, "1"); UU`9UU fprintf (logfile, ",1"); UU`9 } *UU`9P"-else if (mtm0 == MTM0M40 && mtm1 == MTM1M40) ;6UU`9PA{ BUU`9 strcat (out_name, "2"); NUU`9me fprintf (logfile, ",2");  ZUU`9",} fUU`9/else if (mtm0 == MTM0P120 && mtm1 == MTM1P120) `rUU`9UU{ ~UU`9t_ strcat (out_name, "3"); UU`9og fprintf (logfile, ",3"); `UU`9UU} UU`9UU-else if (mtm0 == MTM0P40 && mtm1 == MTM1P40) OUU`9{ UU`9 strcat (out_name, "4"); ƪUU`9 fprintf (logfile, ",4"); ");ҪUU`9} ުUU`9A/* This is a correction to Igor's code, which generates X */ UꪗUU`9else `UU`9t_{ UU`9UU strcat (out_name, "5"); UU`9 fprintf (logfile, ",X"); UU`9UU} &UU`9t_fprintf (logfile, "\n"); 2UU`9og e>UU`9strcat (out_name, ".fits"); JUU`90 } VUU`9MT3 /* Get Temperature values from header */  bUU`9"2fseek (in, 168, 0); menUU`9lefor (i=0; i<10; i++) UzUU`9{ UU`9(m" sprintf (keyname, "TEMP%d", i); UU`9) fread (&stemp, sizeof (short), 1, in); _UU`9 if (!same_end) iUU`9); { UU`9}  swab (&stemp, &swabuf, 2); m0 ªUU`9 = stemp = swabuf.s[0]; ΪUU`9UU } ڪUU`9me sprintf (line, "%d", stemp); pri檂UU`9);, gds_info_newval (gds, keyname, line, ""); UU`9co} UU`9e,/* h g UU`9F * Get offset of real data, skipping header block(s), camera header, ~UU`9"52 * and engineering line; neglect overscan rows "}UU` 9 */ } .|UU` 9t_>gds->data = data = (char *) malloc (data_count * datum_size); :{UU` 9ou%data_start = header_block_ct * 512 + FzUU` 9>(camera_hdr_size + engineering_line_lngth) * sizeof (short); RyUU` 9fsfseek (in, data_start, 0); UU^xUU`9; /* i+jwUU`9E * If extra pixels are to be stripped, must go through line by line SKI`\eeo]]6ix]f\ i6ix[_ UU`9wa */ );UU`9 if (strip) mp UU`9{ *UU`9! for (i=0; idims[1]; i++) n6UU`9ri { BUU`9Y2 fseek (in, (1 + bls_pixels) * datum_size, 1); NUU`90 fread (data, datum_size, gds->dims[0], in); ofZUU`9 s1 fseek (in, overscan_pixels * datum_size, 1); UfUU`9d ' data += gds->dims[0] * datum_size; "}rUU`9 * } |~UU`9gd} UU`9 *else cUU`9um, fread (data, datum_size, data_count, in); deUU`9 gds_put_fits (gds, out_name); UU`9in} UU`=9 ( rSKI`^en,__xHix_f^ UUHix]ari.muh\G b%f07 to FITS conversion script UU`_9#!/bin/csh -f +UU``9set DIR = /home/soi/data/$1 7UU`a9chmod 755 $DIR UUCUU`b9!set LOG = /home/soi/data/logs/$1 OUU`c9{ "@ FC = `/bin/ls $DIR/raw | wc -l` [UU`d9 nif ($FC <= 200) then gUU`e9! foreach F (`/bin/ls $DIR/raw`) ssUU`f9; 6 fz2fits indir=$DIR/raw outdir=$DIR in=$F log=$LOG UU`g9 if ($status == 0) then in,UU`h9 d rm $DIR/raw/$F UU`i9da else dUU`j9e; rm $DIR/$F*.fits  UU`k90 echo "unable to convert file $DIR/raw/$F" UU`l9a, endif ǪUU`m9de end ӪUU`n9puelse (ߪUU`o9 @ FC -- 모UU`p9UU @ MMX = $FC / 1000 UU`q9 @ M = 0 UU`r9 while ($M <= $MMX) UU`s9$ foreach C (0 1 2 3 4 5 6 7 8 9) muUU`t9 b. foreach FN (`/bin/ls $DIR/raw/*.$M$C*`) 'UU`u9in set F = `basename $FN` DI3UU`v9/$: fz2fits indir=$DIR/raw outdir=$DIR in=$F log=$LOG ?UU`w9om if ($status == 0) then `cKUU`x9/l rm $DIR/raw/$F WUU`y9$F else cUU`z9 rm $DIR/$F*.fits rawoUU`{9f4 echo "unable to convert file $DIR/raw/$F" =${UU`|9g endif UU`}9n, end UU`~9 $ end UU`9da @ M ++ UU`9  end UU`9endif êUU`9ho"@ FC = `/bin/ls $DIR/raw | wc -l` ϪUU`9lif ($FC == 0) rmdir $DIR/raw m۪UU`9chmod 555 $DIR e (窓UU`9set MSG = /tmp/msg.$$ UU`9= -echo Message from data capture script: >$MSG UU`9 5echo " Series $1 created on /home/soi/data" >> $MSG  UU`9 94echo " FITS conversion by `whoami` `date`" >> $MSG DIUU`9>echo " see relevant log file on /home/soi/data/logs" >> $MSG #UU`9fz8mail -s "new OP test data set" rick,jayasree,kay < $MSG UU/UU`9ifrm $MSG ==SKI``e aa 6ixaf` 6ix_c@3awxCf5Script for moving data between disk file systems $UU`9g#!/bin/csh -f +UU`9n,H# Script to move a data series from one scratch file system to another ++7UU`9if ($#argv < 2) then UCUU`9F echo "usage: mvser XX [I] J to move series XX from /scrI to /scrJ" OUU`9di exit w m[UU`9endif gUU`9if ($#argv == 2) then sUU`9$ set CUR = `egrep ^$1 catalog`[10] apUU`9  set ORIG = /scr$CUR[10] UU`9d  set DEST = /scr$2 $MUU`9else cUU`9io set ORIG = /scr$2 >>UU`9UU set DEST = /scr$3 reUU`9 /endif ǪUU`9G if ($ORIG == $DEST) exit -ӪUU`9a set OD = $ORIG/mdi_data/$1 UUߪUU`9ifset ND = $DEST/mdi_data/$1 `e모UU`9set MSG = /tmp/msg.$$ UU`9set MD = `cd ~soi/data; pwd` UU`9chmod 755 $OD UU`95# Check if the current link will have to be removed UU`9#!if ($OD == $MD/$1) then 'UU`9 m set UNLINK = 0 r3UU`9e else if (-e $MD/$1) then ?UU`9$# set UNLINK = 1 KUU`9 else sWUU`9J  set UNLINK = 0 fcUU`9" endif oUU`9itif ($UNLINK) rm $MD/$1 en{UU`9# Create the new directory UU`9 if (!(-e $ND)) then atUU`9 mkdir $ND UU`9R[ if ($status) then d UU`9r$* echo "Unable to create directory $ND" UU`9IG" if ($UNLINK) ln -s $OD $MD/$1 êUU`9re exit UϪUU`9 endif ۪UU`9 $endif 窓UU`9# UU`9di>echo "`date` begin $1 $ORIG -> $DEST" >>~rick/soi/optst/notes UU`9*echo Message from data move script: >$MSG  UU`9i/# UU`9c@ FC = `/bin/ls $OD | wc -l` #UU`9heif ($FC < 400) then w/UU`9ov% mv $OD/*.[0-9][0-9][0-9][0-9]* $ND D;UU`9else GUU`9 = @ M = $FC / 1000 9e SUU`9$1 while ($M >= 0) _UU`9LI$ foreach C (0 1 2 3 4 5 6 7 8 9) kUU`9  set D = $C'[0-9][0-9]' SKI`beifccenHixcfb e Hixae@!())UUp9UU mv $OD/*.$M$D* $ND UUp9 3# mv $OD/*.$M$C{0,1,2,3,4,5,6,7,8,9}[0-9]* $ND nabUUp9to end *UUp9 @ M -- ) l6UUp9 end BUUp9 Uendif NUUp9dichmod 555 $ND ZUUp9# Remove the old directory UUfUUp9atchmod 755 $OD rUUp+9k/ rm -r $OD ~UUp,9 # Make a new link if necessary scUUp-9$if ($ND != $MD/$1) ln -s $ND $MD/$1 @ UUp.9| >echo "`date` end $1 $ORIG -> $DEST" >>~rick/soi/optst/notes UUp/9/*Fecho " Series $1 moved from disc file system $ORIG to $DEST" >> $MSG UUp09/ *echo " moved by `whoami` `date`" >> $MSG UUp192mail -s "SOI data moved" rick,jayasree,kay < $MSG ƪUUp29 rm $MSG '[ҪUU`9 SKI`decee6ixefd 6ixcg@3UUx^C $Archive restore script pUU`9/*#!/bin/csh -f +UU`9NDN# Script to restore a data series from archive tape to a scratch file system 7UU`9d if ($#argv < 1) then nCUU`9@ echo "usage: restoreser XX [I] to restore series XX to /scrI" orOUU`99 echo "default is system on which ~soi/data is mounted" [UU`9ak exit inkgUU`9endif sUU`9!=set SD = ~soi/data/$1 UU`9. set RD = `cd ~soi/data; pwd`/$1 ->UU`9i/if ($#argv < 2) then UUU`9Se set DIR = $RD dUU`9RI set RELINK = 0 UU`9/ else "UU`9`  set DIR = /scr$2/mdi_data/$1 9ǪUU`9a  set RELINK = 1 eӪUU`9endif ߪUU`9if (-e $SD) then 모UU` 9eG echo -n "Dataset is online and will be overwritten - proceed ? [n] " UU` 9 set ANS = $< UU` 9 if ($ANS !~ [yY]*) exit UU` 9endif UU` 9sh1set AT = `egrep ^$1 ~soi/data/catalog | head -1` t'UU`9ivset TAP = $AT[11] 3UU`9set FIL = $AT[12] ?UU`9if ($TAP !~ [0-9][0-9]) then :KUU`9 t( echo "No archive tape listed for $1" UUWUU`9de exit syscUU`9daendif oUU`9UUif ($FIL !~ [0-9][0-9]) then U{UU`94 echo "No archive tape file number listed for $1" UU`9 ~ exit pwUU`9UUendif UU`9th:echo -n "Please mount tape # $TAP - type RET when ready " UU`9 = set ANS = $< UUU`9if (!(-e $DIR)) then êUU`9at mkdir $DIR ϪUU`9t  if ($status) then UU۪UU`9+ echo "Unable to create directory $DIR" UU窓UU`9n exit UU`9 o endif -UU` 9endif  UU`!9t :# Remove the existing directory and the link if necessary UU`"9if (-e $SD) then U#UU`#9 ` set CD = `cd $SD; pwd` /UU`$9 chmod 755 $CD TA;UU`%9 rm -r $CD seGUU`&9 if (-e $SD) rm $SD TSUU`'9th mkdir $DIR U_UU`(9"Nendif kUU`)9$1if ($RELINK) ln -s $DIR $SD itSKI`fegg$FHixgff UUHixeiteUU`*9 cd $DIR/..  UU`+9UUmt -f /dev/rmt2h rewind UUUU`,9"P @ FN = $FIL pe*UU`-9 w@ FN-- " 6UU`.9 =#if ($FN) mt -f /dev/nrmt2h fsf $FN ifBUU`/9 tar xf /dev/rmt2h NUU`09mt -f /dev/rmt2h rewoffl aZUU`19 chmod 555 . fUU`29crset MSG = /tmp/msg.$$ rUU`39n -echo Message from data restore script: >$MSG ~UU`49en9echo " Series $1 restored from archive to $DIR" >> $MSG nUU`59ss-echo " restored by `whoami` `date`" >> $MSG UU`69 5mail -s "restored data set" rick,jayasree,kay < $MSG 5UU`*9UUrm $MSG SKI`he ii6ixifh 6ixgk$13$R(b7$S6Script for generating a sorted series of data for @b7 the pipeline -UU`d9#!/bin/csh -f 9UU`e9c set ID = 071 UEUU`f9..!set IDIR = /home/soi/data/opt$ID vQUU`g9set ODIR = /scr8/pt9302/opt$ID *]UU`h9@ # Start & end with single darks $FiUU`i92h@# Sequence: 4 Dark, 6 Calib, 6 Image, 1 Mag, 1 HRMag, 1 HRImage 0uUU`j9mtE# count: 4 30 30 10 10 5 : 89 eUU`k9$ # Start seq at 9n UU`l9m #191184274364454544634 UU`m9 "G# extras 1 4 1 1 1 1 1 0 -ecUU`n9`w4set SKIP = "26 100 134 150 160 211 338 413 458 548" "rUU`o9ri@ DARKREP = 4 UU`p9UU@ CALIBREP = 6 ɪUU`q9e@ IMAGEREP = 6 ժUU`r9 @ MAGREP = 1 ᪪UU`s9@ HMAGREP = 1 UU`t9@ HIMAGEREP = 1 UU`u9@ ISN = 50000 UU`v9er@ CSN = 50000 UU`w9@ DSN = 50000 UU`x9@ MSN = 50000 )UU`y9c@ HSN = 50000 5UU`z9..@ HMSN = 50000 e/sAUU`{9 @ FR = 0 gMUU`|9sc2ln -s $IDIR/*$ID.0000*.fits $ODIR/Dark/N$DSN.fits YUU`}9rk@ DSN++ eUU`~9qu @ FR = 1 kqUU`9, set FN = 0001 }UU`9while ($FR < 723) UU`9 # Darks UU`9 @ DC = 0 UU`9#  while ($DC < $DARKREP) UUU`9845 ln -s $IDIR/*$ID.$FN*.fits $ODIR/Dark/N$DSN.fits sUU`9 1# echo "$FN -> Dark / $DSN" ŪUU`9UU @ DSN++ SKѪUU`950 @ DC++ 413ݪUU`9 @ FR++ @ 骔UU`9 foreach SKIPFR ($SKIP) UU`9e! if ($FR == $SKIPFR) @ FR++ rUU`9  end UU UU`9P  if ($FR < 10) then @ UU`9 set FN = 000$FR %UU`9UU else if ($FR < 100) then 1UU`9N  set FN = 00$FR x=UU`90  else if ($FR < 1000) then IUU`9z set FN = 0$FR /sUUU`9 else gaUU`9sc set FN = $FR 00*mUU`9$D endif SKI`jekk kHixkfj 1 Hixim4UUUU`9  end UU`9DC # Calibs UU`9 @ CC = 0 $DA*UU`9UU set OD = $ODIR/Calib /*$6UU`9/D while ($CC < $CALIBREP) BUU`9o  set SN = $CSN NUU`9 @ P = 0 N+ZUU`9 while ($P < 5) ݪfUU`9 8 ln -s $IDIR/*$ID.$FN*$P.fits $OD/Fgrm$P/N$SN.fits P)rUU`9 @ FR++ (~UU`9FR foreach SKIPFR ($SKIP) nUU`9# if ($FR == $SKIPFR) @ FR++ UU`9 end UU`9 if ($FR < 10) then 0UU`9UU set FN = 000$FR 0$UU`9 else if ($FR < 100) then en ƪUU`9z set FN = 00$FR UҪUU`9 else if ($FR < 1000) then ުUU`9 set FN = 0$FR ꪗUU`9` else UU`9 set FN = $FR UU`91 endif UU`9 @ P++ UUUU`9  end &UU`9# @ CSN++ 2UU`9CC @ CC++ >UU`9t  end JUU`9 # Images DVUU`9CA @ IC = 0 bUU`9se set OD = $ODIR/Image `nUU`9N+ while ($IC < $IMAGEREP) zUU`9 set SN = $ISN UU`9ID @ P = 0 ODUU`9P) while ($P < 5) UU`98 ln -s $IDIR/*$ID.$FN*$P.fits $OD/Fgrm$P/N$SN.fits UU`9$F @ FR++ UU`9UU foreach SKIPFR ($SKIP) UªUU`9 (# if ($FR == $SKIPFR) @ FR++  ΪUU`9$F end ڪUU`9  if ($FR < 10) then 檂UU`9z set FN = 000$FR UU`9  else if ($FR < 100) then UU`9 set FN = 00$FR ꪗ UU`9 else if ($FR < 1000) then ~UU`9  set FN = 0$FR "}UU`9 else .|UU`9 set FN = $FR :{UU`9# endif FzUU`9CC @ P++ RyUU`9t  end ^xUU`9# @ ISN++ jwUU`9IC @ IC++ SKI`legemmN+6ixmfl 6ixkoUU4IDUU`9 end UU`9e # Magnetograms UU`9 @ MC = 0 $ID*UU`9gr set OD0 = $ODIR/RCP 6UU`9  set OD1 = $ODIR/LCP BUU`9h  while ($MC < $MAGREP) UUNUU`9if set SN = $MSN ZUU`9UU @ P = 0 fUU`9 while ($P < 5) 10rUU`9UU9 ln -s $IDIR/*$ID.$FN*$P.fits $OD0/Fgrm$P/N$SN.fits ~UU`9 1 @ FR++ UU`9  foreach SKIPFR ($SKIP) UU`9 # if ($FR == $SKIPFR) @ FR++ UUUU`9se end UU`9 if ($FR < 10) then UUU`9se set FN = 000$FR ƪUU`9 else if ($FR < 100) then ҪUU`9t  set FN = 00$FR `ުUU`9 else if ($FR < 1000) then ꪗUU`9e set FN = 0$FR UU`9 else mfUU`9 set FN = $FR UU`9 endif UUUU`99 ln -s $IDIR/*$ID.$FN*$P.fits $OD1/Fgrm$P/N$SN.fits &UU`9 @ FR++ 2UU`9P  foreach SKIPFR ($SKIP) >UU`9# if ($FR == $SKIPFR) @ FR++ UUJUU`9if end VUU`9UU if ($FR < 10) then bUU`9  set FN = 000$FR nUU`9  else if ($FR < 100) then /FgzUU`9 set FN = 00$FR FRUU`9 else if ($FR < 1000) then UU`9  set FN = 0$FR UU`9 else 9seUU`9 set FN = $FR (UU`9 endif seªUU`9 0 @ P++ ΪUU`9  set ODX = $OD0 nڪUU`9 set OD0 = $OD1 0檂UU`9UU set OD1 = $ODX UU`9 end UUUU`9se @ MSN++  UU`9 @ MC++ ~UU`9  end "}UU`9 # Skip 15 .|UU`9 @ SC = 0 9:{UU`9R/ while ($SC < 15) /FgFzUU`9 @ FR++ 9RyUU`9 foreach SKIPFR ($SKIP) ch ^xUU`9! if ($FR == $SKIPFR) @ FR++ =jwUU`9UU end UUSKI`neoo10Hixofn UUHixmq seUU`9en @ SC++ UUUU`9se end UU`9UUend *UU`9FR2ln -s $IDIR/*$ID.0723*.fits $ODIR/Dark/N$DSN.fits SKI`pe qq6ixqfp 6ixo" h7Register script UU`9#!/bin/csh -f +UU`9 0if ($#argv < 1) then 7UU`9 H echo "usage: register proj (to register all grams under ~soi/pt/proj) CUU`9@  exit ~OUU` 9 endif [UU` 9# set PROJ = /home/soi/pt/$1 gUU` 9set DIRS = `/bin/ls $PROJ` 15)sUU` 9*set REG = /home/soi/src/idl/register_many UU` 9$S-set DARK = $PROJ/Dark/Series_Stats/mean.fits $UU`9wif (!(-e $DARK)) then UU`9` pushd $PROJ/Dark UU`910' @ CT = `/bin/ls N*.fits | wc -l` - 1 iUU`9+ /home/luiz/scripts/ppSSTAT FSN=0 LSN=$CT UU`9 popd 9seǪUU`9UUendif ӪUU`9UU%if (!(-d $PROJ/reg)) mkdir $PROJ/reg OߪUU`9s foreach D ($DIRS) 모UU`9 if (!(-d $PROJ/$D)) continue xUU`9 if ($D == Dark) continue oUU`9  if ($D == Calib) continue isUU`9 set OD = $PROJ/$D shUU`9 set ND = $PROJ/reg/$D 'UU`9 - if (!(-e $PROJ/reg/$D)) mkdir $PROJ/reg/$D 3UU`9t/ foreach N (1 2 3 4) ?UU`9 mkdir $PROJ/reg/$D/Fgrm$N KUU`9PR4 @ CT = `/bin/ls $OD/Fgrm$N/N*.fits | wc -l` - 1 =WUU`95)D $REG in=$OD/Fgrm$N/N.fits out=$ND/Fgrm$N/N.fits fsn=0 lsn=$CT \ cUU` 9DA cmin=300 cut=1 dark=$DARK fioUU`!9 end {UU`"9n end UU`9sh PUU`c9 UUU`9 ` n$SKI`LeftSKI`/sRightpSTSKI`FirstUSKI` ReferenceUenSKI`UUTOC(SKI`2PRIX OSKI`TfoFirst$DISKI`SKI`5ntSKI`7SKI`9ueSKI`; SKI`=isSKI`?t SKI`AUUSKI`C /rSKI`E SKI`G ) SKI`I SKI`K (1SKI`NSKI`QgrSKI`S SKI`X$NSKI`ZSKI`\ iSKI`^D/SKI``T SKI`b SKI`dARSKI`f SKI`henSKI`j PSKI`lSKI`n$SKI`p)=fffSE` I`ff U1Step Step Number S:.\tStep~ fHQ  u`~3HeadingH:\t..\t FirstBody=f EFootnote~~ fE EquationEquation Number E:(EQ )IllfDi I` FirstBody Bodyfffff CBulletf.@ fAbstract1HeadingfS tep CellHeading~;fHQ  u~>.1Heading 1Heading RuleH:\t.\t FirstBodyZ f@E   AuthorPurposell fq n+>Bodyfffff dy CStepofff CBExtract~;fQ  Abu~ng ReferenceHead 1Heading Rule\t\tReferencesAppendix fFA  Figure Table Top Step NumberF:FIGURE .\tBodyfHQ>  u~2HeadingH:\t.\t FirstBody~;fCAQ   u~ AppendixHead 1Heading RuleyA:\tAppendix \tAppendixf@  ffPurpose FirstBodyfffS ;ff fStep Step Number S:.\tf t\t TableFootnotexfT re a TableTitle Table Top Step Number.T:TABLE .\tf@< Title Chapter RuleAuthorf@ 0 aG  pead %ngulVQA:AppendixAppendixf8 CellBody~;f7HQ   uff~ f1Heading 1Heading RulenH:\t.\t FirstBodyf@ Abstract1Heading~;f7AQ  leTue p~ep AppendixHead 1Heading RuleA:\tAppendix \tAppendixf9@ hor0afG %VQAppendixAppendix~;fAQl  xu~ AppendixHead 1Heading RuleA:\tAppendix \tAppendixffuff ffBullet Bullet Symbol \tf6   header firstllf.D f FirstBodyBodyll f. Bodyll f. Body f8Q ~Level2IXffufB  ffadBullet Bullet Symbole\t +>ll f. Bodyff CellBody~~f D  References Referencesf @f  TNTagAbstractfN@< Title Chapter RuleAuthorZ f0@  AuthorTNTagf6   headerfQ    footer leftfQ    footer rightfQ   footer rightfAT  N TableTitle Table Top Step NumberT:TABLE .\tfF Au CellHeading fffff. TNff ff 3HeadingTOCfff-e  ff 2HeadingTOCcfS ~  1HeadingTOCcf.  FigureLOFcf. T TableTitleLOTf8L f SeparatorsIXf8n f SortOrderIX$f8 ffLevel3IXf 8  Level1IX f!6   GroupTitlesIX1Hf"8 IndexIXZ f%0@   AuthorTNTagf&M@  TNTagLAbstract-]E% -  .  /  Bullet Symbol 0  $1Callout= 2 Emphasisff3 Equation Number 4   Run-In Heading 5  6  7  8 tleX 9f :   ; Emphasis <  Step Number$=  First Letter= >  Subscript= ? &M Superscript @AppendixgL A  Step Number- B  Bullet Symbol C    D F  G =Emphasis J   Default Bold K   Default Bold L  Emphasis M <N  O   P  Q  R S  T  euhefg@l@mnoThinMediumDoubleThick@ Very ThinYiZlY6666Format B=$[6l666Format AM]M<CommentN Courier Helvetica Times -Regular Regular BoldRegularObliqueItalicT wZ5hG_(vcḍ)ֳB0ވsCQ~yeJ<4bUhsP# 0-;ԉmIi`W,wE?eZ'=a:';e}/2bQP,UWhOJ+l7;')fbn\IZI- 8dmHxbˌT"9Z.z{N{'Ġh##HM..22z&`Zh3kbWݠ ~BEGxOBA[Z`$oOT 9ֶWw{w"ȴAn܏p-R?K)cqQK8p#0˰o/} ӧ}qH$;Ҭi ָ=ByX}7