Original note by CNF Update to message from esc by CNF 15 Oct 96 Updated on 15 Sep 1998 by EBZ Updated on 28 Jul 1999 by EBZ Updated on 26 Apr 2004 by SEG Checking on a Loss of Telemetry ------------------------------- Below is a message put out by SOHO Science Operations Coordinators (SOCs) and the ECS Staff detailing the steps to take if if there is not telemetry during non working hours. The following is a summary and some further comments. 1. Check to see if there is a station handover at this time, this is often the cause of the problem. The most current SOHO schedule can be found on the mdisas home page : http://sohowww.nascom.nasa.gov/operations/schedule/schedule.html 2. Review the log to see if someone has already mentioned the problem. The log can be found on the health monitoring page or you can type xmonlog if on a computer that supports x-windows or opslog to dump the latest 7 days as ASCII text to your monitor. Alternatively, it is possible to "cd" to /md80/log/history and "more" the latest log file (They are in the form yymmdd.opslog). In addition, mon5klog will display the output of the sci5k processing with updates every few seconds if the telemetry is running. 3. Call the MDI office if anyone is in (301) 286-3233 or 9054. 4. If the problem is with mdisas check to see if the information is coming in on mditlm. 5. Follow the instructions in the SOC memo below to check if telemetry is coming in to the EOF computers. 6. If no telemetry is going to the EOF computers and its during the EOF hours (9:00 am to 10:00 pm) and no one is in the MDI office, call the ECS Operator at 301-286-8985. 7. We are allowed to call the FOT to ask for information. Call the FOT at 301-286-4799 and ask if there is a problem with the S/C. (Note that in the past they thought we were calling and asking why we were not getting data in California and were a bit annoyed, so make sure we make clear that we are concerned if the S/C is not putting down data) WE ARE NOT ALLOWED to discuss any type of operations directly with the FOT. For example, we can not suggest to the FOT that they hold off on a SSR dump in order to minimize the impact to the Dynamics program. This type of request must go through an SOC. If after hours and we think the request is important enough we are allowed to call a SOC at home and make the request. Their numbers are listed above. (I would recommend trying Laura first) SOCs Home Phone Beeper Number --------- ------------- -------------- Joan Hollis (410) 531-3360 (301) 507-9234 8. If the problem requires that someone come in to the EOF the suggestion from Rock is that we call the local person on call and have them come in. The person on call will be listed in the monitoring schedule. If necessary MDI people can also restart the EOF machines after talking with the EOF operators. The MDI team pager numbers are: Sarah 571-236-9929 Julie 301-303-8029 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From: ecs@soc.nascom.nasa.gov Date: Wed, 19 Jun 1996 19:05:14 +0000 (GMT) To: Generic MDI , Piet Martens , soc@soc.nascom.nasa.gov Subject: Checking Telemetry at Odd Hours TO: MDI team FROM: SOHO Science Operations Coordinators and the ECS Staff This memo provides a check list for a remote experimenter to follow when checking that telemetry is properly received during hours when the ECS office is not staffed (see SOHO schedule for coverage). 1) Check the SOHO Daily Operations Schedule to see if a DSN station handover is taking place. At these times you can expect to see some minor losses in telemetry. 2) One can remotely check that ECS is properly receiving telemetry from a "dumb" terminal without X-windows capability as follows: a) FTP to the soc machine, and login as mdi. b) Change directories to the MDI VC01 data, (see below for details) then check the date and modification times on the current "SDU" file -- this file is where the incoming telemetry is written into. This file will later be reformatted into the REL files you usually see. ftp> cd tlm_files/MDISC 250 CWD command successful. ftp> dir *SDU 200 PORT command successful. 150 Opening ASCII mode data connection for /bin/ls. -rw-rw-r-- 1 100 2875392 Jun 18 19:03 MDISC_960618_175022.SDU 226 Transfer complete. Now, wait a minute or so and type the same command again: ftp> dir *SDU 200 PORT command successful. 150 Opening ASCII mode data connection for /bin/ls. -rw-rw-r-- 1 100 2883584 Jun 18 19:04 MDISC_960618_175022.SDU 226 Transfer complete. As you can see in the above example, the file size increased and the modification time was updated -- so the ECS is getting telemetry for this particular APID. If you wanted to check if other APID's are received by the ECS, you could do the above with one of the other directories (instead of MDISC): AOCHK1 CDSSCL ELAHK1 GOLHK OBT SVMHK3 UVCSSC AOCHK2 CDSSCM ELAHK2 GOLSC SUMHK SVMHK4 VIRHK ATTIT1 CELHK ELAHK3 MDIHK1 SUMSCH SW VIRSC ATTIT2 CELSC1 ELASCH MDIHK2 SUMSCL SWAHK CDSHK CEPHK ELASCL MDIHR SVMHK1 SWASC CDSSCH CEPSC EXPHK MDISC SVMHK2 UVCHK Note that MDISC is the MDI 5K science data (VC1) and MDIHR is the MDI-M data (VC2). 2) From the above example, we checked if the ECS is receiving telemetry. If the ECS is receiving telemetry data and the MDI workstation is not, then it is a local MDI problem, contact a local MDI representative. However, if the ECS is not receiving telemetry FOR SEVERAL APIDS, then the problem may be with PACOR and do the following in order: a) Call PACOR (286-3080), PACOR will only be able to tell you if they are sending data to the ECS. PACOR is staffed 24 hrs a day. (However, PACOR will NOT be able to tell you if the ECS is receiving the data properly). Note: There are two streams of telemetry data from JPL to GSFC. One is sent to the FOT and the other is sent to PACOR. PACOR sends telemetry to the ECS, which in turn sends it to the experimenters. So therefore, it is entirely possible for the FOT to see telemetry data and not realize the instrument teams are not getting the data because of the different source. The FOT does not get its data from PACOR but the instrumenters do. b) If PACOR says they are sending data to the ECS and no other teams are receiving telemetry data, then call a SOC. (See last week's weekly meeting minutes for phone numbers). It has been suggested that a member of the FOT could check on the ECS system at odd hours. However, at the present time, the FOT has not been trained to do so. Therefore, call a SOC. We hope this clarifies matters for you. If you have any further questions, comments, let us know.