01:42 UT DPP crashes during Natsuha's long 3c84 scan for polarization calibration. DPP and correlators was rebooted at ~06 UT. Delays were reset at 11:13 UT except for ant 6 (delay update did not clean the phases for some reasons other than tracking issue).
Natsuha noticed that the Hittite LO is supposed to be sweeping but shows Wait4Trg. That must mean that the 1pps signal is missing. I then checked the GPS clock (NTP server) and it looks okay (but I cannot check the 1pps separately). However, when I insert attenuation into the FEMs, the crios acknowledge it but the power level does not change. What happens is that the crios request the change and the rabbit board waits for the next 1pps before actually switching the attenuators. Since that is not happening, I conclude that the 1pps is missing everywhere in the system. This could be the GPS clock (not likely), the SRS Rubidium clock, or the 1pps distribution system (timing module). If the 1pps was present at the Hittite but not at the antennas, I would suspect the Manlight unit, but since that is not the case I think we can assume it is okay.
06:46 UT 1 pps is fixed by Kjell. He found that the 1 pps was not being passed through the Rubidium clock, but was present out of the GPS clock, so he bypassed the Rubidium clock.
The yesterday's procedure may have somehow affected the synchronization of the digitizers in the ROACHes (not sure how this might happen). Anyway, I started a calibration early this morning, measured the delays, and reset them. Just now (about 13:36 UT) I restarted the calibrator scan. We will get about 35 minutes on the calibrator. So please use this scan for the refcal, not the scan before it. Note that Ant 6 is down, so its delay was not checked. Because of the continuing problems with Ant 6 FEM, Kjell will use that antenna for the first installation of the notch filter, so it will be down for quite awhile. Hopefully during that process we will be able to find out what is wrong with it.
Ant 12 front end temperature is still out of wack. I am not sure why.
11:55 UT Dr. Gary took full lohi.fsq scan for X vs. Y delay measurement and band 4 phase calibration, but all but the first one were compromised due to windscram.