02:10 UT After many days of the system running normally, we had a glitch today that I did not notice until this evening. The ROACH clock was messed up...so we took a lot of very short files since 16:03 UT on Aug. 09. Luckily, I was able to get the clock working again, and I should be able to bring the system up and reset delays, etc.
02:17 UT Well, we are on 3C273 and tracking, so it was a fast recovery. I will take data for 20 min or so and set the delays. 3C273 will set soon, but I should be able to check delays on 3C286, at least to see if they are rational. One thing I will need to check is whether the X-Y delays change as a result of rebooting the ROACHes. I will set up to observe 2253+161 and do an X-Y delay calibration.
02:58 UT Not bad! I got the delays reset from the 3C273 observation, and will take my X-Y delay measurements starting at 08:25 UT, both with low and high receivers. However, I have not set the low-receiver delays. Ant 14 may be off by 2 ns as a result (it was on the high receiver).
02:58 UT I updated the delays slightly for both lo and hi receivers (differed only by 2 ns on Ant 14, since none of the others should change). This was after the day's observations, so yesterday's observations (i.e. Aug. 10) can be calibrated with the refcals taken up to now, while today's (i.e Aug. 11) observations will require the new refcals from later this UT day.
10:30 UT The correlator was rebooted in attempt to fix the instabilities in the dynamic spectrum (see any plot from 08/28). Delays were updated (ant8 needed the second iteration). However, this did not solve the issue, so the ACC was rebooted after the first phasecal observation was over.
13:51 UT The reboot of the correlator did not affect the data glitches. However, an analysis from today demonstrates that the error is caused by a timing glitch in applying the DCM attenuations. Once every 10 s, the slot 49 attenuations are applied to slot 1, and the slot 2 attenuations are applied to slot 1. After this 40-ms glitch, the system goes back to normal. This analysis suggests that the data could be corrected, but since it is such a small fraction of the data, it is probably better to just flag it. The analysis also revealed that Ant 12 has fixed attenuations (16 dB), which no doubt is because the DCM attenuation settings were never remeasured after Ant 12 came back into use. Ant 12 very likely suffers from non-linearity.