11:27 UT Started another interferometric pointing run (CALPNT), which has been running since 08:16 UT without error. The wind is low, and Ant 14 is working well.
17:35 UT Ant 14 was found to have a Dec PERMIT error on the current scan started 17:27 UT. All scans prior to this are okay. I will stop this run at 19:19 UT, in order to do a SOLPNT calibration and update the 2-m pointing, then test the result with another SOLPNT observation.
19:23 UT The pointing calibration is complete on schedule. Only one scan was messed up by the Ant14 PERMIT error. A SOLPNT measurement will start at 19:30 UT.
18:52 UT I found several problems today. Ant 9 crio is not working--not sure why. Ants 10, 11 and 13 were not correctly pointed (found this based on SOLPNTCAL), probably because they had an incorrect STOW at some point. I stowed them and sent them back to the Sun just now. Will redo the SOLPNTCAL. Ant 14 control system was off, and when restored I had to restart the Beaglebone computer, reset the FRM, and I also found that Ant 14 HPol is low, with no LNA Drain current. This could not be corrected--may be a hardware failure.
19:08 UT Hmm. After restowing ants 10, 11 and 13, and redoing the SOLPNTCAL, they still show very bad pointing. I am not sure what the problem is.
12:26 UT I finally traced the problem of pointing on ants 11 and 13 (although not 10...) to the fact that the pointing coefficients in their crio.ini files were all set to zero (and other quantities were also reset). I copied the 2016-07-09 crio.ini files back to their /ni-rt/startup directories, but these are old files, so there will need to be some tweaks. Still, hopefully today's SOLPNTCAL will find good positions.
18:40 UT I have continued to work on this problem of SOLPNTCALs, and I think I have finally solved it. The offsets returned from solpntanal() is added for all offsets except RA offsets on the old antennas (9, 10, 11 & 13). On those, it must be subtracted (basically it is a change to HA, which has inverted sign from RA).
23:57 UT I updated the Ant 14 pointing parameters based on the pointing measurements made on 01/27 and 02/01. I will do some additional observations to check it when possible, but right now the wind is high (> 20 mph).
14:11 UT Just started observations with pcal.fsq on 3C273, in order to check the pointing. The wind only just died down a bit after blowing all night, and the high winds may come back, so this will be a short observation. I hope to get an hour.
17:32 UT I just finished analyzing the delays based on today's observation (only the first 30 min was any good), and have updated the delay-center tables as of this time. The next scan should pick up the new values. Note that I did not find a good solution for ant4 and ant12 -- I am not sure why.
01:00 UT I started a run on 3C84, in order to check the delay values from earlier.
20:00 UT The delay centers were okay, but the X vs. Y delays had the wrong sign. I remeasured them today using ND-ON, and verified that they had to be inverted. I wrote a new helper routine, and just now updated the delays (to be incorporated at the next change of scan).
12:54 UT Doing an observation of 3C273 to check delays. This should go about 1.5 hours before the source sets. Using pcal.fsq, and all ants except ant7 are tracking.
00:52 UT During today's solar observations, I made sure that the 27-m was tracking on all three scans. I will now do a test calibration on 3C84 alternating between 3bands.fsq and pcal.fsq. The purpose is to ensure that the phase does not depend on frequency sequence, but I will also use the pcal.fsq data for calibration to apply to the three PHASECAL scans from today's observations (three different sources, 2000-178, 2123+055, and 2348-165) to verify that they can be used to determine phase slopes despite rather poor SNR. The 3C84 observations will begin at 1:30 UT and continue until 04:50 UT.
03:47 UT Just an update to say that the observations have gone without problems. Note that Ants 7 and 12 are not working. At 03:50 UT I sent ants 9-11 and 13 to stow briefly to reset their pointing. It took about 3 minutes.
02:58 UT 30-min scan on 3c84 with pcal.fsq until 03:28 UT. Ant 7 & 12 are down. Windscram during ~03:12-03:18 and ~03:21-03:28(end). Ant 14 HPol experienced a known problem but nothing (including changing the voltage) could fix. Only Vpol (YY) data is usable, but YY and XX seem interchangeable since X vs. Y delays looked okay earlier.
06:40 UT Long (~7 hrs) scan on 3c273 with 3bands.fsq until 13:40 UT. Ant 7 & 12 are down. Delay centers were updated based on the previous 30-min 3c84 scan. Wind affected data until ~07:40 UT. Ant 14 HPol problem continued.
03:47 UT 30-min scan on 3c84 with pcal.fsq until 04:17 UT (originally scheduled from 02:00 UT but delayed due to continuous high wind). Ant 7 & 12 are down. No windscram or significant problem. Data from IDB20170223034837.
19:00 UT The second PHASECAL of the day was mostly missed due to high wind (until ~19:10 UT).
00:12 UT Very impulsive C1.1 flare caught at the end of the day (could be weak though). Ant 9-13 not tracking and 7 & 12 are down as have been.
00:41 UT The last PHASECAL of the day started successfully, and Ant 14 hpol somehow came back right on time. The source 2348-165 was very weak (~1 Jy) though, so another PHASECAL with 0609-157 (8.3 Jy) was attempted from 01:07 UT, but now Ant 14's hpol dropped. Also the windscram until ~01:12 UT.
02:00 UT 30-min scan on 3c84 with pcal.fsq until 02:30 UT. Ant 7 & 12 are down. Quiet wind throughout. Ant 14 hpol not working.
19:00 UT The second PHASECAL source of the day has been changed from the automatically-chosen source (2136+006 to 2253+161). Scan ran successfully, with Ant 14 hpol working. Very stable phase vs. time plot except for Ant 4 which seems to be having some problem.
22:17 UT I noticed that I made a typo in my special schedule for today, and this caused the array to go to the last calibrator source of the day right after the second solar pointing calibration. As a result, the solar data during 21:30 UT - 22:18 UT has been wrongly replaced by 2348-165 phasecal data.
01:41 UT Ant 11 needed manual STOW command after finishing the last phasecal of the day.
02:00 UT 30-min scan on 3c84 with pcal.fsq until 02:30 UT. Ant 7 & 12 are down. Quiet wind throughout. Intermittent Ant 14 hpol drop at ~02:00-02:09 and ~02:14-02:19, working fine otherwise.
12:02 UT Discovered that ROACH 1 is not returning firmware register information, which requires a reload of all of the roaches. I included a powercycle, to make sure everything is reset. Reloading the roaches will necessitate getting new delays. I will attempt to do the required calibrator observations before the Sun comes up.
12:12 UT All ants (except 7 and 12) tracking 3C273. Ant 14 HPol is down. Ant 14 HPol came back at 12:15 UT, so there should be some good HPol data during this and subsequent scans.
12:42 UT There was a bit of a disaster, because the schedule was not properly communicating with the roaches after the roach reboot. The phases were bad on all antennas except ant 8 (no idea why). I restarted the schedule, and the roach connections are okay now. A new scan started 12:42 UT.
13:07 UT I see the same pattern after restarting the schedule. All baselines are bad except ant 8 with 14. In desperation, I rebooted the LO1-A (tunable LO) to see if that is the problem. Another possibility is that Jim's latest update to dppxmp has a problem.
13:30 UT Well, rebooting the LO1-A did not change anything, and 3C273 is now set, so I cannot do more tests. However, I also looked at some solar data from yesterday, and it looks fine, so it does not seem that dppxmp is at fault. I am stymied for now. Further tests will likely have to wait until we are on the Sun.
13:58 UT I checked last night's 3C84 data, and they do not show the problem, so I decided to kill and restart dppxmp. Possibly it could cause the problem, although I am not sure how. I am going to 3C286 (on source at 14:05 UT), to see if that restart makes any difference. This is using band14.fsq to maximize SNR.
14:38 UT Same problem on 3C84. I have rebooted the ACC, which is the last option I have. I am not sure how the ACC could cause the problem I am seeing, but it is the last system not otherwise restarted. Now 3C286 is unreachable, so again I'll wait for the Sun.
22:10 UT The solar data are "okay" except that now ant 8 is showing no coherence, and the pattern of data looks strange, as if baseline order was messed up. I decided to reboot the roaches again, and while I was at it I measured and updated X vs. Y delays. We are on the Sun again, but I am thinking about taking some 3C84 data to see if the earlier problem has been affected by the reboot.
22:52 UT Now everything looks fine, so apparently the reload of the roaches did the trick. I will get the delays and apply them.
23:26 UT I updated the delays, and am going back to 3C84 to check them.
02:28 UT Doh. I just checked, and the delays were put in with the wrong sign. I just updated the delays once more, and the new file is starting 02:28 UT.
13:03 UT I found this morning that Ant 14 stopped tracking at 04:28 UT (possible came to the +51 HA limit). I had also set up the schedule to observe two sources (3C273 and 3C286) all night, but it had a cable tension alarm on the first source, so none of those observations are good. Also, Ant 14 cannot be used until it is fixed, hopefully tomorrow.