19:18 UT Reboot ROACHes to clear fault light and test whether large delays are still present. They were power-cycled and reloaded.
19:24 UT Started roachcal.scd to take packet capture data for checking delays. Still bad!
03:41 UT Found a potential problem in DELAYCAL_END.ctl, so took another two sets of roachcal.scd data. Still bad!
04:14 UT Started roachcal.scd for another set of data. After studying schedule.py, another problem was found. DELAYCAL_END command has to have CIEL-2 as second value on the line, otherwise source id is set to None! Yay! It finally worked!
01:08 UT Set up the schedule to do a long run on 3C84 with the low-frequency receiver, using pcal_lo.fsq (all bands 2.5-6 GHz). One major change is that the PA can be set, so I set it for -30. I would predict that the chi-dependence will shift by 30 degrees (the phase jump will occur at a different hour angle), but the non-intersecting axes issue will cause the same symmetric phase rotation (it depends on elevation only).
04:20 UT 3C84 schedule begins.
12:00 UT 3C84 schedule ends.
03:00 UT Note, the above observations showed no coherence, for reasons unknown. The receiver will be checked for pointing and focus using a total power source. I attempted to observe the Moon at the current time, but it happens to be at -6 Declination, so it is in the geosynchronous satellite belt and could not be peaked up in total power. I will observe Cyg A when it rises at ~ 21:20 UT today.
22:00 UT Began some Cyg A drift scans. Original Z offset was 200, and I could not see any increment on Cyg A, but changing to 50 gave a clear response.
22:20 UT Nominal peak of a drift scan (well seen on baseline 11-14) is 22:20 UT, but it actually occurred at 22:19:55 UT, so original RA offset is right: 0.15 degrees. Also did a Dec drift scan that peaks at 22:46:00 UT, which verifies the DEC offset is also right: 1.03 degrees.
22:49 UT Did a focus scan. This is the time when the focus was best, which corresponds to a focus (Z-axis) setting of 30 on the Lo receiver. Higher settings were significantly worse, so that by a Z-axis setting of 200 the amplitude is going to zero. This explains the lack of phase coherence on the 3C84 run of yesterday.
05:42 UT Attempting to see the Moon with 27-m Hi receiver. I could see nothing at nominal position, but then I got a whopping signal at about 2 degrees RA offset. This might reflect a very wrong X axis position.
05:54 UT Moved X axis setting to 500, and I do get a very high increment at nominal pointing. Best LO Receiver settings X-axis 100, Z-axis 30; Best HI Receiver settings X-axis 500 Y-axis 0.
06:25 UT Began 3C84 observations using HI receiver, with starburst_hi.fsq sequence. Given the much higher signal strength on the Moon, the sensitivity should be very good on 3C84.
06:35 UT Decided to switch to pcal.fsq sequence. This should provide better SNR. Yes! The data look very nice, see Fig. 1 at right.
14:25 UT Going to 3C273, for some PA tests. Oops--used the wrong frequency sequence--was pcal_lo.fsq
14:31 UT Now setting to pcal.fsq
14:37 UT Moved PA position to -35 degrees, which is the current parallactic angle (I will test this and +35 to check the sign).
14:42 UT Began move to PA +35 degrees. I should see a dramatic change in XX and YY amplitudes when feeds are parallel.
15:06 UT Went to PA +29 degrees. Looking at the data, it appears that we need to apply -chi to the PA axis to make the feeds parallel.
15:30 UT Went to PA +23 degrees (tracking chi variation). I did some analysis that verifies -chi is the correct term to apply. See Fig. 2.
15:48 UT Went to PA +15 degrees (a bit ahead of correct -chi, which is +18 at this moment).
16:10 UT Went to PA +10 degrees.
17:51 UT Went to PA -21 degrees. Obviously a lot of rotation between these two... I have continued to make a couple of adjustments at different times--can get this from the stateframe if I need the exact times...
21:51 UT Finished developing a $PA-TRACK command for the schedule, which resulting in many-many restarts of the schedule, so there will be a lot of short solar scans at times before 21:51 UT.
23:30 UT As usual, wind was manageable all day until I wanted to use the 27-m on a calibrator to test the PA rotation. Now the wind is up, and it will probably only get worse. I will stop the observations of 2253+161, which only just started, and go back to it if I can, later. If not, I'll do 3C84 overnight.
00:06 UT I did get back to the source at about 23:45 UT, and so far it is tracking and the wind may be declining. The frequency sequence is band14.fsq.
03:45 UT The data for source 2253+161 are done--very successful observation. There is some clear variation in amplitude that could easily be pointing issues, perhaps with Ant 14. At 03:45 UT I started a 3C84 observation, but then realized that I did not want pcal.fsq, but rather, want to use band14.fsq as I did for 2253+161.
03:55 UT Correct observations on 3C84 begin. These should be interesting due to the large parallactic angle changes, including going past 90 and -90. I am tracking the angle using $PA-TRACK ant1, so we will see how it goes.
12:41 UT The 3C84 observation is done, but there were some glitches. As far as I can tell, the observations went normally until around the time of the discontinuity at HA = 0, then either the PA_adjust() process died, or the Ant14 control program died. In any case, the PA was stuck at -32 degrees after that point. There were other glitches, from time to time, where the Chi value was set to 0, and no doubt this was exacerbated by the schedule lagging badly. I need to find the cause of that.
14:28 UT I was observing 3C273 from about 13:00 UT, but tracking PA for ant14. I realized that the calculation of Chi was wrong for the equatorially mounted dishes, so that whole time the data are taken with the wrong PA. I have updated the stateframe.py code to correct it, and have started a new observation tracking Ant14's Chi value. This one should be okay.
18:51 UT The wind suddenly came up, so the 3C273 observations have ended. The PA rotation seems to have gone without a glitch since 14:28 UT!
22:49 UT I just started a series of observations on sources 2136+006 and 2253+161, alternating between the two, which should continue for about 4 hours. This is to test Jim's change to dppxmp, which is supposed to automatically correct the offset-axes issue. I'll be checking if it has the wrong sign, and update it if so. Argh. WINDSCRAM. --Anyway, Jim's code has a bug--
13:03 UT I started a schedule to test Jim's latest dppxmp. It is cycling among sources 1229+020 (3C273), 1256-057 (3C279), and 1331+305 (3C286), tracking PA, using band14.fsq.
17:18 UT Observations continuing. We are now past the meridian, and the results look good. I think we have the right sign of the correction, but I will want to get some observations on 3C84 before I am entirely convinced.
20:48 UT The observations are coming to an end at 21:55 UT. They went very well, although I notice that the scan starting 20:35 UT ran into a limit on Ant14 as the source set. The last few minutes of that scan will be bad data.
04:00 UT Updated some of the baselines for today's measurements, and now starting an observation on 3C84. This is also using band14.fsq, and rotating the PA on ant14.
12:28 UT Looks like the Ant14 control system stopped sometime shortly after 06:38 UT, which froze the FRM at PA -62. The dish was also pointing in a strange place, so it may also have stopped, but if so it was some time later. The data-taking continued to 11:45 UT, as planned.
14:01 UT Started another run on multiple sources (1229+020, 1256-057, and 1331+305), but with the same baseline setting as for 3C84. I am analyzing the 3C84 scans to update ants 9, 10, 11 and 13.
16:05 UT I updated the baselines on ants 9, 10, 11 and 13, which were way off due to my changes to Ant 14, but also had their own adjustments needed. Hopefully they are better, if I got all the signs right. I have restarted the schedule to implement the changes, and also changed the schedule to eliminate the measurements on 1256-057. The results were precisely the same as for 1229+020, and it was in the GEOSAT belt, so had other problems. Now I am doing 20-min scans alternating between 1229+020 and 1331+305, using pcal.fsq.
16:46 UT After looking at the previous data, I decided the SNR is not high enough, so I restarted using band14.fsq instead. Still tracking PA.
17:05 UT Doh! After seeing no phase coherence with band14.fsq, I noticed that Ant14's radecoff was zero. This happened when I rebooted the Ant14 controller this morning to clear a "permit" alarm. I have reset the offsets to radecoff 0.15 1.03, where they are supposed to be. This should make a bit of a difference! (understatement) --Whew, a check of the data now shows good phase coherence.
18:39 UT Windscram occurred, and the wind is likely to remain high, so observations have ended and I started solar observing.
03:45 UT New observation of 3C84 begins. For this, I am switching from tracking PA to not tracking PA every 10 minutes. Using band14.fsq.
13:50 UT Updated baseline corrections based on latest 3C84 observations (which indicated that I had the wrong sign for my previous update!), so now I am doing a check of these corrections. Starting observations of 1229+020 and 1331+305, with band14.fsq, alternating every 20 minutes. Assuming the Bx and By corrections are okay, these observations should be suitable to some sort of estimate of Bz.
18:35 UT Observations have ended due to windscram.
23:35 UT Started a new observation of 2136+006 and 2253+161 (two_src.scd) using a new frequency sequence called 3bands.fsq (does bands 12, 14 and 16). The data look great, so this is certainly a viable observing mode for a wider frequency range, but due to a transcribing error in the schedule, the observations failed to change sources correctly. This is now corrected, but of course the wind is back and none of the subsequent scans are good. Also, ant11 is acting up, and so is not tracking.
00:40 UT Yesterday's data on 1229+020 and 1331+305 are good. I spent the afternoon looking at the results, but they are rather ambiguous, because their variations do not really match any expected variation. What is interesting is that they all match each other, which suggests that the variations are due to Ant14, either due to a residual baseline error or some other cause. I am giving up on the two_src.scd.
09:30 UT Began observations of 3C84 and another source, 0609-157, alternating every 10 minutes, using 3bands.fsq and tracking PA. Ant 11 is still down.
10:45 UT Observations end. I find that source 0609-157 is much too weak to be much use. However, possible poor pointing at a low declination could be a factor.
12:45 UT Observations start on 1229+020 and 1331+305 (although the latter does not rise for another hour), using 3bands.fsq and tracking PA. Good news--I got Ant 11 working.
15:25 UT Observations end. I will analyze for Bz baseline corrections.
16:45 UT Started additional observations on 1229+020 and 1331+305, using 3bands.fsq and tracking PA. I finally got good measurements of Bz! I have entered the corrections, so these observations should be corrected (if I got the signs right). Note that I could not measure the slope on Ant 11 (data too poor, and phase wind was too much), but from earlier measurements I know it is somewhere near 57 ns (huge!). I went ahead and put in this correction, and hopefully it will be close enough to get a better measurement this time.
17:56 UT Windscram. Looks like the wind will be high for awhile, but I may have gotten my observations that I needed. I can confirm that the Bz corrections are right (although I do not rule out additional adjustments). What is nice is that I can now see delay-center errors that I can confidently ascribe to delay-center.
18:03 UT Going to the Sun, mainly to check peakup on Ant 11.
22:42 UT I have started observations of 2136+006 and 2253+161, after implementing delay-center corrections, using 3band.fsq and tracking PA. I actually started on 2136+006 about 20-min ago, but then discovered that the ra/dec offsets were zeroed on Ant 14, so the good data should only start now. I really expect that the phases should be flat, both in time and frequency, so we shall see...
23:42 UT Stopped observations to go to CIEL-2 with Ant 14. I realized that I had not corrected the delays for that antenna yet. This caused all of the baselines with Ant 14 to have a big phase slope with frequency.
00:17 UT Well, I updated the delays on Ant 14 (and also 13, which was a bit off), but now the wind is up and I have not been able to resume the calibrator observations. Anyway, I have to switch over to the fast correlator, since Jack wants to work on it starting at 01:00 UT.
00:30 UT Actually, the wind died down unexpectedly, so I am resuming observations at least to get one set of data on each source. I'll then have to stop and get the fast correlator design loaded.
03:40 UT Started observation on 3C84, 3bands.fsq, tracking PA.
04:30 UT Stopped observations to test the fast correlator. Jack set the switch (xswitch) to handle jumbo packets, and now the packets are flowing! There are still problems, but at least we can move forward again!
06:49 UT We are done working on the fast correlator. P packets look mostly normal, but X packets are still not coming out, it seems. But at least Jack has something to work with. I restarted the 3C84 observations with 3bands.fsq, but I decided not to track PA. So we will see how that looks.
10:40 UT 3C84 observations end. I hadn't realized it, but apparently the schedule had a few scans of 0609-157 at the end.
12:35 UT Started observations on 1229+020 and 1331+305 (on source at 12:39 UT), after a large update to Ant 11 Bz. I will take data until 13:10 UT without PA tracking, to measure dphi/df on Ant 11, then will likely start tracking PA.
13:33 UT Started PA-track.
15:40 UT Started normal solar observations (unattended, since I am traveling back to NJ today).
21:50 UT Started scheduled observations on 2136+006 and 2253+161, 10-min/source, over entire sky. Checking the tracking of Ant 14, it appears that high wind stowed it from 22:00-23:00 UT, and also the scan at 23:25 UT. All others may be okay.
04:50 UT Observations on 2136+006 and 2253+161 end.
15:42 UT Started normal solar observations. Will repeat first part of calibration observations on 2136+006 and 2253+161 until 0 UT, if wind conditions permit.
21:50 UT Data started on 2136+006 and 2253+161, but after checking the data I find that Ant 14 was not tracking at all, due to a permit condition on the Declination drive. It is too bad, because otherwise the conditions were good the whole night (no wind scrams).
15:31 UT Planned start of solar data. Will do normal solar observing.
20:58 UT Both Bz values and delay centers have been updated. Solar data-taking restarted. I will have to check it to be sure that the sign of the delay-center corrections is right.
01:05 UT The solar data are much better in terms of delay, but still not entirely flat. I just started taking additional data on 2136+006 and 2253+161 to check my Bz corrections.
17:50 UT Somehow I managed to use the wrong sign when calculating my Bz changes, which became immediately apparent on looking at the data. The delay center was right, but needed some additional tweaking, especially for Ant 14 (about three ns more). Once again I have updated both baselines and delays, and am taking new data on the Sun to evaluate. I also found negative delays again on Ant 13 this morning, so I increased the delay center offset for Ant1 to 11000. This may cause overflows in the other direction for some baselines, so I'll have to keep an eye on it. Now taking additional solar data...
02:05 UT Started yet another observation of the two cal sources, to check my update of Bz and delays. The wind had come up earlier, so I could not get started when I wanted, therefore this will only be a couple of hours on source. That should be enough, however. Note, Ant 5 is down due to a hard low elevation limit, and Ant 14 keeps having an El drive issue that throws a permit alarm--not sure what is going on there.
13:40 UT I am doing an experiment. I created a new trajectory file, calpnt.trj, for off-pointing the 27-m antenna. I also created corresponding CALPNTCAL.ctl and calpnt.scd files. It does an off-pointing procedure similar to SOLPNTCAL, but spends 30 s on each spot, and of course is meant to use only a single band (e.g band14.fsq, which I am using for the test). I added P1 and P7 offsets to Ant14, so that the offsets in calpnt.trj are relative to that. One thing I can see is that the 10-min trajectory is going to start while the dish is slewing to another source, so it will miss the first couple of off-pointings. In general, though, it seems to be working!
14:20 UT The experiment is done. It looks like it mainly worked, but I do need intermediate SLEW "scans" that can provide time for the dishes to get where they are going. The macro command for this purpose was later called ACQUIRE instead of SLEW. I will take a look at the data and see if the SNR is high enough for these 30-s pointings. Well--there is no sign of coherence, so either something else went wrong (I could, for example, have the wrong sign for one or both of P1 and P7), or the SNR is woefully inadequate. I will check the pointing parameters later.
15:48 UT Normal solar observing begins.
00:58 UT Started a three-src.scd schedule to observe 2136+006, 2253+161, and 0319+415, without tracking PA, and on band14 only. The pointing offsets are in P1 and P7 on Ant 14, so I will check the quality of the data after the first scan or two, to check that it is pointing well. The wind is on the edge, having been very high earlier, so I hope it dies down further. NB: Later pointing indicates wrong sign on P1, so we were offpointed by 0.3 degrees => low SNR.
01:30 UT I checked the phase coherence, and it looks sort of okay. The wind is causing windscrams, so I am not sure how much of the data will be good, but I'll let it keep trying.
05:55 UT The observations completed, with some windscrams, but I will analyze it tomorrow. Meanwhile, I was messing with calpnt again, on 3C84, and indeed the SNR is woefully inadequate for 30-s pointings. I upped it to 90s, and removed one pointing, so it will take 25 minutes to complete. I'll also check these results tomorrow. I am doing four measurements in a row, 25-min each.
14:50 UT The pointing observations from last night again have low SNR, but I figured out how to sum the results from all n-14 baselines and over frequency to get a good measurement. The measurement strongly indicates that the sign of P1 is wrong, as I suspected it may be. I changed the sign of P1 for Ant 14, and am now doing equivalent pointing measurements (2 each) on 3C273 and 3C286. So far, the system works as desired, which is good.
15:45 UT Yes! The pointing data on 3C273 are great, now. Clearly it needed the sign change in the P1 coefficient. This will work well for the purpose, so now we can proceed with getting pointing measurements over the whole sky and doing the fitting to get all of the pointing coefficients for Ant 14.
16:40 UT I just completed the series of calpnt calibrations, and also wrote the analysis software to reduce it. The results are great, so are applicable to a more complete pointing calibration. Now going to normal solar observing.
21:32 UT The system started having a lot of trouble, restarting files, due to dropped packets, I assume, before the DPP finally crashed. I did not discover it until a few hours later, but anyway Jack wanted to test a new ROACH design, so we rebooted and did tests for while.
02:14 UT Started a repeat of the three-src.scd observation from last night, now that the wind has died down a bit. It is still right on the edge of the limit, though. This should fix the low-SNR problem we had last night.