23:52 UT Kjell got Ant 5 back in the field, with the notch filters installed. When the 3C84 calibration occurred, I found the Ant 5 delays needed adjustment, so I did that, although it will not appear in the data until the solar day ends. The adjustment was only 0.5 ns. Some others were tweaked by 0.1 ns as well. In order to evaluate the new Ant 5 FEM, I need to first measure and set the DCM attenuations in the < 2.5 GHz bands, so that as we tune to those bands we do not saturate the ADCs. My first attempt to do that failed for some reason, but I will attempt it again this evening.
04:35 UT I finally had a chance to check the performance of Ant 5, only to find that it looks just like the other antennas. Looking into it further, I discovered that Kjell had put the notch filter in without taking the 2.5 GHz filter out! It is also in the wrong place. It needs to be between the first (Miteq) LNA and the second amplifier/first variable attenuator. Instead, he had put it between the two attenuators, leaving the 2.5 GHz filter in place. I emailed him to fix it, but I believe he is on a short vacation until next week, so nothing will happen until he returns, just before we all arrive at OVRO on May 12.
03:49 UT Kjell switched the position of the notch filters, and removed the 2.5 GHz filters in Ant 5, so I had a chance to really evaluate it. However, I find that the power is huge in both channels, requiring at least 10 dB more attenuation. Then the DCM power attenuation setting becomes too low for most bands. I am not very happy with it, but I do need to think more carefully and make sure I am making the correct interpretation. I will work on it over the weekend.
04:35 UT I took some packet capture data on bands 1, 2 and 3, and I find that the signal is not really very high on those bands. I verified by looking at the ADC levels. I then tuned to band 35 (which is 600-1000 MHz, aka band 0), and Wow!, the ADC signal was off the charts! I set the attenuation appropriately to allow me to capture some data, and could see that most of the band is filled with strong RFI. Therefore, it is very clear that what is happening is that the RF is being swamped by the <1 GHz part of the band, and is NOT the fault of the notch filter itself. So we need to add a 1 GHz HP filter (or possibly we can get the notch filters changed to cut off 1 GHz). Anyway, we will experiment when we are at OVRO next week.
03:16 UT Kjell found some 1.1 GHz HP filters and installed them in the Ant 5 FEM. I am happy to say that the FEM is performing just fine now. There is a lot of RFI in those bands 1-3, so it is not a panacea, but it will give us additional good science bands interspersed with bad science bands. I am somewhat encouraged that we could proceed with this conversion and get us our full 1-18 GHz range back! I have started a two-src.scd schedule with pcal_hi-all.fsq (will not sample the lower bands).
03:27 UT Actually, I thought better of it, and decided to modify pcal_lo-all.fsq, which was doing bands 4,5,6,7,8,9,10,11,12,12. Now it will do 2,4,5,6,7,8,9,10,11,12, so I added band 2 to the mix. Oops, that is dumb, because band 2 has the top end blanked by the notch filter. I will change to 1,4,5,6,7,8,9,10,11,12, and so will do band 1. Anyway, it is on the hi receiver right now. It will go to the lo receiver at 03:54 UT, and start the first scan with band 1 at 03:56 UT. I am not actually sure if the science bands are correctly defined for band 1, but we'll see. I am running 3C273-hilo.scd, which alternates between hi and lo receivers on 3C273 every half hour or so.
02:05 UT I found Ant 12 was in Maintenance mode, and switching it to Operate brought it to life. It seems to be running fine now. Unfortunately, Ant 14 focus is stuck, and it is nowhere near the focus for the high-frequency receiver, so we will not get good data on that until it is repaired on Monday. We can observe with the low-frequency receiver.
02:05 UT We are observing two_src.scd (3C273 and 3C286) with pcal_lo-all.fsq, which uses the Low-frequency receiver and includes band 1. The main purpose is to find the Ant 12 baseline solution, but we can also evaluate the Ant 5 band-1 data.
03:59 UT Because the 27-m focus rotation mechanism was out due to a power-supply failure, there have been no calibration observations for the past week. I took advantage of that to run the 300 MHz correlator, which has been running since Friday. However, it continues to have its loss of packets, and I also verified that it gets progressively work over time, as if it were losing synchronization more and more badly over time. I was about to do some calibrations with the fast correlator, but find that the antenna (14) is in LOCAL mode. I am not sure why--perhaps it was never set back to REMOTE after the repair was completed last Friday.
16:41 UT The 27-m receiver has been removed from the dish (as of yesterday), and the receiver will be opened and the problems hopefully identified and fixed. I cannot guess how long we will be down. Meanwhile, Jack has verified that the 300 MHz correlator is not producing all of its packets, but has a potential solution for which he is working on a new compile. I have high hopes that it may solve the lost-packets problem. There is also the synchronization problem that may or may not be due to the same lost-packet issue. We are taking normal solar data today, but of course without calibration...
11:15 UT It looks like Jack's change, and a setting of 15000 for inter-packet delay, has completely solved the missing packets problem! Jack loaded a new design in the wee hours of May 25, and now about 30 h later it is still putting out the full number of packets. However, his change did not affect the oscillations we see in the data. I spent quite a bit of time yesterday looking into it, and I reported to Jack that I strongly suspect that the problem is incorrect timing of the accumulation window. An accumulation has mostly the target frequency, but with a non-trivial admixture of the following frequency in the sequence. The result is a contamination by data oscillating at a 500 MHz fringe rate (the difference frequency between two tunings). I verified this by looking at band 10 data, which is followed by band 25, and hence has a much larger frequency difference. Indeed, the contamination oscillated at a far higher fringe rate, as expected. Today I will do a test, by capturing packet data during tuning in the pcal.fsq sequence, and then do some other tests where I adjust the accumulation window to be shorter in an attempt to reduce the contamination. For this, I need the Sun, so I'll have to wait for a few hours.
23:48 UT I finally got around to adjusting the accumulation window. I reduced it by 512 samples, so it will be 2048 long. Hmm--looks like the design does not tolerate that, or else I made some other error. I am not getting all the packets out. I gave up and went back to the full 2560-length.
17:15 UT I got a late start today. I noticed that not all packets were coming out of the correlator, so I reloaded it and started observations just now.
15:20 UT We are now running the 200 MHz correlator design. Jack is away for a week, so no further progress is expected on the 300 MHz design for awhile. We will continue with the 200 MHz design until some progress is made on finding the cause of loss of synchronization. The 27-m receiver is still in the lab--hopefully will be back this week! No calibrations possible until it is restored to service.
12:50 UT We are struggling to get control of the receiver in the lab, using a different control computer. Some assumptions about hostnames has made for some problems, so I made some changes to the starburstControl system that appear to have helped. However, I still cannot bias the amplifiers in the receiver, and the ACC cannot communicate with the control system on feanta. We will work more on it today.