Owen's Notes

List of Routine Tasks

Daily Tasks

Check the following pages:

http://ovsa.njit.edu/status.php (the all-day plot from the previous day is at the bottom of the page)

http://www.ovsa.njit.edu/browser/ (this defaults to showing pipeline output from two days previous)

http://ovsa.njit.edu/pointing/ (the latest pointing plot(s) at the bottom)

http://ovsa.njit.edu/phasecal/ (the latest phasecal vs. frequency is somewhere in the middle, i.e. the bottom of the list of pcf*.png files).

http://ovsa.njit.edu/EOVSA/ant_status.php (Current antenna status).

Ensure that the schedule is running and that the stateframe is updating.

Check that the data files are being updated. Goto the user@dpp terminal and type:

ls /data1/IDB |tail

Perform a Phase Calibration

Check that the network monitoring function (fix_packets()) is working on the DPP. It can also be seen as dropouts on the spectrogram (http://ovsa.njit.edu/status.php).

Check the Front End Power on the stateframe display. These values should be between about 0 and 3dBm. If the front end power looks suspicious, you can check it by plotting the front end power vs time (see here for instructions: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Plotting_Stateframe_Data).

If there are any problems with any of the antennas goto http://ovsa.njit.edu/EOVSA/ant_status/ant_status_form.php and fill out the appropriate form.

Weekly Tasks

Perform a Gain Calibration

Check the NJIT Vehicle:

Ensure fluid levels are good.

Check the rodent baits and replace as needed.

Check the engine compartment for signs of rodents.

Drive the vehicle for at least 1/2 hour per week.

Monthly Tasks

Perform a Pointing Calibration

Login Procedures

Remote Login

The computers can be accessed remotely using SSH and VNC. Since I use Linux (Ubuntu) these instruction are for that OS.

Helios

To VNC into Helios, open a terminal and type the following command:

ssh -L 5902:helios.solar.pvt:20000 oweng@ovsa.njit.edu

Note replace oweng with your gateway login name)

Then open Remmina Remote Desktop Client. Make sure VNC is selected at the top left. Type in:

localhost:2

and enter the password when prompted.

The Helios display should now be visible.

Occasionally, the VNC connection will be lost. If this occurs the following procedure need to be performed.

From a new terminal window type:

ssh -L 8887:helios.solar.pvt:22 oweng@ovsa.njit.edu

From another terminal issue the command:

ssh -p 8887 sched@localhost

Now type in:

x11go

Now restart the VNC client.

Note that the VNC connection issue should now be resolved. If you notice it dropping out let Owen know what you were doing at the time.

Pipeline

To log in to pipeline, from a terminal enter the command:

ssh -L 8888:pipeline.solar.pvt:22 oweng@ovsa.njit.edu

Then from a new terminal enter:

ssh -p 8888 user@localhost

Win1

To log in to the win1 machine, from a terminal enter the command:

ssh -L 8889:win1.solar.pvt:5900 oweng@ovsa.njit.edu

Then from a new VPN client enter the hostname:

localhost:8889

DPP

To login to the DPP machine, from a terminal enter the command:

ssh -L 8890:dpp.solar.pvt:5900 oweng@ovsa.njit.edu

Then from a new terminal enter:

ssh -p 8900 user@localhost

Local Login

If logging into a computer locally on site then the following procedures are used:

Helios

Open the Remmina Remote Desktop Client. Make sure VNC is selected at the top left. Type in:

helios:20000

and enter the password when prompted.

The Helios display should now be visible.

To access a helios terminal, from a new terminal window type:

ssh sched@helios

Pipeline

To log in to pipeline, from a terminal enter the command:

ssh user@pipeline

Win1

To log in to the win1 machine, from a new VPN client enter the hostname:

win1

DPP

To login to the DPP machine, from a terminal enter the command:

ssh user@dpp

Setting up new accounts

The most likely computers on which a new user will need to be added will be on the Gateway computer and pipeline. As they are both Ubuntu systems the procedure is identical for both.

Login to an account with superuser priveliges on the computer where the new account is to be created.

From a terminal issue the command:

sudo useradd -m -d /opt/username username

where username is the name of the new account.

You will be prompted to enter a password. If creating the account for someone else, ensure they change it after their first login. They can issue the command

passwd</p"

If creating the account on pipeline, they will probably need access to the python files located in :/common/python/current/

To set this up, edit the :~/.bash_profile using nano:

nano ~/.bash_profile

Add the following line:

export PYTHONPATH="/common/python/current":"/common/python/mapping"

Press <CRTL> X to exit and when prompted press Y to save the file.

To immediately make the changes to effect issue the command:

source ~/.bash_profile

If a user requires superuser privileges, from an account with superuser access enter the following:

sudo usermod -aG sudo user

where user is the name of the account you are granting the privileges.

Helios Displays

Schedule Window

The Schedule window shows the currently running schedule as shown below.

The left side of the window shows each of the scripts that is to be run. The orange bar shows the task that is currently being run.

The right side of the window shows each command that is to be run in the current script.

At the bottom of the window are a series of buttons. The STOP button will stop at the current script. Once stopped, the remaining buttons will become active. To restart the schedule click on the GO button.

Starting the schedule window

If the schedule window gets closed it will need to be restarted. Follow the procedure below:

From a helios terminal enter the following commands:

cd ~/Dropbox/PythonCode/Current

screen -S schedule

python /common/python/current/schedule.py &

Press <CRTL>AD to exit from screen.

Stateframe Display

The Stateframe display shows the current status of the system as shown below:

There are problems with any antennas or systems that are highlighted in red. Lines that are marked in yellow designate a warning, but may not be critical.

Lines marked in purple are collapsible sections. Clicking on them will expand or collapse the section.

The tabs near the top of the display bring up different sections and enable graphs to be plotted.

Note that the Log Stateframe box should be marked in blue. If there is a second Stateframe display running this box will be unmarked on the second display.

Restarting the Stateframe Display

To restart the Stateframe display enter the following commands at a helios terminal:

cd ~/Dropbox/PythonCode/Current

screen -S stateframe

python /common/python/current/sf_display.py &

Press <CRTL>AD to exit from screen.

Note that it will take several minutes for the display to come up.

Computer Restart

Helios

To restart Helios, at a terminal enter the command

sudo reboot now

and wait about 5 minutes for it to restart.

Open a terminal and start the dropbox server:

python /home/sched/Downloads/dropbox.py start

Check to see if the time service is running. Enter the command:

chronyc sourcestats

If nothing is displayed, then the time service will need to be started by typing:

sudo systemctl restart chrony.service

Start the x11vnc server by typing:

x11go

Start the schedule as described here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Starting_the_schedule_window

Start the Stateframe display as follows as described here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Restarting_the_Stateframe_Display

Open up a new terminal tab and log into the DPP by typing:

ssh user@dpp

Now issue the command:

ls /data1/IDB | tail 10

Open up a new terminal tab and log into the DPP as above. View the SMP_AFFINITY.sh by typing the command:

cat /home/user/test_svn/shell_scripts/SMP_AFFINITY.sh

Note down the two CPU's that are in use. Issue the following commands:

ipython --pylab

import dpp_plot_packets as dpp

dpp.fixpackets(cpu = [cpu1, cpu2])

where cpu1 and cpu2 are the CPUs that were noted down earlier.

DPP

First, check that the 10 Gb ethernet interfaces are present, by typing the command:

ifconfig

which should show a list of interfaces including enp5s0 and enp7s0. If these are NOT present, type:

sudo modprobe myri10ge

to start the device driver. Then check again that these interfaces are present.

Resetting the Interrupt Priority

When the DPP is rebooted, or there are lost packets, its interrupt priority needs to be reconfigured. This is done via the following, typed at a terminal command prompt:

sudo /home/user/test_svn/shell_scripts/SMP_AFFINITY.sh

Comment out the current two CPUs and uncomment out the next two. If CPU's 22 and 23 were currently in use, then uncomment CPU's 18 and 19.

If the DPP was rebooted and the dppxmp program was running at the time of the reboot, then after the reboot it will also be necessary to delete the lock file:

rmlock

From a helios terminal log into the dpp

ssh user@dpp

and then issue the following commands:

ipython --pylab

import dpp_plot_packets as dpp

dpp.fixpackets(cpu = [cpu1, cpu2])

Mouting the dppdata1 disk

Note also that if the DPP reboots it is likely necessary to mount its disk on Pipeline, otherwise the pipeline task will fail. To check, log in to Pipeline and type df. The dppdata1 disk should be present. If it isn't, issue the command from pipeline:

sudo mount -a

Calibrations

Reference Calibration

Before a phase calibration can be done, a reference calibration needs to be performed, which will be subsequently used as the reference for the phase calibration.

Either access Helios via VNC computer or login to pipeline. If using helios access a pipeline terminal.

At the prompt type the following command:

python /common/python/current/calwidget.py

The following window will open up:

In the date box near top right of the window type in or use the up and down arrows to select the previous days date and hit enter.

Click on the first available scan that does not suffer from a Windscram. Windscrams are marked in red. If a windscram occurred for less than 20% of the scan then it will be marked in yellow and may still be usable. The example below shows a windscram at 22:14:05.

Select the earliest calibration that does not have a windscram, and then click on 'Analyze as Refcal'. The Sigma map should now be populated as shown below:

The above chart is for the low-band (channels 1 to 12). Yellow blocks indicate bad or unavailable data. Bands 3 and 4 are excluded due to intense RFI at these frequencies. Antenna 5 currently is not operating well, and will be excluded from any analysis. The green blocks indicate that the data may be alright if some points are excluded. Purple blocks indicate the data is good.

Also shown in the above plot is the power and phase for Antenna 1 and band 12. The blue points are the X-polarization and the orange points are the Y-polarization. These should be close to straight lines.

In this case we need to exclude all of Antenna 5 from the calibration. First click on any block for antenna 5 in the range of band 1 to 12. Next check the box 'Apply to all bands'. Move the mouse over to either the Power or Phase plots (either works). We need to exclude all points. Move the mouse slightly before the first point in the chosen plot and press the 'A' key. A green line will appear. next move the mouse slightly past the last point and press the 'B' key. A red line will appear. The screen should look like the figure below.

Now click on 'Apply time flagging. All of antenna 5's bands should turn yellow.

Once complete, select the associated high band calibration and then click on 'Analyse as refcal'. In this case bands 7 to 52 will be shown on the sigma map as shown in the figure below.

Antenna 5 once again needs to be excluded using the procedure described above. In addition antenna 13 has problems from band 13 to 52. We will remove these channels from the analysis. Ensure 'Apply to all bands is unchecked and select 'Apply to all bands above selected one'. Select band 13. As before select all of the points by pressing 'a' and 'b' at the appropriate points (start and end of the plot). Click on 'Apply Time Flagging'. All of the bands from the selected band up should turn yellow as shown in the figure below.

As can be seen, there are still 3 green squares. We can make these good by selecting each one and flagging bad samples. In this example I have selected Antenna 4, Band 23.

Ensure all checkboxes are unchecked. Slelect the box for the antenna and band the you wish to time flag. select (using mouse and a and b keys) up to two ranges of points to exclude. pressing the x key will remove the most resent line chosen. Once you have flagged the ranges, click on 'Apply Time Flagging'. Hopefully the box will turn purple. If not you may need to select different ranges.

The result of this procedure is shown below.

Repeat this procedure for the remaining bad antennas and bands.

Once completed, select one of the bands of the refcals from list on the left and then click on 'Set as Refcal'. The click on extend selection. NOw select the other refcal. Make sure both are highlighted. Click on 'Set as extended refcal'. Click on 'Save to SQL'.

Phase Calibration

After the reference calibration has been done, the phase calibration can be performed. Click on a good calibration for the high band (one that has not had a windscram and has a reasonable duration). Not that Phase Cals are not performed using the low band.

Then click on 'Analyze as Phasecal'. At the top of the screen click on the 'Sum Pha' tab. The important thing here is that the slopes look good. Below is an example display of the slopes. These slopes are with reference to the Refcal.

In the above display, we can see that all of the slopes look good except for Antenna 4. note that we have excluded Antenna 5 from the observations as these have no reference cal. By moving the mouse over points on antenna 4 the channel is displayed as the x value in the bottom right of the window. In this case, channel 24 and above will be excluded. Ensure 'Apply to all bands above selected one' is checked and click on the antenna 4 and channel 24 box. Select all points and click on 'Apply time flagging' Once done, click on the 'Sum Pha' tab again to ensure the slope now looks okay (See figure below). Once satisfied click on 'Save to SQL'.

System Gain Calibration

This procedure is designed to adjust the system gains. This should be performed weekly or whenever the calibrations become poor. The full details can be found at EOVSA System Gain Calibration. The steps below are a summary of that page.

The best time to perform this calibration is immediately after a Solar Point Calibration, as the antennas need to be taken off the sun.

The dBm values in the power and attenuation section of the Stateframe display should be between 1 and 4.

Initially click on the Stop button in the schedule window.

From the Schedule window issue the following commands:

$scan-stop

stow

femauto-off

femattn 0

$fem-init

Wait approximately 1 minute for the new settings to update. The frontend power levels should be between 1-4 dBm. If they aren't then reissue the $fem-init command

Then proceed with:

$subarray1 ant1-14

dcmauto-off ant1-14

fseq-file solar.fsq

fseq-on

$capture-1s dcm

Click on the Clear button and then the Go button.

From a pipeline terminal issue the following commands

ipython --pylab

import roachcal

tbl=roachcal.DCM_calnew('/dppdata1/PRT/PRTyyyymmddhhmmssdcm.dat',dcmattn=10,missing='ant15')

In the above command yyyymmddhhmmss is the year, month, day, hour minute and second of the dcm file. Pressing tab will complete the filename. for the missing field, antenna 15 must be included. Add any other antennas that are not to be included on the gain calibration separated by spaces.

Enter

tbl

to view the contents of the tbl variable that was just created.

Band 1 has recently had bad attenuations. To use older good values for this band issue the command:

tbl = roachcal.override(tbl, bandlist=[1])

Other bands can be excluded by adding the bands to the bandlist (space separated).

It is useful to compare the new table with an existing table to check for errors/glitches. To do this issue the command:

roachcal.compare_tbl(tbl, t=Time('2020-05-08'))

Replace the time with the previous days gain calibration. All of the differences should be close to zero.

Finally:

import cal_header

cal_header.dcm_master_tablesql(tbl)

From the Schedule window click the Stop button and then the Go button.

In the schedule command box enter the command

$scan-start

Total Power Calibration

The Power calibration is performed whenever components are replaced in the receiver. For a description of how to service the receiver see [[1]].

Setup

Connect the power meter to the receiver as described here [2].

Find the GPIB cable and connect it to the ACC. Run the cable into the lab and connect it to the power meter.

Performing the Calibration

Access the windows (win1) computer.

From the Quicklaunch menu select the E4418B Measurement program.

This should bring up a screen as shown below:

Change Enum to the appropriate antenna number.

Change Reading to the appropriate polarisation.

From the Helios computer open up the gedit program.

Open the file /home/sched/Dropbox/PythonCode/Current/FEMbenchtest.scd

Edit the first line so that the correct antenna is present. For example, if antenna 5 is to be calibrated then the first line should read similar to:

1 2019-11-08 12:07:00 FEMBENCHTEST ant5

Once edited, save and close the file.

On the Schedule window click on Stop.

Click on File and then Open

Select the /home/sched/Dropbox/PythonCode/Current/FEMbenchtest.scd file and open it.

Click on Today

From the Window computer, on the E4418B program click on Start Logging Data

The computer should go through four sequences which display 4 block patterns on the plots in the E4418B program.

Once completed click on Stop

Repeat for the other polarisation if required.

Remember to reload and restart the Solar.scd schedule.

Loading the Calibration to the CRIO

The files created are located in the Dropbox/EOVSA M&C/Test Data/ folder.

Find the most recently created text files and open the first of them in a text editor. An example of a part of the file is shown below:

This file need to be edited to remove all but one of the 5 lines for each attenuation. Typically choose the third entry in each data block.

Once the file has been edited and saved, from pipeline change to the folder where the edited files are located.

From ipython, enter the following commands:

import fem_cal as fc

import crio

fc.fem_cal(ant)

where ant is the antenna number for the particular calibration.

The latest files for that antenna will be read, and a plot will appear for each polarization with both points and quintic fit. The parameters of the fits will be printed to the screen in a form that can be cut and pasted directly into the crio.ini file for the appropriate antenna

To retrieve the crio.ini file for a given antenna issue the following command in ipython:

crio.save_crio_ini(ant_str='ant#')

where # is the antenna number. Press CRTL Z to exit ipython temporarily.

Open the downloaded ini file in a text editor and copy the new coefficients into this file. Enter fg to get back into ipython.

Finally enter the command

crio.reload_crio_ini(ant_str='ant#')

Update Antenna Pointing

2m Antenna

Access the Pipeline Computer.

From the terminal, start ipython:

ipython --pylab

Import the calibration and Time libraries:

import calibration as cal

from util import Time

Load up the solar pointing scans for a given date and time.

x,y,qual=cal.solpntanal(Time('2020-03-17 18:30'))

The above command will load the scans from the 17 March 2020 18:30 UT.

To view the beam widths as a function of frequency for each antenna and polarisation:

cal.sp_bsize(x,y)

An example of the X-beam widths for each antenna

An example of the Y-beam widths for each antenna

To obtain the pointing offsets:

xoff,yoff,dx,dy=cal.sp_offsets(x,y)

This will display the offsets for each antenna and feed. Ideally the offsets should be at zero. If particular antennas significantly deviate from zero then a correction can be made.

For example, the following command will adjust the pointing offsets for antennas 6 to 8, 10 and 13:

cal.offsets2ants(Time('2020-03-17 18:30'),xoff,yoff,ant_str='ant6 ant7 ant8 ant10 ant13')

The newer antennas with the alt-az drives that have had tracking updated will need to be rebooted. From the Schedule Window issue the commands:

reboot 1 ant6-8

tracktable sun_tab.radec 1 ant6-8

track ant6-8

The antenna pointing adjustment is typically performed once per month.

27m Antenna

Only do this if absolutely required. The 27m pointing should only need to be performed at most on an annual basis. It takes 24 hours to complete. If possible, schedule the pointing when there are few active regions on the sun

From a helios terminal, view the calpnt48.scd file by issuing the command:

cat /home/sched/Dropbox/PythonCode/Current/calpnt48.scd

Below are the first few lines of this file:

2019-06-22 10:26:00 ACQUIRE 2136+006
2019-06-22 10:30:00 CALPNTCAL 2136+006 band21.fsq calpnt.trj
2019-06-22 10:55:00 ACQUIRE 2253+161
2019-06-22 10:59:00 CALPNTCAL 2253+161 band21.fsq calpnt.trj
2019-06-22 11:24:00 ACQUIRE 2136+006
2019-06-22 11:28:00 CALPNTCAL 2136+006 band21.fsq calpnt.trj
2019-06-22 11:53:00 ACQUIRE 2253+161
2019-06-22 11:57:00 CALPNTCAL 2253+161 band21.fsq calpnt.trj

The dates and times need to be edited. Use the following equation to compute the new times:

OFFSET = NST - OST

NT = OT + OFFSET

where NST is the sidereal time when you wish to start the calibration, OST is the sidereal time for the first entry in the calpnt48.scd file, NT is a new UT time in the schedule, and OT is the old UT time in the schedule.

Note that I am in the process of writing python code to automate the above procedure

Adjusting Antenna Delays

Note: Only perform this procedure if absolutely necessary! This should be done either after the evening calibration or before the morning calibration.

Using the Delay Widget

From pipeline, start the delay widget as follows:

python /common/python/current/delay_widget.py

A window similar to the one shown below should appear:

At the top right, click on File and then Open npz file

Select the most recent file.

You will see a screen that looks similar to that below:

Note that if you had a file loaded and different antenna selected, the program will default to antenna 1 but not change the number in the Antenna box.

Select the antenna for which the delays are to be checked.

For the four panels on the left, each cluster of points represents a band. Each cluster in the XY and YX plots needs to be as flat as possible. Also the XX-YY plot on the right need to have all points as flat as possible. The slopes can be adjust by incrementing or decrementing the X delays on the left of the screen.

Repeat this process for any other antennas.

The following should be noted if adjusting the delays on antenna 1:

Adjusting the delays on antenna 1 is essentially the same as adjusting the delays on antenna 14 as antenna 1 is used as a reference for the other antennas. If delays on antenna 1 are adjusted, they will need to be adjusted on all other antennas.

You will also need to adjust the low frequency delays on antenna 1, but no other antennas.

Once done the new delays can be save from the File menu.

Only save the new delays if you are absolutely certain changes need to be made!

Click on Exit to close the application.

Checking that the delays were updated

To see if the delays were updated, perform the following procedure.

From pipeline access ipython:

ipython --pylab

Load the cal_header and and Time modules:

import cal_header as ch

from util import Time

We want to use the dla_censql2table function to create a table of the delays. This is done by:

ch.dla_cen2sqltable(t=Time('yyyy-mm-dd hh:mm'),acc=False)

where yyyy-mm-d hh:mm is the date and time for the last delays (it will look for the first time prior to that specified) and acc=False tells the function not to send values to the ACC.

For example to view delays from the 3rd of November 2020 (or the first date prior to this) you would issue the command:

ch.dla_cen2sqltable(Time('2020-11-01 01:00'),acc=False)

Note that this writes the data to a text file: /tmp/delay_centers.txt

To view this file issue the command:

!cat /tmp/delay_centers.txt

To view an older file, use the ch.dla_cen2sqltable with an earlier time and reissue the cat command. The delays that were changed should be different for the given antennas. Note that if antenna 1 delays were changed, this change will be shown in antenna 14.

Trouble Shooting

General Problems

Temperature Controller

Occassionally a front end temperature will show an error. This is usually because the temperature controller that reports the temperature has locked up. To fix this type the following command in the Schedule window:

tec$bc ant#

where # is the antenna number that needs the temperature controller rebooted.

To clear the temperature error history (that updates every minute) issue the command:

tec$sc ant#

If the above steps do not work, the following command can be issued:

$pcycle fem ant#

This will recycle power to the front end controller. If this does not work you can check the log file from a helios terminal by typing

tail -20 /tmp/schedule.log

A display similar to that below will be shown:

2020-08-17 13:56:03.002 Ant4 attempt #1 to login.
2020-08-17 13:56:03.002 Ant4 Login failed with status code: 401
2020-08-17 13:56:04.002 Ant4 attempt #2 to login.
2020-08-17 13:56:04.002 Ant4 Login successful.
2020-08-17 13:56:04.002 Ant4 Fronttend now off
2020-08-17 13:56:19.002 Ant4 attempt #1 to login.
2020-08-17 13:56:19.002 Ant4 Login successful.
2020-08-17 13:56:20.002 Ant4 Frontend now on

Note that the second line says that the login failed. It will attempt a login in three times. If this fails you can attempt a recycle to issue the command again. If this does not work the front end needs to be reset at the auxiliary box at the antenna. This is done by switching off the two breakers, waiting 10 seconds and then switching them back on. The breakers are shown below.

Packet Loss on DPP

If the packet monitoring program on the DPP computer shows lines similar to:

2020-05-28 15:43:42.001 0.0 0.0

then data packets are being lost. To fix this problem follow the following steps:

Terminate the fix_packets() program by pressing [CTRL] C.

Type !nano /home/user/test_svn/shell_scripts/SMP_AFFINITY.sh

Comment out the current cpu's and uncomment the following ones. Exit nano.

Type !./home/user/test_svn/shell_scripts/SMP_AFFINITY.sh to restart the SMP_AFFINITY.sh program.

Restart the fix_packets program by entering dpp.fix_packets(cpu=[C1,C2]) where C1 and C2 are the two CPUs selected in the SMP_AFFINITY.sh script.

Occasionally the glitch will occur after the above procedure has been performed. In this case the DPP will need to be rebooted. Follow the DPP reboot instructions here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#DPP_3

Data Recording on the DPP has Stopped

To Check if the DPP is recording data, from a DPP terminal issue the command:

ls /data1/IDB |tail

This will display the last 10 IDB files that were created. They should have a time stamp about 10 minutes apart and the last file should only be 10 minutes or so old.

If current IDB files are not displayed then firstly check to see if The STATE is On in the phase tracking section of the stateframe is displayed as shown below.

If the state is off then simply click Stop on the schedule window and then click on Go. The state should return to On. Check to see if the DPP is creating files after about a minute.

If the DPP is still not recording data or the State was already on then follow the instruction below.

Log in to the DPP computer (user@dpp).

Enter top . A display similar to that below should be shown.

Look for dppxmp under “command” column (shown here in the red box). If this is present not the PID# (shown here in the green box. Note that this number will change if dppxmp is restarted).

Press 'q' to exit the top program

If dppxmp is present issue the command:

kill pid#

where PID# is the number noted down previously. This will stop the dppxmp program.

Enter the command rmlock

Check again to see if data recording has recovered

Type “rmlock" on DPP terminal. Check if the data recording has recovered by sending ls /data1/IDB |tail

Restarting the Schedule

If the schedule needs to be restarted then follow the steps below:

Click Stop and then Clear.

Click Today and then Go.

Pipeline Problems

If there is no data showing at http://www.ovsa.njit.edu/browser/ then something has gone wrong with the pipeline processing. Below is a image of what the webpage looks like when an error has occurred.

Clicking on the Newest button will refresh the page and upload the latest data. If this does not correct the problem take a note of the date in the web address of the page. Then click on 'Today's Data'. A list of fits files should appear as below:

To check to see why the images are not visible, the log files can be inspected. Log on to pipeline and type the command:

crontab -l

The image below show a part of the output. The portion in the red box is the script that generates the full disk image.

To view the programs run by this script issue the command:

cat /common/python/suncasa/shellScript/pipeline_fdimg.bash

and an output similar to that below should be produced:

Note that the are three log files produced: /tmp/pipeline.log, /tmp/pipeline_plt.log and /tmp/pipeline_compress.log.

By inspecting these files you should be able to determine at what point an error occurred. To view these files type:

cat [logfile]

where [logfile] is the name of the logfile you wish to view.

Bad Front End Attenuation Values

The front end power values on both the horizontal and vertical channels should be between about +1 dBm and +4 dBm. These can be found on the stateframe display as shown below.

If the power deviates significantly from the 1 to 3 dBm range (unless there is a burst in progress or a gaincal test being performed) then the attenuations can be adjusted by issuing the following commands from the Schedule window:

hattn first second ant#

where first and second are the attenuations of the first and second attenuators for the horizontal polarization and ant# is the antenna number. Note that first should always be zero.

Similarly to adjust the vertical attenuators:

vattn first second ant#

Note that no warning will be shown if the power deviates significantly from the acceptable range.

Warning on Power and Attenuation

On the state frame display, occasionally an antenna will turn yellow (warning). This will usually be because the 1st attenuator value is not zero. This will occur during a gaincaltest or a burst. The value should be even. If the value is 9, then the CRIO file will most likely need to be be reloaded.

Follow the instructions here http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Reloading_the_crio.ini_File to relaod the crio.ini file. If Antenna 11's CRIO is to be updated then see the special case here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Special_Case:_Antenna_11

Schedule Commands not Working

If schedule commands are not working then the ACC may need to be restarted.

To determine if the ACC needs restarting issue a command from Schedule. At the top right of the Stateframe the 'Task' should be updated. if it is not then follow the following procedure:

Go to the Windows computer and from the Quicklaunch menu select 'EOVSA E&C'.

Once running click on 'Reset ACC' and wait about 10 seconds.

Queue Overflow Error

On the Stateframe display, a 'Queue Overflow' Error may occur under the Frequency Tuning bar. To correct this issue type the following command into the Raw Command box on the schedule:

lo1a-reboot

Stop and then restart the schedule.

Older Antennas Not Tracking

The older antennas (9-11 and 13) will occasionally seem to stop tracking. This is usually because it has lost where it is pointing. It will move incredibly slowly until it reaches its limit before returning to regular tracking. This process cant take up to about 20 minutes. If it has not resolved it tracking problems in this time, proceed with normal trouble shooting.

Antenna Trouble Shooting

If an antenna in the State-frame Antenna Tracking section is highlighted red, go through the following list.

Reloading the track table

Reload the track table as by issuing the following commands from the schedule

tracktable sun_tab.radec ant#

track ant#

Note that omitting the ant# will resend the commands to all antennas. You can specify more that on antenna.

Stowing the antennas

Attempt to stow the antenna by issuing:

stow ant#

If there are still errors, from the state-frame display, click on the antenna display and then click on the tab for the appropriate antenna.

Antenna at its limits (New Antennas)

If a limit is highlighted you will need to move the antenna from its limit. Firstly issue the command:

runmode 2 ant#

Issue the appropriate command according to the following table:

Co-ord	Limit	Command
Azimuth	High	azimuthvelocity -5000 ant6
	Low	azimuthvelocity 5000 ant6
Elevation	High	elevationvelocity -5000 ant3
	Low	elevationvelocity 5000 ant6

Once the limit indicator goes green issue the command:

azimuthvelocity 0 ant# or elevationvelocity 0 ant#

Return the antenna to tracking:

track ant#

Antenna at its limits (Old Antennas)

Initially try to stow the antenna:

stow ant#

It may take a while for the antenna to stow.

If the antenna fails to stow, it will need to be manually driven off its limit as described here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Moving_the_Old_Antennas_Manually and then try to stow the antenna.

Once stowed, return the antenna to tracking:

track ant#

Bad values for antenna position

If any of the following is true:

on antennas 1 to 8 or 12 the azimuth is outside the range of 0 to 360 degrees,

on antennas 1 to 8 or 12 the elevation is outside the range of 0 to 90 degrees,

on antennas 9 to 11 or 13 the hour angle is outside the range of -180 to 180 degrees,

on antennas 9 to 11 or 13 the declination is outside the range of -90 to 90 degrees,

you can try to issue the following commands:

stop ant#

sync-motion ant#

The antenna should then go to its stow position. Once it is stowed send the commands:

tracktable sun_tab.radec ant#

track ant#

The antenna should resume tracking.

Rebooting the antenna

On the newer antennas (antennas 1 to 8 and 12), access the window computer, click on then Quick launch icon at bottom right and then click on the antenna that is problematic.

A screen window to that below will come up. Ensure that the Enable box at the lower right is checked.

One or both of the Permit boxes may be red.

Click on the Reboot Drives button. and wait about 10 seconds. The click on the Drives On Button. It should turn Green. Click on the Operate Button. If this does not turn green you may need to reboot the edge13b server.

Click Exit.

Rebooting the antenna interface on the edge13b server

Whenever an antenna link seems to be down (indicated by the lack of a blinking link light on the edge13b router), note the port that is down (The antenna is labelled on each fibre pair).

Refer to it as

fa0/#

where # is the number of the port. E.g., if port 1 is down, it will be referred to as fa0/1.

From Helios, telnet to the switch:

telnet edge13b.admin.pvt

Enter the the password when prompted.

Enter the command:

show inter fa0/1

The first line will show the status. If it is listed as Connected then nothing needs to be done. Enter:

quit

to exit from telnet

If the connection is listed as err-disabled then proceed with the following steps. At the telnet terminal issue the commands:

enable

and enter the password.

configure ter

interface fa0/#

where # is the port number.

shutdown

no shutdown

end

You can then reissue the show inter command to see if the interface is running. The port light should also be blinking on the server.

Issue then command

quit

to exit telnet.

Resetting antenna parameters

If the antenna still will not start goto the following webpage:

http://ant#.solar.pvt

Click Login and then enter the username and password.

Click Parameters and the change Option 10.00 to 1070 and Option 10.38 to 100. Click on change.

Then click on logout.

After the above procedures are completed you will need to reload the track table. This is done from the State Frame display using the following commands (again ensure that ant5 is replaced with the appropriate antenna):

tracktable pcal_tab.radec ant5

track ant5

Antenna Communications Failure

If an antenna is highlighted in the Communications section of the State-Frame display then the following procedure can be used to correct it.

From the Schedule on Helios enter the command

$pcycle crio ant2

and wait about 2 minutes.

From the Schedule on Helios issue the command

sync ant#

where # is the antenna number.

If this does not work, then try the command:

sync-motion ant#

If this does not resolve the communications problem, then proceed with the procedure below.

Access the win1 computer.

Click on Start then select NI Max.

Goto Remote Systems and then select the CRIO that is not communicating.

Click the System Settings tab at the bottom of the window. Status should be listed as Connected - Running

Click on Refresh to reconnect.

The CRIO will have an IP address in the range of 192.168.24.41 to 192.168.24.53. If it is not working this IP address will be strange. Click Restart.

From the Schedule on Helios issue the command

sync ant#

where # is the antenna number.

If the communications are still not restored, you will need open Labview.

If it is not already open, Click Start then NI Labview 2015.

The LabView start screen will appear after a few seconds.

Click on the Operate menu option and then select Debug Application or Shared Library.

Once the new window opens type in the name of the CRIO in the Machine Name or IP Address box. Then click Refresh. The file startup.rtexe should appear. Click on Connect.

Note that if a message similar to 'No debuggable application found' is displayed you may need to click on refresh several times. You may need to reset the CRIO at the viking unit (http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Restarting_the_Viking_Unit). When it first reboots it will say it cannot find any debuggable applications, and then at some point it comes back with startup.exe. But if you wait too long it ends up in software safe mode and you essentially have to start over. Just click repeatedly until you see startup.exe, wait for a few seconds and then hit the button to load the debug software.

In the new window that opens check that the time is updating and that the time is correct an updating and that the Run index is updating. The Exec States should all be set to Run.

At the top of the screen click on the Stop Icon and then the Go icon (the arrow).

From the Schedule on Helios issue the command

sync ant#

or

sync-motion ant#

where # is the antenna number.

To exit from the Debug session, right click on an empty part of the form which will bring up Remote Debugging. Select Quit Debug Session.

Once again try sync ant#

If this does not work try sync-motion ant#

If all else fails, restart the CRIO from the Viking unit.

Restarting the Viking Unit

The Viking units control power to the various systems (Antenna, Front End and CRIO). To recycle power to a particular system, goto the webpage

http://vik#.solar.pvt

where # is the antenna number. Enter the username and password. The following screen should appear:

Note that when the relays are off, power is being supplied to the indicated system. In the image above, all relays are off so each system is powered.

To recycle power to a particular system, click on the circle next to the system. If the circle was red initially, it will go green, and vice-versa. If you are turning the relay on then wait ten seconds before turning the relay off again. Ensure that the relay is Off (red) before logging out.

Receiver Trouble Shooting and Repair

Positioning the Antenna

If working on a newer antenna follow the instructions here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Moving_the_New_Antennas_Manually

If working on an older antenna follow the instructions here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Moving_the_Old_Antennas_Manually

Removing the Receiver

Switch off the power to the receiver at the auxiliary box by pulling down the two breakers.

Below is an image of the tools required for removing the receiver.

Whenever touching the receiver ensure that the Anti-static strap is worn and grounded!

Loosen the nuts on bolts that attach the receiver to the feed almost the entire way but leave them on the bolts. The feed can be pressed forward to give enough room to detach the SMA cables.

Carefully disconnect the SMA cables from the feed. Note the position of the horizontal and vertical polarisation cables.

Place caps on the end of the SMA cables.

Carefully remove all of the cables from the back of the receiver.

Place a cap on the fibre optic port and the fibre optic cable.

Remove the nuts entirely. Remove the 3 allen bolts.

The receiver can now be taken into the lab for testing or repair.

Opening the Receiver

Ensure you are grounded with the anti-static strap!

Remove removing the bolts from the stoppers, but keep the stoppers in place. Once removed the triangular front mounting bracket will slide off.

Recessed into the case are 6 screws that need to be undone. the location of 4 of these are shown in the image below. Loosen the screws using a Phillips head screwdriver. The will no come all the way out.

Place the receiver on it side and cautiously partially open the case. The SMA cable connecting the two halves together needs to be disconnected before the case can be fully opened. Carefully loosen the SMA cable on the Horizontal polarisation side and disconnect the cable.

Cautiously fully open the case. There is a full replica of the Auxiliary box under the bench. If testing is to be performed then this will need to be connected. Note that the CRIO out at the antenna will also need to be powered on. Ensuring that the power is off, connect the cables to the receiver.

Whenever disconnecting or connecting components to the receiver, ensure that the power is off!

Reconnect the SMA cable between the two halves of the receivers.

Connecting the Power Meter

If required for testing, the power meter will need to be connected to the either the horizontal or vertical polarisation. The power meter is normally kept in the correlator room.

Connect the power meter to the appropriate polarisation as shown below.

Replacing the LNA

Ensure that power is off to the receiver. The LNA locations is shown below. It is in the same place on the other half of the receiver for the other polarisation.

Carefully unsolder the black and red wires connected to the LNA.

Undo the nuts on either side of the LNA. To undo they will move towards the LNA.

Remove the allen bolts fixing the LNA to the case.

Carefully remove the LNA.

Add some silicone heat sink compound to the base of the new LNA.

Repeat the process in reverse to install the new LNA.

Antenna A Power Outage and Reset

Recycling Power to Antenna A

To recycle power to Antenna A goto http://pduanta.solar.pvt and enter the username and password.

There will be eight devices that can be chosen from. Click Cycle to recycle the power to the appropriate option.

You may need to follow the procedure below to bring Antenna A back to operation.

Recovering Antenna A from a Power Outage

On the Stateframe display click on the CryoRX tab. This will bring up the display shown below.

Recycling Power to the FEM

Login to the feanta machine as follows:

ssh antctl@feanta

If the FEM Outlets on the CryoRX window are marked as Off then from a Helios terminal:

Then type in the command:

/home/antctl/starburst/femTools/src/starburstControl start

Otherwise recycle the power to the FEM as follows:

/home/antctl/starburst/femTools/src/starburstControl stop

Wait about 10 seconds and the issue the start command:

/home/antctl/starburst/femTools/src/starburstControl start

Once done type in:

exit

Recycling Power to the Brick

If faults are showing on the Motor section (4th green line down) the Brick power will need to be recycled. To do this issue the following commands from the Schedule Window:

outlet 3 off ant14

outlet 3 on ant14

Restarting the LNAs

From a Helios terminal:

ssh root@lna14.solar.pvt

If you get an error "No route to host" you will need to restart the computer at the 27m antenna by issuing the commands below from the Schedule Window:

outlet 6 off ant14

outlet 6 on ant14

Then issue the command from the terminal:

./killAll python

Then press the up arrow until the most recent python command is found and press enter. Text will scroll up continuously Just type exit even with the text scrolling to exit the ssh session.

From Schedule issue the command:

$lna-init ant14

The Drain Voltage for Hi H will be incorrect (~.61 when it should be 1.2). To correct this, from Schedule issue:

lna drain hh 2.4 ant14

Aligning the Receivers

The receivers must be properly aligned. From Schedule send the command:

frm-home ant14

It will take a few minutes for the FRM to find its home position. Once it has send the command:

frm-set-pa 0 ant14

to zero the positions.

Set the Hi Receiver:

rx-select hi ant14

Ant A Maths Error

Goto the webpage

style="font-family:courier">https://ant14.solar.pvt

. You should get to the screen as shown below:

Ensure that on the left under the SP image that there is a green 'Ready' message.

If there is a Maths Error then follow the following procedure.

Click on the Login tab and enter the username and password.

Click on the Parameter tab to see the following screen:

Click on view next to #10.

At the next screen click on view next to Item #10.00:

In the Update Value box enter 1070 and click on Change

Click on the arrow near the top of the page to return to menu 10.

Click on View next to Item #10.38

In the Update Value box enter 100 and click on Change

Site Wide Power Outage

Most of the procedures to recover from a site wide power outage are covered above. Below are the the steps required in order.

27m Antenna (Antenna A)

Refer to the procedure here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Recovering_Antenna_A_from_a_Power_Outage

2m Antennas (Antennas 1-13)

For any antennas that have a communications failure (highlighted in red in the communications section of the Stateframe Display) follow the procedures here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Antenna_Communications_Failure

It may take some time for the CRIOs to restart. If after 20 minutes a CRIO has not restarted, redo the procedure.

Stop the schedule by clicking on the stop button.

Stow the antennas by issuing the command

stow

If some of the antennas did not stow the refer to Antenna trouble shooting: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Antenna_Trouble_Shooting

If an antenna is highlighted in the Power and Attenuation section on the stateframe display then you may need to reload the .ini file to the CRIO as described here: http://www.ovsa.njit.edu/wiki/index.php/Owen%27s_Notes#Warning_on_Power_and_Attenuation

Schedule Command Description

NOTE: All Schedule commands accept as optional arguments a list of antennas to which the command should selectively sent. Accepted syntaxes are:

Command ant1 ant2 ant3

Command ant1-3

Command subarray1

Command subarray2

If no antenna list is indicated, the command is sent to all antennas currently in subarray1.

These commands can be found in the ~/Dropbox/EOVSA\ M\&C/EOVSA\ Schedule\ Commands.docx and ~/Dropbox/OVSA\ Expansion/Design/Antennas/Ant_A\&B_Commands.pdf files.

Commands handled directly by ACC

ABORT

Aborts all commands sent to ACC but not yet executed

CLEAROFF [antenna list]

Sets all AzEl or RaDec offsets to zero

DCMATTN HPOLattn VPOLattn [antenna list]

Indicates the VPOL and HPOL attenuations for the DCM modules indicated in the list. The attenuations are applied on the next second start.

Warning: The command is executed only if the DCM modules are in manual mode (see the DCMAUTO-ON and DCMAUTO-OFF command below)

DCMAUTO-OFF [antenna list]

Sets the DCM modules in the manual attenuation mode. This mode allows the attenuations requested by the DCMATTN to be applied.

DCMAUTO-ON [antenna list]

Sets the DCM modules in the automatic attenuation mode. This mode ignores the manual DCMATTN commands and applies the attenuation set through the DCMOFFSET mechanism.

DCMOFFSET inc1 inc2…..inc50

Sets the 50 DCM attenuation offset increments to be added to the current list of DCM offset attenuations to be cycled during each second, starting with the next second boundary. The command expects up to 50 arguments. If fewer arguments are provided, the existing set is repeated in the same sequence until all 50 slots are filled. This command is only intended for testing. During normal operations, the DCM offsets states are requested by the DPP computer when needed.

Warning: The command is executed only if the DCM modules are in auto mode (see the DCMAUTO-ON and DCMAUTO-OFF command below).

NOTE: The DCM offsets are changed on the edge of the next second after a TCP/IP message is received by ACC at the TCP.dpp.port (default port: 6344-set in ACC.ini file ). The message is expected to be a fixed length binary string having the following format: 50 32bit integers (I32)

DCMOFFSET-CLEAR

Clears the current list of 50 DCM offset attenuations that are applied when DCMAUTO-ON is set.

Warning: This command clears the offset attenuations for all DCM modules.

DCMTABLE [antennalist] filename

Request the antenna list portion of a specific DCM base attenuation table to be uploaded on the respective DCM modules. The DCM table must have 30 columns and 50 rows, each pair of adjacent columns listing the HPOL-VPOL pairs for each of the 50 slots of a tuning sequence. The DCM table text file must be located in the ACC “c:\parm” directory and its default name is “DCM.txt”.

The DCM/table_timeout key in the ACC.ini files have to be manually edited in order to modify the default timeout (ms) of the DCM modules when a table is broadcasted.

The actual DCM attenuations for a given tuning slot are calculated based on the following formula

DCM_atten[pol,tuning_index]=DCM_base_atten[pol,tuning_index]+DCM_offset[broadcasted]

DPPOFFSET-OFF

Ignores DCM offset attenuation recommendations sent by DPP but still allows setting the offsets via the DCMOFFSET schedule command.

DPPOFFSET-ON

Applies DCM offset attenuation recommendations sent by DPP but still allows setting the offsets via the DCMOFFSET schedule command.

NOTE: This is the default behavior of the system after each hardware ACC reboot or after a REBOOT or RESET command is sent via TCP/IP to the emergency 6543 acc.solar.ini port

FEM-INIT

Initialize all front-end module base attenuations with the values read from the ACC initialization file: “c:\ni-rt\startup\acc.ini”.

FEMATTN level [antennalist]

Manually selects the front-end attenuation levels for the antennas indicated in the list to be applied on the next second boundary. The attenuation level selects the corresponding entry in the “c:\parms\FEMATTN.txt” attenuation FEM table located on each the cRIOs.

The attenuation level set by this command is saved in the [FEM AGC] section of the “c:\ni-rt\startup\crio.ini” and it is automatically applied when the module is restarted.

Warning: The attenuation level setting may be overwritten by the AGC loop, if active.

The format of the table is as shown below (column heads are added here for clarity)

FEMATTN level	First attenuation	Second attenuation	Total attenuation
0	0	0	0
1	0	3	3
2	0	6	6
3	0	9	9
4	0	12	12
5	0	15	15
6	0	18	18
7	9	12	21
8	9	15	24
9	9	18	27
10	9	21	30
11	9	24	33
12	9	27	36
13	18	21	39
14	18	24	42
15	31	31	62

The actual FEM attenuations applied as the result of a FEMATTN command are given by

HPOL_ATTENUATION=HPOL_BASE+FEMATTN.txt(level)

VPOL_ATTENUATION=VPOL_BASE+FEMTTTN.txt(level)

Where Attenuation, Base, and FEMATTN.txt(level) are two elements vectors indicating the settings for the First and Second attenuators.

HATTN first second [antennalist]

Sets the FEM HPOL_BASE first and second attenuators for the antennas in the list, and overwrites the corresponding sections in the global “c:\ni-rt\startup\acc.ini” file, as well as in the local “c:\ni-rt\startup\crio.ini” files.

VATTN first second [antennalist]

Sets the FEM VPOL_BASE first and second attenuators for the antennas in the list, and overwrites the corresponding sections in the global “c:\ni-rt\startup\acc.ini” file, as well as in the local “c:\ni-rt\startup\crio.ini” files.

FSEQ-FILE filename

Tells ACC to upload to the Hittite synthesizer the frequency sequence located at “c:\parm\filename”.

FSEQ-INIT

Initializes the Hittite synthesizer according to the settings defined in the [LO Configuration] section of the “c:\ni-rt\startup\acc.ini” file.

FSEQ-ON

Starts the tuning sequence

FSEQ-OFF

Stops the tuning sequence

FSEQ-SCRIPT filename

Sends, line by line, to the Hittite synthesizer the command sequence listed in the “c:\parm\filename” script file.

LO1A-REBOOT

Commands the PDU controller to recycle the power on the LO1A Hittite synthesizer

LO1B-REBOOT

Not implemented yet

LO1A-WRITE command

Sends the specified command to the LO1A Hittite synthesizer

LO1B-WRITE command

Sends the specified command to the LO1B Hittite synthesizer

SERVICE [antennalist]

Takes the antennas in the list out of subarray1 or subarray2 and drives them to the service position.

Warning: Once in service position, the serviced antenna should be put in local operation mode in order to avoid it being controlled by a subsequent schedule command.

SUBARRAY1 antennalist

Puts all antennas in the list into subarray1. All antennas than are not listed, but currently in subarray1, are taken out from subarray1.

NOTE: As of Dec 03 2015, this command no longer switches by default the LO connector to an alternative source. To force the LO connector to switch switches, one must use the new explicit command SUBARRAY1_SWITCH antennalist

SUBARRAY2 antennalist

Puts all antennas in the list, which are not already in subarray1, into subarray2. In order to move a given antenna from subarray1 to subarray2, one should first redefine subarray1.

NOTE: As of Dec 03 2015, this command no longer switches by default the LO connector to an alternative source. To force the LO connector to switch switches, one must use the new explicit command SUBARRAY2_SWITCH antennalist

SYNC [antennalist]

Restarts the real-time executable on all cRIOs corresponding to the antenna list.

NOTE: Unlike the general convention, not providing a list of antennas commands all cRIOs to restarts their execution.

TRACKTABLE filename [antennalist]

Uploads to all antennas in the list the tracking table located at “c:\parm\filename”

TRAJ-FILE filename [antennalist]

Uploads to all cRIOs corresponding to antennas in the list the trajectory file table located at “c:\parm\filename”

UNLISTEDCOMMAND [antennalist]

Any command not listed above is sent for local execution to all cRIOs corresponding to antennas in the list.

Commands handled by cRIOs

NOTE: These commands are selectively sent by ACC to all antennas in the optional argument antennalist, or, if not present, to all antennas in subarray1

AGC active [low [high [samples]]]

Activates (1) or deactivates (0) the front end automatic gain control loop, which adjust the FEMATTN level when both of the HPOL and VPOL voltages, averaged over the given number of consecutive samples , cross the low and high voltage limits. Missing parameter values are uploaded from the [FEM AGC] section of the local “c:\ni-rt\startup\crio.ini” file. All parameters provided when an AGC command is issued are saved in the same file for further use.

NOTE: At startup, the AGC loop is activated or not according with the latest active value written in the initialization file.

AZELOFF azoff eloff

Sets the Azimuth and Elevation offsets indicated in the argument list

RADECOFF raoff decoff

Sets the RA and DEC offsets indicated in the argument list

FLUSH

Flush the existing tracktable from the associated antenna controller

ND-ON and ND-OFF

Set on/off the local noise diode.

POSITION azimuth elevation

Requests a given azel position

RESTART

Reboots the cRIO

DRIVE-RESET

Resets Antenna controller

STOP

Stops the antenna

STOW

Stows the antenna

TRACK

Sets antenna in track mode

TEC-LOG

Dumps all TEC register values to the local log file “c:\tec.txt” for debugging purposes

TEC-INIT

Initializes the TEC controller with register values hard-coded in the crio code. These were read from a working controller at some time in the past.

TEC$BC

Reboot the TEC controller to recover from a bad or stuck state.

TEC$SC

Clear the error status of the TEC controller. Does nothing other than this, and is purely asthetic.

TRAJ-ON and TRAJ-OFF

Starts/Ends the execution of the trajectory script

NOTE: At startup, the WINDSCRAM loop is activated or not according with the latest active written in the initialization file.

BSCRAM-ON

Activate the BRIGHTSCRAM Monitor

BSCRAM-OFF

Deactivate the BRIGHTSCRAM Monitor

BSCRAM-CLEAR

Takes the antenna out of the BRIGHTSCRAM Active State independently of the brightness sensor state.

If the BRIGHTSCRAM Monitor’s operation mode is set to STOW, clearing the BRIGHSCRAM ACTIVE state does not have any effect on the mechanical state of the antenna. If the BRIGHTSCRAM Monitor’s operation mode is set to OFFSET, clearing the BRIGHSCRAM ACTIVE state results in removing the AZEL offsets

Note: If the BRIGHTSCRAM Monitor is active, the antenna may get back in the Active State depending on the state of the brightness sensor.

BSCRAM-SET

Force the antenna into the BRIGHTSCRAM Active State independently of the brightness sensor state.

BSCRAM-WAIT seconds

Sets the BRIGHTSCRAM Monitor waiting time before clearing the BRIGHSCRAM ACTIVE state.

Note: This setting is saved in the crio.ini file and remains persistent until changed

BSCRAM-OFFSET

Sets the BRIGHTSCRAM operation mode to stow the antenna when in active state.

Note: This setting is saved in the crio.ini file and remains persistent until changed

BSCRAM-OFFSET

Sets the BRIGHTSCRAM operation mode to apply offsets

Note: This setting is saved in the crio.ini file and remains persistent until changed

BSCRAM-AZELOFF azoff eloff

Sets the Azimuth and Elevation Offsets (degrees) for the BRIGHTSCRAM offset operation mode.

Note: This setting is saved in the crio.ini file and remains persistent until changed. Default values re AZOFF=0 and ELOFF=10

Warning: No action is taken if less than 2 arguments are provided

WSCRAM-ON

Activate the WINDSCRAM Monitor

WSCRAM-OFF

Deactivate the WINDSCRAM Monitor

WSCRAM-LIMIT value

Sets the wind speed threshold.

Note: This setting is saved in the crio.ini file and remains persistent across reboots until changed.

WSCRAM-WAIT seconds

Sets the WINDSCRAM Monitor waiting time between checking the wind speed.

Note: This setting is saved in the crio.ini file and remains persistent across reboots until changed.

UpdateElevationDiagnostics 1

Requests an update of the elevation trip register.

UpdateAzimuthDiagnostics 1

Requests an update of the azimuth trip register.

REGWRITE address value

Writes to a specific (new type) antenna register address the specified value.  This command is ignored by the old type antennas.

For example,

REGWRITE 23386 2 [antlist]

may be used to set the controller in velocity mode. Note that the velocity rate registers must also be set to control the velocity of motion. The appropriate registers are:

AzimuthVelocity, or 23585

ElevationVelocity, or 23605

Alternatively, one can write to a specific register address by directly sending a command formed by the name of the register (as defined in the M&C document Controller_registers.xlsx) followed the desired value to be written.

For example one may alternatively set the controller in velocity mode by issuing the command

RUNMODE 2 [antlist]

A possible need for velocity mode is to drive an antenna off a hard limit. For safety, the only way to do that is to set the controller in velocity mode and provide a velocity in the direction to drive off the limit. For example, if Ant 6 is on an azimuth low hard limit, one can drive it off by the following sequence:

RUNMODE 2 ANT6

AZIMUTHVELOCITY 5000 ANT6

Wait for hard limit to clear.

AZIMUTHVELOCITY 0 ANT6

TRACK ANT6

Antenna A (Antenna 14) Commands

Note: Only antenna A is currently in use. In the commands below the value of [ant] is ant14, the value of [rx] is hi or lo (for the receiver), and the value of [lna] can be lv, lh, hv or hh.

FRM-HOME [ant]

Homes the x-axis (receiver select) and position angle axis.

FRM-RX-SEL [rx] [ant]

Move x and z axes to select receiver.

FRM-SET-PA [pa] [ant]

Set the position angle, [pa] for polarization. Pointing to the south is 0 degrees. The PA can range from -90 to 90 degrees.

FRM-X-OFFSET [xoff] [ant]

Offset in x relative to the selected reciever. In effect until a new value is set or the Geo Brick is reset.

FRM-Z-OFFSET [zoff] [ant]

Offset in z relative to the selected reciever. In effect until a new value is set or the Geo Brick is reset.

FRM-ABS-X [xpos] [ant]

Set to an absolute x position. Not used.

FRM-ABS-Z [zpos] [ant]

Set to an absolute z position. Not used.

FRM-KILL [ant]

Kill all motors. They will be unpowered.

FRM-ENABLE [ant]

Enable all motors after a kill.

OUTLET [out] [state] [ant]

Turns the selected outlet on or off. [out] is a value from 1 to 8 and [state] is either on or off.

ND-ON [ant]

Turns the noise diode on via the power strip.

ND-OFF [ant]

Turns the noise diode off via the power strip.

RX-SELECT [rx] [ant]

Moves the requested receiver to focus and sets the RF switch appropriately.

LNA-GATE1 [lna] [vg1] [ant]

Sets the first gate bias on the selected amplifier to the request voltage ([vg1] in range of -2.0 to 2.0). Will only operate if that amplifier is enabled.

LNA-GATE2 [lna] [vg2] [ant]

Sets the second gate bias on the selected amplifier to the request voltage ([vg2] in range of -2.0 to 2.0). Will only operate if that amplifier is enabled.

LNA-DRAIN [lna] [vd] [ant]

Sets the drain bias ([vd] in range 0.0 to 2.0) on the selected amplifier. Will only operate if that amplifier is enabled.

LNA-ENABLE [lna] [state] [ant]

Turn the requested amplifier on or off.

Local Commands

These commands are all executed locally. Any command with a '$' in front of it is a local command.

$MK_TABLES [fname] [source]

Creates the tracktable for the named [source], writes it to the filename given by [fname], for example, sun_tab.radec, pcal_tab.radec, or geosat_tab and stores it in the /parm directory on the ACC. This file can then be sent to the antennas via the TRACKTABLE command. This must be performed at the start of each scan in order for the antennas to track the source.

$FEM_INIT <antlist>

Invokes a sequence of commands to (1) read the current front-end power levels, (2) calculate any attenuation change needed to adjust the power levels to the range 1-4 dBm, and (3) send the attenuation adjustments to the antennas. The result is that the front end power levels are adjusted to this range. It acts on the antennas given in the antenna list <antlist>, or on all antennas in the current subarray if <antlist> is omitted. This command changes the gain calibration, so should only be done prior to the start of a day's observations if possible. This is part of the System Gain Cal procedure.

$PLUSDELAY [nsec]

Relay specific command that adds given delay argument (in nanoseconds) to the Ant 14 low frequency receiver. Delays both X and Y in the current delay center table. This writes the changed record as a new record in the SQL database.

$CAPTURE-1S <stem>

Capture 1 second of data on the dpp. <stem> is an optional string to add to the end of the capture filename. This will take a few seconds to complete.

$WSCRAM-LIMIT [limit]

Update the 27-m windscram limit. the limit is in miles per hour.

$SCAN-START <NODATA>

Starts data collection. Command by itself is normal scan start, while $SCAN-START NODATA means set up scan, but do not take data. The NODATA option is used for some calibrations.

$SCAN-RESTART

This command is for restarting a scan with the same setup as the previously running scan, where only the scan state must be turned on.

$SCAN-STOP

This command stops a scan.

$PA-SWEEP [PA] [RATE]

Rotate 27-m focus rotation mechanism to sweep through a given angle range (does nothing if PA adjustment routine is already running). The angle is swept from -[PA] to [PA] at rate of 1-degree-per-[RATE] (s)

$PA-TRACK ant14 [CROSSED]

Track 27-m focus rotation mechanism to correct for parallactic angle of given antenna (does nothing if antenna not correctly specified). [CROSSED] should have value TRUE or FALSE.

$PA-STOP

Aborts a running position-angle sequence (started by $PA-SWEEP or $PA-TRACK commands) that is controlling the Antenna 14 focus rotation mechanism.

$TRIPS

Send commands to update antenna trip information.

$DLASWEEP [ant] [dla] [dlastop] <pol>

Starts a relative delay sweep on antenna [ant] (an integer, i.e. ant1), <pol> ('X', 'Y', or if omitted, both) ranging from delay step [dla] to [dlastop]. This will sweep the delay by 1 step per second for delays from [dla] to [dlastop] relative to the current nominal delay. For example, $DLASWEEP 2 5 10 X will sweep the X delay on ant2 by adding 5 steps up to 10 steps, one step per second. The start and/or stop delays can be negative, but [dla] must be less than [dlastop]. Note, if [ant] is 0, the delay of the specified polarization is swept for all antennas. Used for some total power tests.

$WAIT [secs]

Waits the number of specified seconds before executing the next command.

$PCYCLE [dev] [antlist]

Cycle the power of some device (antenna controller, fem, or crio). If [dev] is omitted then the antenna controller is cycled. The options for [dev] are ANT, FEM or CRIO.

$KATADC_GET

Get the standard deviation for each KatADC (assumes frequency tuning is static).

Note, takes about 0.2 s for each active ROACH.

$REWIND

Get date of first line of current schedule and increment by 1 day, then auto-generate a new schedule.

$LNA-INIT

Get LNA_settings.txt file from ACC and send the series of commands needed to set the LNA voltages.

$SUBARRAY' [antlist]

Run the SUBARRRAY1 command if this is the master schedule, otherwise run the SUBARRAY2 command.

Useful Tools and Utilities

Screen Command

On linux systems, the screen command allows a program to be run in the background. This is particularly useful for processes that take a long time to complete. In addition it will display output when the screen is activated.

To start a screen session, open up a terminal and issue the command:

screen

Text will appear and you will be prompted to press space or return. Just press return. Type in the command that you want to be performed. The process will then start. You can exit the screen by pressing <CRTL> A D The command will still be running. To re-open the screen at any time simply type:

screen -r

You can have more than one screen running at a time. In this case it is advisable to give a screen a name. For example:

screen -S schedule

will assing the name 'schedule' to the screen. To access this screen after it has been closed, use the command:

screen -r schedule

To list the currently active screens use the command:

screen -ls

To terminate a program running in a screen, from within the screen press <CRTL> Z

In the example below the ping command was stopped and fives the following output:

The [1] is the job number. To completely terminate it issue the command:

kill %1

To close a screen session type exit

df Command

On linux systems, the df command lists all of the file systems currently mounted on computer.

The command is simply:

df

More usefully, issuing the command:

df -h

will display the file sizes in 'human readable' format. Below is an example from pipeline of the output of df -h:

Whatup Program

The whatup program will display a plot of sources that are currently available. A sample plot is shown below.

The program must be run from helios and ipython. To run ensure you are logged into a helios terminal and issue the following commands

cd /common/python/current

ipython --pylab

import whatup

whatup.whatup()

This will display a plot of sources (Altitude vs Time) visible to the 27m Antenna from the current time for the next 24 hours. The following optional arguments can be supplied:

dur - the duration of the plot in hours

t - the UT start time of the plot

For example, the following commands

from util import Time

whatup.whatup(dur=6, t=Time("2020-10-01 14:00:00"))

will display a plot from 2020-10-01 14:00:00 for a duration of 6 hours as shown below:

renice: Changing priority of a program

If, on a linux system, a program is using to many resources, it is possible to use the renice> command to set it to a different priority.

From a terminal, type the command:

ps -elf | grep [program]

where [program] is the name of the program whose priority you wish to change.

Below is an example with output:

user@dpp:/$ ps -elf | grep rsync

4 S root 22708 21761 0 80 0 - 16638 - 18:52 pts/3 00:00:00 sudo rsync -av --exclude=/dev --exclude=/data1/IDB --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/nas1 --exclude=/proc --exclude=/sbdata --delete / /sbdata/dpp-backup/

4 S root 22709 22708 2 80 0 - 47670 - 18:52 pts/3 00:01:25 rsync -av --exclude=/dev --exclude=/data1/IDB --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/nas1 --exclude=/proc --exclude=/sbdata --delete / /sbdata/dpp-backup/

1 S root 22710 22709 1 80 0 - 24727 - 18:52 pts/3 00:00:34 rsync -av --exclude=/dev --exclude=/data1/IDB --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/nas1 --exclude=/proc --exclude=/sbdata --delete / /sbdata/dpp-backup/

1 D root 22711 22710 9 80 0 - 34724 - 18:52 pts/3 00:04:38 rsync -av --exclude=/dev --exclude=/data1/IDB --exclude=/lost+found --exclude=/media --exclude=/mnt --exclude=/nas1 --exclude=/proc --exclude=/sbdata --delete / /sbdata/dpp-backup/

Here we looked for the rsync command and found 4 occurrences. The 8th column (marked here in red) is the priority.

The priority ranges from -19 (highest priority) to 19 (lowest priority)

To change the priority of a program issue the command:

sudo renice -n [pr] -p [PID]

where [pr] is the new priority number and [PID] is the process identification number. The PID is the 4th column across in the display above (marked in green).

Plotting Stateframe Data

It is often useful to be able to access and plot Stateframe data.

Plotting Stateframe Data using the Stateframe Display

On the stateframe display, click on the Create tab shown below:

A screen similar to that below will be displayed. If there is already a graph present click on the Clear Graph tab shown below.

By selecting items one at a time from left to write next to 'Select item to add' you can choose what to plot. The click on 'Add Selected Item' By repeating this process more than on data set can be plotted on a graph at a time.

Using Python to plot Stateframe Data

This section describes, with an example, the procedure for plotting Stateframe data using python. This is useful as it allows time periods longer than an hour to be plotted.

Suppose we wish to plot the vertical polarization power over a 1 hour period from 2020-10-05 00:00 to 2020-10-05 01:00.

Firstly, from a pipeline terminal, access ipython:

ipython --pylab

Next import the required libraries:

import dbutil as db

from util import Time

import numpy as np

Now set up the cursor:

cursor = db.get_cursor()

The time range is set as follows (Just replace the times in quotes with new start and end times):

trange=Time(['2020-10-05 00:00','2020-10-05 01:00']).lv.astype(int)

Now we set up a query string which will search the data base:

query = 'select Timestamp,Ante_Fron_FEM_VPol_Power from fV66_vD15 where (I15 % 15) = 3 and Timestamp between '+str(trange[0])+' and '+str(trange[1])+' order by Timestamp'

Some explanation here is required.

Timestamp is the time for each data point.

Ante_Fron_FEM_VPol_Power is the variable for the VPOL power. The name of each variable can be retrieved by looking at the variables an the Create tab on the stateframe display. Note that except for the last variable, it is only the first 4 characters (or fewer if there are less than 4) that make up the name.

fV66_vD15 is the database from which the data is to be retrieved.

(I15 % 15) = 3 will give the data for the antenna. I15 denotes that there are 15 antennas. This takes the record number mod 15 and will only extract a record when this equals 3 (antenna 4, as the values here are 0 indexed).

It will also only extract values in the specified time range.

Finally we want the data ordered by timestamp.

Now we retrieve the requested data:

data, msg = db.do_query(cursor,query)

The data is stored in a dictionary. We can view the keys as follows:

data.keys()

This will produce an output as follows:

[u'Timestamp', u'Ante_Fron_FEM_VPol_Power']

We convert the timestamps to numpy times:

dt = np.array(Time(data['Timestamp'].astype(float), format = 'lv').isot,dtype = 'datetime64')

The labels for the plot can be set if desired:

title('VPOL Power vs Time')

xlabel('Universal Time')

ylabel('Power (dB)')

Finally the data is plotted:

plot(dt,data['Ante_Fron_FEM_VPol_Power'])

A sample plot is shown below:

This plot looks rather cluttered. The number of points can be reduced if we modify our query:

query = 'select Timestamp,Ante_Fron_FEM_VPol_Power from fV66_vD15 where (I15 % 15) = 3 and Timestamp between '+str(trange[0])+' and '+str(trange[1])+' and (cast(Timestamp as bigint) % 10) = 0 order by Timestamp'

We then request the data again,convert the timestamps to numpy times and replot the data producing:

Suppose we want to plot both the HPol Power and VPol power on the same plot. This is not difficult to do. We simply modify our query again:

query = 'select Timestamp,Ante_Fron_FEM_VPol_Power,Ante_Fron_FEM_HPol_Power from fV66_vD15 where (I15 % 15) = 3 and Timestamp between '+str(trange[0])+' and '+str(trange[1])+' and (cast(Timestamp as bigint) % 10) = 0 order by Timestamp'

Now HPol has been added and we just send this query as above. If we look at the keys now we see:

[u'Timestamp', u'Ante_Fron_FEM_HPol_Power', u'Ante_Fron_FEM_VPol_Power']

To plot this data as a scatter plot we use the following commands:

hpol, = plot(dt,data['Ante_Fron_FEM_HPol_Power'].astype(float),'o', label='HPOL')

vpol, = plot(dt,data['Ante_Fron_FEM_VPol_Power'],'o', label='VPOL')

legend(handles=[hpol,vpol])

which produces this plot:

Other

Reloading the crio.ini File

DO THIS WITH EXTREME CAUTION. BE ABSOLUTELY CERTAIN THAT RELOADING THE CRIO FILE IS REQUIRED.

For antenna 11, see the special case here:

After uploading the new crio.ini file the gains and pointing may also need to be updated.

Manually uploading the crio.ini file

From helios, open a terminal and type cd /Dropbox/PythonCode/Current/crio_inis

For the antenna that needs to be reset, find the most recent '.ini' file. For example, for antenna 10 this may be 'crio10-2019-09-28.ini'. Make a note of this file.

Enter the command ftp crio10.solar.pvt. Replace crio10 with the apropriate CRIO number.

Enter the username and password when prompted.

Enter the following commands:

cd ni-rt/startup

put crio10-2019-09-28.ini crio.ini

bye

replacing 'crio10-2019-09-28.ini' with the appropriate file for the CRIO that needs to be reset.

The above commands have reloaded the crio.ini file. The CRIO now needs to be reset. For example, Antenna 10 will be reset by issuing the following commands from the schedule window:

sync ant10

tracktable sun_tab.radec ant10

track

Using ipython to Upload the crio.ini File

From helios, open a terminal and access ipython:

ipython --pylab

Load the crio.py file by entering:

import crio

Simply issue the command:

crio.reload_crio_ini('ant#')

where # is the antenna number. The most recent crio.ini file will be uploaded for that antenna.

Special Case: Antenna 11

Antenna 11 has a new model CRIO. It has a different file structure than the other antenna's CRIOs and uses sftp rather than ftp. The procedure below details how to reload the crio.ini file for antenna 11

Power Meter Setup

The user manual for the power meter can be found at https://literature.cdn.keysight.com/litweb/pdf/E4418-90032.pdf

Before use the power meter will need to be zeroed and calibrated.

1. Ensure that the sensor is not connected to anything.

2. Press the ZERO/CAL button.

3. Press the button next to [Zero] displayed on the screen. It will take several seconds to zero.

4. Next remove the SMA adaptor from the end of the sensor and connect the sensor to the POWER REF socket on the power meter as shown below.

5. Press the button next to [CAL] displayed on the screen. It will take several seconds.

6. The SMA power sensor can now be disconnected from the POWER REF socket and the SMA adaptor reconnected.

Moving the New Antennas Manually

Power off the antenna by turning the power switch at the bottom right of the Antenna Control Box. Switch the Maint/Norm Switch to Maint. Switch the Remote Switch to Remote.

Remove the block from the hand controller port. Remember to place the block back after you have finished working on the antenna. It will not run in remote without it!

Connect the hand controller to the port.

Turn the power switch back on. The antenna can now be moved to a suitable position using the azimuth and elevation controls on the hand controller.

Moving the Old Antennas Manually

At the lower right of the control box, there is a panel for controlling the antenna manually as shown below.

Move the bottom center switch to Local. The antenna can now be positioned using the HA and Dec switches.

Antenna Stowing at End of Day

Antennas 1 to 8 are not stowed after observations. Therefore, the antennas will be highlighted as red as they will still be tracking (see image below). There is no need to take any action for these antennas.

Antenna 11 will need to be stowed manually after observations are complete. To do this, enter the commands below:

stop ant11

stow ant11

Service Stow

Use this procedure to place the antenna into its service stow position.

NOTE: do NOT use the service command. This seems to also reset the CRIO.

From the schedule window issue the coommand:

position 225 15 ant#

where ant# is the antenna to place into service stow.

Stopping and Restarting Helios VNC

If the helios VNC server needs to be stopped then you will need to find the process ID number (PID#). From a helios terminal issue the following command:

ps -A | grep vnc

You will see a line that looks something like:

26626 ? 00:00:00 x11vnc

The number on the left is the PID#, in this case 26626. To stop VNC issue the command

kill PID#

where in this case PID# would be 26626

Note that doing this will cut the VNC connection to any users that are currently logged in.

To restart VNC issue the command:

x11vnc -shared -display :0 -forever -noxrecord -clip 1920x1200+0+0 -bg -usepw -rfbport 20000 -auth /home/sched/.Xauthority

Note that VNC is started when Helios is started. The startup script for VNC is located in: /etc/init/x11vnc.conf

Helios Bash Aliases

The aliases for various bash command can be found in the .bash_aliases file

Shutting off the 27m Control System

At times during maintenance on the 27m antenna it may be necessary to turn off the control system to prevent anyone from moving the antenna. To do this perform the following steps.

Log into the feanta machine from helios:

ssh antctl@feanta

Issue the command (note you can use the up arrow until you find the command):

/home/antctl/starburst/femTools/src/starburstControl stop

Log out of feanta by issuing the command exit

There is a crontask that periodically restarts the control system on helios. Issue the command:

crontab -e

This will allow you to edit the crontab file. Scroll down to find the line that reads:

1 1,5,9,13,17,21 * * * ssh -t antctl@feanta /home/antctl/starburst/femTools/src/starburstControl start

Place a # in front of this line so that it reads:

#1 1,5,9,13,17,21 * * * ssh -t antctl@feanta /home/antctl/starburst/femTools/src/starburstControl start

Press CRTL x and then answer Y to save the file.

Note that when maintenance has finished you will need to edit the crontab file again and remove the # from that line.

Data Analysis

This section is mostly to do with how to load an view EOVSA synoptic FITS images.

From pipelne change to the /common/python/mapping folder.

From ipython, issue the following commands to load the required libraries:

from fits2map import *

from plot_map import *

The files can be found at http://www.ovsa.njit.edu/fits/synoptic/. To see what files are available click on the appropriate folder.

To load a particular file you would issue the command:

eomap1 = fits2map("eomap1=fits2map("http://www.ovsa.njit.edu/fits/synoptic/2020/05/19/eovsa_20200519.spw06-10.tb.disk.fits")

In this case it would load a full disk image from 2020-05-19 created from bands 6 to 10 into eomap1.

To view the image issue the command:

plot_map(eomap1)

This will produce a plot similar to that below.

The image parameters are stored as a dictionary and can be viewed by:

eomap1.keys()

For more information see http://www.ovsa.njit.edu/wiki/index.php/Mapping_Software