EOVSA Wiki - User contributions [en]

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:18:58Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

=====4.1=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====4.4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

=====4.5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:18:35Z

Sshaik: /* 4.1 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

*=====4.1=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====4.4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

=====4.5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:17:48Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles
</pre>

=====4.1=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====4.4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

=====4.5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:13:06Z

Sshaik: /* Step 5 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====4.4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

=====4.5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:12:56Z

Sshaik: /* Step 4 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====4.4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:12:44Z

Sshaik: /* Step 3 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====4.3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T22:12:28Z

Sshaik: /* Step 2 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====4.2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:56:24Z

Sshaik: /* 3. Software requirements and installation */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=11. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:56:02Z

Sshaik: /* Other useful links */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=11. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:39:23Z

Sshaik: /* Step 2 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask] # Not used?
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:33:42Z

Sshaik: /* Step 3 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.pause(0.5) # Allows viewing of each image as it is plotted.
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:32:16Z

Sshaik: /* Step 5 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:31:39Z

Sshaik: /* Step 3 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1: # if sp < 0: # For post-2020 data (50 spws)
sp_i = 1 # sp_i = 0 # For post-2020 data (50 spws)
if sp_e > 30: # if sp_e > 49: # For post-2020 data (50 spws)
sp_e = 30 # sp_e = 49 # For post-2020 data (50 spws)
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:29:45Z

Sshaik: /* Step 3 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)
plt.pause(0.5) # Allows viewing of the plot

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:27:23Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)] ###spws=[str(s+1) for s in range(49)] # For post-2020 data
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.)) # fig = plt.figure(figsize=(14, 7)) # For post-2020 data (50 spws)
gs = gridspec.GridSpec(5, 6) # gs = gridspec.GridSpec(5, 10) # For post-2020 data (50 spws)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1) # spbg=max(int(sp)-2,0) # For post-2020 data (50 spws)
sped=min(int(sp)+2,30) # sped=min(int(sp)+2,49) # For post-2020 data (50 spws)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>
=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:19:54Z

Sshaik: /* Step 2: Concatenate all the 10 mins data, if there are any */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|right|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>
=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:18:23Z

Sshaik: /* 12. Installation of SunCASA (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=11. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:18:15Z

Sshaik: /* 11. Spectral Fitting with GSFIT (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:17:59Z

Sshaik: /* 10. Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T21:17:32Z

Sshaik: /* 9. Downloading fits files to your local system */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control of all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:15:37Z

Sshaik: /* Step 3 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>
=====Step 4=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]
=====Step 5=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:12:02Z

Sshaik: /* Step2 */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step 2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.
=====Step 3=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:11:21Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]
=====Step2=====
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:10:40Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

=====Step2=====
# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:06:52Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna=antennas,
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas=antennas
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

Tohban EOVSA Imaging Tutorial A-Z

2022-07-07T21:03:18Z

Sshaik: /* Step 4: Self-calibration */

This tutorial describes the step-by-step procedure to download EOVSA IDB data, obtain and calibrate the measurement sets (.ms), and, transfer them to the inti server for self-calibration and further analysis.

'''Pre-requisites:''' Accounts on Pipeline and Inti or Baozi servers.

=== Connection details to pipeline server ===
One can use the Mobaxterm platform to connect to the Pipeline server through a Windows machine or use SSH through a Mac machine.

==== Step 1: Downloading raw data (IDB) on Pipeline machine====
On CASA of the Pipeline machine, which has the complete EOVSA SQL database.

If you are not using mobaxterm, directly SSH to Pipeline through your Linux or Mac computer and start tcsh to run CASA.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from astropy.time import Time
import os
trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:25:00']) ###Change accordingly###
#### (Optional) change output path, default current directory "./" #####
outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
######################################################
msfiles = importeovsa(idbfiles=trange, ncpu=1, visprefix=outpath)
</pre>

NOTE: The above step currently gives an error with importeovsa. This is because the data location was changed and the code doesn't know where to look for the IDB files. Sijie is looking into the problem. We will update the page when it is fixed. For now, please use the method below for all time ranges.

OR (for a time range longer than 10 minutes)

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.tasks import task_calibeovsa as calibeovsa
from suncasa.tasks import task_importeovsa as timporteovsa
from split_cli import split_cli as split
import dump_tsys as dt
from util import Time
import numpy as np
import os
from glob import glob
from eovsapy import util

trange = Time(['2017-08-21 20:15:00', '2017-08-21 20:35:00']) ###Change accordingly###
idbdir = util.get_idbdir(trange[0])

info = dt.rd_fdb(trange[0])
for k, v in info.iteritems():
info[k] = info[k][~(info[k] == '')]

sidx = np.where(
np.logical_and(info['SOURCEID'] == 'Sun', info['PROJECTID'] == 'NormalObserving') & np.logical_and(
info['ST_TS'].astype(np.float) >= trange[0].lv,
info['ST_TS'].astype(np.float) <= trange[
1].lv))
filelist = info['FILE'][sidx]

outpath = './msdata/' ###Change accordingly###
if not os.path.exists(outpath):
os.makedirs(outpath)
inpath = idbdir + '{}/'.format(trange[0].datetime.strftime("%Y%m%d"))
ncpu = 1

msfiles = timporteovsa.importeovsa(idbfiles=[inpath + ll for ll in filelist], ncpu=ncpu, timebin="0s", width=1,
visprefix=outpath,
nocreatms=False, doconcat=False,
modelms="", doscaling=False, keep_nsclms=False, udb_corr=True)
</pre>

If you come across errors with calibeovsa, add following lines to your ~/.casa/init.py file.
<pre style="background-color: #FCEBD9">
import sys
sys.path.append('/common/python')
sys.path.append('/common/python/packages/pipeline_casa')
execfile('/common/python/suncasa/tasks/mytasks.py')
</pre>

==== Step 2: Concatenate all the 10 mins data, if there are any====
Follow this step if there are more than one .ms files, if not run step 3 directly. If doimage=True, a quicklook image will be produced (by integrating over the entire time) as shown below.
<pre style="background-color: #FCEBD9">
# This is to set the path/name for the concatenated files
concatvis = os.path.basename(msfiles[0])[:11] + '_concat_cal.ms'
vis = calibeovsa.calibeovsa(msfiles, doconcat=True, concatvis=concatvis, caltype=['refpha','phacal'], doimage=True)
</pre>

[[file:Figure1_imagingtutorial.png|frame|center|800px|'''Figure 1:''' Quick-look full-Sun image after the initial calibration.]]

==== Step 3: Calibration ====
This will calibrate the input visibility, write out calibration tables under /data1/eovsa/caltable/, and apply the calibration.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
vis = calibeovsa.calibeovsa('IDB20170821202020.ms', caltype=['refpha','phacal'], doimage=True) ###Change the vis filename accordingly###
# Append '_cal' to the ms filename and split the corrected column to the new caled ms
vis_str = str(' '.join(vis))
caled_vis=vis_str.replace('.ms','_cal.ms')
split(vis=' '.join(vis),outputvis=caled_vis,datacolumn='corrected',timerange='',correlation='XX')
</pre>

One needs to transfer the created caled ms data files (xxx_concat_cal.ms or xxx_cal.ms) from pipeline to inti server, which has all the casatasks installed, in order to run the rest of the imaging steps.

=== Connection details to Inti server ===
For Windows, on Mobaxterm,

#With your NJIT VPN connected, connect to one of the afsconnect servers (for example, afsaccess3.njit.edu) using your UCID and password.
#ssh -X UCID@inti.hpcnet.campus.njit.edu

To avoid typing the full inti address each time you attempt for ssh, you may wish to add the following lines with your username to C:\Program Files\Git\etc\ssh\ssh_config on Windows and /.ssh/config on Mac and Linux machines.

<pre style="background-color: #FCEBD9">
Host inti
Hostname inti.hpcnet.campus.njit.edu
User USERNAME ###Insert UCID/inti username here###
</pre>

To have sufficient disk space with EOVSA data analysis over Inti, use your dedicated directory YOURDIRECTORY at the location given below. If you do not have a directory, Please take help from Sijie in creating one.
<pre style="background-color: #FCEBD9">
cd /inti/data/users/YOURDIRECTORY
</pre>

For Linux or Mac machine,
ssh -X UCID@inti.hpcnet.campus.njit.edu

=== Transferring data files between servers ===
For directly transferring your calibrated .ms data between the Pipeline and Inti servers, follow the below given steps.
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
1. Log into Inti using your username.
ssh -X USERNAME@inti.hpcnet.campus.njit.edu

Then create a tunnel into Pipeline from Inti.
ssh -L 8888:pipeline.solar.pvt:22 guest@ovsa.njit.edu

2. Log into Inti again from a new terminal.

Change to your working directory and give this command to copy your data on Pipeline.
scp -v -C -r -P 8888 USERNAMEofPipeline@localhost:PATHofMSDATA/MSfilename ./
Eg: scp -v -C -r -P 8888 shaheda@localhost:/data1/shaheda/IDB20220118173922_cal.ms ./
where, MSfilename, PATHofMSDATA are your .ms data and its path on Pipeline machine.
</pre>

Alternatively, one can follow the below procedure to do the transfer.

For Windows, on mobaxterm, drag and drop the ms file to your local machine or use scp command as given below.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C userid@pipeline:/your/folder/msfile /localmachine/destination/folder/
</pre>

On mobaxterm and on your local terminal, use the following command to finally copy the ms file to Inti.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
scp -r -C /localmachine/destination/folder/msfile UCID@inti.hpcnet.campus.njit.edu:/inti/data/users/YOURDIRECTORY
</pre>

For Mac and Linux, SCP can be used in the same way. Add the following lines to your SSH config file, to bypass ovsa.njit.edu from copying.
vi ~/.ssh/config
<pre style="background-color: #FCEBD9">
Host ovsa
HostName ovsa.njit.edu
User guest
Host pipeline
Hostname pipeline.solar.pvt
User userid
ProxyCommand ssh -W %h:%p guest@ovsa.njit.edu
</pre>

=== Software details on the servers ===
On Inti,
when logging in for the first time, please add the following lines to your accounts ~/.bashrc file.

>>vi ~/.bashrc
Insert the text given below and save it.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
#### setting start ####
if [ $HOSTNAME == "baozi.hpcnet.campus.njit.edu" ]; then
source /srg/.setenv_baozi
fi
if [ $HOSTNAME == "inti.hpcnet.campus.njit.edu" ]; then
source /inti/.setenv_inti
fi
if [ $HOSTNAME == "guko.resource.campus.njit.edu" ]; then
source /data/data/.setenv_guko
fi
#### setting end ####
</pre>

Both CASA 5 and 6 are available on Inti.

Enter the bash environment on inti, and load the desired casa environment.
To load CASA 5, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa5
>> suncasa #This should load the software making you ready for the analysis
</pre>

Or to load CASA 6, enter bash environment by giving >>bash
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
>> loadcasa6
>> ipython #This should load the software making you ready for the analysis
</pre>
Here, for example, to use clean, first start ipython as given above, then type in >>from casatasks import tclean

====Step 4: Self-calibration====
[[file:Figure3_imagingtutorial.png|thumb|center|500px|Figure 2: Cotton-Schwab clean major and minor cycles. [Source: http://www.aoc.nrao.edu/~rurvashi/ImagingAlgorithmsInCasa/node2.html].]]
Follow the below given steps to run the self-calibration of the imaging data and produce the calibrated images in .fits format. https://github.com/binchensun/casa-eovsa/blob/master/slfcal_example.py

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import helioimage2fits as hf
import os
import numpy as np
import pickle
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
from split_cli import split_cli as split
import time

# =========== task handlers =============
dofullsun = 1 # initial full-sun imaging ###Change accordingly###
domasks=1 # get masks ###Change accordingly###
doslfcal=1 # main cycle of doing selfcalibration ###Change accordingly###
doapply=1 # apply the results ###Change accordingly###
doclean_slfcaled=1 # perform clean for self-calibrated data ###Change accordingly###

# ============ declaring the working directories ============
workdir = os.getcwd()+'/' #main working directory. Using current directory in this example
slfcaldir = workdir+'slfcal/' #place to put all selfcalibration products
imagedir = slfcaldir+'images/' #place to put all selfcalibration images
maskdir = slfcaldir+'masks/' #place to put clean masks
imagedir_slfcaled = slfcaldir+'images_slfcaled/' #place to put final self-calibrated images
caltbdir = slfcaldir+'caltbs/' # place to put calibration tables
# make these directories if they do not already exist
dirs = [workdir, slfcaldir, imagedir, maskdir, imagedir_slfcaled, caltbdir]
for d in dirs:
if not os.path.exists(d):
os.makedirs(d)

# ============ Split a short time for self-calibration ===========
# input visibility
ms_in = workdir + 'IDB20170821202020_cal.ms' ###Change the initial calibrated (through calibeovsa) vis accordingly###
# output, selfcaled, visibility
ms_slfcaled = workdir + os.path.basename(ms_in).replace('cal','slfcaled')
# intermediate small visibility for selfcalbration
# selected time range for generating self-calibration solutions
trange='2017/08/21/20:21:10~2017/08/21/20:21:30' ###Change accordingly###
slfcalms = slfcaldir+'slfcalms.XX.slfcal'
slfcaledms = slfcaldir+'slfcalms.XX.slfcaled'
if not os.path.exists(slfcalms):
split(vis=ms_in,outputvis=slfcalms,datacolumn='data',timerange=trange,correlation='XX')

# ============ Prior definitions for spectral windows, antennas, pixel numbers =========
spws=[str(s+1) for s in range(30)]
antennas='0~12&0~12'
npix=512
nround=3 #number of slfcal cycles

# =========== Step 1, doing a full-Sun image to find out phasecenter and appropriate field of view =========
if dofullsun:
#initial mfs clean to find out the image phase center
im_init='fullsun_init'
os.system('rm -rf '+im_init+'*')
tclean(vis=slfcalms,
antenna='0~12',
imagename=im_init,
spw='1~15',
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['5arcsec'],
niter=1000,
gain=0.05,
stokes='I',
restoringbeam=['30arcsec'],
interactive=False,
pbcor=True)

hf.imreg(vis=slfcalms,imagefile=im_init+'.image.pbcor',fitsfile=im_init+'.fits',
timerange=trange,usephacenter=False,verbose=True)
clnjunks = ['.flux', '.mask', '.model', '.psf', '.residual','.sumwt','.pb','.image'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(im_init + clnjunk):
os.system('rm -rf '+im_init + clnjunk)

from sunpy import map as smap
from matplotlib import pyplot as plt
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
eomap=smap.Map(im_init+'.fits')
#eomap.data=eomap.data.reshape((npix,npix))
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
plt.show()
viewer(im_init+'.image.pbcor')
</pre>
[[file:Figure2_imagingtutorial.png|thumb|center|500px|Figure 3: Full-Sun image after initial clean to find the flare location.]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# parameters specific to the event (found from step 1)
phasecenter='J2000 10h02m59 11d58m07' ###Change accordingly###
xran=[280,480] ###Change accordingly###
yran=[-50,150] ###Change accordingly###

# =========== Step 2 (optional), generate masks =========
# if skipped, will not use any masks
if domasks:
clearcal(slfcalms)
delmod(slfcalms)
antennas='0~12'
pol='XX'
imgprefix=maskdir+'slf_t0'

# step 1: set up the clean masks
img_init=imgprefix+'_init_ar_'
os.system('rm -rf '+img_init+'*')
#spwrans_mask=['1~5','6~12','13~20','21~30']
spwrans_mask=['1~12']
#convert to a list of spws
spwrans_mask_list = [[str(i) for i in (np.arange(int(m.split('~')[0]),int(m.split('~')[1])))] for m in spwrans_mask]
masks=[]
imnames=[]
for spwran in spwrans_mask:
imname=img_init+spwran.replace('~','-')
try:
tclean(vis=slfcalms,
antenna='0~12',
imagename=imname,
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=1000,
gain=0.05,
stokes='XX',
restoringbeam=['20arcsec'],
phasecenter=phasecenter,
weighting='briggs',
robust=1.0,
interactive=True,
datacolumn='data',
pbcor=True,
savemodel='modelcolumn')
imnames.append(imname+'.image')
masks.append(imname+'.mask')
clnjunks = ['.flux', '.model', '.psf', '.residual'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(imname + clnjunk):
os.system('rm -rf '+ imname + clnjunk)
except:
print('error in cleaning spw: '+spwran)

pickle.dump(masks,open(slfcaldir+'masks.p','wb'))
</pre>
[[file:Figure4_imagingtutorial.PNG|thumb|center|800px|Figure 4: Interactive clean window to create masks over the source. The white outline surrounding the source is the mask selected by the polygon drawing option.]]

The outlines drawn for masks can be created by any of the icons with the letter 'R' in the Viewer window. The instructions for doing this can be found by hovering over those icons.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 3, main step of selfcalibration =========
if doslfcal:
if os.path.exists(slfcaldir+'masks.p'):
masks=pickle.load(open(slfcaldir+'masks.p','rb'))
if not os.path.exists(slfcaldir+'masks.p'):
print 'masks do not exist. Use default mask'
masks=[]
os.system('rm -rf '+imagedir+'*')
os.system('rm -rf '+caltbdir+'*')
#first step: make a mock caltable for the entire database
print('Processing ' + trange)
slftbs=[]
calprefix=caltbdir+'slf'
imgprefix=imagedir+'slf'
tb.open(slfcalms+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
# starting beam size at 3.4 GHz in arcsec #Change accordingly
sbeam=40.
strtmp=[m.replace(':','') for m in trange.split('~')]
timestr='t'+strtmp[0]+'-'+strtmp[1]
refantenna='0'
# number of iterations for each round
niters=[100, 300, 500]
# roburst value for weighting the baselines
robusts=[1.0, 0.5, 0.0]
# apply calibration tables? Set to true for most cases
doapplycal=[1,1,1]
# modes for calibration, 'p' for phase-only, 'a' for amplitude only, 'ap' for both
calmodes=['p','p','a']
# setting uvranges for model image (optional, not used here)
uvranges=['','','']
for n in range(nround):
slfcal_tb_g= calprefix+'.G'+str(n)
fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
print 'processing spw: '+sp
cfreq=cfreqs[int(sp)]
# setting restoring beam size (not very useful for selfcal anyway, but just to see the results)
bm=max(sbeam*cfreqs[1]/cfreq, 6.)
slfcal_img = imgprefix+'.spw'+sp.zfill(2)+'.slfcal'+str(n)
# only the first round uses nearby spws for getting initial model
if n == 0:
spbg=max(int(sp)-2,1)
sped=min(int(sp)+2,30)
spwran=str(spbg)+'~'+str(sped)
print('using spw {0:s} as model'.format(spwran))
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print('using mask {0:s}'.format(mask))
findmask = True
if not findmask:
print('mask not found. Do use any masks')
else:
spwran = sp
if 'spwrans_mask' in vars():
for m, spwran_mask in enumerate(spwrans_mask):
if sp in spwran_mask:
mask = masks[m]
print 'using mask {0:s}'.format(mask)
findmask = True
if not findmask:
print('mask not found. Do use any masks')
try:
tclean(vis=slfcalms,
antenna=antennas,
imagename=slfcal_img,
uvrange=uvranges[n],
spw=spwran,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['2arcsec'],
niter=niters[n],
gain=0.05,
stokes='XX', #use pol XX image as the model
weighting='briggs',
robust=robusts[n],
phasecenter=phasecenter,
mask=mask,
restoringbeam=[str(bm)+'arcsec'],
pbcor=False,
interactive=False,
savemodel='modelcolumn')
if os.path.exists(slfcal_img+'.image'):
fitsfile=slfcal_img+'.fits'
hf.imreg(vis=slfcalms,imagefile=slfcal_img+'.image',fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False,overwrite=True)
clnjunks = ['.mask','.flux', '.model', '.psf', '.residual', '.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for clnjunk in clnjunks:
if os.path.exists(slfcal_img + clnjunk):
os.system('rm -rf '+ slfcal_img + clnjunk)
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfile)
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
#eomap.draw_grid()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
os.system('rm -f '+ fitsfile)

except:
print 'error in cleaning spw: '+sp
print 'using nearby spws for initial model'
sp_e=int(sp)+2
sp_i=int(sp)-2
if sp_i < 1:
sp_i = 1
if sp_e > 30:
sp_e = 30
sp_=str(sp_i)+'~'+str(sp_e)
try:
tget(tclean)
spw=sp_
print('using spw {0:s} as model'.format(sp_))
tclean()
except:
print 'still not successful. abort...'
break

gaincal(vis=slfcalms, refant=refantenna,antenna=antennas,caltable=slfcal_tb_g,spw=sp, uvrange='',\
gaintable=[],selectdata=True,timerange=trange,solint='inf',gaintype='G',calmode=calmodes[n],\
combine='',minblperant=4,minsnr=2,append=True)
if not os.path.exists(slfcal_tb_g):
print 'No solution found in spw: '+sp
figname=imagedir+'slf_t0_n{:d}.png'.format(n)
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.savefig(figname)
time.sleep(10)
plt.close()

if os.path.exists(slfcal_tb_g):
slftbs.append(slfcal_tb_g)
slftb=[slfcal_tb_g]
os.chdir(slfcaldir)
if calmodes[n] == 'p':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='phase',\
subplot=431,plotrange=[-1,-1,-180,180],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
if calmodes[n] == 'a':
plotcal(caltable=slfcal_tb_g,antenna='1~12',xaxis='freq',yaxis='amp',\
subplot=431,plotrange=[-1,-1,0,2.],iteration='antenna',figfile=slfcal_tb_g+'.png',showgui=False)
os.chdir(workdir)

if doapplycal[n]:
clearcal(slfcalms)
delmod(slfcalms)
applycal(vis=slfcalms,gaintable=slftb,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='nearest',flagbackup=False,applymode='calonly',calwt=False)

if n < nround-1:
prompt=raw_input('Continuing to selfcal?')
#prompt='y'
if prompt.lower() == 'n':
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
break
if prompt.lower() == 'y':
slfcalms_=slfcalms+str(n)
if os.path.exists(slfcalms_):
os.system('rm -rf '+slfcalms_)
split(slfcalms,slfcalms_,datacolumn='corrected')
slfcalms=slfcalms_
else:
if os.path.exists(slfcaledms):
os.system('rm -rf '+slfcaledms)
split(slfcalms,slfcaledms,datacolumn='corrected')
print 'Final calibrated ms is {0:s}'.format(slfcaledms)
</pre>

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 4: Apply self-calibration tables =========
if doapply:
import glob
os.chdir(workdir)
clearcal(ms_in)
clearcal(slfcalms)
applycal(vis=slfcalms,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
applycal(vis=ms_in,gaintable=slftbs,spw=','.join(spws),selectdata=True,\
antenna=antennas,interp='linear',flagbackup=False,applymode='calonly',calwt=False)
if os.path.exists(ms_slfcaled):
os.system('rm -rf '+ms_slfcaled)
split(ms_in, ms_slfcaled,datacolumn='corrected')
</pre>
[[file:Slf.G0.png|thumb|center|500px|Figure 1: Phase before self-calibration]]
[[file:Slf.G1.png|thumb|center|500px|Figure 2: Phase after self-calibration]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
# =========== Step 5: Generate final self-calibrated images (optional) =========
if doclean_slfcaled:
import glob
pol='XX'
print('Processing ' + trange)
img_final=imagedir_slfcaled+'/slf_final_{0}_t0'.format(pol)
vis = ms_slfcaled
spws=[str(s+1) for s in range(30)]
tb.open(vis+'/SPECTRAL_WINDOW')
reffreqs=tb.getcol('REF_FREQUENCY')
bdwds=tb.getcol('TOTAL_BANDWIDTH')
cfreqs=reffreqs+bdwds/2.
tb.close()
sbeam=30.
from matplotlib import gridspec as gridspec
from sunpy import map as smap
from matplotlib import pyplot as plt
fitsfiles=[]
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
bm=max(sbeam*cfreqs[1]/cfreq,4.)
imname=img_final+'_s'+sp.zfill(2)
fitsfile=imname+'.fits'
if not os.path.exists(fitsfile):
print 'cleaning spw {0:s} with beam size {1:.1f}"'.format(sp,bm)
try:
tclean(vis=vis,
antenna=antennas,
imagename=imname,
spw=sp,
specmode='mfs',
timerange=trange,
imsize=[npix],
cell=['1arcsec'],
niter=1000,
gain=0.05,
stokes=pol,
weighting='briggs',
robust=2.0,
restoringbeam=[str(bm)+'arcsec'],
phasecenter=phasecenter,
mask='',
pbcor=True,
interactive=False)
except:
print 'cleaning spw '+sp+' unsuccessful. Proceed to next spw'
continue
if os.path.exists(imname+'.image.pbcor'):
imn = imname+'.image.pbcor'
hf.imreg(vis=vis,imagefile=imn,fitsfile=fitsfile,
timerange=trange,usephacenter=False,toTb=True,verbose=False)
fitsfiles.append(fitsfile)
junks=['.flux','.model','.psf','.residual','.mask','.image','.pb','.image.pbcor','.sumwt'] #Do not run the next 4 lines if needed to view and assess the subset of clean process images
for junk in junks:
if os.path.exists(imname+junk):
os.system('rm -rf '+imname+junk)
else:
print('fits file '+fitsfile+' already exists, skip clean...')
fitsfiles.append(fitsfile)

fig = plt.figure(figsize=(8.4,7.))
gs = gridspec.GridSpec(5, 6)
for s,sp in enumerate(spws):
cfreq=cfreqs[int(sp)]
ax = fig.add_subplot(gs[s])
eomap=smap.Map(fitsfiles[s])
eomap.plot_settings['cmap'] = plt.get_cmap('jet')
eomap.plot(axes = ax)
eomap.draw_limb()
ax.set_title(' ')
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.set_xlim(xran)
ax.set_ylim(yran)
plt.text(0.98,0.85,'{0:.1f} GHz'.format(cfreq/1e9),transform=ax.transAxes,ha='right',color='w',fontweight='bold')
plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=0, hspace=0)
plt.show()

</pre>
The .fits files of the self calibrated images at each frequency for the given time are saved at /slfcal/images_slfcaled in your working directory.

[[file:Slf_t0_n0.png|thumb|center|500px|Figure 3: Multi-frequency images before self-calibration]]
[[file:Slf_t0_n1.png|thumb|center|500px|Figure 4: Multi-frequency images after self-calibration]]

====Step 5: Quick-look imaging ==== 
For spectral imaging analysis of the event, follow [http://www.ovsa.njit.edu/wiki/index.php/EOVSA_Data_Analysis_Tutorial#Spectral_Imaging_with_SunCASA this tutorial] using the self-calibrated data obtained from the previous step or use [https://colab.research.google.com/drive/1lSLLxgOG6b8kgu9Sk6kSKvrViyubnXG6?usp=sharing#scrollTo=xbXyyLmCFCGL this link].

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
from suncasa.utils import qlookplot as ql ## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.
## set the time interval
from suncasa import dspec as ds
import time
visibility_data = 'IDB20170821202020_slfcaled.ms'
specfile = visibility_data + '.dspec.npz'
d = ds.Dspec(visibility_data, bl='4&9', specfile=specfile)

timerange = '19:02:00~19:02:10' ## select frequency range from 2.5 GHz to 3.5 GHz
spw = '2~5' ## select stokes XX
stokes = 'XX' ## turn off AIA image plotting, default is True
plotaia = False
xycen = [375, 45] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel size
imsize=[128] ## x and y image size in pixels.
fov = [100,100] ## field of view of the zoomed-in panels in unit of arcsec
spw = ['{}'.format(s) for s in range(1,31)]
clevels = [0.5, 1.0] ## contour levels to fill in between.
calpha=0.35 ## now tune down the alpha
restoringbeam=['6arcsec']
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, stokes=stokes, \
restoringbeam=restoringbeam,imsize=imsize,cell=cell, \
xycen=xycen,fov=fov,clevels=clevels,calpha=calpha)
</pre>

EOVSA Data Analysis Tutorial 2022

2022-07-07T20:10:10Z

Sshaik: /* 1. Introduction to EOVSA and Datasets */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

===Level 0 (raw visibility) Data===
Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data. However, to be useful for imaging, these files must be first calibrated and then converted to a CASA measurement set. At present, this can only be done with software at the OVRO site although we are working on software to allow this processing to be done elsewhere. For now, please request this processing to be done for you by emailing Dale Gary, Bin Chen, Sijie Yu, or others you may know in the solar radio group.

===Level 1 (calibrated visibility) Data===
The images you see in the browser are generated by an automated pipeline that also produces calibrated CASA ms datasets, but at 1-min integration. If a 1-min cadence is sufficient for your purposes, you may wish to download the calibrated data and perform your own imaging. The 1-min cadence calibrated data are at [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ this link]. You should be aware that a model disk has been subtracted from the visibilities in these files, which is what you want if you are interested in imaging active regions or flares. It is possible to add the model back, but this is not available (yet) in standard software, so please contact someone in the NJIT solar radio group for help with that.

===Other useful links===

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files. Please

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:34:35Z

Sshaik: /* 11. Spectral Fitting with GSFIT (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files. Please

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer [http://ovsa.njit.edu/wiki/index.php/GSFIT_Help this page] for a detailed description of further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:33:42Z

Sshaik: /* 9. Downloading fits files to your local system */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files. Please

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer this page for the detailed description on further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:32:03Z

Sshaik: /* 11. Spectral Fitting with GSFIT (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] can be read as an input to the GSFIT package. The screenshot of that is shown in Figure 9. Refer this page for the detailed description on further GSFIT analysis.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:28:01Z

Sshaik: /* 11. Spectral Fitting with GSFIT (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure 9.

[[file:GSFIT_20220507.PNG|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

File:GSFIT 20220507.PNG

2022-07-07T08:27:37Z

Sshaik:

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:27:12Z

Sshaik: /* 11. Spectral Fitting with GSFIT (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure 9.

[[file:GSFIT_20220507.png|thumb|right|400px|Figure 9: Image and the corresponding spectrum on GSFIT GUI.]]

=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:23:42Z

Sshaik: /* 10. Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Figure 8: Multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure
=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:22:56Z

Sshaik: /* 8. Quick-Look Imaging */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5: EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure
=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:22:44Z

Sshaik: /* 8. Quick-Look Imaging */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|Figure 5:EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|Figure 6: EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|Figure 7: EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure
=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:21:46Z

Sshaik: /* 11. Installation of SunCASA (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Spectral Fitting with GSFIT (optional)=
The .fits images obtained in the can be taken as input to the GSFIT package. The screenshot of that is shown in Figure
=12. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:16:02Z

Sshaik: /* Installation of SunCASA (optional) */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=11. Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:15:50Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=10. Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:15:36Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

=Plotting Images in SSWIDL=
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:15:24Z

Sshaik: /* 9. Downloading fits files to your local system */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] for an example on downloading image fits files.

===Plotting Images in SSWIDL===
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T08:14:01Z

Sshaik: /* 9. Downloading fits files to your local system */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. Please follow the

===Plotting Images in SSWIDL===
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:50:27Z

Sshaik: /* 9. Downloading fits files to your local system with Batch-Mode Imaging */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.

===Plotting Images in SSWIDL===
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:49:34Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

===Plotting Images in SSWIDL===
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|400px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:48:42Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

====Plotting Images in SSWIDL====
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.PNG|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

File:Batch mode images.PNG

2022-07-07T07:46:41Z

Sshaik:

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:46:18Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

====Plotting Images in SSWIDL====
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:Batch_mode_images.png|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]

An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:44:18Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

====Plotting Images in SSWIDL====
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:fig-specimg_idl.png|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]
An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-05-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:39:53Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

====Plotting Images in SSWIDL====
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro.

IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles, eomaps [calcrms];, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:fig-specimg_idl.png|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]
An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-0-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.

EOVSA Data Analysis Tutorial 2022

2022-07-07T07:38:10Z

Sshaik: /* Plotting Images in SSWIDL */

=1. Introduction to EOVSA and Datasets=
[[file:Eovsa1.png|center|500px|EOVSA]]
EOVSA (Expanded Owens Valley Solar Array) is a solar-dedicated radio interferometer operated by the New Jersey Institute of Technology. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at *451 science frequency channels each second, in four polarization products (XX, YY, XY, YX), as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system whose data flow is shown in the block diagram (Figure 1). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays).

EOVSA delivers the radio interferometry data on the following three levels:
[[file:EOVSA_data_levels.PNG|thumb|right|500px|Figure 1: Pipeline block diagram]]
* '''Level 0''' - Raw visibility data - This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
* '''Level 1''' - Calibrated visibility data - After applying calibration and other preliminary processing to level 0 data, we create the calibrated visibility data in CASA ms format (second column in Figure 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. The following tutorial will guide you through how to make EOVSA images and spectrogram from visibility data. We provide a set of standard ms’s for each day (pink solid boxes in Figure 1) for users who wish to start with visibility data. You can retrieve EOVSA 1-min averaged visibility data in CASA ms format from this [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ page]. Full-resolution EOVSA visibility data in CASA ms format will be provided per request. Please contact the *EOVSA team if you wish to have Level 1 visibility data for a specific event.
* '''Level 2''' - Images and spectrogram data in standard FITS format - Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

=2. Browsing and Obtaining EOVSA data=
[[file:EOVSA_Browser.PNG|thumb|right|500px|Figure 2: EOVSA Browser]]
[[file:RHESSI_browser.png|thumb|right|500px|Figure 3: RHESSI Browser]]

====EOVSA Browser====
The most convenient way for browsing '''Level 2''' EOVSA data is through the [http://ovsa.njit.edu/browser EOVSA Browser]. The overview EOVSA dynamic spectra on the top-left panel are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare emission can be seen in the dynamic spectra, which usually appears as vertical bright features across many frequency bands. The pipeline quicklook full-disk images are also displayed for the given frequencies along with multi-wavelength full-disk maps such as SDO AIA, HMI, BBSO H-alpha by hovering on the buttons on the bottom of each map.. More information can be found on this [http://ovsa.njit.edu/data-browsing.html page]. The figure on the right shows an example of the overview EOVSA dynamic spectrum for 2021 May 07.

====RHESSI Browser====
EOVSA data can also be browsed through [http://sprg.ssl.berkeley.edu/~tohban/browser/ RHESSI Browser]. Check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also, note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at [http://www.ovsa.njit.edu/fits/IDB/ this link]. Each data file is usually 10 minutes in duration. The name convention is YYYYMMDD (folder name) /IDBYYYYMMDDHHMMSS (file name), where the time in the file name indicates the start time of the visibility data.

====Other useful links====

* [https://join.slack.com/t/eovsatutorial-2021/shared_invite/zt-sob838f4-zvs_qH3M4yTbWGlPIiw6og EOVSA User Forum]
* [http://www.ovsa.njit.edu/wiki/index.php/Recent_Flare_List_(2021) EOVSA Flare list]
* [http://ovsa.njit.edu/browser EOVSA browser]
* [http://www.ovsa.njit.edu/wiki/index.php/Expanded_Owens_Valley_Solar_Array EOVSA wiki]
* [http://www.ovsa.njit.edu/fits/UDBms_slfcaled/ EOVSA 1-min averaged CASA ms database]
* [https://github.com/suncasa/suncasa-src SunCASA on Github] and [https://pypi.org/project/suncasa/ PyPI]
* Flare pipeline?

=3. Software requirements and installation=
EOVSA data processing and analysis are developed over two packages:
* '''[https://github.com/suncasa/suncasa SunCASA]''' A Python wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (3.6, 3.10*) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).

* '''[https://github.com/Gelu-Nita/GSFIT GSFIT]''' A IDL-widget (GUI)-based spectral fitting package called GSFIT, which provides a user-friendly display of EOVSA image cubes and an interface to fast-fitting codes (via platform-dependent shared-object libraries).

For this tutorial, we will demonstrate using SunCASA to create EOVSA image and spectrogram from the visibility data observed for an M class flare on 2021 May 07. There are two approaches to accessing the SunCASA package:

# Through [https://colab.research.google.com/ Google Colaboratory (colab)] which hosts a free virtual environment that requires no setup and runs entirely in the cloud. For example, the [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA workspace] of this tutorial explains the analysis setup.
# Install on your own machine, and run the notebook as a regular Jupyter Notebook. See [http://ovsa.njit.edu//wiki/index.php/EOVSA_Data_Analysis_Tutorial_2022#Installation_of_SunCASA_(optional) SunCASA Installation] section below, also [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation this page] and [http://www.ovsa.njit.edu/wiki/index.php/GSFIT_Installation GSFIT Installation] for instructions.

=4. Download Sample EOVSA Data for the Tutorial=

For this tutorial, we use an M4 class flare on 07 May 2021, around 19:00 UT occurred near the solar east limb as viewed from Earth, and was well observed by Solar Orbiter (including the STIX instrument). For other recent EOVSA events, check here.
Here, we provide a calibrated and self-calibrated EOVSA visibility dataset in CASA ms format (IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms) which is ready for science analyses purposes,

=5. Information on the EOVSA Visibility Data (CASA Measurement Set)=

With the pip installation, the CASA modules may be used in a standard Python manner. For example, CASA tasks can be invoked using import, while CASA tools are Python classes that first need to be instantiated to create usable objects. A useful first task to run is [https://casa.nrao.edu/docs/taskref/listobs-task.html listobs], which provides a summary of the measurement set.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from casatasks import listobs
import os
msfile = 'IDB20210507_1840-1920XXYY.cal.10s.slfcaled.ms'
rc = listobs(vis=msfile, listfile = msfile + '.listobs', overwrite=True)

print(os.popen("cat " + msfile + ".listobs").read())
</pre>
[[file:Listobs_20210507.png|thumb|center|1400px|Figure 4: Output of listobs task]]

=6. Self-calibration (optional)=
Self-calibration is often needed in bringing out details of the flare at multiple frequencies. An example script, run in SunCASA, for doing self-calibration of the 2017 Aug 21 flare at ~20:20 UT can be found at [http://www.ovsa.njit.edu/wiki/index.php/Tohban_EOVSA_Imaging_Tutorial_A-Z here]. The EOVSA workspace discussed along with this tutorial anyway uses the self-calibrated dataset for analysis.

=7. Cross-Power Dynamic Spectrum=
The first module we introduce is [https://github.com/suncasa/suncasa/blob/main/suncasa/utils/dspec.py''dspec'']. This module allows you to generate a cross-power dynamic spectrum from an MS file, and visualize it. You can select a subset of data by specifying a [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu112.html time range], [https://casaguides.nrao.edu/index.php/Selecting_Spectral_Windows_and_Channels spectral windows/channels], [https://casaguides.nrao.edu/index.php/Antenna/Baseline_Selection_Syntax_with_or_without_Autocorrelations antenna baseline], or [https://casa.nrao.edu/Release3.3.0/docs/UserMan/UserMansu113.html uvrange]. The selection syntax follows the ''CASA'' convention. More information of CASA selection syntax may be found in the above links or the [https://casa.nrao.edu/casadocs/casa-5.4.0/data-selection/data-selection-in-a-measurementset Measurement Selection Syntax].

[[file:Dynspec_20210507.png|thumb|right|300px|Figure 4: EOVSA cross power dynamic spectrum at stokes XX polarization]]

<pre style="background-color: #FCEBD9;">
## in SunCASA
from suncasa import dspec as ds
import time
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## The example below shows the cross-power spectrogram from a baseline selected using the parameter "bl".
## bl = '4&9' means selecting a baseline from Antenna ID 4 (Antenna Name "eo05") correlating with Antenna ID 9
## (Antenna Name "eo10") - refer listobs output.
## you can also use the "bl" parameter to select multiple baseline(s), i.e., bl='0&2;4&9;8&11'.

## Hover over "Dspec" in the next line to see additional arguments such as frequency range ("spw"), and time range ("timeran")
## or use help(ds.Dspec)
## this step generates a dynamic spectrum and saves it to specfile as a numpy .npz file
d = ds.Dspec(msfile, bl='4&9', specfile=specfile)
d.plot(vmin=None, vmax=None, pol='XX')
print(d.data.shape) # The shape gives the dimensions of polarization, baselines, frequencies, time
</pre>

Alternative methods of using the "dspec" task are discussed in the EOVSA workspace.

NOTE: If you are using your own machine, the plotting should be in interactive mode. In that mode, hovering your mouse over the dynamic spectrum allows you to read the time, frequency, and flux information under the cursor at the bottom of the window.

=8. Quick-Look Imaging=
We use CASA's CLEAN algorithm for EOVSA imaging. As the default coordinate system is equatorial, solar coordinate transformation and image registration need to be done to place the image into the usual Helioprojective Cartesian Coordinates (X- and Y-axes are Solar X and Solar Y in arcsecond unit, respectively). We have bundled a number of these steps into a module named [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images.

Now, let us start with making a summary plot of the EOVSA image at the spectral windows 2 to 5 (2.4–3.7 GHz).

[[file:EOVSAimg_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image]]

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
## in SunCASA
from suncasa.utils import qlookplot as ql
## (Optional) Supply the npz file of the dynamic spectrum from previous step.
## If not provided, the program will generate a new one from the visibility.

timerange = '19:02:00~19:02:10' ## set the time interval
spw = '2~5' ## select frequency range
stokes = 'XX' ## select stokes XX
plotaia = False ## Initially, turn off AIA image plotting, default is True

ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=plotaia, restoringbeam=['60arcsec'], robust=0.5)
</pre>

The empty panels are there as only one polarization is selected for imaging/spectroscopy.

Feel free while working in [https://colab.research.google.com/drive/19NQb6Emb9HvKX4QHq9ZYCP3RM6nT7sDL#scrollTo=cLdDVptBGG-X EOVSA Workspace] to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

====Quick-look Imaging with AIA data====

With [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''], it is easy to engage solar data from SDO/AIA in the summary plot. Setting ''plotaia=True'' in the [https://github.com/suncasa/suncasa/blob/master/utils/qlookplot.py ''qlookplot''] command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

[[file:EOVSA_AIA_20210507.png|thumb|right|400px|EOVSA full-Sun single-band quicklook image with SDO/AIA as background]]

<pre style="background-color: #FCEBD9">
## in SunCASA
outfits = 'EOVSA_20210507T190205.000000.outim.image.fits'
ql.qlookplot(vis=msfile, specfile=specfile, timerange=timerange, spw=spw, \
stokes=stokes, plotaia=True)
</pre>

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file Instrument_yymmddTHHMMS.ffffff.outim.image.fits under your working directory. An alternate name for the output fits file can be specified using the outfits parameter. In this example, we combine all selected frequencies (specified in keyword spw) into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, the fourth axis only has one plane as well (XX). Here is the relevant information (first few lines) in the header of the resulting FITS file.

<pre>
In [1]: from astropy.io import fits
In [2]: hdu = fits.open('EOVSA_20210507T190230.000000.outim.image.fits')[0]
In [3]: hdu.header
Out[3]:
SIMPLE = T / conforms to FITS standard
BITPIX = -64 / array data type
NAXIS = 4 / number of array dimensions
NAXIS1 = 512 / Nx
NAXIS2 = 512 / Ny
NAXIS3 = 1 / number of frequencies
NAXIS4 = 1 / number of polarizations
</pre>

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center (xycen), pixel scale (cell), and image field of view (fov). Here we show an example that images for each spectral window in this data set (from spw 0 to 49) at the same time interval around 19:02 UT. At high frequencies, the microwave source concentrates near the flare site, corresponding pretty well with the flare kernel in SDO/AIA. At low frequencies, the microwave source extends to higher altitudes in the corona. Again, feel free to change some of these parameters to explore what they do.

====Quick-look Imaging with AIA data and all spw====
Next, we will make images for every single spectral window in this data set (from spw 0 to 47).

[[file:EOVSA_AIA_allspw_20210507.png|thumb|right|400px|EOVSA multi-band quicklook image]]

<pre style="background-color: #FCEBD9">
## in SunCASA
from suncasa.utils.mstools import time2filename
timerange = '19:02:00~19:02:10' ## set the time interval
spw = ['{}'.format(l) for l in range(48)]. ## select (almost) all spectral windows from spw id #0 to #47
outfits = time2filename(visibility_data,timerange=timerange)+'.outim.image.allbd.fits'

stokes = 'XX'. ## select stokes XX
xycen = [-900, 280] ## image center for clean in solar X-Y in arcsec
cell=['2.0arcsec'] ## pixel scale
imsize=[128] ## number of pixels in X and Y. If only one value is provided, NX = NY
fov = [300,300] ## field of view of the zoomed-in panels in unit of arcsec

plotaia = True ## turn off AIA image plotting, default is True
aiawave = 1600 ## AIA passband in Å. The options are [171,131,304,335,211,193,94,1600,1700]
acmap = 'gray_r' ## Choose the coloar map for AIA images. If not provided, the program will use default AIA colormap.

ql.qlookplot(vis=visibility_data, specfile=specfile, timerange=timerange,
spw=spw, stokes=stokes, plotaia=plotaia, aiawave=aiawave,
restoringbeam=['60arcsec'], robust = 0.5, acmap=acmap,
imsize=imsize,cell=cell,xycen=xycen,fov=fov,
outfits=outfits,overwrite=False)
</pre>

=9. Downloading fits files to your local system with Batch-Mode Imaging=
This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.
====Producing a 30-band map cube for a given time====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.
<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_example.py ./
</pre>

[[file:fig-specimg.png|thumb|right|300px|Example multi-frequency images at a single time integration]]

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.
<pre>
################### USER INPUT GOES IN THIS BLOK ################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
trange = '2017/08/21/20:21:00~2017/08/21/20:21:30' # time range for imaging
xycen = [380., 50.] # center of the output map (in solar X and Y, Unit: arcsec)
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # Use all 13 EOVSA 2-m antennas by default
npix = 256 # number of pixels in the image
cell = '1arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
outdir = './images' # Specify where to save the output fits files
outimgpre = 'EO' # Something to add to the output image name
</pre>

Then, run the script in SunCASA via
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_example.py')
</pre>

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.
<pre>
NAXIS = 4 / number of array dimensions
NAXIS1 = 128
NAXIS2 = 128
NAXIS3 = 30
NAXIS4 = 2
</pre>

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to [[#Plotting Images in SSWIDL|Section 3.3.3.3]].

====Producing a time series of 30-band maps (a 4-d cube)====
An example script can be found at [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/imaging_timeseries_example.py this Github link]. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

<pre style="background-color:#FCEBD9;">
# in SunCASA
CASA <##>: cd your_working_directory
CASA <##>: !cp /common/data/eovsa_tutorial/imaging_timeseries_example.py ./
</pre>
Similar as the previous script, change inputs in the following block of your copy of the "imaging_timeseries_example.py" script and save the changes.

[[file:fig-specmovie.png|thumb|right|300px|Example multi-frequency images at one time frame in the [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html output javascript movie]]]
<pre>
################### USER INPUT GOES IN THIS BLOK ####################
vis = 'IDB20170821201800-202300.4s.slfcaled.ms' # input visibility data
specfile = vis + '.dspec.npz' ## input dynamic spectrum
nthreads = 1 # Number of processing threads to use
overwrite = True # whether to overwrite the existed fits files.
trange = '' # time range for imaging, default to all times in the data
twidth = 1 # make one image out of every 2 time integrations
xycen = [380., 50.] # center of the output map in solar X and Y. Unit: arcsec
xran = [340., 420.] # plot range in solar X. Unit: arcsec
yran = [10., 90.] # plot range in solar Y. Unit: arcsec
antennas = '' # default is to use all 13 EOVSA 2-m antennas.
npix = 128 # number of pixels in the image
cell = '2arcsec' # pixel scale in arcsec
pol = 'XX' # polarization to image, use XX for now
pbcor = True # correct for primary beam response?
grid_spacing = 5. * u.deg # grid spacing in degrees
outdir = './image_series/' # Specify where to save the output fits files
imresfile = 'imres.npz' # File to write summary of the imaging results
outimgpre = 'EO' # Something to add to the image name
</pre>

Then, run the script in SunCASA by
<pre style="background-color:#FCEBD9;">
CASA <##>: execfile('imaging_timeseries_example.py')
</pre>

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as [[#Producing a 30-band map at a given time|the previous section]]. The summary yields the time, frequency, and path to every image.
<pre style="background-color:#FCEBD9;">
CASA <##>: imres = np.load(outdir + imresfile)['imres'].item()
CASA <##>: imres.keys()
['FreqGHz', 'ImageName', 'Succeeded', 'BeginTime', 'EndTime']
CASA <##>: ls image_series/*.fits | head -5
image_series/EO20170821T201800.500_ALLBD.fits
image_series/EO20170821T201804.500_ALLBD.fits
image_series/EO20170821T201808.500_ALLBD.fits
image_series/EO20170821T201812.500_ALLBD.fits
image_series/EO20170821T201816.500_ALLBD.fits
</pre>
The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a [https://web.njit.edu/~sjyu/download/eovsa-tutorial/movie.html javascript movie].

====Plotting Images in SSWIDL====
All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with [https://hesperia.gsfc.nasa.gov/rhessidatacenter/complementary_data/maps/ the SSWIDL map suite] which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the [https://github.com/binchensun/eovsa-tutorial/tree/master/rhessi18 Github repository of our tutorial]. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_readfits.pro casa_readfits.pro]: read an array of multi-frequency FITS files into FITS header (index) and data.
* [https://github.com/binchensun/eovsa-tutorial/blob/master/rhessi18/casa_fits2map.pro casa_fits2map.pro]: convert an array of multi-frequency FITS files into an array of SSWIDL map structure (adapted from P. T. Gallagher's hsi_fits2map.pro)

First, start SSWIDL from command line:
<pre style="background-color:#FCEBD9;">
sswidl

file = 'EOVSA_20210507T190205.000000.outim.image.allbd.fits'
fits2map,file,eomaps
window,0,xs=1000,ys=800
loadct,3
!p.multi=[0,6,5]
for i=0, 29 do plot_map,eomaps[i],grid=5.
</pre>

Find and convert the fits files:
<pre style="background-color:#FCEBD9;">
; cd to your path that contains casa_readfits.pro and casa_fits2map.pro. If on Virgo, just add the path
IDL> add_path, '/common/data/eovsa_tutorial/'
IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles[45:54], eomaps [, /calcrms, rmsxran=[260., 320.], rmsyran=[-70., 0.]]
</pre>

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]
<pre style="background-color:#FCEBD9;">
IDL> help,eomaps
EOMAPS STRUCT = -> <Anonymous> Array[30, 1, 10]
</pre>
They have compatible keywords recognized by, e.g., plot_map.pro. Example of the output map keywords:
<pre style="background-color:#FCEBD9;">

IDL> help,omaps,/str
** Structure <cd28140>, 24 tags, length=132272, data length=132272, refs=2:
DATA DOUBLE Array[128, 128]
XC DOUBLE -901.02022
YC DOUBLE 278.99427
DX DOUBLE 2.0000000
DY DOUBLE 2.0000000
TIME STRING ' 7-May-2021 19:02:00.000'
ID STRING 'EOVSA XX 1.256 GHz'
DUR DOUBLE 9.9999998
XUNITS STRING 'arcsec'
YUNITS STRING 'arcsec'
ROLL_ANGLE DOUBLE 0.00000000
ROLL_CENTER DOUBLE Array[2]
FREQ DOUBLE 1.2557989
FREQUNIT STRING 'GHz'
STOKES STRING 'XX'
DATAUNIT STRING 'K'
DATATYPE STRING 'Brightness Temperature'
BMAJ DOUBLE 0.021222222
BMIN DOUBLE 0.021222222
BPA DOUBLE -337.28110
RSUN DOUBLE Array[48]
L0 FLOAT Array[48]
B0 DOUBLE Array[48]
COMMENT STRING 'Converted by CASA_FITS2MAP.PRO'
</pre>

[[file:fig-specimg_idl.png|thumb|right|300px|Example multi-frequency EOVSA images at one time plotted in SSWIDL]]
An example for plotting all images at a selected time:
<pre style="background-color:#FCEBD9;">
; choose a time pixel
IDL> plttime = anytim('2021-0-07T19:02:00')
IDL> dt = min(abs(anytim(eomaps[0,0,*].time)-plttime),tidx)
; use maps at this time, first polarization, and all bands
IDL> eomap = reform(eomaps[*,0,tidx])
IDL> window,0,xs=1000,ys=800
IDL> !p.multi=[0,6,5]
IDL> loadct, 3
IDL> for i=0, 29 do plot_map,eomap[i],grid=5.,$
tit=string(eomap[i].freq,format='(f5.2)')+' GHz', $
chars=2.0,xran=[340.,420.],yran=[10.,90.]
</pre>

=Installation of SunCASA (optional)=
If you want to install SunCASA on your local machine follow this section. If not, run SunCASA directly on the Colab Workspace.

PIP wheels for SunCASA and its CASA dependencies are available as a Python 3 module from [https://pypi.org/ PyPI]. This allows simple installation across most of the UNIX-BASED PLATFORMS (Linux & macOS) and import into standard *Python 3.6* environments. The RHEL 6/7 are the officially supported platforms for the casa modules. We have also tested the SunCASA/CASA packages under other Linux-based platforms such as Ubuntu 18, Scientific Linux 6/7, CentOS 7, as well as macOS Big Sur to some extent. YMMV for other versions of Linux or macOS. We do not recommend the use of Conda until CASA's compatibility with Conda is better understood.

====Requirements====
The following prerequisites must be present on the host machine before installing SunCASA:

* '''Python 3.6''' (3.7 may also work, its compatibility with CASA is not fully understood yet)
* '''For Linux:''' libgfortran3 ([https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum yum] or [https://help.ubuntu.com/community/AptGet/Howto) apt-get] install)
* '''For MacOS:''' gcc ([https://brew.sh/) brew install], [https://www.xquartz.org/ XQuartz] and [https://apps.apple.com/us/app/xcode/id497799835?mt=12 Xcode])

====Installating SunCASA====
Installation instructions are as follows (from a UNIX terminal window). First create a Python virtual environment named suncasaenv under the $HOME directory:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ cd $HOME
$ python3.6 -m venv suncasaenv
</pre>

'''NOTE:''' We strongly recommend using a Python virtual environment to prevent breaking any packages within a pre-existing Python environment.

Then use [https://pypi.org/project/suncasa/ pip] to install SunCASA within the newly-created virtual environment:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
$ source suncasaenv/bin/activate
(suncasaenv) $ pip install --upgrade pip
(suncasaenv) $ pip install suncasa
</pre>

'''NOTE:''' If this does not work, it could be due to unsuccessful installation of some dependencies. Running these commands should address this.

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install casatasks
(suncasaenv) $ pip install casatools
(suncasaenv) $ pip install casadata
(suncasaenv) $ pip install PyQt5
(suncasaenv) $ pip install sunpy[all]
(suncasaenv) $ pip install suncasa
</pre>

To exit the python venv, type "deactivate" from the terminal. However, the rest of this tutorial '''assumes the venv is active''' (to reactivate, type source $HOME/suncasaenv/bin/activate)

====Updating a previous installation of SunCASA====
You can update suncasa to its latest version by running:
<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ pip install --upgrade suncasa
</pre>

====Sanity check====
With the pip installation, SunCASA, as well as CASA, may be used in a standard Pythonic manner. For example, SunCASA modules and CASA tasks can be invoked using “import”, while CASA tools first need to be instantiated:

<pre style="background-color: #FCEBD9;overflow: auto;width: auto;">
(suncasaenv) $ ipython
Python 3.6.9 (default, Jun 16 2021, 22:21:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import suncasa
In [2]: help(suncasa)
In [3]: import casatasks
In [4]: help(casatasks)
</pre>

The use of python3 venv is a simple built-in method of containerizing the pip install such that multiple versions of SunCASA can be kept on a single machine in different environments. In addition, SunCASA is built and tested using standard (python 3.6) libraries which can be replicated with a fresh venv, keeping the libraries needed for SunCASA isolated from other libraries which may already be installed on your machine.