Difference between revisions of "EOVSA Imaging Workshop"

From EOVSA Wiki
Jump to: navigation, search
(Set up the software)
(Set up the software)
Line 28: Line 28:
 
* add your installed anaconda "site-packages" to ~/.casa/init.py. On a Mac, the default path is ~/anaconda/lib/python2.7/site-packages
 
* add your installed anaconda "site-packages" to ~/.casa/init.py. On a Mac, the default path is ~/anaconda/lib/python2.7/site-packages
 
<pre>
 
<pre>
sys.path.append('FULL PATH TO YOUR CONDA site-packages') #note: "~" for home directory does not work here. You need to make it the full path, or use os.path.expanduser()
+
sys.path.append('FULL PATH TO YOUR CONDA site-packages')  
 
</pre>
 
</pre>
 +
#note: "~" for home directory does not work here. You need to make it the full path, or use os.path.expanduser()
  
Another way to install SunPy or AstroPy is as following
+
Another way to install SunPy or AstroPy is as following -- this may fail if the CASA version does not match the most updated sunpy version, unfortunately.
 
<pre>
 
<pre>
 
from setuptools.command import easy_install
 
from setuptools.command import easy_install

Revision as of 03:39, 4 January 2018

Sample EOVSA Data

To get started, we have made some calibrated flare and active region data in CASA measurement set format, available here. Here are some things to consider when working with the data:

  • Antennas 1-8 and 12 have a wide sky coverage, while antennas 9-11 and 13 are the older equatorial mounts that can only cover +/- 55 degrees from the meridian. All antennas were tracking for the flare, but for portions of the all-day scan (roughly before 1600 UT or after 2400 UT) only 9 antennas will be tracking. Data for non-tracking antennas is NOT flagged, so if you try to image these periods, you'll need to manually omit ants 9-11 and 13.
  • Because of the 2.5 GHz high-pass filters, data are taken only for bands 4-34. These will be labeled spectral window (spw) 0-31 in the measurement sets. However, band 4 (spw 0) is currently not calibrated, so only the 31 spectral windows 1-31 (bands 5-34, or frequencies 3-18 GHz) are valid data. You'll need to omit spw 0 when making images.
  • In principle, circular polarization should be valid and meaningful, but no polarization calibration has been done (yet) so no non-ideal behavior (leakage terms, etc.) have been accounted for.

Here are some basic information on CASA and the software we are developing:

  • Obtaining CASA; CASA Guides
  • suncasa A CASA/python-based package being developed for imaging and visualizing spectral imaging data. A possibly useful (but perhaps buggy) routine (helioimage2fits.py) is available for registering EOVSA/VLA (and possibly ALMA) CASA images.

Set up the software

  • Download latest version of CASA CASA. I'm using version 4.7.0, but latest version should work.
  • Download the suncasa package from the github page, and put it under YOURPATH as suncasa
  • Create ~/.casa/init.py, and put the following lines in:
import sys
sys.path.append('YOURPATH')
  • Install SunPy (and Astropy, if CASA version < 5.0) into CASA. The fastest way is to install SunPy and AstroPy with Anaconda, and add the site-package path to the ~/.casa/init.py file. If you don't have Anaconda, refer to Conda installation guide. With Anaconda installed, open a system command prompt and install sunpy and astropy as following
conda config --add channels conda-forge
conda install sunpy
conda install astropy

Note: if you are using CASA 5.0 or higher, there is no need for installing AstroPy for CASA.

  • add your installed anaconda "site-packages" to ~/.casa/init.py. On a Mac, the default path is ~/anaconda/lib/python2.7/site-packages
sys.path.append('FULL PATH TO YOUR CONDA site-packages') 
  1. note: "~" for home directory does not work here. You need to make it the full path, or use os.path.expanduser()

Another way to install SunPy or AstroPy is as following -- this may fail if the CASA version does not match the most updated sunpy version, unfortunately.

from setuptools.command import easy_install
easy_install.main(['--user', 'pip'])
import pip
pip.main(['install', 'astropy'])
pip.main(['install', 'sunpy''])
  • Try if you can call suncasa.utils.helioimage2fits, in casa
from suncasa.utils import helioimage2fits as hf

Flare Data

Folder Flare_20170910 contains two measurement sets at full 1-s time resolution, each with 10-min duration:

  1. IDB20170910154625.corrected.ms (171 MB) [1] containing the first 10 minutes of the flare (15:46:25-15:56:25 UT), calibrated but no self-calibration
  2. IDB20170910154625.corrected.ms.xx.selfcaled (66 MB) [2] self-calibrated data for polarization XX
  3. IDB20170910155625.corrected.ms (232 MB) [3] containing the second 10 minutes of the flare (15:56:25-16:06:25 UT)
  4. IDB20170910155625.corrected.ms.xx.selfcaled (77 MB) [4] self-calibrated data for polarization XX

All Day Quiet Sun/Active Region Data

Folder AR_20170710 contains 8 calibrated measurements sets at 60-s time resolution.

  1. The seven UDB20170710*.corrected.ms are files for each scan separately, ranging in size from 7 - 150 MB depending on the length of the scan.
  2. UDB20170710_allday.ms (268 MB) [5] is the same data concatenated into a single file. However, each scan has a separate source ID.

The AR data are also calibrated but not self-calibrated. In fact, we need to create a strategy for self-calibration of such data. You can get an overview of the AR coverage on that day by looking at the plot at http://ovsa.njit.edu/flaremon/daily/2017/XSP20170710.png. The separate scan files above are labeled with the time of the start of the scan, and each scan covers the time of continuous solar data between each calibration scan.

Make an EOVSA Flare Image

Download this CASA script. It does basic clean on the full Sun using this slfcaled measurement set data near the flare peak (1600 UT) with 10-s integration, as well as image registration (into standard FITS file with helioprojective coordinates), and plotting using sunpy.

Fig. 1: Example EOVSA image for 2017 Sep 10 flare at around 1600 UT.

In CASA, cd to your data directory and do the following

execfile('imaging_example_20170910T1600.py')

You are supposed to see an image popped up on your screen.

Possible Topics

We think that the best approach to the imaging workshop is not for everyone to work on the same topic, but rather to consider (and perhaps prioritize) different topics and then put together small teams of people who might be interested in investigating them. Here is a list of some possible topics, but in no prioritized order. Many topics will be somewhat interconnected, but we should avoid time-wasting situations where people working on one topic have to wait until another, uncompleted topic is done.

Calibration

Most calibrations needed for basic imaging are well in hand, so additional efforts are not really needed for an imaging workshop. Here is a list of (semi-)automatic calibrations we are doing so far. All can be improved, but such improvements will not make any major qualitative difference in the images.

  1. Gain corrections: To account for changes of the attenuation settings (esp. during flares). See this page for details.
  2. Correcting for polarization mixing due to feed rotation: See this page for details.
  3. Antenna-based amplitude calibration: made by referencing total power and auto-correlation amplitudes to RSTN/Penticton total solar flux density measurements. See this page for details.
  4. Reference phase calibration: antenna-based, band-averaged phases derived from "reference" observations of strong celestial radio sources every night. See this page for details]
  5. Daily phase calibration: antenna-based multi-band delays resulting from the phase difference as a function of frequency between those derived from phase calibrators (usually observed 3 times during the day) and the reference calibrator. See this page for details.

There are, however, two other calibrations still being developed:

  1. Polarization calibration: To calibrate absolute degree of circular polarization. We believe that the information already exists in available data, but the procedures need to be worked out. Additional observations are possible during this week, if needed.
  2. Per-channel bandpass (or single-band-delay) calibration within individual bands: To calibrate the phase/amplitude variations from one science channel to another. Procedures need to be worked out.
  3. Real-time spectral-kurtosis flagging of RFI, and other forms of RFI excision.

Pipeline Imaging

We have developed the basis for pipeline imaging of data on a daily basis, several examples of which can be seen here. These 10-min averages at a few bands might be okay, but lots of improvement is needed. Mainly, these improvements should be in the form of better imaging strategies (which are the subject of some of the topics below), and in better data-quality checks. In addition to these 10-min average images, we also envision making shorter (10-s?) images at a few bands during flares. Some of the RHESSI experience might come in handy here. Improving and finalizing the pipeline imaging is a high priority for this meeting!

Flare Imaging

We have succeeded in making flare multi-frequency movies, which are extremely interesting and revealing, but are quite time-consuming and human-intensive. We need strategies to reduce the number of parameters for different imaging cases, and perhaps to automate or aid in clean-box choices. When converting data-cubes to spatially-resolved spectra, issues of total power calibration come into play.

Self-calibration Strategies

(Bin) Currently I am using a sliding window of five bands (band_to_selfcal +- 2 bands) to create a multi-frequency synthesized image as the input model for doing self-calibration at the band in question. Any alternative strategies? How about channel-by-channel selfcal?

Imaging Strategies

Clean? MEM? Optimum way to do spectral imaging? Here is an example made for the Sep 10 flare (Bin).

Quiet Sun and Active Region Imaging

We desperately need help in this area. We have made simple AR images that are quite promising, but full-disk images are still quite bad. Stephen can probably do wonders, and hopefully teach us to do so also. We should explore the use of additional data (say full-disk images from GAVRT) and how that might be used. How do we handle the issue of solar rotation, which can move an active region by multiple arcminutes over a 12-h day?

Self-calibration Strategies

We can probably do some self-calibration based on bright AR, but can we self-calibration on the solar disk? Gordon would like to explore the self-cal corrections vs. time, but it requires someone succeeding in doing an all-day self-calibration in the first place.

Imaging Strategies

Given solar rotation, and the limitations of EOVSA uv coverage, what are the best imaging strategies for full-disk and AR imaging?

Software Development

Currently we are using Python for analyzing calibration data and CASA for script-based imaging.

  1. What is the best approach to make an interactive imaging tool to serve the community (for both experts and non-experts)?
  2. How about the tool to visualize/analyze/make use of the resulted 4-D dynamic spectral imaging data?

Notes

Imaging Priority order

  1. flare selfcal Tim
  2. active region rotation Stephen
  3. solar disk model Stephen
  4. pipeline heuristics Group (Thurs. morning)
  5. pipeline quicklook Bin, Sijie, Sa"m
  6. pipeline parallelization optimization
  7. visibility weighting
  8. user interface Jim, Sa"m

Calibration Priority order

  1. polarization calibration Sam
  2. total power calibration
  3. relative vs spectrum Gordon
  4. gain calibration Gordon
  5. RFI excision Gelu, Gordon, Jim