Difference between revisions of "EOVSA Data products"

From EOVSA Wiki
Jump to: navigation, search
(Level 0 - Raw visibility data from the instrument)
(All-day XP spectrum)
(34 intermediate revisions by the same user not shown)
Line 1: Line 1:
Welcome to the EOVSA data products documentation. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. During January, coverage will be 16-24 UT daily, 7 days/week. EOVSA will provide microwave spectra and imaging at up to 451 frequencies between 1 and 18 GHz. We deliver the radio interferometry data in level 0-2.  
+
Welcome to the EOVSA data products documentation. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. During January, coverage will be 16-24 UT daily, 7 days/week. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system (the block diagram in Figure 1 shows the data flow in the pipeline). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays). Many different CASA ms’s can be created from the raw visibility data by selecting time ranges, frequencies, integration times, polarizations, and so on.  
[[File:pipeline_flowchart.jpg|center|600px]]
+
[[File:pipeline_flowchart.jpg|center|600px|EOVSA pipeline block diagram/flow chart ]]
  
=Data Definition=
 
Our working definition is that Level 0 is the raw visibility data from the instrument, whereas Level 1 corresponds to the calibrated visibility data, which have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. All of these data products are archived and available online, accessed from the EOVSA server.
 
  
==Level 0 - Raw visibility data from the instrument==
+
We deliver the radio interferometry data on the following three levels:
TBD
+
  
==Level 1 - Calibrated visibility data==
 
Level 1 data have all of the required content to produce Level 2 data products.
 
===CASA 1-min averaged ms===
 
EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:
 
  
http://www.ovsa.njit.edu/fits/UDBms_slfcaled
+
=Level 0 - Raw visibility data from the instrument=
  
===CASA Event ms===
+
As outlined in Figure 1, the real-time pipeline creates raw data products in the left-hand column labeled Level 0. This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
TBD
+
  
==Level 2 - Images and spectrogram data in standard FITS format ==
+
=Level 1 - calibrated visibility data=
 +
After applying calibration and other preliminary processing to level 0 data, we create the CASA ms’s in the second column in Figure 1, labeled Level 1. These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format.  We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who wish to start with visibility data, while the full-resolution, raw visibility data are made available through links (TBD) to our server.
  
===Getting level 2 data===
+
==CASA 1-min averaged ms==
 +
EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:
  
EOVSA Level 2 data products can be retrieved from the "synoptic data" page:
+
http://www.ovsa.njit.edu/fits/UDBms_slfcaled
  
[http://www.ovsa.njit.edu/fits/synoptic http://www.ovsa.njit.edu/fits/synoptic]
+
==CASA Event ms==
 +
TBD
  
'''Example - get the synoptic data on December 25, 2019'''
+
=Level 2 - Images and spectrogram data in standard FITS format =
* go to [http://ovsa.njit.edu/browser/ EOVSA browser] page.
+
 
* Browse to December 25, 2019.
+
Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).
* Click "synoptic fits" button next to the calendar tool.
+
* Select the data product you want to download based on the name in the following list.
+
  
 
'''List of Level 2 data products'''
 
'''List of Level 2 data products'''
Line 49: Line 42:
 
|-
 
|-
 
| Synoptic 1.4 GHz image
 
| Synoptic 1.4 GHz image
| eovsa_20191225.spw00-01.tb.disk.fits
+
| eovsa_yyyymmdd.spw00-01.tb.disk.fits
 
|-
 
|-
 
| Synoptic 3.0 GHz image
 
| Synoptic 3.0 GHz image
| eovsa_20191225.spw02-05.tb.disk.fits
+
| eovsa_yyyymmdd.spw02-05.tb.disk.fits
 
|-
 
|-
 
| Synoptic 4.5 GHz image
 
| Synoptic 4.5 GHz image
| eovsa_20191225.spw06-10.tb.disk.fits
+
| eovsa_yyyymmdd.spw06-10.tb.disk.fits
 
|-
 
|-
 
| Synoptic 6.8 GHz image
 
| Synoptic 6.8 GHz image
| eovsa_20191225.spw11-20.tb.disk.fits
+
| eovsa_yyyymmdd.spw11-20.tb.disk.fits
 
|-
 
|-
 
| Synoptic 10.2 GHz image
 
| Synoptic 10.2 GHz image
| eovsa_20191225.spw21-30.tb.disk.fits
+
| eovsa_yyyymmdd.spw21-30.tb.disk.fits
 
|-
 
|-
 
| Synoptic 13.9 GHz image
 
| Synoptic 13.9 GHz image
| eovsa_20191225.spw31-43.tb.disk.fits
+
| eovsa_yyyymmdd.spw31-43.tb.disk.fits
 
|}
 
|}
  
===Reading level 2 data===
+
==Getting level 2 data==
====Software====
+
 
 +
EOVSA Level 2 data products can be retrieved with the following steps:
 +
* go to [http://ovsa.njit.edu/browser/ EOVSA browser] page.
 +
* Browse to the date of interest.
 +
* Click "synoptic fits" button next to the calendar tool.
 +
* Select the data product you want to download based on the names listed in the table above.
 +
 
 +
==Reading level 2 data==
 +
===Software===
 
[https://github.com/suncasa/suncasa suncasa]
 
[https://github.com/suncasa/suncasa suncasa]
  
====All-day TP spectrum====
+
===All-day TP spectrum===
 
Daily total power full-Sun-integrated spectrogram calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution.
 
Daily total power full-Sun-integrated spectrogram calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution.
  
Line 77: Line 78:
 
from suncasa.eovsa import eovsa_dspec as ds
 
from suncasa.eovsa import eovsa_dspec as ds
 
## define the dynamic spectrum file.
 
## define the dynamic spectrum file.
tpfits = 'EOVSA_TPall_20191225.fts'
+
tpfits = 'EOVSA_TPall_20190504.fts'
 
## display the dynamic spectrum.  
 
## display the dynamic spectrum.  
 
ds.dspec(tpfits)
 
ds.dspec(tpfits)
Line 83: Line 84:
 
[[File:TPSP.jpeg|center|500px]]
 
[[File:TPSP.jpeg|center|500px]]
  
====All-day XP spectrum====
+
===All-day XP spectrum===
 
Daily cross power spectrogram (mean flux from baselines with "intermediate" lengths) calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution. XP spectrogram has much better sensitivity to weak bursts.
 
Daily cross power spectrogram (mean flux from baselines with "intermediate" lengths) calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution. XP spectrogram has much better sensitivity to weak bursts.
  
Line 89: Line 90:
 
from suncasa.eovsa import eovsa_dspec as ds
 
from suncasa.eovsa import eovsa_dspec as ds
 
## define the dynamic spectrum file.
 
## define the dynamic spectrum file.
tpfits = 'EOVSA_XPall_20191225.fts'
+
tpfits = 'EOVSA_XPall_20190504.fts'
 
## display the dynamic spectrum.  
 
## display the dynamic spectrum.  
 
ds.dspec(tpfits)
 
ds.dspec(tpfits)
Line 96: Line 97:
 
[[File:XPSP.jpeg|500px| center]]
 
[[File:XPSP.jpeg|500px| center]]
  
====Synoptic 6-band Images====
+
===Synoptic 6-band Images===
 
Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature. [[File:synoptic_image.jpg| center |400px]]
 
Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature. [[File:synoptic_image.jpg| center |400px]]
  
====10-min 6-band Images====
+
===10-min 6-band Images===
====Event images====
+
===Event images===

Revision as of 00:29, 24 March 2020

Welcome to the EOVSA data products documentation. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. During January, coverage will be 16-24 UT daily, 7 days/week. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system (the block diagram in Figure 1 shows the data flow in the pipeline). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays). Many different CASA ms’s can be created from the raw visibility data by selecting time ranges, frequencies, integration times, polarizations, and so on.

EOVSA pipeline block diagram/flow chart


We deliver the radio interferometry data on the following three levels:


Level 0 - Raw visibility data from the instrument

As outlined in Figure 1, the real-time pipeline creates raw data products in the left-hand column labeled Level 0. This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.

Level 1 - calibrated visibility data

After applying calibration and other preliminary processing to level 0 data, we create the CASA ms’s in the second column in Figure 1, labeled Level 1. These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format. We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who wish to start with visibility data, while the full-resolution, raw visibility data are made available through links (TBD) to our server.

CASA 1-min averaged ms

EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/UDBms_slfcaled

CASA Event ms

TBD

Level 2 - Images and spectrogram data in standard FITS format

Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

List of Level 2 data products

category data product name
Dynamic Spectrum All-day TP spectrum EOVSA_TPall_yyyymmdd.fts
All-day XP spectrum EOVSA_XPall_yyyymmdd.fts
Image
Synoptic 1.4 GHz image eovsa_yyyymmdd.spw00-01.tb.disk.fits
Synoptic 3.0 GHz image eovsa_yyyymmdd.spw02-05.tb.disk.fits
Synoptic 4.5 GHz image eovsa_yyyymmdd.spw06-10.tb.disk.fits
Synoptic 6.8 GHz image eovsa_yyyymmdd.spw11-20.tb.disk.fits
Synoptic 10.2 GHz image eovsa_yyyymmdd.spw21-30.tb.disk.fits
Synoptic 13.9 GHz image eovsa_yyyymmdd.spw31-43.tb.disk.fits

Getting level 2 data

EOVSA Level 2 data products can be retrieved with the following steps:

  • go to EOVSA browser page.
  • Browse to the date of interest.
  • Click "synoptic fits" button next to the calendar tool.
  • Select the data product you want to download based on the names listed in the table above.

Reading level 2 data

Software

suncasa

All-day TP spectrum

Daily total power full-Sun-integrated spectrogram calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution.

from suncasa.eovsa import eovsa_dspec as ds
## define the dynamic spectrum file.
tpfits = 'EOVSA_TPall_20190504.fts'
## display the dynamic spectrum. 
ds.dspec(tpfits)
TPSP.jpeg

All-day XP spectrum

Daily cross power spectrogram (mean flux from baselines with "intermediate" lengths) calibrated in solar flux units are provided at 451 frequencies and 1 s time resolution. XP spectrogram has much better sensitivity to weak bursts.

from suncasa.eovsa import eovsa_dspec as ds
## define the dynamic spectrum file.
tpfits = 'EOVSA_XPall_20190504.fts'
## display the dynamic spectrum. 
ds.dspec(tpfits)
XPSP.jpeg

Synoptic 6-band Images

Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature.
Synoptic image.jpg

10-min 6-band Images

Event images