EOVSA Data products: Difference between revisions

From EOVSA Wiki
Jump to navigation Jump to search
 
(53 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Welcome to the EOVSA data products documentation. EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon, which is season dependent and ranges from 7-12 hours duration centered on 20 UT. During January, coverage will be 16-24 UT daily, 7 days/week. Like any radio interferometer, the fundamental measurement for imaging is the correlated amplitude and phase between each pair of antennas, which is called a “complex visibility.” EOVSA’s 13 antennas form 78 such visibilities at any frequency and instant of time, i.e. 78 measurements of the spatial Fourier transform of the solar brightness distribution. EOVSA records these visibilities at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. These data are then processed through a pipeline processing system (the block diagram in Figure 1 shows the data flow in the pipeline). One of the outputs of the pipeline is a visibility database in a widely used open-standard format called a CASA measurement set (or “ms”; CASA is the Common Astronomy Software Applications package used by many modern interferometer arrays). Many different CASA ms’s can be created from the raw visibility data by selecting time ranges, frequencies, integration times, polarizations, and so on.  
EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon (season dependent and ranges from 7-12 hours duration centered on 20 UT). EOVSA records data at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. Raw data in the visibility domain are processed through a pipeline processing system to produce images and spectrograms in the time- and frequency-dependent image plane (the block diagram in Figure 1 shows the data flow in the pipeline).  
[[File:pipeline_flowchart.jpg|center|600px|EOVSA pipeline block diagram/flow chart ]]
[[File:pipeline_flowchart.jpg|center|600px|EOVSA pipeline block diagram/flow chart ]]


We deliver the radio interferometry data on the following three levels:
We deliver the radio interferometry data on the following three levels:
Line 8: Line 7:
=Level 0 - Raw visibility data from the instrument=
=Level 0 - Raw visibility data from the instrument=


As outlined in Figure 1, the real-time pipeline creates raw data products in the left-hand column (labeled Level 0). This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.
As outlined in Figure 1, EOVSA creates raw data products in the left-hand column (labeled Level 0). This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.


==raw full resolution visibility database==
==Raw "Interim" Database (IDB) visibility data==
EOVSA full resolution visibility database can be retrieved from the following page:
Full-resolution raw "Interim" Database (IDB) visibility data. They are stored in Miriad format, and hence may not be that useful for most people. Be patient after clicking the link--this is a very long list of directories, one for each available date. These data can be retrieved from the following page:


http://www.ovsa.njit.edu/fits//IDB/
http://www.ovsa.njit.edu/fits/IDB/


==raw 1-min averaged visibility database==
==Raw 1-min-averaged visibility data==
EOVSA 1-min averaged visibility database can be retrieved from the following page:
This is the same as for the IDB data, except with 1-minute time integration applied. This is typically not useful for flares, but is perfectly fine for imaging active regions and full Sun. These data can be retrieved from the following page:


http://www.ovsa.njit.edu/fits/UDB/
http://www.ovsa.njit.edu/fits/UDB/


=Level 1 - calibrated visibility data=
=Level 0.5 - Calibrated visibility data=
After applying calibration and other preliminary processing to level 0 data, we create the CASA ms’s in the second column in Figure 1 ( labeled Level 1). These visibility data have all of the required content to produce Level 2 images and spectrogram data in standard FITS format.  We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who wish to start with visibility data.  
After applying calibration and other preliminary processing to the raw (level 0) data, we create the CASA ms’s in the second column in Figure 1 (labeled "level 0.5"). These visibility data are in the Fourier domain of the true images in the plane of the sky and are not immediately ready for spectral imaging analysis yet. However, they have all of the required content to produce images and spectrogram data in standard FITS format (level 1.0).  We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who know how to deal with visibility data.  


==CASA Event ms==
==Calibrated full-resolution visibility data for flare events==
EOVSA event data products in boxes with dashed outlines in Figure 1 will typically be available within 30 days after they are taken.
Calibrated and self-calibrated visibility data for flare events (purple boxes in Figure 1) will typically be available within 7 days after they are taken. They will be released at our flare list site soon: https://ovsa.njit.edu/flarelist


Link: TBD.
==Self-calibrated 1-min-averaged visibility data==
 
==CASA 1-min averaged ms==
EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:
EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:


http://www.ovsa.njit.edu/fits/UDBms_slfcaled
http://www.ovsa.njit.edu/fits/UDBms_slfcaled


=Level 2 - Images and spectrogram data in standard FITS format =
=Level 1.0 - Images and spectrogram data in standard FITS format =


Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power and a sum of amplitudes over intermediate-length baselines (cross power). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).
Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power (TP) and a sum of amplitudes over intermediate-length baselines (cross power or XP). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).


'''List of Level 2 data products'''
We provide the following data products:
* Synoptic products:
** All-day total-power spectrograms:
** All-day cross-power spectrograms:
** All-day synoptic images:
* Event-based products:
** Flare spectrograms: These are full time and frequency resolution spectrograms produced from the median of calibrated cross-power visibilities in FITS format, cropped to cover the flare duration. Pre-flare background is also subtracted. Compared to total-power spectrograms, these spectrograms have the advantage of revealing details of the flare evolution by "filtering out" the large-scale, continuous background from the visibilities (as they are poorly correlated by baselines with intermediate lengths). Note that for flares that have a large source size, the flux shown by the cross-power spectrograms can be lower than its true flux (as a fraction of the flux will be "resolved out"). In this case, one might compare the spectrograms with our total-power spectrogram products.
** Pipeline-produced spectral images: We also have a semi-automated flare imaging pipeline to produce calibrated (and self-calibrated) images at 12-s cadence in up to 10 frequency bands. They are saved in standard FITS format and have been registered into Helioprojective coordinates. They can be read by SSWIDL or astropy/sunpy. These data have already been calibrated to physical units and are usually good to be compared with context data. But please be cautious when using them for quantitative spectral analysis.
 
'''List of Level 1 data products currently provided'''
{| class="wikitable"
{| class="wikitable"
|-
|-
! scope="col"| category
! scope="col"| Category
! scope="col"| data product
! scope="col"| Data Product
! scope="col"| name
! scope="col"| Naming Convention
! scope="col"| Download Link
|-
|-
! rowspan="2" | Dynamic Spectrum
! rowspan="2" | Synoptic Spectrograms
| All-day TP spectrum
| All-day TP Spectrograms
| EOVSA_TPall_yyyymmdd.fts
| EOVSA_TPall_yyyymmdd.fts
!rowspan="9" | https://ovsa.njit.edu/browser
|-
|-
| All-day XP spectrum
| All-day XP Spectrograms
| EOVSA_XPall_yyyymmdd.fts
| EOVSA_XPall_yyyymmdd.fts
|-
|-
! rowspan="7" | Image
! rowspan="7" | Synoptic Images
|-
|-
| Synoptic 1.4 GHz image
| Synoptic 1.4 GHz images
| eovsa_yyyymmdd.spw00-01.tb.disk.fits
| eovsa_yyyymmdd.spw00-01.tb.disk.fits
|-
|-
| Synoptic 3.0 GHz image
| Synoptic 3.0 GHz images
| eovsa_yyyymmdd.spw02-05.tb.disk.fits
| eovsa_yyyymmdd.spw02-05.tb.disk.fits
|-
|-
| Synoptic 4.5 GHz image
| Synoptic 4.5 GHz images
| eovsa_yyyymmdd.spw06-10.tb.disk.fits
| eovsa_yyyymmdd.spw06-10.tb.disk.fits
|-
|-
| Synoptic 6.8 GHz image
| Synoptic 6.8 GHz images
| eovsa_yyyymmdd.spw11-20.tb.disk.fits
| eovsa_yyyymmdd.spw11-20.tb.disk.fits
|-
|-
| Synoptic 10.2 GHz image
| Synoptic 10.2 GHz images
| eovsa_yyyymmdd.spw21-30.tb.disk.fits
| eovsa_yyyymmdd.spw21-30.tb.disk.fits
|-
|-
| Synoptic 13.9 GHz image
| Synoptic 13.9 GHz images
| eovsa_yyyymmdd.spw31-43.tb.disk.fits
| eovsa_yyyymmdd.spw31-43.tb.disk.fits
|-
! rowspan="1" | Flare Spectrograms
| Full-resolution cross-power Spectrogram
| eovsa.spec.flare_id_YYYYMMDDHHMMSS.fits
!rowspan="2" | https://ovsa.njit.edu/flarelist
|-
! rowspan="1" | Flare Spectral Images
| Pipeline-produced spectral images
| eovsa.lev1_mbd_12s.YYYY-MM-DDTHHMMSSZ.image.fits
|-
|}
|}


==Getting level 2 data==  
==Getting level 1 data==  
[[File:eovsa_browser.jpg|150px|thumb]]
===Synoptic level 1 data===
EOVSA Level 2 data products can be retrieved with the following steps:
EOVSA Level 1 synoptic data products can be retrieved with the following steps:
* go to [http://ovsa.njit.edu/browser/ EOVSA browser] page.
* Go to [http://ovsa.njit.edu/browser/ EOVSA browser] page.
 
* Browse to the date of interest.
* Browse to the date of interest.
* Click "synoptic fits" button next to the calendar tool.
* Click "synoptic fits" button next to the calendar tool.
* Select the data product based on the names listed in the table above.
* Select the data product based on the names listed in the table above.
[[File:eovsa_browser.jpg|800px|center]]
===Flare level 1 data===
EOVSA flare list with spectrograms and spectral images can be queried and downloaded at https://ovsa.njit.edu/flarelist.


==Reading level 2 data==
==Reading level 1 data==
===Software===
===Software===
We have developed a package for EOVSA data processing and analysis:
We have developed a package for EOVSA data processing and analysis:
[https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA).  
[https://github.com/suncasa/suncasa SunCASA] A wrapper around [https://casa.nrao.edu/ CASA (the Common Astronomy Software Applications package)] for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the [https://public.nrao.edu/ National Radio Astronomy Observatory]. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on [https://casa.nrao.edu/ NRAO's CASA website ]. Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA). We are developing a new version of SunCASA based on CASA 6 (which offers a modular approach) so that users have the flexibility to build CASA tools and tasks in their Python environment.
====SunCASA installation====
 
Please [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation follow this link] for details regarding the installation of SunCASA on your own machine (only available on Unix-bases OS). This will take you to another page.
Please [http://www.ovsa.njit.edu/wiki/index.php/SunCASA_Installation follow this link] for details regarding the installation of SunCASA on your own machine (only available on Unix-bases OS). This will take you to another page.
===Flare Spectrograms and Multi-Frequency Images ===
* An example of how to read and plot these FITS data in Python (with Astropy and SunPy) can be accessed at [https://colab.research.google.com/drive/1Y3ONWCxLPYvWda5_LqFNxafJtwZDNJBD?usp=sharing#scrollTo=ueiMoHbdxfo- this Google Colab Jupyter notebook].
* We are working on an example with SSWIDL and will release it soon.


===All-day TP spectrum===
===All-day TP spectrum===
Line 94: Line 119:


<pre style="background-color: #FCEBD9;">
<pre style="background-color: #FCEBD9;">
from astropy.io import fits
from suncasa.eovsa import eovsa_dspec as ds
import astropy.table
from astropy.time import Time
from astropy.time import Time
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
import matplotlib.colors as colors
## Read EOVSA Dynamic Spectrum FITS file <filename>
from matplotlib.dates import DateFormatter
filename = 'EOVSA_TPall_20170713.fts'
import numpy as np
s = ds.get_dspec(filename, doplot=True, cmap='gist_heat', norm=LogNorm(vmax=2.1e3, vmin=40))
from mpl_toolkits.axes_grid1 import make_axes_locatable
## To access the data in the spectrogram object, use
 
spec = s['spectrogram']                    ## (Array of amplitudes in SFU, of size nfreq,ntimes)
 
fghz = s['spectrum_axis']                  ## (Array of frequencies in GHz, of size nfreq)
def get_dspec(filename, doplot=False, vmax=None, vmin=None, norm=None, cmap=None):
tim = Time(s['time_axis'], format='mjd')  ## (Array of UT times in astropy.time object, of size ntimes)
    """
    Read EOVSA Dynamic Spectrum FITS file <filename> and return a spectrogram dictionary.
    Optionally show an overview plot if doplot switch is set.
 
    Example:
    --------
    >>> from suncasa.eovsa import eovsa_dspec as ds
    >>> from astropy.time import Time
    >>> filename='EOVSA_TPall_20170713.fts'
    >>> s = ds.dspec(filename,doplot=True,vmax=200)
    ## To access the data in the spectrogram object, use
    >>> filename='EOVSA_TPall_20170713.fts'
    >>> s = ds.dspec(filename,doplot=True,vmax=200)
    >>> spec = s['spectrogram']                    ## (Array of amplitudes in SFU, of size nfreq,ntimes)
    >>> fghz = s['spectrum_axis']                  ## (Array of frequencies in GHz, of size nfreq)
    >>> tim = Time(s['time_axis'], format='mjd')  ## (Array of UT times in astropy.time object, of size ntimes)
 
    Parameters
    ----------
    filename :  filename of the spectrogram fits file
 
    doplot : Boolean, optional
 
    vmin, vmax : scalar, optional
    When using scalar data and no explicit norm, vmin and vmax
    define the data range that the colormap covers. By default,
    the colormap covers the complete value range of the supplied data.
    vmin, vmax are ignored if the norm parameter is used.
 
    norm : `matplotib.colors.Normalize` or str, optional
    The Normalize instance used to scale scalar data to the [0, 1]
    range before mapping to colors using cmap. By default, a linear
    scaling mapping the lowest value to 0 and the highest to 1 is used.
    This parameter is ignored for RGB(A) data.
 
    cmap : `matplotib.colors.Colormap` or str
    A colormap instance or the name of a registered colormap.
 
    Returns
    -------
    spectrogram : dictionary
    """
 
    hdulist = fits.open(filename)
    spec = hdulist[0].data
    fghz = np.array(astropy.table.Table(hdulist[1].data)['sfreq'])
    tim = astropy.table.Table(hdulist[2].data)
    tim = Time(np.array(tim['mjd']) + np.array(tim['time']) / 24. / 3600 / 1000, format='mjd')
    timplt = tim.plot_date
    ntim = len(timplt)
    nfreq = len(fghz)
    if doplot:
        fig, ax = plt.subplots(figsize=(9, 4))
        # if vmax is None:
        #    vmax = np.nanmax(spec)
        # if vmin is None:
        #    vmin = 0.1
        # if norm is None:
        #    norm = colors.Normalize(vmax=vmax, vmin=vmin)
        pcm = ax.pcolormesh(timplt, fghz, spec, norm=norm, vmax=vmax, vmin=vmin, cmap=cmap)
        ax.xaxis_date()
        ax.xaxis.set_major_formatter(DateFormatter("%H:%M"))
        ax.set_ylim(fghz[0], fghz[-1])
        ax.set_xlim(tim[0].plot_date, tim[-1].plot_date)
        ax.set_xlabel('Time [UT]')
        ax.set_ylabel('Frequency [GHz]')
        ax.set_title('EOVSA Dynamic Spectrum for ' + tim[0].datetime.strftime('%Y-%m-%d'))
        divider = make_axes_locatable(ax)
        cax_spec = divider.append_axes('right', size='1.5%', pad=0.05)
        clb_spec = plt.colorbar(pcm, ax=ax, cax=cax_spec, label='Flux density [sfu]')
 
        def format_coord(x, y):
            col = np.argmin(np.absolute(timplt - x))
            row = np.argmin(np.absolute(fghz - y))
            if col >= 0 and col < ntim and row >= 0 and row < nfreq:
                timstr = tim[col].isot
                flux = spec[row, col]
                return 'time {0}, freq = {1:.3f} GHz, flux = {2:.2f} sfu'.format(timstr, y, flux)
            else:
                return 'x = {0}, y = {1:.3f}'.format(x, y)
 
        ax.format_coord = format_coord
        fig.tight_layout()
        plt.show()
    return {'spectrogram': spec, 'spectrum_axis': fghz, 'time_axis': tim.mjd}


</pre>
</pre>


The code is available in [https://github.com/suncasa/suncasa/blob/master/eovsa/eovsa_dspec.py suncasa].
The '''get_dspec''' function is accessible on [https://github.com/suncasa/suncasa-src/blob/master/suncasa/eovsa/eovsa_dspec.py GitHub]. For comprehensive guidance, please refer to suncasa's [https://suncasa-src.readthedocs.io/en/latest/autoapi/suncasa/eovsa/eovsa_dspec/index.html ReadtheDocs page].
[[File:TPSP.jpeg|center|500px]]
[[File:TPSP.jpeg|center|500px]]
'''TODO''': change the example in py section to 13-jul-17. add software installation from zero. add return in the eovsa_dspec. add sunpy/radiospectra after that.




Line 212: Line 150:
   ;    spec = s.get(/spectrogram)    (Array of amplitudes in SFU, of size ntimes, nfreq)
   ;    spec = s.get(/spectrogram)    (Array of amplitudes in SFU, of size ntimes, nfreq)
   ;    fghz = s.get(/spectrum_axis)  (Array of frequencies in GHz, of size nfreq)
   ;    fghz = s.get(/spectrum_axis)  (Array of frequencies in GHz, of size nfreq)
   ;    ut = s.set(/time_axis)        (Array of UT times in anytim format, of size ntimes)
   ;    ut = s.get(/time_axis)        (Array of UT times in anytim format, of size ntimes)
    
    
   default,doplot,0
   default,doplot,0
Line 235: Line 173:
</pre>
</pre>
[[File:IDL_TPSP.png|center|500px]]
[[File:IDL_TPSP.png|center|500px]]


===Synoptic 6-band Images===
===Synoptic 6-band Images===
Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature. [[File:synoptic_image.jpg| center |400px]]
Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature. [[File:synoptic_image.jpg| center |400px]]


===10-min 6-band Images===
The EOVSA full disk image FITS files are compressed with the RICE algorithm implemented in the FITS file handling module (astropy.io.fits) in Astropy. EOVSA FITS files are very similar to the compressed SDO/AIA FITS files from JSOC. Popular coding languages can easily read compressed images directly.
===Event images===
 
In IDL, you can use read_sdo in the ONTOLOGY package, which should be installed by default in SolarSoftWare (SSW), to read compressed EOVSA FITS files. The following code will read the EOVSA image FITS file in SSWIDL:
 
<pre style="background-color: #FCEBD9;">
read_sdo,'eovsa_20191225.spw11-20.tb.disk.fits',header,data,/UNCOMP_DELETE
index2map,header,data,eomap
plot_map,eomap
</pre>
 
[[File:eovsa_20191225_image_sswidl.jpg| center |250px]]
 
In Python, you can use SunPy map module to read EOVAS FITS files. SunPy is an open solar data analysis environment for Python. The installation instruction can be found at [https://sunpy.org/ SunPy official website].
<pre style="background-color: #FCEBD9;">
import matplotlib.pyplot as plt
from sunpy import map as smap
eofile='eovsa_20191225.spw11-20.tb.disk.fits'
eomap = smap.Map(eofile)
eomap.plot()
plt.show()
</pre>
[[File:eovsa_20191225_image_py.jpg| center |250px]]
 
Although the Sunpy Python package already exists for doing analysis of solar data, it has a significant learning curve and lacks the generality of the Mapping routines written by Dominic Zarro for the IDL-based Solarsoft (SSW). We provided IDL Mapping routines available in Python that may help those IDL users who have been avoiding learning Python. '''The mapping routines work equally well in both Python 2.7 and Python 3'''.
Get the [http://www.ovsa.njit.edu/wiki/index.php/Mapping_Software Mapping routines]
The following code will read the EOVSA image FITS file in python:
<pre style="background-color: #FCEBD9;">
from mapping.plot_map import plot_map
from mapping.fits2map import fits2map
eofile='eovsa_20191225.spw11-20.tb.disk.fits'
eomap, h = fits2map(eofile,header=True)
plot_map(eomap, grid=15, limb=True,cmap='gray')
</pre>
[[File:eovsa_20191225_image_py_ssw-mapping.jpg| center |250px]]

Latest revision as of 20:53, 28 March 2024

EOVSA observes the full disk of the Sun at all times when the Sun is >10 degrees above the local horizon (season dependent and ranges from 7-12 hours duration centered on 20 UT). EOVSA records data at 451 science frequency channels each second, in four polarization products, as well as additional total flux measurements from each individual antenna. Raw data in the visibility domain are processed through a pipeline processing system to produce images and spectrograms in the time- and frequency-dependent image plane (the block diagram in Figure 1 shows the data flow in the pipeline).

EOVSA pipeline block diagram/flow chart

We deliver the radio interferometry data on the following three levels:


Level 0 - Raw visibility data from the instrument

As outlined in Figure 1, EOVSA creates raw data products in the left-hand column (labeled Level 0). This includes observations of cosmic sources for phase calibration, and gain and pointing observations required for total power calibration.

Raw "Interim" Database (IDB) visibility data

Full-resolution raw "Interim" Database (IDB) visibility data. They are stored in Miriad format, and hence may not be that useful for most people. Be patient after clicking the link--this is a very long list of directories, one for each available date. These data can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/IDB/

Raw 1-min-averaged visibility data

This is the same as for the IDB data, except with 1-minute time integration applied. This is typically not useful for flares, but is perfectly fine for imaging active regions and full Sun. These data can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/UDB/

Level 0.5 - Calibrated visibility data

After applying calibration and other preliminary processing to the raw (level 0) data, we create the CASA ms’s in the second column in Figure 1 (labeled "level 0.5"). These visibility data are in the Fourier domain of the true images in the plane of the sky and are not immediately ready for spectral imaging analysis yet. However, they have all of the required content to produce images and spectrogram data in standard FITS format (level 1.0). We provide a set of standard ms’s for each day (red boxes in Figure 1), for use by researchers who know how to deal with visibility data.

Calibrated full-resolution visibility data for flare events

Calibrated and self-calibrated visibility data for flare events (purple boxes in Figure 1) will typically be available within 7 days after they are taken. They will be released at our flare list site soon: https://ovsa.njit.edu/flarelist

Self-calibrated 1-min-averaged visibility data

EOVSA 1-min averaged visibility data in CASA ms format can be retrieved from the following page:

http://www.ovsa.njit.edu/fits/UDBms_slfcaled

Level 1.0 - Images and spectrogram data in standard FITS format

Most users, however, will prefer to work with spectrogram (frequency-time) and image data, which are also outputs of the pipeline system shown in Figure 1 (orange boxes). Spectrograms are provided as standard FITS tables containing the frequency list, list of times, and data in both total power (TP) and a sum of amplitudes over intermediate-length baselines (cross power or XP). Likewise, image data products are in FITS format with standard keywords and are converted into the Helioprojective Cartesian coordinate system compatible with the World Coordinate System (WCS) convention, along with correct registration for the spatial, spectral, and temporal coordinates. Both the spectrogram and image data products are calibrated properly and have physical radio intensity units (sfu for spectrograms and brightness temperature for radio images).

We provide the following data products:

  • Synoptic products:
    • All-day total-power spectrograms:
    • All-day cross-power spectrograms:
    • All-day synoptic images:
  • Event-based products:
    • Flare spectrograms: These are full time and frequency resolution spectrograms produced from the median of calibrated cross-power visibilities in FITS format, cropped to cover the flare duration. Pre-flare background is also subtracted. Compared to total-power spectrograms, these spectrograms have the advantage of revealing details of the flare evolution by "filtering out" the large-scale, continuous background from the visibilities (as they are poorly correlated by baselines with intermediate lengths). Note that for flares that have a large source size, the flux shown by the cross-power spectrograms can be lower than its true flux (as a fraction of the flux will be "resolved out"). In this case, one might compare the spectrograms with our total-power spectrogram products.
    • Pipeline-produced spectral images: We also have a semi-automated flare imaging pipeline to produce calibrated (and self-calibrated) images at 12-s cadence in up to 10 frequency bands. They are saved in standard FITS format and have been registered into Helioprojective coordinates. They can be read by SSWIDL or astropy/sunpy. These data have already been calibrated to physical units and are usually good to be compared with context data. But please be cautious when using them for quantitative spectral analysis.

List of Level 1 data products currently provided

Category Data Product Naming Convention Download Link
Synoptic Spectrograms All-day TP Spectrograms EOVSA_TPall_yyyymmdd.fts https://ovsa.njit.edu/browser
All-day XP Spectrograms EOVSA_XPall_yyyymmdd.fts
Synoptic Images
Synoptic 1.4 GHz images eovsa_yyyymmdd.spw00-01.tb.disk.fits
Synoptic 3.0 GHz images eovsa_yyyymmdd.spw02-05.tb.disk.fits
Synoptic 4.5 GHz images eovsa_yyyymmdd.spw06-10.tb.disk.fits
Synoptic 6.8 GHz images eovsa_yyyymmdd.spw11-20.tb.disk.fits
Synoptic 10.2 GHz images eovsa_yyyymmdd.spw21-30.tb.disk.fits
Synoptic 13.9 GHz images eovsa_yyyymmdd.spw31-43.tb.disk.fits
Flare Spectrograms Full-resolution cross-power Spectrogram eovsa.spec.flare_id_YYYYMMDDHHMMSS.fits https://ovsa.njit.edu/flarelist
Flare Spectral Images Pipeline-produced spectral images eovsa.lev1_mbd_12s.YYYY-MM-DDTHHMMSSZ.image.fits

Getting level 1 data

Synoptic level 1 data

EOVSA Level 1 synoptic data products can be retrieved with the following steps:

  • Go to EOVSA browser page.
  • Browse to the date of interest.
  • Click "synoptic fits" button next to the calendar tool.
  • Select the data product based on the names listed in the table above.
Eovsa browser.jpg

Flare level 1 data

EOVSA flare list with spectrograms and spectral images can be queried and downloaded at https://ovsa.njit.edu/flarelist.

Reading level 1 data

Software

We have developed a package for EOVSA data processing and analysis: SunCASA A wrapper around CASA (the Common Astronomy Software Applications package) for synthesis imaging and visualizing solar spectral imaging data. CASA is one of the leading software tools for "supporting the data post-processing needs of the next generation of radio astronomical telescopes such as ALMA and VLA", an international effort led by the National Radio Astronomy Observatory. The current version of CASA uses Python (2.7) interface. More information about CASA can be found on NRAO's CASA website . Note, CASA is available ONLY on UNIX-BASED PLATFORMS (and therefore, so is SunCASA). We are developing a new version of SunCASA based on CASA 6 (which offers a modular approach) so that users have the flexibility to build CASA tools and tasks in their Python environment.

Please follow this link for details regarding the installation of SunCASA on your own machine (only available on Unix-bases OS). This will take you to another page.

Flare Spectrograms and Multi-Frequency Images

  • An example of how to read and plot these FITS data in Python (with Astropy and SunPy) can be accessed at this Google Colab Jupyter notebook.
  • We are working on an example with SSWIDL and will release it soon.

All-day TP spectrum

Daily total power full-Sun-integrated spectrogram calibrated in solar flux units are provided at 451 frequencies (134 frequencies prior to 2019 Feb 22) and 1 s time resolution.

To read a spectrogram file in Python using the suncasa library:

from suncasa.eovsa import eovsa_dspec as ds
from astropy.time import Time
from matplotlib.colors import LogNorm
## Read EOVSA Dynamic Spectrum FITS file <filename>
filename = 'EOVSA_TPall_20170713.fts'
s = ds.get_dspec(filename, doplot=True, cmap='gist_heat', norm=LogNorm(vmax=2.1e3, vmin=40))
## To access the data in the spectrogram object, use
spec = s['spectrogram']                    ## (Array of amplitudes in SFU, of size nfreq,ntimes)
fghz = s['spectrum_axis']                  ## (Array of frequencies in GHz, of size nfreq)
tim = Time(s['time_axis'], format='mjd')   ## (Array of UT times in astropy.time object, of size ntimes)

The get_dspec function is accessible on GitHub. For comprehensive guidance, please refer to suncasa's ReadtheDocs page.

TPSP.jpeg


The following code will read the spectrogram file in IDL:

function dspec,filename,doplot=doplot
  ; Read EOVSA Dynamic Spectrum FITS file <filename> and return a spectrogram object.
  ; Optionally show an overview plot if doplot switch is set
  ; 
  ; Usage:
  ;    s = dspec(<filename>)            ; Returns spectrogram object
  ;    s = dspec(<filename>,/doplot)    ; Plots spectrum and returns spectrogram object
  ;
  ; To access the data in the spectrogram object, use
  ;    spec = s.get(/spectrogram)    (Array of amplitudes in SFU, of size ntimes, nfreq)
  ;    fghz = s.get(/spectrum_axis)  (Array of frequencies in GHz, of size nfreq)
  ;    ut = s.get(/time_axis)        (Array of UT times in anytim format, of size ntimes)
  
  default,doplot,0
  spec = mrdfits(filename,0)
  freq = mrdfits(filename,1)
  time = mrdfits(filename,2)
  fghz = freq.sfreq
  ut = anytim(time)
  s = spectrogram(spec,ut,fghz)
  if doplot then begin
    window,/free,xsiz=1024,ysiz=600
    ; Find min and max of data from 5% to 95% of sorted array (eliminates outliers)
    sarr = sort(spec)
    dlim = minmax(spec[sarr[n_elements(sarr)*0.05:n_elements(sarr)*0.95]])
    ; Set drange with margin factor of 2 on low end and 5 on high end
    s.set,drange=dlim*[0.5,5]
    loadct,3
    s.plot,/log,/xsty,/ysty,ytitle='Frequency [GHz]',charsize=1.5
  endif
return,s
end
IDL TPSP.png

Synoptic 6-band Images

Full disk images at 6 selected frequency bands centered at 1.4, 3.0, 4.5, 6.8, 10.2, and 13.9 GHz are provided once per day, calibrated in brightness temperature.

Synoptic image.jpg

The EOVSA full disk image FITS files are compressed with the RICE algorithm implemented in the FITS file handling module (astropy.io.fits) in Astropy. EOVSA FITS files are very similar to the compressed SDO/AIA FITS files from JSOC. Popular coding languages can easily read compressed images directly.

In IDL, you can use read_sdo in the ONTOLOGY package, which should be installed by default in SolarSoftWare (SSW), to read compressed EOVSA FITS files. The following code will read the EOVSA image FITS file in SSWIDL:

read_sdo,'eovsa_20191225.spw11-20.tb.disk.fits',header,data,/UNCOMP_DELETE
index2map,header,data,eomap
plot_map,eomap
Eovsa 20191225 image sswidl.jpg

In Python, you can use SunPy map module to read EOVAS FITS files. SunPy is an open solar data analysis environment for Python. The installation instruction can be found at SunPy official website.

import matplotlib.pyplot as plt
from sunpy import map as smap
eofile='eovsa_20191225.spw11-20.tb.disk.fits'
eomap = smap.Map(eofile)
eomap.plot()
plt.show()
Eovsa 20191225 image py.jpg

Although the Sunpy Python package already exists for doing analysis of solar data, it has a significant learning curve and lacks the generality of the Mapping routines written by Dominic Zarro for the IDL-based Solarsoft (SSW). We provided IDL Mapping routines available in Python that may help those IDL users who have been avoiding learning Python. The mapping routines work equally well in both Python 2.7 and Python 3. Get the Mapping routines The following code will read the EOVSA image FITS file in python:

from mapping.plot_map import plot_map
from mapping.fits2map import fits2map
eofile='eovsa_20191225.spw11-20.tb.disk.fits'
eomap, h = fits2map(eofile,header=True)
plot_map(eomap, grid=15, limb=True,cmap='gray')
Eovsa 20191225 image py ssw-mapping.jpg