Tohban OVRO-LWA Imaging Tutorial - Revision history

Dgary: /* What Happens When You Run the Script */

2024-01-16T15:29:03Z

What Happens When You Run the Script

← Older revision		Revision as of 15:29, 16 January 2024
Line 157:		Line 157:

	== What Happens When You Run the Script ==		== What Happens When You Run the Script ==
	One way to run ~~this script~~ is to cut-and-paste into a file, say process.py, and then in an iPython session type		One way to run these scripts is to cut-and-paste the first script into a file, say process.py, cut-and-paste the second script into another file, say mk_imgs.py, and then in an iPython session type
	<pre>		<pre>
	import sys		import sys
	sys.path.append('/home/dgary/ovro-lwa-solar') # Change to your path where you cloned the git repository		sys.path.append('/home/dgary/ovro-lwa-solar') # Change to your path where you cloned the git repository
	run 'process.py'		run 'process.py' # Wait until all final ms's are created--could take many hours!
			run 'mk_imgs.py'
	</pre>		</pre>
	If all goes well, after many hours you will have all of your images. If you examine the script, you will see that there are two loops, an inner one over frequency and an outer one over time. The inner loop will create a subdirectory for the frequency it is working on (first will be subdirectory named 27MHz), then do the calibration for that frequency and create a subfolder caltables with a .bcal file in it. Luckily, this only has to be done once and then the .bcal file will be used for subsequent times so its creation will be skipped. Other files with .gcal extension will be created for the first data time, and also will be reused for subesquent times up to one hour later. When a new .gcal file is needed, the pipeline will create it automatically for you. The gain files take about 10 min for each frequency, but again is only done once for an hour of data. After the calibration is complete, <code>wsclean</code> is used to create images (in 10 subbands of each 4.5 GHz band, plus an MFS image integrated over the whole band). They are converted to heliographic coordinates and you will find them in 27MHz/images when done. This takes another 10 minutes or so.		If all goes well, after many hours you will have all of your images. If you examine the script, you will see that there are two loops, an inner one over frequency and an outer one over time. The inner loop will create a subdirectory for the frequency it is working on (first will be subdirectory named 27MHz), then do the calibration for that frequency and create a subfolder caltables with a .bcal file in it. Luckily, this only has to be done once and then the .bcal file will be used for subsequent times so its creation will be skipped. Other files with .gcal extension will be created for the first data time, and also will be reused for subesquent times up to one hour later. When a new .gcal file is needed, the pipeline will create it automatically for you. The gain files take about 10 min for each frequency, but again is only done once for an hour of data. After the calibration is complete, <code>wsclean</code> is used to create images (in 10 subbands of each 4.5 GHz band, plus an MFS image integrated over the whole band). They are converted to heliographic coordinates and you will find them in 27MHz/images when done. This takes another 10 minutes or so.

	When all of that is done for the first frequency, the whole process starts again for the next, and so one until all images for the first time are done. In this example, then, it will take about 20/min per frequency * 13 frequencies = 260 minutes (> 4 hours!) to make all 143 images for the first time (10 images per band + 1 MFS image). For subsequent times, though, the calibration step is skipped so each subsequent time will take 10 min * 13 frequencies (around 2 hours). That means the entire script will run in about 8 hours and produce 429 images.		When all of that is done for the first frequency, the whole process starts again for the next, and so one until all images for the first time are done. In this example, then, it will take about 20/min per frequency * 13 frequencies = 260 minutes (> 4 hours!) to make all 143 images for the first time (10 images per band + 1 MFS image). For subsequent times, though, the calibration step is skipped so each subsequent time will take 10 min * 13 frequencies (around 2 hours). That means the entire script will run in about 8 hours and produce 429 images.

Dgary: /* Calibration and Imaging Script */

2024-01-16T15:26:29Z

Calibration and Imaging Script

← Older revision		Revision as of 15:26, 16 January 2024
Line 112:		Line 112:
	After the final ms's are created, run the code below to create fits-wrapped image cubes of the data.		After the final ms's are created, run the code below to create fits-wrapped image cubes of the data.
	<pre>		<pre>
	~~import sys~~
	~~sys.path.append('/home/dgary/ovro-lwa-solar') # ***Change to your folder (or wherever you cloned ovro-lwa-solar)~~
	from solar_realtime_pipeline import run_imager		from solar_realtime_pipeline import run_imager
	from suncasa.utils import helioimage2fits as hf		from suncasa.utils import helioimage2fits as hf

Dgary: /* Calibration and Imaging Script */

2024-01-16T15:24:38Z

Calibration and Imaging Script

← Older revision		Revision as of 15:24, 16 January 2024
Line 96:		Line 96:
	os.system("rm -rf .ms .fits .gcal .cl .badants")		os.system("rm -rf .ms .fits .gcal .cl .badants")
	# Make 10 images for this band (integrates over 19 or 20 subchannels, bandwidth ~0.4545 MHz)		# Make 10 images for this band (integrates over 19 or 20 subchannels, bandwidth ~0.4545 MHz)
	os.system('wsclean -no-dirty -size 1024 1024 -scale 1arcmin -weight uniform -minuv-l 10 -name '+imagename+' -niter 10000 -mgain 0.8 -beam-fitting-size 1 -pol I -join-channels -channels-out 10 final_ms/'+msname)		#os.system('wsclean -no-dirty -size 1024 1024 -scale 1arcmin -weight uniform -minuv-l 10 -name '+imagename+' -niter 10000 -mgain 0.8 -beam-fitting-size 1 -pol I -join-channels -channels-out 10 final_ms/'+msname)
	# Convert images to heliocentric, move them to the final image folder, and delete all fits files		# Convert images to heliocentric, move them to the final image folder, and delete all fits files
	files = glob.glob('*-image.fits')		#files = glob.glob('*-image.fits')
	for imgfile in files:		#for imgfile in files:
	#utils.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])		# #utils.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])
	helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)		# helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)
	os.system('mv '+helio_image+' '+image_fold)		# os.system('mv '+helio_image+' '+image_fold)
	os.system('rm *.fits')		#os.system('rm *.fits')
	except:		except:
	pass		pass

Dgary: /* Calibration and Imaging Script */

2024-01-16T15:23:35Z

Calibration and Imaging Script

← Older revision		Revision as of 15:23, 16 January 2024
Line 108:		Line 108:
	os.chdir(home)		os.chdir(home)

			</pre>

			After the final ms's are created, run the code below to create fits-wrapped image cubes of the data.
			<pre>
			import sys
			sys.path.append('/home/dgary/ovro-lwa-solar') # ***Change to your folder (or wherever you cloned ovro-lwa-solar)
			from solar_realtime_pipeline import run_imager
			from suncasa.utils import helioimage2fits as hf
			from suncasa.io import ndfits
			from astropy.time import Time
			import glob
			import os

			freqs = [27,32,36,41,46,50,55,59,64,69,73,78,82] # ***Change to the bands you want to image
			solar_times = ['192030'] # ***Change to the times to use for solar imaging -- these times must exist!
			datstr = '20231009_' # ***Change to the date of your event
			for solar_time in solar_times:
			tref = Time(datstr[:4]+'-'+datstr[4:6]+'-'+datstr[6:]+' '+solar_time[:2]+':'+solar_time[2:4]+':'+solar_time[4:])
			ephem = hf.read_horizons(tref, dur=1./60./24., observatory='OVRO_MMA')
			if not os.path.exists('imagedir_allch'):
			os.makedirs('imagedir_allch')
			if not os.path.exists('fits'):
			os.makedirs('fits')
			outfits_helio = []
			# Make all the images (by calling run_imager)
			for freq in freqs:
			folder = str(freq)+'MHz'
			msname = folder+'/final_ms/'+datstr+solar_time+'_'+str(freq)+'MHz_final.ms'
			outfits_helio += run_imager(msname, imagedir_allch='imagedir_allch/', ephem=ephem)

			fitsfiles_mfs = []
			fitsfiles_fch = []
			for f in outfits_helio:
			if 'MFS' in f:
			fitsfiles_mfs.append(f)
			else:
			fitsfiles_fch.append(f)
			## Wrap images
			timestr_iso = tref.isot[:-4].replace(':','')+'Z'
			fits_mfs = 'fits/ovro-lwa.lev1_mfs_10s.' + timestr_iso + '.image.fits'
			fits_fch = 'fits/ovro-lwa.lev1_fch_10s.' + timestr_iso + '.image.fits'
			# multi-frequency synthesis images
			fitsfiles_mfs.sort()
			ndfits.wrap(fitsfiles_mfs, outfitsfile=fits_mfs)

			# fine channel spectral images
			fitsfiles_fch.sort()
			ndfits.wrap(fitsfiles_fch, outfitsfile=fits_fch)
	</pre>		</pre>

Binchen: /* Python Environment */

2024-01-04T16:26:38Z

Python Environment

← Older revision		Revision as of 16:26, 4 January 2024
Line 11:		Line 11:
	import sys		import sys
	sys.path.append('/home/dgary/ovro-lwa-solar') # Replace with your own home directory		sys.path.append('/home/dgary/ovro-lwa-solar') # Replace with your own home directory
	import solar_pipeline		from ovrolwasolar import solar_pipeline
	</pre>		</pre>
	If that succeeds, you should be ready to proceed.		If that succeeds, you should be ready to proceed.

Dgary: /* Calibration and Imaging Script */

2024-01-03T16:40:15Z

Calibration and Imaging Script

← Older revision		Revision as of 16:40, 3 January 2024
Line 47:		Line 47:
	<pre>		<pre>
	import os, glob		import os, glob
	import utils		from ovrolwasolar import solar_pipeline, utils
	from time import time		from time import time
	~~import solar_pipeline~~

	freqs=[27,32,36,41,46,50,55,59,64,69,73,78,82] # ***Change to the bands you want to image		freqs=[27,32,36,41,46,50,55,59,64,69,73,78,82] # ***Change to the bands you want to image

Dgary: /* Calibration and Imaging Script */

2024-01-02T20:58:54Z

Calibration and Imaging Script

← Older revision		Revision as of 20:58, 2 January 2024
Line 101:		Line 101:
	files = glob.glob('*-image.fits')		files = glob.glob('*-image.fits')
	for imgfile in files:		for imgfile in files:
	utils.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])		#utils.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])
	helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)		helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)
	os.system('mv '+helio_image+' '+image_fold)		os.system('mv '+helio_image+' '+image_fold)

Dgary: /* Calibration and Imaging Script */

2024-01-02T18:38:50Z

Calibration and Imaging Script

← Older revision		Revision as of 18:38, 2 January 2024
Line 78:		Line 78:
	# Copy the calibration data (will be deleted later)		# Copy the calibration data (will be deleted later)
	print('Copying calibration data to frequency folder')		print('Copying calibration data to frequency folder')
	os.system("cp -r "+os.path.join(~~datapath~~,calib_ms)+"* "+freq_fold+"/")		os.system("cp -r "+os.path.join(calpath,calib_ms)+"* "+freq_fold+"/")

	os.chdir(freq_fold)		os.chdir(freq_fold)

Dgary: /* Calibration and Imaging Script */

2023-11-22T19:36:42Z

Calibration and Imaging Script

← Older revision		Revision as of 19:36, 22 November 2023
Line 101:		Line 101:
	files = glob.glob('*-image.fits')		files = glob.glob('*-image.fits')
	for imgfile in files:		for imgfile in files:
	~~solar_pipeline~~.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])		utils.correct_primary_beam('final_ms/'+msname, imgfile.split('-image.fits')[0])
	helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)		helio_image = utils.convert_to_heliocentric_coords('final_ms/'+msname, imgfile)
	os.system('mv '+helio_image+' '+image_fold)		os.system('mv '+helio_image+' '+image_fold)

Dgary: /* Calibration and Imaging Script */

2023-11-09T16:41:37Z

Calibration and Imaging Script

← Older revision		Revision as of 16:41, 9 November 2023
Line 87:		Line 87:
	if not os.path.isdir('caltables'):		if not os.path.isdir('caltables'):
	os.mkdir('caltables')		os.mkdir('caltables')
			if not os.path.isdir('final_ms'):
			os.mkdir('final_ms')

	try:		try:
Line 108:		Line 110:

	</pre>		</pre>

	== What Happens When You Run the Script ==		== What Happens When You Run the Script ==
	One way to run this script is to cut-and-paste into a file, say process.py, and then in an iPython session type		One way to run this script is to cut-and-paste into a file, say process.py, and then in an iPython session type