Automation using meqtree-pipeliner.py

<<Previous Up Next>>

Automation using meqtree-pipeliner.py

Logged on 12/02/13 23:34:05

We've got the Measurement Sets, the sky model and the FITS beams so we could just repeat the above simulations for the next five frequency channels by hand. It'll be a lot easier to just script them though.

MeqTrees is perfectly happy to operate in 'headless' mode or batch mode, i.e. you can run it directly from the command line. This makes it highly scriptable, and this feature is basically essential for large calibration jobs. The software that makes the images from within the Meqbrowser is also a standalone command line tool called lwimager (based on the CASA imaging libraries) so we can script that too.

Before we begin however, the imager can sometimes grumble if appropriate columns aren't added to the MS, and in previous batch mode simulations for some reason this error pops up whereas it doesn't if we run a sim from the browser. It's easily avoided though. Attached below is a tiny Python script called add_imager_cols.py (courtesy of Tony Willis) that invokes a Pyrap function to do this task.

Before we proceed, pay your terminal a visit and in the folder that contains our Measurement Sets, run this script on all of them via:

for f in *.ms ; do python add_imager_cols.py $f; done

Now we're ready to proceed.

The plan here is to invoke both MeqTrees and lwimager from a script in order to do the following steps:

1) Simulate the intrinsic sky.

2) Make a cleaned image of the intrinsic sky and write it to a FITS file with a *.intrinsic.fits suffix.

3) Apply the beam patterns to simulate the apparent sky.

4) Make a cleaned image of the apparent sky and write it to a FITS file with a *.apparent.fits suffix.

There is a very handy script that runs batch jobs called meqtree-pipeliner.py, which should be in your path. If you run

meqtree-pipeliner.py -h

from the terminal you should get a bit of info about it. Essentially you provide it with a .tdl.conf file, a TDL script and the name of the TDL job you want it to run and it sets up a meqserver and executes the job. The real flexibility emerges when you realise you can tweak the input parameters between specificying the .tdl.conf file and the TDL script. We have an obvious use case for it in this simulation: in step (1) above we want to run the same simulation job on a series of Measurement Sets, and in point (3) above we need to do the same task, plus apply the appropriate simulated beam pattern.

I have attached a Python script (batch_beam_sim.py) which will hopefully perform steps 1 - 4 for you for all six Measurement Sets, including the one we just did by hand. The script is heavily annotated so you are encouraged to open it up and take a look inside and figure out what it's doing. I'm not the neatest programmer in the world (even when it comes to Python, I never let anyone see my Perl scripts), for which I apologize. I'm sure I'm not the most efficient either, so if you see something that can be done better please do implement it and send me your version. You might also want to investigate the --mt switch for meqtree-pipeliner.py if you are running this on a powerful machine.

Here's an example of one of the meqtree-pipeliner.py invocations from the script:

meqtree-pipeliner.py -c tdlconf.profiles \[source_colourization_apparent\] ms_sel.msname=KAT7_1000_1x50_12h.ms pybeams_fits.filename_pattern=beam_patterns/fitsbeam_L1000_$(xy)_$(realimag).fits turbo-sim.py =_tdl_job_1_simulate_MS

This loads the TDL parameters from tdlconf.profiles (also attached), specifically those from the [source_colourization_apparent] section. Following this, it changes the ms_sel_msname and pybeams_fits.filename_pattern parameters to the ones that the batch_beam_sim.py script has generated for that particular iteration. Finally it passes these parameters to turbo-sim.py and run the '1 simulate MS' job that we're now familiar with from the browser.

If you put the batch_beam_sim.py script in the same folder as your Measurement Sets and run it it should spend a minute or two running the simulations and generating the images for you.

It's debateable in our example which one is faster, doing this by hand or spending the time writing this script, but if you're still not convinced of the awesome power of batch mode then tonight's homework is to repeat this simulation for a thousand channels.

I'm not going to explain the ins and outs of the lwimager software. If you look up the CASA 'clean' task online or in the cookbook it should explain most of the parameters that you have access to with lwimager ('lwimager -h' is your friend here).

If all is well you should end up with the following in your terminal:

Finished!

I have made the following products:

KAT7_1000_1x50_12h.ms:

KAT7_1000_1x50_12h.ms.intrinsic.fits

KAT7_1000_1x50_12h.ms.apparent.fits

KAT7_1200_1x50_12h.ms:

KAT7_1200_1x50_12h.ms.intrinsic.fits

KAT7_1200_1x50_12h.ms.apparent.fits

KAT7_1400_1x50_12h.ms:

KAT7_1400_1x50_12h.ms.intrinsic.fits

KAT7_1400_1x50_12h.ms.apparent.fits

KAT7_1600_1x50_12h.ms:

KAT7_1600_1x50_12h.ms.intrinsic.fits

KAT7_1600_1x50_12h.ms.apparent.fits

KAT7_1800_1x50_12h.ms:

KAT7_1800_1x50_12h.ms.intrinsic.fits

KAT7_1800_1x50_12h.ms.apparent.fits

KAT7_2000_1x50_12h.ms:

KAT7_2000_1x50_12h.ms.intrinsic.fits

KAT7_2000_1x50_12h.ms.apparent.fits

Data products

788bA.gif

788bA.gif

add_imager_cols.py

tdlconf.profiles

batch_beam_sim.py