SSSC

Frequently Asked Questions

(If you have more questions, or require clarifications, contact noordam@astron.nl)

MeqTrees Technical Support

For MeqTrees technical support please consult the MeqWiki, or send an email to: meqtrees-support@astron.nl

Where has the spigot2sink page gone?

PURR Logs

The PURR log is a rather effective and low-threshold way of exchanging detailed information, with others and with yourself. It is a tool that watches the subdirectory in which you are working, and asks you to do something every time a new file is generated. This then gives you the opportunity to collect all the relevant plots, images, parameter sets and commands in the right order, together with your comments. The HTML result can be easily displayed (e.g. on a website), and exchanged by email. Examples and instructions are on the SSSC page.

What are we trying to achieve with the Nancay workshop?

Click here for the workshop web page, and here for the SKADS webpage.

The qualification process for participation in the Nancay workshop is not only a way to select talented people, but also to bring them up to speed before coming to Nancay. Once there, we will devote time to all aspects of 3GC, but most of all we will try to use this valuable "face-time" to create the bonds and common understanding that is needed to continue working together remotely afterwards.

What are Creative Commons?

This is a group that has the tools and the incentive to work together remotely. Of course they exist in all kinds of forms, but the name has been coined by some organization (google it). We are doing something slightly different, but we like the stimulating name.

Developing software together is an old dream, but it has been very difficult in practice (see the AIPS++ project). However, we feel that this time we have the tools (MeqTrees, PURR logs) and the incentive (the urgent need for 3GC for the new telescopes).

What kind of community do we hope to create?

Ideally, we hope to create a "can-do" atmosphere around all the 3GC issues, but first of all around our two selected themes (see also below). These subjects happen to be the most urgent for many of the new (and existing) telescopes, but solutions are emerging much too slowly. We propose to change all that. First of all by answering the questions that are already floating out there, and then by generating the next level of questions (and answers) ourselves.

What kind of people are we looking for?

All the existing radio telescopes perform way below their potential because it is too difficult to experiment with new ideas. This is caused partly by the design of the existing software, but most of all because neither users nor observatories have the skills or the inclination to experiment. The sustained success of the existing software packages (AIPS, MIRIAD, DIFMAP, NEWSTAR) has caused the old class of user-developers to disappear.

Our goal is to recreate this class. Therefore, we are looking primarily (in this stage) for people who have demonstrated unusual productivity in creating software. We expect them to find a place somewhere between the extremes of "user" and "developer", depending on their tastes and talents. But all of them should be able to move more deftly and more easily than their current counterparts.

Our guiding model here is Jodrell Bank, where user-developers were the norm once. Things have changed now, of course, but it should be noted that a remarkable number of these Jodrellites have reached leading positions at radio observatories all over the world.

We predict that there will be lots of jobs for user-developers when "the shit hits the fan", i.e. when it becomes clear that the thickly flowing data from the new telescopes cannot be reduced.

What is 3rd Generation Calibration (3GC)?

Until ~1980, 1GC relied on the stability on the instrument between observations of bright calibrator sources. The interval could be up to 12 hours, and produced a Dynamic Range of 1:1000 at best. All of the great discoveries of the 1970's were made with this DR.

After the invention of selfcal ~1980, the DR shot up to more than 1:100.000 (the WSRT is world champion with 1:5.000.000). However this 2GC solves for instrumental effects that are valid for the entire FOV, i.e. for Direction-Independent Effects (DIE). It is assumed that all station beams are identical, so "the" beam may be divided out from the image.

The new telescopes (and the upgraded existing ones) require 3GC, which deals with Direction-Dependent Effects (DDE). The most important are the ionosphere, and station beams that are no longer identical. One problem is that solving for many more parameters requires a huge increase in processing. An additional problem is that there might not be enough information to solve for so many parameters. That is why it is imperative to make maximum use of "a priori" information about the continuity (smoothness) of parameter values in time, freq and direction. And finally, it is much more difficult to apply DDE's, either in the forward (imaging) or the backward (prediction) direction.

Since 3GC requires so much processing, more efficient 4GC techniques will probably be needed to reach the noise. So, in the case of highly demanding applications like the detection of the Epoch of Reionization (EoR), 3GC is needed to create the conditions in which 4GC statistical analysis of the residuals will work sufficiently well.

What is the Measurement Equation (M.E.)?

A Measurement Equation is a mathematical model of a measuring instrument. In our case it is a matrix formalism that describes a generic radio telescope in full polarization. One of its most important elements is the 2x2 Jones matrix, which fully describes the instrumental effects of a single station on the signal. By assiging a separate Jones matrix to each station, arrays with rather dissimilar stations may be accommodated. Each station Jones matrix may be the product of a succession of Jones matrices, each of which describes a different instrumental effect.

The M.E. may be used to predict the values of the uv-data when a particular radio telescope observes a particular brightness distribution. One application of this is the generation of simulated uv-data, for instance for the SSSC. The Oxford group has taken a strong lead in this area. Another application is calibration, i.e. solving for the parameters of the M.E. by minimizing the difference between predicted and measured uv-data.

The radio-astronomical M.E. was first written down in a closed form by Hamaker et al. in Dwingeloo (1996). This was rather fortunate, because 3GC, and thus the calibration of the new radio telescopes, would have been unthinkable without an explicit M.E.

What is MeqTrees?

MeqTrees has been designed for implementing an arbitrary M.E., and to solve for (arbitrary subsets of) its parameters. At this moment, it is the only package that can do this.

MeqTrees implements the M.E. in the form of a tree (or rather graph) of software nodes, each of which performs an elementary function. The may range from simple to very complex. With its simple Request/Result interface, nodes can be connected into trees of arbitrary complexity.

In addition, MeqTrees has a number of features that are crucial for "increasing the rate of evolution" of data reduction software:

-) Thanks to the Tree Definition Language (TDL, Python), it has a very short turn-around time for implementing and testing new ideas: hours rather than months.

-) It offers unprecedented visualization at all levels.

-) It is highly modular, so many people can contribute specialized nodes without without having to know about the rest of the system. An example is the UVBrick, written by Filipe Abdalla (UCL).

-) Because processing scripts (Python) can be easily exchanged by email, it is much easier to collaborate in experimentation.

MeqTrees interfaces with the AIPS++/CASA Measurement Set (uv-data file), and also other AIPS++/CASA modules like fitting, coordinate transformations and imaging.

Theme A: Station Beamshapes (EJones)

Most of the radio telescopes (new or existing) have a beamshape problem. The sophisticated calibration that is made possible by an explicit M.E. requires a better knowledge of the full-polarization response of the main lobe(s), and even the sidelobes. Even the simplest of them all (the WSRT) is only known approximately.

After the Nancay workshop, we should have an active Creative Commons on this subject. Its goal will be to generate EJones matrices for all the existing and new radio telescopes in the world, including exotic ones like parabolic cylinders, and including phased arrays like EMBRACE and APERTIF.

In addition, it will offer MeqTrees processing scripts to process data for these telescopes. The same scripts may be used for all, with only a different Jones matrix for each telescope.

A particular case for treatment is the (E)VLA, with its offset R and L beams that rotate on the sky. A proper EJones matrix will make all the difference in wide-field polarization calibration.

Theme B: Ionosphere (ZJones, FJones)

Over the last 10 years or so, ionospheric calibration has evolved from 2GC via "Field-Based Calibration" (FBC, Cotton) to "Source Peeling Atmospheric Modelling" (SPAM, Intema). Each represented a clear improvement, but the applicability was limited to small arrays (FBC) or a single-layer ionosphere (SPAM). We now propose to take the last step to the fully general Minimum Ionospheric Model (MIM).

The word "minimum" refers to a minimum number of parameters, and a minimum number of assumptions about the ionosphere. For estimating the MIM parameters, it is possible to mix astronomical selfcal phases with GPS TEC measurements. The latter helps to increase the density of the sampling function, especially for large arrays like LOFAR and SKA.

For imaging, a relative TEC is sufficient. But for Faraday rotation, the absolute TEC is required. The latter is difficult to measure, but Ger de Bruyn has pointed out an intriguing possibility to use the differential refraction of widely separated sources in the sky. This requires multi-beam observations, which only LOFAR can do.

After the Nancay workshop, we should have an active Creative Commons on this subject. Its goal will be to generate ZJones (ionos. phase) and FJones matrices (Faraday rotation), based on the MIM model. After thorough verification, for instance by the LOFAR Long Baseline Working Group (LLWBG), the MeqTrees implementation will serve as working prototype for implementation in production systems like the LOFAR BBS.

Further Evolution of the SSSC

The SKADS Set of Standard Challenges (SSSC) is meant to become a proving ground for new algorithms for radio astronomical data reduction. Its core will be a set of Closed Challenges, i.e. sets of simulated (or real) uv-data, with an unknown source model, and unknown instrumental errors. Any data reduction package may be used to reduce the data, and unlock its secrets. Prizes will be awarded, and glory, by an international panel of experts.

Ideally, no claims for new algorithms should be accepted by anyone, unless they have proved themselves in one or more Challenges. This, and the detailed information in their accompanying PURR log, will greatly speed up their implementation in a form that actually reaches the user.

While developing the SSSC idea, we realized that it is also a great teaching device, in the form of Open Challenges. From there to using it as a Teaching and Qualification tool for the Nancay workshop was but a small step. This will also allow us to gain some experience before opening the SSSC in earnest.