Frequently Asked Questions
(If you have more questions, or require clarifications, contact noordam@astron.nl)
MeqTrees Technical Support
For MeqTrees technical support please consult the
MeqWiki, or send an email to: meqtrees-support@astron.nl
Where has the spigot2sink page gone?
It's here!
PURR Logs
The PURR log is a rather effective and low-threshold way of exchanging
detailed information, with others and with yourself. It is a tool that
watches the subdirectory in which you are working, and asks you to do
something every time a new file is generated. This then gives you the
opportunity to collect all the relevant plots, images, parameter sets
and commands in the right order, together with your comments. The HTML
result can be easily displayed (e.g. on a website), and exchanged by
email. Examples and instructions are on the SSSC page.
What are we trying to achieve with the Nancay workshop?
Click
here for the workshop web page, and
here for the SKADS webpage.
The qualification process for participation in the Nancay workshop is
not only a way to select talented people, but also to bring them up to
speed before coming to Nancay. Once there, we will devote time to all
aspects of 3GC, but most of all we will try to use this valuable
"face-time" to create the bonds and common understanding that is
needed to continue working together remotely afterwards.
What are Creative Commons?
This is a group that has the tools and the incentive to work together
remotely. Of course they exist in all kinds of forms, but the name has
been coined by some organization (google it). We are doing something
slightly different, but we like the stimulating name.
Developing software together is an old dream, but it has been very
difficult in practice (see the AIPS++ project). However, we feel that
this time we have the tools (MeqTrees, PURR logs) and the incentive (the urgent need for 3GC for the new
telescopes).
What kind of community do we hope to create?
Ideally, we hope to create a "can-do" atmosphere around all the 3GC
issues, but first of all around our two selected themes (see also
below). These subjects happen to be the most urgent for many of the
new (and existing) telescopes, but solutions are emerging much too
slowly. We propose to change all that. First of all by answering the
questions that are already floating out there, and then by generating
the next level of questions (and answers) ourselves.
What kind of people are we looking for?
All the existing radio telescopes perform way below their potential
because it is too difficult to experiment with new ideas. This is
caused partly by the design of the existing software, but most of all
because neither users nor observatories have the skills or the
inclination to experiment. The sustained success of the existing
software packages (AIPS, MIRIAD, DIFMAP, NEWSTAR) has caused the old
class of user-developers to disappear.
Our goal is to recreate this class. Therefore, we are looking
primarily (in this stage) for people who have demonstrated unusual
productivity in creating software. We expect them to find a place
somewhere between the extremes of "user" and "developer", depending on
their tastes and talents. But all of them should be able to move more
deftly and more easily than their current counterparts.
Our guiding model here is Jodrell Bank, where user-developers were the
norm once. Things have changed now, of course, but it should be noted
that a remarkable number of these Jodrellites have reached leading
positions at radio observatories all over the world.
We predict that there will be lots of jobs for user-developers when
"the shit hits the fan", i.e. when it becomes clear that the thickly
flowing data from the new telescopes cannot be reduced.
What is 3rd Generation Calibration (3GC)?
Until ~1980, 1GC relied on the stability on the instrument between
observations of bright calibrator sources. The interval could be up to
12 hours, and produced a Dynamic Range of 1:1000 at best. All of the
great discoveries of the 1970's were made with this DR.
After the invention of selfcal ~1980, the DR shot up to more than
1:100.000 (the WSRT is world champion with 1:5.000.000). However this
2GC solves for instrumental effects that are valid for the entire FOV,
i.e. for Direction-Independent Effects (DIE). It is assumed that all
station beams are identical, so "the" beam may be divided out from the
image.
The new telescopes (and the upgraded existing ones) require 3GC, which
deals with Direction-Dependent Effects (DDE). The most important are
the ionosphere, and station beams that are no longer identical. One
problem is that solving for many more parameters requires a huge
increase in processing. An additional problem is that there might not
be enough information to solve for so many parameters. That is why it
is imperative to make maximum use of "a priori" information about the
continuity (smoothness) of parameter values in time, freq and
direction. And finally, it is much more difficult to apply DDE's,
either in the forward (imaging) or the backward (prediction)
direction.
Since 3GC requires so much processing, more efficient 4GC techniques
will probably be needed to reach the noise. So, in the case of highly
demanding applications like the detection of the Epoch of Reionization
(EoR), 3GC is needed to create the conditions in which 4GC statistical
analysis of the residuals will work sufficiently well.
What is the Measurement Equation (M.E.)?
A Measurement Equation is a mathematical model of a measuring
instrument. In our case it is a matrix formalism that describes a
generic radio telescope in full polarization. One of its most
important elements is the 2x2 Jones matrix, which fully describes the
instrumental effects of a single station on the signal. By assiging a
separate Jones matrix to each station, arrays with rather dissimilar
stations may be accommodated. Each station Jones matrix may be the
product of a succession of Jones matrices, each of which describes a
different instrumental effect.
The M.E. may be used to predict the values of the uv-data when a
particular radio telescope observes a particular brightness
distribution. One application of this is the generation of simulated
uv-data, for instance for the SSSC. The Oxford group has taken a
strong lead in this area. Another application is calibration,
i.e. solving for the parameters of the M.E. by minimizing the
difference between predicted and measured uv-data.
The radio-astronomical M.E. was first written down in a closed form by
Hamaker et al. in Dwingeloo (1996). This was rather fortunate, because
3GC, and thus the calibration of the new radio telescopes, would have
been unthinkable without an explicit M.E.
What is MeqTrees?
MeqTrees has been designed for implementing an arbitrary M.E., and to
solve for (arbitrary subsets of) its parameters. At this moment, it is
the only package that can do this.
MeqTrees implements the M.E. in the form of a tree (or rather graph)
of software nodes, each of which performs an elementary function. The
may range from simple to very complex. With its simple Request/Result
interface, nodes can be connected into trees of arbitrary complexity.
In addition, MeqTrees has a number of features that are crucial for
"increasing the rate of evolution" of data reduction software:
-) Thanks to the Tree Definition Language (TDL, Python), it has a
very short turn-around time for implementing and testing new ideas:
hours rather than months.
-) It offers unprecedented visualization at all levels.
-) It is highly modular, so many people can contribute specialized
nodes without without having to know about the rest of the system. An
example is the UVBrick, written by Filipe Abdalla (UCL).
-) Because processing scripts (Python) can be easily exchanged by
email, it is much easier to collaborate in experimentation.
MeqTrees interfaces with the AIPS++/CASA Measurement Set (uv-data
file), and also other AIPS++/CASA modules like fitting, coordinate
transformations and imaging.
Theme A: Station Beamshapes (EJones)
Most of the radio telescopes (new or existing) have a beamshape
problem. The sophisticated calibration that is made possible by an
explicit M.E. requires a better knowledge of the full-polarization
response of the main lobe(s), and even the sidelobes. Even the
simplest of them all (the WSRT) is only known approximately.
After the Nancay workshop, we should have an active Creative Commons
on this subject. Its goal will be to generate EJones matrices for all
the existing and new radio telescopes in the world, including exotic
ones like parabolic cylinders, and including phased arrays like
EMBRACE and APERTIF.
In addition, it will offer MeqTrees processing scripts to process data
for these telescopes. The same scripts may be used for all, with only
a different Jones matrix for each telescope.
A particular case for treatment is the (E)VLA, with its offset R and L
beams that rotate on the sky. A proper EJones matrix will make all the
difference in wide-field polarization calibration.
Theme B: Ionosphere (ZJones, FJones)
Over the last 10 years or so, ionospheric calibration has evolved from
2GC via "Field-Based Calibration" (FBC, Cotton) to "Source Peeling
Atmospheric Modelling" (SPAM, Intema). Each represented a clear
improvement, but the applicability was limited to small arrays (FBC)
or a single-layer ionosphere (SPAM). We now propose to take the last
step to the fully general Minimum Ionospheric Model (MIM).
The word "minimum" refers to a minimum number of parameters, and a
minimum number of assumptions about the ionosphere. For estimating the
MIM parameters, it is possible to mix astronomical selfcal phases with
GPS TEC measurements. The latter helps to increase the density of the
sampling function, especially for large arrays like LOFAR and SKA.
For imaging, a relative TEC is sufficient. But for Faraday rotation,
the absolute TEC is required. The latter is difficult to measure, but
Ger de Bruyn has pointed out an intriguing possibility to use the
differential refraction of widely separated sources in the sky. This
requires multi-beam observations, which only LOFAR can do.
After the Nancay workshop, we should have an active Creative Commons
on this subject. Its goal will be to generate ZJones (ionos. phase)
and FJones matrices (Faraday rotation), based on the MIM model. After
thorough verification, for instance by the LOFAR Long Baseline Working
Group (LLWBG), the MeqTrees implementation will serve as working
prototype for implementation in production systems like the LOFAR BBS.
Further Evolution of the SSSC
The SKADS Set of Standard Challenges (SSSC) is meant to become a
proving ground for new algorithms for radio astronomical data
reduction. Its core will be a set of Closed Challenges, i.e. sets of
simulated (or real) uv-data, with an unknown source model, and unknown
instrumental errors. Any data reduction package may be used to reduce
the data, and unlock its secrets. Prizes will be awarded, and glory,
by an international panel of experts.
Ideally, no claims for new algorithms should be accepted by anyone,
unless they have proved themselves in one or more Challenges. This,
and the detailed information in their accompanying PURR log, will
greatly speed up their implementation in a form that actually reaches
the user.
While developing the SSSC idea, we realized that it is also a great
teaching device, in the form of Open Challenges. From there to using
it as a Teaching and Qualification tool for the Nancay workshop was
but a small step. This will also allow us to gain some experience
before opening the SSSC in earnest.